reconcile-inject.sh wrote an orphan under a key whose uuid differed
from the id it printed, so the demo could only compare counts; it now
uses one uuid for both, and the dry-run check compares the found and
the injected sets per class. The heal step returns the original bytes
under one quarantined key and random bytes of the same length under
another, and checks that only the first leaves the quarantine. The
load phase sums rechecked over its passes, and a new step stops a pass
hanging on a paused S3 with docker stop and checks exit code 143 within
seconds and a free lock afterwards.
reconcile_track.py cost clears the run files of the previous series and
measures 1k, 10k, 30k and 100k with --deep everywhere. The scenarios
record date, commit and docker inspect of the replicas or of S3 at the
moment they are down. make load-evidence and make clean-clone-evidence
rerun the message load and the clean clone block with raw output, and
make upload-memory passes ARGS to reproduce the run directories.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
A pass without --deep released the quarantine of a missing or resized
attachment as soon as an object of the same size appeared under its
key, so other bytes of the same length went back to /content. The pass
now streams and hashes the object of every quarantined row whose size
matches again, deep or not, and releases it only when the SHA-256
matches; other bytes keep the row quarantined. Consistent data still
costs no GET.
The keyset reads page + 1 rows, so a pass takes exactly ceil(N/page)
SQL pages instead of one more for the empty tail. The reconciler is
assembled by dishka from PlatformProvider and the attachment storage
provider that the API uses too, and it publishes the events of the
intents it drops, so audit.attachment_discarded reason=reconcile
reaches the journal. Any failure of a pass, not only Unavailable, ends
as reconcile.failed with the error type and exit code 1.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
The review of track 2 found that a pass without --deep released the
quarantine of a missing or resized attachment as soon as an object of
the same size reappeared, without checking its SHA-256; that the keyset
read an empty last page, one SQL request more than the spec promised;
that the discarded intents of a pass never reached the audit journal;
that the entry point duplicated the S3 client wiring and printed a
traceback for any failure other than Unavailable; and that the cost
matrix stopped at 30k while the preregistered H2 was about 100k.
The change hashes the object of a quarantined row before releasing it,
reads page + 1 rows, assembles the reconciler with dishka and publishes
its events, turns any pass failure into reconcile.failed, links the
injected orphans to the found ones by key and measures up to 100k.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Upload checks the token before the declared size, a slow but steady
transfer outlives the request timeout while a stalled one is still cut,
the API and the reconciler hold separate minimal S3 keys, and the
one-off services run as non-root with an init process. On the stand a
3 MiB upload at 256 KiB/s took 12.1 s at the replica and got 201, a
3 MiB download at 192 KiB/s took 12.8 s and matched the SHA-256, the
API key is denied a bucket listing, and docker stop ends a hanging
reconciler pass in 0.13 s with exit code 143. student-test (345),
test-faults (19), persistence, failover and the lab 4 scenarios pass.
The requirements are synced into lab4-attachments, lab4-upload-resilience
and lab4-object-storage.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
The API key now has only GetObject, PutObject and DeleteObject on the
attachment objects: the server never lists the bucket, never runs
multipart uploads and has its region set. The reconciler gets its own
user messenger-reconciler with ListBucket, GetObject and DeleteObject,
created by s3-init from a secret of its own, so the API replicas can no
longer list the bucket. The cost seeding writes objects and therefore
runs in a one-off API container under the API key. s3-check proves the
API key cannot list and checks the reconciler key; its snapshot drops
dated fields with a JSON filter instead of grep, which left a dangling
comma, and its first line carries the date and the commit.
s3-init and s3-cli run as uid 65534, and migrate and reconciler get an
init process, so SIGTERM reaches python and docker stop ends a pass in
a fraction of a second instead of waiting for SIGKILL. limits.py and CI
check the users of the one-off services. Stack tests move a 3 MiB file
up and down through the balancer at 256 and 192 KiB/s, longer than the
request timeout, and check 401 before the declared size.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
The review of lab 4 found four gaps at the attachment boundaries: an
anonymous upload with a too large Content-Length got 413 before 401,
the single request deadline cut slow but steady uploads and downloads,
the API S3 key could list the bucket and run multipart actions, and the
one-off services ran as root or as PID 1 without an init process.
The change checks the token first, paces the request deadline by the
bytes moved, splits the S3 keys of the API and of the reconciler and
runs the one-off services as non-root with an init process.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Track 2 of lab 4 is closed. One reconciler pass merges the sorted bucket
listing with keyset pages of attachments by object key, finds objects
without a row, ready rows without an object or with other bytes and
abandoned intents, deletes only old objects without a row after a
second look at the row and old intents under SKIP LOCKED, and puts
damaged ready rows into a reversible quarantine instead of deleting
them. On the stand the dry-run found exactly the 22 injected
discrepancies of seven classes, apply fixed the 18 fixable ones, the
repeat did nothing, and 75 passes during 3418 uploads lost no ready
attachment. A shallow pass costs 0.105 / 1.000 / 3.729 s for 1k / 10k /
30k objects at a flat 83 MiB, 95 % of it in the RustFS listing; --deep
is 4.3-5 times dearer per object.
For the defence: make reconcile-demo shows injection, dry-run, apply,
the repeat, quarantine through the API, healing, the held lock, a
stopped S3 and the run under uploads in about four minutes; the cost
matrix is evidence/reconcile/cost/summary.txt (make reconcile-cost
repeats it on a fresh stand in about seven).
The lab4-storage-reconciler spec is added and the quarantine rule of
lab4-attachments is updated in the main specs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
REPORT.md gets the track 2 section: what the reconciler does and why it
is safe, the hypotheses written before the first run, the method of the
demo and the cost matrix with its screening, both result tables, the
verdicts and the limits. H1 holds. H2 holds for the pass cost: time is
linear (t(30k)/t(10k) = 3.73, 3.05 in the series of 29.10), the listing
is 95 % of it, list requests are ceil(N/1000) and SQL one more for the
empty last keyset page, RSS stays at 82.5-83.8 MiB; --deep is 4.3-5
times dearer per object, not ten. The criteria table, the failure table,
the limits, AGENTS.md and the openspec context name the track rows, and
evidence/reconcile/README.md describes every file.
On a stand from a clean state make test, test-persistence,
test-failover, restore, reconcile, lint, check, validate and
student-test (342 passed) all pass; the dry-run after these scenarios
finds 3 objects, 3 rows and no discrepancy.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
make reconcile-cost on a fresh stand at b191623: RustFS 2 CPU / 3 GiB on
cpus 11-12, PostgreSQL on 13-14, the reconciler on 15, three dry-run
passes per point; no run was disturbed, foreign CPU stayed at most 0.2
cores in the VM and 0.01 on the pinned cores. Medians:
objects mode t, s lists sql gets rss, MiB
1000 shallow 0.105 1 2 0 82.5
10000 shallow 1.000 10 11 0 83.6
30000 shallow 3.729 30 31 0 83.8
10000 page100 2.058 100 101 0 81.0
1000 deep 0.493 1 2 1000 83.0
10000 deep 5.041 10 11 10000 84.1
30000 deep 16.150 30 31 30000 84.1
The listing takes 95 % of a shallow pass. The first full series of
29.10 (5ca2cdc, not screened, another lab under k6 at the time) is kept
in cost-2026-10-29 for comparison between days.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
make reconcile-demo on a fresh stand at 5c3c238, 12 of 12 checks true.
The dry-run found exactly what was injected on both sides (3 stale
orphans, 3 missing objects, 3 shorter and 3 same-size rewritten objects,
6 stale intents with and without an object, 2 fresh orphans and 2 fresh
intents) and changed nothing; apply fixed the same 18 and the second
apply did nothing. The independent check outside the API found no
violation and no lost ready attachment; the 9 damaged rows sit in
quarantine with their message links, the metadata answers 200 and the
content 404, and a restored object left quarantine. A pass during a held
lock was skipped, a pass with s3 stopped exited with 1 in 1.6 s and
changed nothing. 75 apply passes during 75 s of uploads through the
balancer removed only the 10 injected leftovers, saw 247 objects of
uploads in flight as fresh orphans and kept them: all 3418 uploads are
ready with matching SHA-256.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
The cost matrix shares the Docker VM with the stands and builds of other
labs. Each run now reads the VM-wide /proc/stat and the cpu.stat of the
s3 and postgres cgroups before and after the pass, and the reconciler
reports its own CPU time. Foreign CPU over the window is the busy time
minus ours minus the cost of starting a container (1.5 core-s in the VM
and 0.5 on the pinned cores, measured on the idle stand). A run with
more than 2 foreign cores in the VM or 0.25 on the pinned cores goes to
cost/disturbed and is repeated after 15 s of quiet, up to three times;
docker stats of the whole host are saved before and after the series.
The demo also runs a pass with s3 stopped: it must exit with code 1 in
bounded time, log reconcile.failed and leave PostgreSQL and the bucket
as they were.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
The merge loop runs inside the TaskGroup that holds the --deep checks,
so a listing or page failure in the middle of a pass left the group as
an ExceptionGroup. The entry point catches Unavailable only: the pass
still exited with code 1, but through an uncaught traceback instead of
the reconcile.failed record. The pass now unwraps the group and raises
the Unavailable with its original cause; the API test fails the listing
on the second page of a deep apply pass and expects Unavailable.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Sync the delta specs into the main specs: zones are linked by the
schema and the ports main.py assembles, the event bus and the network
dependencies name S3 and Valkey, an outsider does not wait for chat
locks, an empty cursor is rejected, a request cancelled at shutdown gets
503 in the contract format, the body limit counts received bytes per
route, logout withdraws the revocation mark on a database failure,
detection runs three times, failover-each records the stopped replica,
lost answers are retried on the stand, posting checks its input in a
fixed order and make up waits for every replica in the balancer.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Bring the system changes of lab 3 from 61c0c28..72a1e57 onto the lab 4
code, resolving each conflict by meaning and keeping everything lab 4
added:
- a message is posted with one SQL statement under the chat row lock,
and the attachment links go into the same statement; a foreign key
violation on the attachment link still answers 404 and leaves nothing
behind. Text and the attachment list are checked before the write, an
outsider gets 404 for both, as before the carry over;
- the request pipeline counts the body bytes actually received against
the limit of the route, so a chunked body over 1 MiB gets 413 and the
upload route keeps its MAX_UPLOAD_BYTES limit;
- a request cancelled at shutdown gets 503 unavailable in the contract
format, the worker waits a second less than the rest of the shutdown
budget, SHUTDOWN_TIMEOUT_SECONDS becomes 6 s and the stop hooks 7 s;
- readiness sits behind a Readiness port with the draining flag in
shared infrastructure, and the drain window is logged through Logger;
- logout revokes through SessionRevocations and withdraws the mark when
PostgreSQL rejects the delete; an empty cursor is 400;
- EventBus.project is gone, the sweep test subscribes and waits on a
queue; domain tests check attachment_list instead of Chat.post;
- make up waits for every replica in the balancer, BASE_URL follows the
lb port, the image is tagged by the compose project, retry-check and
faults-evidence are added, detection runs three times, failover-each
and rolling-update record the stand usage, CI checks numeric uids.
down also takes the tools and reconcile profiles of lab 4.
Track 1 tooling of lab 3 (scaling.py, its samplers, tools/ and the CPU
pinning of api, lb, valkey and k6) stays out; stand-usage.sh queries the
database size itself instead of the track 1 sampler.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Lab 4 started from the lab 3 commit 61c0c28, and lab 3 changed up to
72a1e57 after its track and two reviews. The change lists what moves
over and how it meets the lab 4 code: the one-statement message write
takes the attachment links along, the body limit counts received bytes
per route, the shutdown, readiness and logout fixes come as they are,
and the stand tooling keeps the lab 4 profiles and CPU pinning. Delta
specs bring the requirements lab 3 changed, adapted to three zones and
the object store; track 1 tooling stays out.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Filling RustFS 1.0.4 to 100k objects under one prefix did not finish in
two attempts: a write gets slower as the prefix grows (about 8 ms in an
empty prefix against 32 ms at 28k objects, 13 writes per second with 16
parallel writers), with enough memory and no background scanner load.
The cost matrix now stops at 30k objects with four parallel writers,
which still spans a factor of 30 for the linearity check; the design
and the spec of the change say so, and both aborted attempts with their
1k and 10k points and the diagnosis are kept in
evidence/reconcile/cost-aborted.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
reconcile-inject.sh and reconcile_check.py come from the prepared kit:
the first breaks either side on purpose (an object without a row, a
removed object, another size, the same size with other bytes, an old
intent with and without its object), the second is an oracle outside
the API that classifies every row and object from psql and rc, hashes
objects with --deep and reports ready attachments lost against a saved
set.
reconcile_track.py demo uploads through the balancer, links some
attachments to messages, injects each class on both sides with old and
fresh variants, and checks that a deep dry-run finds exactly what was
injected and changes nothing, that apply fixes exactly that and a
second apply does nothing, that broken rows and their message links
survive in quarantine with metadata 200 and content 404, that a
restored object heals, that a second pass is skipped while the lock is
held, and that passes in a loop during 75 s of uploads lose nothing.
Intents are injected 300 s old, between the reconciler grace and the
API sweep grace, so only the reconciler removes them.
reconcile_track.py cost snapshots docker stats of the whole host, pins
S3, PostgreSQL and the reconciler to their own CPUs, fills the stand to
each N with seed_storage.py run in the API image (parallel PutObject and
COPY), and records three dry runs per N and mode with a median table.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
The reconciler service under the reconcile profile runs the API image
with the reconciler module as entrypoint and --dry-run as the default
command, so docker compose run --rm --no-deps reconciler takes only
flags. It gets the database URL and the limited API key of S3, which
already allows listing, reading and deleting objects, a read-only root,
1 CPU and 256 MiB, and a longer S3 read timeout for listing pages.
S3, PostgreSQL and the reconciler accept an optional cpuset from the
environment so measurements can pin them; the default is no pinning.
make reconcile ARGS=... runs a pass, make reconcile-demo and make
reconcile-cost drive the track experiments, and the limits policy now
covers the new service.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
StorageReconciler walks the sorted bucket listing and keyset pages of
attachments ordered by object key side by side, so a pass is linear
and keeps one page of each side in memory. An object without a row is
deleted only when it is older than grace plus skew and a lookup by key
right before the delete still finds no row; a ready row whose object
is missing, has another size or, with the deep check, another digest
is put in quarantine after a HEAD recheck and is never deleted; a stale
intent is claimed with SKIP LOCKED and dropped object first; a
quarantined row whose object matches again is released. Deep checks
hash objects in a bounded task group, each action uses its own unit of
work, and failures of single actions are counted instead of aborting.
An advisory lock lets one pass run at a time. Content of a quarantined
attachment answers 404 without touching S3, its metadata is unchanged.
The storage port gains listing pages and HEAD, the repository keyset
pages, lookup by key, claim and quarantine marking; the platform logger
can write to another stream. The one-off entry point
python -m messenger.attachments.entrypoints.reconcile is a dry run
unless --apply is given, assembles the adapters with the API key of S3
and prints a JSON report with counts per class, samples and the cost
of the pass to stdout while the journal goes to stderr.
Tests run the reconciler against its own migrated database with every
class, apply and repeat, the row that appears after the listing, the
busy lock and the release, and check paged byte-order listing and HEAD
against RustFS.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Migration 0004 adds a nullable quarantine reason (missing, size or
digest mismatch) and the time it was set, both null or both present.
Nullable columns without a default only touch the catalogue, so the
running version keeps working while the migration is applied.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Track 2 of grade 5: a one-off reconciler that merges the sorted bucket
listing with keyset pages of the attachments table, so a pass is linear
in time and holds one page per side in memory. It finds objects without
a row, ready rows without an object or with another size or digest, and
abandoned intents; by default it only reports, with --apply it deletes
old orphans after a row recheck, drops stale intents object first, and
quarantines broken ready rows instead of deleting them, so referenced
attachments survive and a restored object heals. A quarantined
attachment answers 404 on its content. An advisory lock keeps a single
pass at a time.
The design states the hypotheses on correctness and on the cost of a
pass, the reproducible method behind make reconcile-demo and make
reconcile-cost, and the acceptance criteria.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Grade 4 of lab 4 is closed. A failed ready commit after the object
write is compensated at once under a row lock, intents abandoned by a
crash or a lost write are swept with their objects after a 900 s grace
period by every replica with SKIP LOCKED, and nothing younger, nothing
ready and nothing referenced is touched. A client abort is a 400 and a
stalled body a 503 after 8 s, both leaving nothing behind. Three upload
slots per worker, capped socket buffers in the balancer and the API
and a fixed glibc mmap threshold keep 40 concurrent 10 MiB uploads at
330 of 512 MiB per replica with every upload succeeding, while without
the slots the same load hits the limit and loses 37 of 80. 36 user
errors through the balancer give no 5xx, and 23 checks prove the
bucket private, the API key limited, the console closed and the
bootstrap idempotent.
For the defence: run make upload-faults to show the negative suite,
the compensation, the killed replica with its leftover swept after the
grace period while the fresh one stays, and the abort and stall; show
the memory summary and make s3-check.
The lab4-upload-resilience spec is added, and the write order of
lab4-attachments and the console and API key requirements of
lab4-object-storage are updated in the main specs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Evidence comes from the running stand at a774a2f: upload-faults.txt
with the negative suite through the balancer (36 user errors, no 5xx
in the replica logs) and the compensation, crash, sweep, abort and
stall checks; upload-memory with per-run JSON and a summary for 64
and 3 slots, 1 and 10 MiB and 10, 20 and 40 concurrent uploads plus
the mmap threshold comparison; s3-security.txt with 23 checks; and
regression-level4.txt showing that smoke, persistence, failover of
each replica, the upload failover, S3 outage and abort scenarios, the
limits policy and all 15 stand checks still pass. The lint and own
test log now counts 335 tests. The upload-faults target stamps its
output with the date and commit.
REPORT.md gains rows 4.1 to 4.4 and a regression row, the compensation
and sweep mechanism with its grace period, where each upload limit and
buffer cap applies, the new variables, level 4 failure rows and the
memory experiment with hypothesis, method, results and limits. AGENTS
lists the new targets.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
upload_memory.py recreates the replicas and the balancer for every run
of 10, 20 and 40 concurrent uploads of 1 and 10 MiB with 64 slots
(effectively no limit) and with 3, drives them through lab4_seed.py
from the prepared tooling, and keeps per-run JSON plus a summary with
the cgroup memory.peak of each replica and of the balancer and the
oom_kill counters. The seed script now also records memory.current
before the load and reads X-Instance-Id regardless of case.
The first run showed that slots alone only move a waiting body out of
the worker: with 3 slots and 40 uploads of 10 MiB the kernel grew the
socket buffers of HAProxy to about 110 MB and the 128 MiB balancer was
killed. HAProxy now caps its client and server socket buffers at 64
KiB and the API replicas cap tcp_rmem and tcp_wmem at 256 KiB, which
keeps the balancer at 30 MiB. glibc raised its mmap threshold after
the first large free and kept freed 10 MiB bodies in the heap, so the
API fixes MALLOC_MMAP_THRESHOLD_ at 128 KiB.
With these settings 40 concurrent uploads of 10 MiB peak at 323 MiB of
512 per replica and all 80 succeed, while without the slot limit the
same load reaches 512 MiB, a worker is killed and only 31 of 80
succeed. UPLOAD_CONCURRENCY stays at 3.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
test_attachment_faults.py runs through the balancer against the real
PostgreSQL and RustFS. The negative suite sends a body one byte over
the limit with Content-Length (answered before any body byte) and
chunked, a decompression bomb, truncated and disguised images, missing
and foreign content types, dangerous names, a stranger's reads and
references, an attachment of another chat and invalid lists, checks
each code and error body, and finds no 5xx in the replica logs and no
extra rows.
Faults come from triggers installed for the module and dropped after
it: an AFTER INSERT trigger remembers the id and key of each fault
intent, a BEFORE UPDATE trigger rejects the ready commit for
fault-fail names and holds it for 1.8 s for fault-crash names, while
the test finds the waiting backend's client address and kills that
replica with SIGKILL. The rejected commit leaves neither row nor
object right away and an audit line says compensation; a crash leaves
a pending row and an object, the one aged past the grace period is
swept with its object while the fresh one stays, and an aged ready
attachment in a message still downloads. A body cut after 256 KiB is
logged as 400 with nothing written, and a stalled body gets 503 after
8 s with nothing written.
make upload-faults stores the output in evidence/upload-faults.txt
without tracebacks of locals, and the CI acceptance job runs it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
The API replicas get UPLOAD_CONCURRENCY, UPLOAD_GRACE_SECONDS,
UPLOAD_SWEEP_INTERVAL_SECONDS and UPLOAD_SWEEP_BATCH through the S3
anchor of compose.yaml with the same defaults as the settings, and
.env.example documents them. A value in the shell overrides the
default, which the memory experiment uses to recreate the replicas with
a different number of slots.
s3-check.sh now also asks the console port of the S3 service from a
separate container on the stand network and expects a refused
connection, so 23 checks cover ports, the console inside and outside
the container, the repeated bootstrap, the rights of the API key and
anonymous access.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
When the ready commit fails after the object was written, the upload
handler now locks the row, keeps it if it turned out ready, and
otherwise deletes the object first and the pending row second before
answering 503. A ready update that touches no row means the intent was
already swept, so it fails as unavailable and the object goes too;
a 201 without a row cannot happen.
Every worker runs an UploadSweeper that every 30 s removes pending
intents older than UPLOAD_GRACE_SECONDS (900 s, at least 60 s) one row
at a time under FOR UPDATE SKIP LOCKED over the partial pending index,
object first, and logs an audit event for each discard. Bodies are
buffered only inside one of UPLOAD_CONCURRENCY slots per worker and
released before the ready commit, and a client that disconnects in the
middle of a body becomes a 400 instead of an unhandled 500.
The tests run against a throwaway PostgreSQL: a trigger rejects the
ready commit and the compensation leaves neither row nor object, a
row removed during the write gives 503, a failed write stays until the
grace ends, fresh intents and an aged ready attachment in a message
survive a sweep, two concurrent sweeps share five intents exactly, the
background loop runs by itself, a second upload waits for the slot
without reading its body, and aborted and stalled bodies leave nothing.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Grade 4 of lab 4 asks that a failure between the S3 write and the
PostgreSQL commit leaves no eternal orphans, that concurrent uploads
cannot take unbounded memory, that user errors never become a 500 and
that the API key and the private bucket are proven limited outside the
API. The change compensates a failed ready commit right away under a
row lock, sweeps abandoned intents after a grace period in every
replica with SKIP LOCKED, maps a client abort to a client error, bounds
buffered bodies per worker with upload slots, and checks all of it
against the stand: a negative suite through the balancer, faults
injected with a PostgreSQL trigger and a killed replica, and a memory
experiment with 10, 20 and 40 concurrent uploads.
The new lab4-upload-resilience capability holds these requirements;
the write order of lab4-attachments and the console and key checks of
lab4-object-storage are updated by deltas.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Grade 3 of lab 4 is closed. RustFS with a private bucket runs in the
stand, metadata lives in PostgreSQL and the bytes only in the bucket,
uploads answer 201 ready after the object write and the commit, and
the original is served byte for byte to chat members only. The smoke
test passes from a fresh clone; limits of 10 MiB and 40 MP, content
type by content and dangerous names are covered by 325 own tests; an
upload survives the stop of the replica that took it, both replicas
fail over under writes in under 3.1 s, a full restart keeps metadata
and bytes, and with S3 stopped uploads and downloads answer 503 within
2.6 s while texts, metadata and readiness keep working. 22 checks show
the bucket is private outside the API and the API key is limited.
For the defence: upload the fixture, send it in a message, download it
as the second member and compare SHA-256, get 404 as a stranger and
403 for an anonymous GET in S3, then run persistence, failover and the
upload failover scenario.
The lab4-attachments and lab4-object-storage specs are added, and
message sending, history, idempotent replay, the restart hook and stand
persistence are updated in the main specs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Evidence comes from the stand started from an empty state and from a
fresh clone: the public smoke, the contract checks, lint and 325 own
tests, the storage layout with an out-of-API comparison of a stored
object against its metadata, failover of each replica under writes, the
upload failover scenario, persistence with StartedAt of every service
including S3, the S3 outage scenario and 22 S3 privacy and bootstrap
checks. The storage evidence script now covers the attachment tables
and the S3 secrets.
REPORT.md follows the lab 4 template: launch from a clean clone,
architecture with the source of truth for each fact, the write order
between PostgreSQL and S3 and what each failure leaves behind, where
each upload limit applies, roles and services, the .env.example
mapping, grade 3 rows and the grade 3 failure table. The design notes
the botocore Expect header that hung uploads behind the port proxy.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
lab4_scenarios.py drives the public contract through tests/client.py:
failover-upload stops the replica that answered the upload with the
stop-one hook and downloads through another one comparing SHA-256, and
s3-outage stops S3 and records status and time of upload, download,
metadata, a text message and readiness before bringing it back. Both
print JSON with an ok field. s3-check.sh checks published ports, the
console, two more bootstrap runs without state changes, the allowed
and denied operations of the API key and anonymous access including a
plain HTTP GET from a separate container. evidence-attachments.sh
shows where the bytes live and compares a stored object with its
metadata outside the API.
Makefile gains scenario, s3-check and attachments-evidence, ruff covers
the new script, and the CI acceptance job runs both scenarios and the
S3 check after the smoke test.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
MinIO community images can no longer be pulled on this arm64 host, so
the stand runs RustFS 1.0.4 pinned by digest as non-root with a read
only root, no published port, the console off and data on the s3data
volume. A one-off s3-init with rc v0.1.36 creates the bucket, keeps it
private for anonymous clients, creates the API user with a policy on
the objects and listing of that one bucket, imports a lifecycle rule
for incomplete multipart uploads and checks itself; the API replicas
start only after it succeeded. Root keys reach only s3, s3-init and the
operator s3-cli; the API gets its own key and the S3 settings through
an anchor merged into its environment.
secrets.sh creates hex S3 keys, the restart hook restarts S3 next to
PostgreSQL and Valkey and waits until it is healthy, the limits policy
covers the S3 services, and the stack evidence lists the new packages.
The object storage adapter is tested against a throwaway RustFS of the
same image: a 10 MiB round trip, a missing object, an idempotent delete
and an unreachable endpoint failing within the request budget.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A message now carries attachment_ids in the request, in every response,
in the idempotent replay and in history, in the order they were sent.
The chat aggregate checks membership first, then refuses whitespace
text, more than ten or repeated attachments, and an empty text without
attachments, so an outsider still gets 404 before any 400.
The send statement inserts the links next to the message and the
idempotency key, and the foreign key from message_attachments to
attachments of the same chat turns an unknown or foreign attachment
into 404 and rolls the whole send back, without the messaging zone
reading the attachments table. Replay and history read the list with a
subquery on the primary key. The request fingerprint covers the list,
so a missing attachment_ids and [] are the same body while a different
order is a conflict.
The zone dependency test now covers three zones, and the SQL ownership
test checks that every zone touches only its own tables.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Add the adapters behind the attachments ports: PostgreSQL for the
pending and ready rows, with a foreign key violation on the chat member
turned into 404; Pillow that opens only PNG and JPEG, refuses a type
that differs from Content-Type, checks the pixel count from the header
before any decoding, verifies PNG chunks and decodes JPEG at 1/8 scale
in a worker thread under a semaphore; and S3 on aiobotocore with path
style, SigV4, short timeouts and checksums only when required. botocore
sends Expect: 100-continue for streamed bodies and RustFS behind the
Docker port proxy never answers it, so the PUT hung until the read
timeout; the handler that adds the header is unregistered.
The HTTP layer streams the raw body into the use case, returns the
contract Attachment and streams the original back with its content
type and length and no redirect. The request pipeline takes a body
limit per route, so uploads may carry 10 MiB while JSON stays at 1 MiB.
main.py wires the zone and answers its membership port from the
messaging chat directory.
Tests run the API with an in-memory object store: download by another
member, 404 for strangers, the exact 10 MiB limit and one byte over it
with Content-Length and chunked, 40 MP and above, spoofed and truncated
images, unsupported types, dangerous names, the variant, a failing
store and a lost object.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The zone owns the Attachment aggregate, which is staged as pending with
a server-assigned object key and becomes ready only when stored, the
Filename value that refuses empty, blank, overlong names and names with
a path separator or a control character, and the declared image type.
UploadAttachment checks membership first through the ChatMembership
port, then the name and the declared type, reads the body stream only
up to the limit while hashing it, inspects the image through a port,
commits the pending row, writes the object through the ObjectStorage
port and commits the ready row. GetAttachment and OpenAttachment answer
404 to strangers and to a ready row whose object is gone.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Migration 0003 creates attachments with a pending or ready status, a
server-assigned object key in byte order collation and a partial index
for pending rows, and message_attachments that keeps the order of the
attachments of a message. Foreign keys hold the links between zones:
an attachment belongs to a member of its chat, and a message may only
reference an attachment of its own chat, with RESTRICT so a referenced
attachment can never be deleted. The text check is relaxed to at most
4000 code points because an empty text is legal next to an attachment;
an empty message without attachments is still refused by the domain.
The migration test covers the new version, a rerun without changes and
both foreign keys.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Plan grade 3 of lab 4 as one change: a third domain zone for
attachments that takes the raw bytes of one PNG or JPEG, applies the
size limit before reading the whole body and the pixel limit before
decoding, checks the type by content, writes a pending row, the object
and then the ready row and only then answers 201, and serves metadata
and the original only to chat members through the API. Messages gain
attachment_ids, linked by a foreign key to attachments of the same chat
so the messaging zone never reads the attachments table, and the zone
asks messaging about membership through a port wired in main.py.
The stand gains RustFS with a private bucket, a one-off bootstrap with
a separate API key, and a restart hook that waits for S3. The specs add
lab4-attachments and lab4-object-storage and modify message sending,
history, idempotent replay, the restart hook and stand persistence.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Bring the Claude Code settings, the lint hook, the opsx commands and
skills, the architecture notes, the lab 1 to lab 3 specs and the
archived lab 1 to lab 3 changes into the lab 4 skeleton, so lab 4
continues the same spec history instead of starting from scratch. The
files come from the lab 3 solution commit 61c0c28.
Rewrite AGENTS.md and the openspec context and rules for lab 4: image
attachments with metadata in PostgreSQL and bytes in a private RustFS
bucket inside the stand, a third domain zone for attachments that asks
the messaging zone about chat membership through a port wired in
main.py, upload limits applied before unbounded reading or decoding,
201 only after both the object write and the commit, Out of scope moved
to labs 5-7 plus presigned uploads, and rules on the write order between
the two stores and on root S3 keys staying out of the API.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>