Carry source f591fa64e5 forward from 72a1e57. Preserve lab4 attachment transactions, reconciliation locks, quarantine, and existing schema data through verified legacy adoption. Keep report and evidence work in progress untouched.
overload-leftovers.txt was assembled by hand with summarised reconciler
reports. make overload-leftovers now runs the dry-run, the status
counts, the wait for the sweep after the grace period, the audit and
swept counts of the replicas and the dry-run afterwards, and writes
their raw output.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
reconcile-inject.sh wrote an orphan under a key whose uuid differed
from the id it printed, so the demo could only compare counts; it now
uses one uuid for both, and the dry-run check compares the found and
the injected sets per class. The heal step returns the original bytes
under one quarantined key and random bytes of the same length under
another, and checks that only the first leaves the quarantine. The
load phase sums rechecked over its passes, and a new step stops a pass
hanging on a paused S3 with docker stop and checks exit code 143 within
seconds and a free lock afterwards.
reconcile_track.py cost clears the run files of the previous series and
measures 1k, 10k, 30k and 100k with --deep everywhere. The scenarios
record date, commit and docker inspect of the replicas or of S3 at the
moment they are down. make load-evidence and make clean-clone-evidence
rerun the message load and the clean clone block with raw output, and
make upload-memory passes ARGS to reproduce the run directories.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
The level 3 and 4 evidence was taken on 24f9d61 and a774a2f, before the
lab 3 fixes were carried over and before the reconciler. Take all of it
again in one series on 81dd816, whose application and stand code is the
final one: a fresh clone runs the launch block of the report from secrets
to reset-data, then one stand from clean volumes gives the smoke, stack,
limits, lb-check, failover of each replica and of the upload replica,
persistence, S3 outage, S3 privacy, lost-answer retry, upload faults, all
stack checks, storage layout and three runs of the upload memory matrix.
The three memory runs keep the peak with three slots at 297-345 MiB
against 512 MiB at the limit without them, but at 40 uploads of 10 MiB
the wait for a slot nears the 8 s request timeout and one run of three
gave 7 of 80 a 503. Those timeouts left 67 invisible intents; the record
shows the reconciler seeing them as fresh and the sweep removing all of
them after the grace period.
Message write load is taken with tests/load.py from a container on the
edge network: 2 106 RPS, no errors, replicas and PostgreSQL at their CPU
limits. The first try from the macOS host is kept as a discarded attempt,
it ran out of ephemeral ports while the replicas answered every request.
The report gets the defense answers, the load section, the evidence
commits, the upload limit order as the pipeline applies it, medians in
the track 2 cost table and the proposals over the word limit; the evidence
index lists every file with its command and commit.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
REPORT.md gets the track 2 section: what the reconciler does and why it
is safe, the hypotheses written before the first run, the method of the
demo and the cost matrix with its screening, both result tables, the
verdicts and the limits. H1 holds. H2 holds for the pass cost: time is
linear (t(30k)/t(10k) = 3.73, 3.05 in the series of 29.10), the listing
is 95 % of it, list requests are ceil(N/1000) and SQL one more for the
empty last keyset page, RSS stays at 82.5-83.8 MiB; --deep is 4.3-5
times dearer per object, not ten. The criteria table, the failure table,
the limits, AGENTS.md and the openspec context name the track rows, and
evidence/reconcile/README.md describes every file.
On a stand from a clean state make test, test-persistence,
test-failover, restore, reconcile, lint, check, validate and
student-test (342 passed) all pass; the dry-run after these scenarios
finds 3 objects, 3 rows and no discrepancy.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Bring the system changes of lab 3 from 61c0c28..72a1e57 onto the lab 4
code, resolving each conflict by meaning and keeping everything lab 4
added:
- a message is posted with one SQL statement under the chat row lock,
and the attachment links go into the same statement; a foreign key
violation on the attachment link still answers 404 and leaves nothing
behind. Text and the attachment list are checked before the write, an
outsider gets 404 for both, as before the carry over;
- the request pipeline counts the body bytes actually received against
the limit of the route, so a chunked body over 1 MiB gets 413 and the
upload route keeps its MAX_UPLOAD_BYTES limit;
- a request cancelled at shutdown gets 503 unavailable in the contract
format, the worker waits a second less than the rest of the shutdown
budget, SHUTDOWN_TIMEOUT_SECONDS becomes 6 s and the stop hooks 7 s;
- readiness sits behind a Readiness port with the draining flag in
shared infrastructure, and the drain window is logged through Logger;
- logout revokes through SessionRevocations and withdraws the mark when
PostgreSQL rejects the delete; an empty cursor is 400;
- EventBus.project is gone, the sweep test subscribes and waits on a
queue; domain tests check attachment_list instead of Chat.post;
- make up waits for every replica in the balancer, BASE_URL follows the
lb port, the image is tagged by the compose project, retry-check and
faults-evidence are added, detection runs three times, failover-each
and rolling-update record the stand usage, CI checks numeric uids.
down also takes the tools and reconcile profiles of lab 4.
Track 1 tooling of lab 3 (scaling.py, its samplers, tools/ and the CPU
pinning of api, lb, valkey and k6) stays out; stand-usage.sh queries the
database size itself instead of the track 1 sampler.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Evidence comes from the running stand at a774a2f: upload-faults.txt
with the negative suite through the balancer (36 user errors, no 5xx
in the replica logs) and the compensation, crash, sweep, abort and
stall checks; upload-memory with per-run JSON and a summary for 64
and 3 slots, 1 and 10 MiB and 10, 20 and 40 concurrent uploads plus
the mmap threshold comparison; s3-security.txt with 23 checks; and
regression-level4.txt showing that smoke, persistence, failover of
each replica, the upload failover, S3 outage and abort scenarios, the
limits policy and all 15 stand checks still pass. The lint and own
test log now counts 335 tests. The upload-faults target stamps its
output with the date and commit.
REPORT.md gains rows 4.1 to 4.4 and a regression row, the compensation
and sweep mechanism with its grace period, where each upload limit and
buffer cap applies, the new variables, level 4 failure rows and the
memory experiment with hypothesis, method, results and limits. AGENTS
lists the new targets.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Bring the Claude Code settings, the lint hook, the opsx commands and
skills, the architecture notes, the lab 1 to lab 3 specs and the
archived lab 1 to lab 3 changes into the lab 4 skeleton, so lab 4
continues the same spec history instead of starting from scratch. The
files come from the lab 3 solution commit 61c0c28.
Rewrite AGENTS.md and the openspec context and rules for lab 4: image
attachments with metadata in PostgreSQL and bytes in a private RustFS
bucket inside the stand, a third domain zone for attachments that asks
the messaging zone about chat membership through a port wired in
main.py, upload limits applied before unbounded reading or decoding,
201 only after both the object write and the commit, Out of scope moved
to labs 5-7 plus presigned uploads, and rules on the write order between
the two stores and on root S3 keys staying out of the API.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>