No failover run ever lost an answer, so the retry of an unconfirmed
write was only covered in process. make retry-check holds the chat row
with SELECT ... FOR UPDATE for 1.5 s (below lock_timeout), sends a
message with a 0.5 s client timeout, finds the replica whose request
waits on the lock in pg_stat_activity and then retries with the same
key and text through the LB, once with that replica alive and once
after docker kill. It checks the 201, a single copy with the id of the
retry answer and, for the live replica, the 201 it logged without a
client; created_at tells which attempt wrote the message. CI runs it.
failover-each and rolling-update now record the database size and a
docker stats snapshot in the background of the action, rolling-update
names the tag and id of every image it replaces, and scaling.py writes
started and the compose-up log marks in UTC with Z; the track README
states that the recorded rows are host time (MSK) and what
host-others.csv covers.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Brings the system changes of lab 2 from 8cddb09..102579e onto the lab 3
code, resolving each conflict by meaning:
- the request pipeline counts the body bytes actually received, so a
chunked body over the limit gets 413 instead of reaching the app;
- logout revokes through a SessionRevocations port and withdraws the
mark when PostgreSQL rejects the delete, so 503 means the session
still works; PostgresSessionDirectory loses its no-op invalidate;
- an empty cursor is 400 on history and on the chat list;
- EventBus.project and the test that allowed importing an event across
zones are gone, the config rules speak of the schema foreign key;
- the worker waits one second less than the rest of the shutdown
budget, so it cancels a stuck request with 500 and exits 0 instead
of being killed by the gunicorn master. Lab 3 keeps 3 s after the
drain window for the database timeouts, so SHUTDOWN_TIMEOUT_SECONDS
becomes 6 s and the stop hooks wait 7 s, still under the 8 s grace;
- the lab 3 single statement post already locks the chat only for a
member, so lab 2's LOCK_CHAT edit is not needed; its test is kept;
- lab 2 tests come along with Idempotency-Key where lab 3 needs it;
the short TTL after restart runs without api-shortttl by shortening
the session in the database; test_drain adds a stuck request;
- make validate uses .venv with openapi-spec-validator, PYTEST_ARGS,
faults-evidence, down with the bench profile, the image is tagged by
the compose project, start probes every 200 ms, cache evidence fails
without a tombstone and checks the key expiry, kdf.txt shows the
seed stub apart.
The track 2 tooling of lab 2 (track.py, postgres-restore and the
database argument of migrate) and its evidence scripts stay out.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
scripts/rolling-update.sh runs migrate once before touching any
replica, then for each old replica scales the service to N+1 without
recreating the others, waits until the new container is healthy and
the balancer sees N+1 UP servers, and only then stops the old one
with docker stop -t 6 (so it drains) and removes it. The log names
the hostname and image of the old and the new replica of every step,
which is what the report quotes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>