Commit Graph
3 Commits
Author SHA1 Message Date
ArtyomandClaude Opus 5.5 e049c265b8 Reproduce a lost answer on the stand and enrich the run records
No failover run ever lost an answer, so the retry of an unconfirmed
write was only covered in process. make retry-check holds the chat row
with SELECT ... FOR UPDATE for 1.5 s (below lock_timeout), sends a
message with a 0.5 s client timeout, finds the replica whose request
waits on the lock in pg_stat_activity and then retries with the same
key and text through the LB, once with that replica alive and once
after docker kill. It checks the 201, a single copy with the id of the
retry answer and, for the live replica, the 201 it logged without a
client; created_at tells which attempt wrote the message. CI runs it.

failover-each and rolling-update now record the database size and a
docker stats snapshot in the background of the action, rolling-update
names the tag and id of every image it replaces, and scaling.py writes
started and the compose-up log marks in UTC with Z; the track README
states that the recorded rows are host time (MSK) and what
host-others.csv covers.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 05:43:44 +03:00
ArtyomandClaude Opus 5.5 a59e3a39a8 Carry over the lab 2 fixes made after 8cddb09
Brings the system changes of lab 2 from 8cddb09..102579e onto the lab 3
code, resolving each conflict by meaning:

- the request pipeline counts the body bytes actually received, so a
  chunked body over the limit gets 413 instead of reaching the app;
- logout revokes through a SessionRevocations port and withdraws the
  mark when PostgreSQL rejects the delete, so 503 means the session
  still works; PostgresSessionDirectory loses its no-op invalidate;
- an empty cursor is 400 on history and on the chat list;
- EventBus.project and the test that allowed importing an event across
  zones are gone, the config rules speak of the schema foreign key;
- the worker waits one second less than the rest of the shutdown
  budget, so it cancels a stuck request with 500 and exits 0 instead
  of being killed by the gunicorn master. Lab 3 keeps 3 s after the
  drain window for the database timeouts, so SHUTDOWN_TIMEOUT_SECONDS
  becomes 6 s and the stop hooks wait 7 s, still under the 8 s grace;
- the lab 3 single statement post already locks the chat only for a
  member, so lab 2's LOCK_CHAT edit is not needed; its test is kept;
- lab 2 tests come along with Idempotency-Key where lab 3 needs it;
  the short TTL after restart runs without api-shortttl by shortening
  the session in the database; test_drain adds a stuck request;
- make validate uses .venv with openapi-spec-validator, PYTEST_ARGS,
  faults-evidence, down with the bench profile, the image is tagged by
  the compose project, start probes every 200 ms, cache evidence fails
  without a tombstone and checks the key expiry, kdf.txt shows the
  seed stub apart.

The track 2 tooling of lab 2 (track.py, postgres-restore and the
database argument of migrate) and its evidence scripts stay out.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 02:32:49 +03:00
ArtyomandClaude Opus 5.5 572204950a Replace replicas one by one under load
scripts/rolling-update.sh runs migrate once before touching any
replica, then for each old replica scales the service to N+1 without
recreating the others, waits until the new container is healthy and
the balancer sees N+1 UP servers, and only then stops the old one
with docker stop -t 6 (so it drains) and removes it. The log names
the hostname and image of the old and the new replica of every step,
which is what the report quotes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 19:52:19 +03:00