Tie the grade 5 claim to the final SQLAlchemy revalidation and measured scaling results. Clarify clean-clone setup, failure timings, historical evidence, and operational constraints; record report-stage checks and evidence audits.
Keep the message and idempotency receipt in one Core CTE statement with chat-first locking. Preserve rollback, bounded dependency failures, and domain ports.
Replace the SQL runner with Alembic. Adopt legacy databases only after validating their journal and PostgreSQL schema, retain all data, and verify migrations on fresh and populated databases.
User requirement (2026-11-05, labs 3+): all PostgreSQL access goes through
SQLAlchemy 2.x async and migrations run through Alembic. The proposal keeps
every port, zone boundary and observable API behavior in place, moves the
checksum-based migration journal scenario to Alembic revision semantics and
pins the evidence that depends on SQL behavior for regeneration.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The deltas are synced to openspec/specs: a request that outlives the
graceful shutdown gets 503 unavailable in the common error format with
X-Request-Id and X-Instance-Id instead of the bare 500; a lost answer is
retried with the same key both on a live replica and after the replica
holding the request was killed, with one copy and the same id; make up
returns once every running replica is UP in the LB.
Before archiving, on e049c26 from a reset stand: make test, both
failover runs (first write after 2.858 and 2.682 s, 0 failed attempts),
persistence, retry-check, faults and make level4 pass, the clean clone
run follows the launch block verbatim; make check, make lint, 243 own
tests and openspec validate --all --strict pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
The header names the code and harness commits of the second review
cycle and states that every level 3 and 4 record and the clean clone
run are on e049c26. Row 4.1 now carries the detection ranges and the
SIGTERM window of the current records, the 503 answer to a request
cancelled at shutdown replaces the 500 everywhere, and the launch block
loses the manual LB wait now done by make up and gains retry-check and
storage-evidence, so the clean clone run follows it verbatim. Rows 3.4
rest on the stand reproduction of a lost answer instead of failover runs
that never failed, a new row proves the contract, public tests and task
untouched against upstream and names the CI job, and the level 4
experiments state their repeats, data volume and resource snapshots.
Track 1: errors are attributed to the replicas that returned them, with
a failure column in the pool control table; the pool control states its
fixed variant order and growing base, pool 4 is a lower bound and no
longer compared with 4x1; the write command times are call-weighted over
all statement rows; the 3 600 rps comparison names the uneven foreign
load and rests on lock waits and CPU per request; the limits name the
database sizes behind the series A peaks and what host-others.csv
covers. AGENTS.md lists retry-check and the LB wait of make up.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Every level 3 and 4 record is taken anew on e049c26 from a reset stand:
checks with the upstream diff of the contract, the public tests and the
task (empty) and of .github (only the own CI job), student tests with
the 503 answer to a request cancelled at shutdown, smoke, stack,
storage on the filled database, both failover runs with the
service.draining event, persistence, hooks, faults, limits, lb-check,
the new retry-check, three detection runs, failover-each and both
rolling updates with the database size, a docker stats snapshot and the
tag and id of every replaced image. The clean clone run executes the
launch block of the report verbatim and marks the observations made
between its steps.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
No failover run ever lost an answer, so the retry of an unconfirmed
write was only covered in process. make retry-check holds the chat row
with SELECT ... FOR UPDATE for 1.5 s (below lock_timeout), sends a
message with a 0.5 s client timeout, finds the replica whose request
waits on the lock in pg_stat_activity and then retries with the same
key and text through the LB, once with that replica alive and once
after docker kill. It checks the 201, a single copy with the id of the
retry answer and, for the live replica, the 201 it logged without a
client; created_at tells which attempt wrote the message. CI runs it.
failover-each and rolling-update now record the database size and a
docker stats snapshot in the background of the action, rolling-update
names the tag and id of every image it replaces, and scaling.py writes
started and the compose-up log marks in UTC with Z; the track README
states that the recorded rows are host time (MSK) and what
host-others.csv covers.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
compose up --wait returns once lb is healthy, and lb is healthy with one
replica in rotation; the second one joins after rise 2 checks, about
2.5 s later, so make test-failover right after make up could see a
single X-Instance-Id. make up now runs scripts/lb-wait.sh, which waits
for as many UP servers as there are running api containers.
The CI non-root step compared the third column of docker compose top
with 0, but the daemon prints the user name there (root), so it could
never fail. It now reads the numeric uid of every process of every
container with docker top -eo pid,uid,comm (Docker needs the PID
column); a throwaway root container makes the check fail.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
The /health/ready route imported the PostgreSQL adapter and kept the
drain flag in the HTTP layer, which gunicorn_conf imported to raise it.
Readiness is now a port of the shared application layer; its adapter in
shared/infrastructure checks the per-process drain flag and pings
PostgreSQL, and PlatformProvider assembles it. The route only knows the
port, and gunicorn_conf raises the flag through the infrastructure
module. Another critical dependency now changes the adapter and the
provider, not the HTTP layer.
The drain window was logged by a uvicorn logger with a printf string;
it is now the service.draining event of a StructlogLogger with the same
fields as the application, and failover-evidence looks for it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
After the drain window uvicorn waits for accepted requests for three
seconds and then cancels their tasks. CancelledError is a BaseException,
so neither the pipeline nor ServerErrorMiddleware saw it and uvicorn
answered with its own text/plain 500 without X-Request-Id and
X-Instance-Id. The pipeline now handles the cancellation next to the
request timeout: it logs request.cancelled, sends 503 unavailable in the
common error format when the response has not started and re-raises,
so uvicorn only closes the transport after the buffered answer.
The declared Content-Length check joins the guard as a rejection to keep
the call within the statement limit; behaviour is unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
The second review found that a request cancelled at the end of the
graceful shutdown gets uvicorn's plain 500 without the contract headers,
that readiness reaches the PostgreSQL adapter and a global flag of the
HTTP layer past the ports, that the drain window is logged past the
Logger port, that make up returns with one replica in rotation and that
the CI non-root check can never fail. The retry of an unconfirmed write
was never exercised on the stand either.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
The CPU per request and the lock waiters of the placement control are
taken at 4560 rps; the header now says so instead of leaving the reader
to guess between the two offered rates.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
The deltas are synced to openspec/specs: the order of checks on posting
a message (401, then 400 for the request shape without touching the
chat, then 404 for access, then 400 for the text); three detection runs
with a 0.25 s measurement slack and the replica states after each stop
in failover-each; the upper bound of the track 1 grid with a ceiling on
the first grid point stated as a lower bound; control runs of the pool,
the LB queue and the placement.
Before archiving, on the running stand: make test passes, failover
recovers the first write in 2.78 s with 0 failed attempts, persistence
passes; make check, make lint and 243 own tests pass. All level 3 and 4
records were taken on b1c80ee, after the lab 2 carry over.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
The LB queue does not move the peak of 8x1 (4526, 4677 and 4525 rps for
maxconn 16, 32 and 64, overlapping ranges), and the pool moves it the
other way: 4 connections per worker give 5358 rps, 16 give 3789, while
the lock waiters grow from 14.7 to 102. So the pool is not short, extra
connections only add writers to the hot chat row, which confirms the
coordination cost of the USL fit. On the same CPUs 8x0.5 peaks 7 % below
4x1 and costs 15 % more CPU per request, so the retrograde 8x1 comes from
the number of processes and connections, not from slower vCPUs.
The report also names the cause of the flaky compose up, the WalSync
stall of the dropped attempt, the disturbed control runs, and the
evidence index gets a commit column for track 1. The scenario of the
open change no longer presumes its own outcome.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
make track1-control on b1c80ee: 8x1 at 5400 rps with a pool of 4, 8 and
16 connections per worker and an LB maxconn of 16, 32 and 64 per
replica, then 4x1 against 8x0.5 on the same CPUs 7-10, three repeats
with the variants interleaved.
The first placement attempt is kept in aborted/ only as a summary: four
4x1 runs had the write command at 7-34 ms instead of 0.2-0.6 ms, waiting
on WalSync, which the CPU based rejection does not see. The rerun right
after it is the one in control-cpus. The full output of the two failed
compose up attempts names the cause of the flaky switch: all replicas
are recreated at once and lb turns unhealthy without a ready server.
Plots and capacity.md now include both control series.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
The report now names a59e3a3 as the last change of the application code
and b1c80ee as the commit of every level 3 and 4 record, with the new
replica IDs and numbers: three detection runs within the formula without
the measurement slack, failover-each and both rolling updates without a
failed request, 12 stack tests and 243 own tests. The shutdown section
and the timeout chain follow the carried over budget: drain 2 s, then
3 s for accepted requests and a 500 for a stuck one, graceful timeout
6 s, stop hooks 7 s. make validate runs from .venv, faults evidence has
its own target, AGENTS.md lists the control runs of track 1.
The clean clone is taken again on f5fea5c without an explicit BASE_URL.
The track 1 part keeps the review fixes of the H2 verdict, the grid
bounds and the rounded numbers; the control runs follow.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
The carry over of the lab 2 fixes changed the request pipeline, logout,
the worker shutdown budget and the stop hooks, so every level 3 and 4
record is taken again on b1c80ee: checks, own tests, smoke, stack,
storage, failover of both replicas, persistence, hooks, the lab 2 stack
tests, limits and the injected violations, lb-check, three detection
runs, failover-each and both rolling updates under k6.
All of them pass: 243 own tests, 12 stack tests, no failed request under
k6 in failover-each and in both rolling updates.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
make faults-evidence wrote the pytest rootdir with the local home
directory, so the committed faults.txt was cleaned by hand and could not
be reproduced by the command from the report. The target now replaces
the repository path with <repo> and still fails when a stack test fails,
since the pipe through sed hides the pytest exit code.
The open review change also records that the lab 2 carry over moved the
default image tag to the compose project.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Lab 3 now stands on the finished lab 2. The deltas are synced to
openspec/specs: the zones meet only in the schema and Valkey is needed
only to confirm a logout; an outsider does not wait for a chat lock;
an empty cursor is 400 on history and on the chat list; a stuck request
at SIGTERM gets 500 from the server itself within the 6 s budget, which
still leaves 3 s after the drain window for the database timeouts; a
chunked body is limited by the bytes received; a logout rejected by
PostgreSQL withdraws its revocation mark. The event publication purpose
follows lab 2.
On a stand rebuilt from empty volumes: make test passes, failover
recovers the first write in 3.1 s with 0 failed attempts, persistence
passes, test-faults 12 passed including the full restart with logout
and the shortened session after an API restart, detection passes all
four scenarios with SIGTERM detected in 0.82 s and the stop done in
2.44 s, and the cache evidence finds a tombstone and a key that does
not outlive its session.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Brings the system changes of lab 2 from 8cddb09..102579e onto the lab 3
code, resolving each conflict by meaning:
- the request pipeline counts the body bytes actually received, so a
chunked body over the limit gets 413 instead of reaching the app;
- logout revokes through a SessionRevocations port and withdraws the
mark when PostgreSQL rejects the delete, so 503 means the session
still works; PostgresSessionDirectory loses its no-op invalidate;
- an empty cursor is 400 on history and on the chat list;
- EventBus.project and the test that allowed importing an event across
zones are gone, the config rules speak of the schema foreign key;
- the worker waits one second less than the rest of the shutdown
budget, so it cancels a stuck request with 500 and exits 0 instead
of being killed by the gunicorn master. Lab 3 keeps 3 s after the
drain window for the database timeouts, so SHUTDOWN_TIMEOUT_SECONDS
becomes 6 s and the stop hooks wait 7 s, still under the 8 s grace;
- the lab 3 single statement post already locks the chat only for a
member, so lab 2's LOCK_CHAT edit is not needed; its test is kept;
- lab 2 tests come along with Idempotency-Key where lab 3 needs it;
the short TTL after restart runs without api-shortttl by shortening
the session in the database; test_drain adds a stuck request;
- make validate uses .venv with openapi-spec-validator, PYTEST_ARGS,
faults-evidence, down with the bench profile, the image is tagged by
the compose project, start probes every 200 ms, cache evidence fails
without a tombstone and checks the key expiry, kdf.txt shows the
seed stub apart.
The track 2 tooling of lab 2 (track.py, postgres-restore and the
database argument of migrate) and its evidence scripts stay out.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Lab 3 was carried over from lab 2 at 8cddb09, and lab 2 changed up to
102579e after two reviews. The change brings those system fixes here:
the chunked body limit, a worker that cancels a stuck request before
the master kills it, a logout that withdraws its revocation mark when
PostgreSQL rejects it, the empty cursor, the zone rule kept to the
schema and the toolchain and stand tests. To keep the lab 3 rule that
draining leaves the database timeouts their full 3 s, the shutdown
budget grows from 5 to 6 s and the stop hooks from 6 to 7 s.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
The launch block of the report now works verbatim with a changed
LB_HTTP_PORT: in a fresh clone with port 18083 make test, both
failovers, persistence and storage-evidence pass without BASE_URL, the
second failover stops the other replica after one shifting request, and
storage-evidence lists only the volumes of the clone project.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Every level 3 and level 4 file in evidence was taken on e00eb6c, before
track 1 rewrote the message write into one statement under the chat
lock. The runs are repeated on a02d442: smoke, stack, storage, both
failovers (the second after one shifting request), persistence, hooks,
the lab 2 fault tests, limits and injected violations, 20 same-key races,
three detection runs, failover-each under k6 and both rolling updates.
All pass. checks.txt adds the real output of make check, make validate
and make lint, and student-tests.txt includes the new access order test.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
The openspec rules keep a proposal within 400 words; the archived track 1
proposal had 405. Three phrasings outside the hypotheses are shortened,
the hypotheses stay as they were written before the runs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
BASE_URL defaults to the published port of lb from docker compose, so
the targets work with a changed LB_HTTP_PORT. rolling-update FROM_REV
and track1-compare tag the old revision by its commit instead of the
shared mtusi-messenger tag, which a parallel clone could overwrite.
detection runs three times to show the spread, failover-each records
docker ps -a of the replicas right after the stop, and failover-evidence
takes SHIFT to move round robin to the other replica. track1-control
varies the pool and the balancer queue at 8x1 and compares 8x0.5 with
4x1 on the same CPUs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
The orchestrator printed only the last 500 characters of stderr before
retrying, which hid the cause of a flaky up on eight replicas. The whole
output of each failed attempt now goes to compose-up-failed.log next to
summary.csv.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27