The deltas are synced to openspec/specs: a request that outlives the
graceful shutdown gets 503 unavailable in the common error format with
X-Request-Id and X-Instance-Id instead of the bare 500; a lost answer is
retried with the same key both on a live replica and after the replica
holding the request was killed, with one copy and the same id; make up
returns once every running replica is UP in the LB.
Before archiving, on e049c26 from a reset stand: make test, both
failover runs (first write after 2.858 and 2.682 s, 0 failed attempts),
persistence, retry-check, faults and make level4 pass, the clean clone
run follows the launch block verbatim; make check, make lint, 243 own
tests and openspec validate --all --strict pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
The deltas are synced to openspec/specs: the order of checks on posting
a message (401, then 400 for the request shape without touching the
chat, then 404 for access, then 400 for the text); three detection runs
with a 0.25 s measurement slack and the replica states after each stop
in failover-each; the upper bound of the track 1 grid with a ceiling on
the first grid point stated as a lower bound; control runs of the pool,
the LB queue and the placement.
Before archiving, on the running stand: make test passes, failover
recovers the first write in 2.78 s with 0 failed attempts, persistence
passes; make check, make lint and 243 own tests pass. All level 3 and 4
records were taken on b1c80ee, after the lab 2 carry over.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Lab 3 now stands on the finished lab 2. The deltas are synced to
openspec/specs: the zones meet only in the schema and Valkey is needed
only to confirm a logout; an outsider does not wait for a chat lock;
an empty cursor is 400 on history and on the chat list; a stuck request
at SIGTERM gets 500 from the server itself within the 6 s budget, which
still leaves 3 s after the drain window for the database timeouts; a
chunked body is limited by the bytes received; a logout rejected by
PostgreSQL withdraws its revocation mark. The event publication purpose
follows lab 2.
On a stand rebuilt from empty volumes: make test passes, failover
recovers the first write in 3.1 s with 0 failed attempts, persistence
passes, test-faults 12 passed including the full restart with logout
and the shortened session after an API restart, detection passes all
four scenarios with SIGTERM detected in 0.82 s and the stop done in
2.44 s, and the cache evidence finds a tombstone and a key that does
not outlive its session.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
The openspec rules keep a proposal within 400 words; the archived track 1
proposal had 405. Three phrasings outside the hypotheses are shortened,
the hypotheses stay as they were written before the runs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Track 1 of grade 5 is closed. At a fixed 4 CPU / 2 GiB the 1x4, 2x2
and 4x1 layouts peak at 5 375, 4 964 and 4 724 rps: within the
hypothesised 15 percent, yet smaller replicas cost a third more CPU
per request and reach a long tail earlier. Growing one-CPU replicas
gives 1 720, 2 936, 4 724 and only 4 260 rps at eight, a retrograde
curve with USL sigma 0.014 and kappa 0.038. The saturation point is
the chat row lock in PostgreSQL: at eight replicas 47.5 of 64
connections wait for it while the API, the balancer and the database
CPU still have room. Writing a message with one statement removed the
application round trips from under the lock: p95 at 3 600 rps fell
from 64.5 to 3.7 ms and the peak rose by 15 % with the same resources.
The new lab3-scaling-cost spec is synced to openspec/specs, with the
compare scenario stated as measured: below the knee fewer lock waits
and a lower p95, past it a higher peak. AGENTS.md lists make track1
and make plots. The neighbour CPU sampler is stopped and its file
closed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Grade 4 of lab 3 is closed. A replica on SIGTERM first answers 503 on
readiness for a two second window, longer than the 1.5 s worst
detection, and only then closes its listener and finishes accepted
requests within five seconds; the balancer lets a replica back only
after two passed checks. A test pins the timeout chain to compose,
HAProxy and the stop hook. make detection measured 0.71 s for SIGTERM
(before the process exited at 2.49 s), 0.67 s for a killed process,
2.76 s for a hung one and 1.23 s until no replica was left with
PostgreSQL down. make failover-each stopped and restored each replica
under k6 at 100 rps with no errors and no balancer retries, make
lb-check ran 20 same-key races of 8, make limits checks limits and
isolation against a policy, and make rolling-update replaced the
level 3 image with the current one under load with capacity kept at
two and no errors, migrate running once. The new
lab3-failover-operations spec is synced, and the lab 1 operability
and lab 2 dependency failure specs carry their deltas.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Grade 3 of lab 3 is closed: the stand runs two API replicas of one
compose service behind HAProxy as the only published entry point,
one token reaches both replicas by X-Instance-Id, stopping either
replica with scripts/stop-one.sh keeps writes going within 1.2-1.7 s
without errors or duplicates, eight concurrent posts with one key give
one message, and make test, the persistence scenario with
scripts/restart.sh, the failover scenario, lint, own tests, check,
validate and the lab 2 stack fault tests pass. The new
lab3-load-balancing and lab3-idempotent-messages specs are synced, and
the lab 1 history and operability specs and the lab 2 storage spec
carry their deltas.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Bring the Claude Code settings, the lint hook, the opsx commands and
skills, the architecture notes, the lab 1 and lab 2 specs and the
archived lab 1 and lab 2 changes into the lab 3 skeleton, so lab 3
continues the same spec history instead of starting from scratch.
Rewrite AGENTS.md and the openspec context and rules for lab 3:
HAProxy in front of several stateless API replicas, X-Instance-Id on
every response, a mandatory Idempotency-Key for messages shared by all
replicas through PostgreSQL, explicit stop-one and restart hooks, Out
of scope moved to labs 4-7 plus Swarm and HA storage, and rules on
retry policy, timeout chains and replica-local state.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>