The deltas are synced to openspec/specs: the order of checks on posting
a message (401, then 400 for the request shape without touching the
chat, then 404 for access, then 400 for the text); three detection runs
with a 0.25 s measurement slack and the replica states after each stop
in failover-each; the upper bound of the track 1 grid with a ceiling on
the first grid point stated as a lower bound; control runs of the pool,
the LB queue and the placement.
Before archiving, on the running stand: make test passes, failover
recovers the first write in 2.78 s with 0 failed attempts, persistence
passes; make check, make lint and 243 own tests pass. All level 3 and 4
records were taken on b1c80ee, after the lab 2 carry over.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
The LB queue does not move the peak of 8x1 (4526, 4677 and 4525 rps for
maxconn 16, 32 and 64, overlapping ranges), and the pool moves it the
other way: 4 connections per worker give 5358 rps, 16 give 3789, while
the lock waiters grow from 14.7 to 102. So the pool is not short, extra
connections only add writers to the hot chat row, which confirms the
coordination cost of the USL fit. On the same CPUs 8x0.5 peaks 7 % below
4x1 and costs 15 % more CPU per request, so the retrograde 8x1 comes from
the number of processes and connections, not from slower vCPUs.
The report also names the cause of the flaky compose up, the WalSync
stall of the dropped attempt, the disturbed control runs, and the
evidence index gets a commit column for track 1. The scenario of the
open change no longer presumes its own outcome.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
The report now names a59e3a3 as the last change of the application code
and b1c80ee as the commit of every level 3 and 4 record, with the new
replica IDs and numbers: three detection runs within the formula without
the measurement slack, failover-each and both rolling updates without a
failed request, 12 stack tests and 243 own tests. The shutdown section
and the timeout chain follow the carried over budget: drain 2 s, then
3 s for accepted requests and a 500 for a stuck one, graceful timeout
6 s, stop hooks 7 s. make validate runs from .venv, faults evidence has
its own target, AGENTS.md lists the control runs of track 1.
The clean clone is taken again on f5fea5c without an explicit BASE_URL.
The track 1 part keeps the review fixes of the H2 verdict, the grid
bounds and the rounded numbers; the control runs follow.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
make faults-evidence wrote the pytest rootdir with the local home
directory, so the committed faults.txt was cleaned by hand and could not
be reproduced by the command from the report. The target now replaces
the repository path with <repo> and still fails when a stack test fails,
since the pipe through sed hides the pytest exit code.
The open review change also records that the lab 2 carry over moved the
default image tag to the compose project.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Lab 3 now stands on the finished lab 2. The deltas are synced to
openspec/specs: the zones meet only in the schema and Valkey is needed
only to confirm a logout; an outsider does not wait for a chat lock;
an empty cursor is 400 on history and on the chat list; a stuck request
at SIGTERM gets 500 from the server itself within the 6 s budget, which
still leaves 3 s after the drain window for the database timeouts; a
chunked body is limited by the bytes received; a logout rejected by
PostgreSQL withdraws its revocation mark. The event publication purpose
follows lab 2.
On a stand rebuilt from empty volumes: make test passes, failover
recovers the first write in 3.1 s with 0 failed attempts, persistence
passes, test-faults 12 passed including the full restart with logout
and the shortened session after an API restart, detection passes all
four scenarios with SIGTERM detected in 0.82 s and the stop done in
2.44 s, and the cache evidence finds a tombstone and a key that does
not outlive its session.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Brings the system changes of lab 2 from 8cddb09..102579e onto the lab 3
code, resolving each conflict by meaning:
- the request pipeline counts the body bytes actually received, so a
chunked body over the limit gets 413 instead of reaching the app;
- logout revokes through a SessionRevocations port and withdraws the
mark when PostgreSQL rejects the delete, so 503 means the session
still works; PostgresSessionDirectory loses its no-op invalidate;
- an empty cursor is 400 on history and on the chat list;
- EventBus.project and the test that allowed importing an event across
zones are gone, the config rules speak of the schema foreign key;
- the worker waits one second less than the rest of the shutdown
budget, so it cancels a stuck request with 500 and exits 0 instead
of being killed by the gunicorn master. Lab 3 keeps 3 s after the
drain window for the database timeouts, so SHUTDOWN_TIMEOUT_SECONDS
becomes 6 s and the stop hooks wait 7 s, still under the 8 s grace;
- the lab 3 single statement post already locks the chat only for a
member, so lab 2's LOCK_CHAT edit is not needed; its test is kept;
- lab 2 tests come along with Idempotency-Key where lab 3 needs it;
the short TTL after restart runs without api-shortttl by shortening
the session in the database; test_drain adds a stuck request;
- make validate uses .venv with openapi-spec-validator, PYTEST_ARGS,
faults-evidence, down with the bench profile, the image is tagged by
the compose project, start probes every 200 ms, cache evidence fails
without a tombstone and checks the key expiry, kdf.txt shows the
seed stub apart.
The track 2 tooling of lab 2 (track.py, postgres-restore and the
database argument of migrate) and its evidence scripts stay out.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Lab 3 was carried over from lab 2 at 8cddb09, and lab 2 changed up to
102579e after two reviews. The change brings those system fixes here:
the chunked body limit, a worker that cancels a stuck request before
the master kills it, a logout that withdraws its revocation mark when
PostgreSQL rejects it, the empty cursor, the zone rule kept to the
schema and the toolchain and stand tests. To keep the lab 3 rule that
draining leaves the database timeouts their full 3 s, the shutdown
budget grows from 5 to 6 s and the stop hooks from 6 to 7 s.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
The launch block of the report now works verbatim with a changed
LB_HTTP_PORT: in a fresh clone with port 18083 make test, both
failovers, persistence and storage-evidence pass without BASE_URL, the
second failover stops the other replica after one shifting request, and
storage-evidence lists only the volumes of the clone project.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Every level 3 and level 4 file in evidence was taken on e00eb6c, before
track 1 rewrote the message write into one statement under the chat
lock. The runs are repeated on a02d442: smoke, stack, storage, both
failovers (the second after one shifting request), persistence, hooks,
the lab 2 fault tests, limits and injected violations, 20 same-key races,
three detection runs, failover-each under k6 and both rolling updates.
All pass. checks.txt adds the real output of make check, make validate
and make lint, and student-tests.txt includes the new access order test.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
The openspec rules keep a proposal within 400 words; the archived track 1
proposal had 405. Three phrasings outside the hypotheses are shortened,
the hypotheses stay as they were written before the runs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
BASE_URL defaults to the published port of lb from docker compose, so
the targets work with a changed LB_HTTP_PORT. rolling-update FROM_REV
and track1-compare tag the old revision by its commit instead of the
shared mtusi-messenger tag, which a parallel clone could overwrite.
detection runs three times to show the spread, failover-each records
docker ps -a of the replicas right after the stop, and failover-evidence
takes SHIFT to move round robin to the other replica. track1-control
varies the pool and the balancer queue at 8x1 and compares 8x0.5 with
4x1 on the same CPUs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
The orchestrator printed only the last 500 characters of stderr before
retrying, which hid the cause of a flaky up on eight replicas. The whole
output of each failed attempt now goes to compose-up-failed.log next to
summary.csv.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
storage-evidence listed volumes by the name mtusi2026lab3, so a clone in
another directory recorded the volumes of the main stand; it now takes
the project name from docker compose config and filters by its label.
make limits wrote only the date, while every evidence file promises the
date and the commit in its first line; the commit is added.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
FastAPI rejects a missing Idempotency-Key, extra fields and wrong types
with 400 before the handler runs, so an outsider gets the same 400 for
an existing and a missing chat; only a bad text, checked by the domain,
makes the handler look up membership and answer 404. The new test pins
that order so it is a recorded rule rather than an accident.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
An independent review of the grade 5 submission found that the level 4
measurements under load and the fault tests were taken on the code
before track 1 rewrote the message write, that the claim that the pool
and the balancer are not the bottleneck rests on reasoning alone, and
that the retrograde 8x1 result is not separated from running on other
vCPUs. Some evidence tooling also writes data of another compose
project or cannot reproduce a committed file.
The change records the order of checks on posting a message (401,
request shape 400, access 404, text 400), requires three detection runs
with the 0.25 s measurement tolerance, a docker ps of replicas after
each stop in failover-each, an upper bound of the rate grid, and control
runs that vary the pool, the balancer queue and the CPU placement one at
a time.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Rerun the clean clone pass on the final commit in its own compose project:
secrets, config, up with migrations on empty volumes, readiness, smoke,
replica spread and same key races, failover, persistence, repeated
migrations, down keeping volumes and reset-data removing them. Record the
student tests by name and the limits policy on the final code.
Check every figure of the report against evidence files and fix the ones
that drifted: data set sizes, the share of the chat lock statement, the
WAL waits, the aborted series, the disturbed steps, the restart timing
and the image ID of the rolling update. State which commit each evidence
file was taken on, answer the defense questions, index evidence/ and
bring AGENTS.md and the openspec context up to date for lab 3.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Track 1 of grade 5 is closed. At a fixed 4 CPU / 2 GiB the 1x4, 2x2
and 4x1 layouts peak at 5 375, 4 964 and 4 724 rps: within the
hypothesised 15 percent, yet smaller replicas cost a third more CPU
per request and reach a long tail earlier. Growing one-CPU replicas
gives 1 720, 2 936, 4 724 and only 4 260 rps at eight, a retrograde
curve with USL sigma 0.014 and kappa 0.038. The saturation point is
the chat row lock in PostgreSQL: at eight replicas 47.5 of 64
connections wait for it while the API, the balancer and the database
CPU still have room. Writing a message with one statement removed the
application round trips from under the lock: p95 at 3 600 rps fell
from 64.5 to 3.7 ms and the peak rose by 15 % with the same resources.
The new lab3-scaling-cost spec is synced to openspec/specs, with the
compare scenario stated as measured: below the knee fewer lock waits
and a lower p95, past it a higher peak. AGENTS.md lists make track1
and make plots. The neighbour CPU sampler is stopped and its file
closed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
make plots draws throughput and latency against offered load for both
series and the comparison, CPU per component, peak RPS against API
CPU with the USL fit and the improved 8x1 point, and the utilisation of
every link at each peak; capacity.md tabulates the ceilings.
after-checks.txt holds the level 3 and 4 checks on the new image:
student tests 240 passed, smoke PASS, 20 same-key races of 8 through
the balancer with 160 x 201 and one copy each, failover with 131
writes and no failure in 2.73 s, persistence PASS.
REPORT.md gains the track section: hypotheses as stated before the
runs, the method and its equal-resource rules, both series in tables,
the saturation point with database, balancer and pool figures, the
USL reading, the before and after table, conclusions per hypothesis
and the limits of a shared single host. The idempotent send section
now describes the one-statement write, the evidence table has the
track rows and the commands are in the clean clone section.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
8x1 with the image of edc091d (before) and the one-statement write
(after), alternating per repeat, three repeats at 3 600, 5 400 and
8 100 rps. At 3 600 rps, below the knee, p95 falls from 64.5 ms
(35.7..142.5) to 3.7 ms (2.8..5.4) and p99 from 122 to 13 ms; the API
spends 391 instead of 457 % CPU and PostgreSQL 102 instead of 137 %;
connections waiting for a chat lock drop from 12.8 to 0.15 and none is
idle in transaction any more. Past the knee the peak rises from 4 252
(4 054..4 396) to 4 894 rps (4 494..5 104), +15 %, and at 8 100 rps
from 4 104 to 4 841 (+18 %); the ranges do not overlap. Saturated, the
lock holders now wait for the WAL flush (LWLock WALWrite) instead of
the application.
The switch to the after image for the third repeat failed in compose
while another lab was running tests on the shared VM, so that repeat
was run alone with --first-repeat 3 on the same database; apply now
retries compose up twice and prints its stderr. Two after rows were
still marked disturbed after three reruns and are kept; they are the
lower ones.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
make track1-compare builds the image of BEFORE_REV, the last commit
without the one-statement write, builds the current one, and runs 8x1
at 3 600, 5 400 and 8 100 rps with the two images alternating per
repeat, so drift of the shared VM affects both alike. make track1 runs
the host snapshot, both series, the comparison and the plots.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Sending a message used to be BEGIN, a SELECT ... FOR UPDATE of the
chat with its members, a lookup of the idempotency key, the insert CTE
and COMMIT. Three of those round trips ran under the chat row lock, and
each one waited for the replica's event loop, so under load the lock
holders sat idle in transaction while everyone else queued behind
them.
The new MessageLog port has one PostgreSQL adapter that does it all in
a single statement with an implicit transaction: lock the chat row
joined with the sender's membership, stamp the message with the later
of the replica clock and the last message plus one microsecond, claim
the key (overwriting it only if it has expired), insert the message
only when the key was claimed, and move last_message_at. The foreign
key from the key to the message is checked at the end of the statement,
so the key may be written first. When the key is already taken the
adapter reads the stored result with a second statement outside the
lock, which also covers a concurrent duplicate that waited for the
first one; the handler then replays it or answers 409 as before.
The chat aggregate no longer takes part in posting, so its post,
membership and last_message_at members and their unit tests are gone;
Message.posted() builds the event. An invalid text is still answered
404 for an outsider: only then the handler reads the membership. A new
API test pins the clock rule against a real database: a lagging
replica clock gets the last stamp plus one microsecond, a clock ahead
is kept. The concurrency, same-key race, expiry and access tests pass
unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Series B located the bottleneck. At eight replicas the API has 40 % of
its CPU spare and the balancer 60 % of a core, yet three quarters of
the pool connections wait for a chat row lock: the locking SELECT of
the chat averages 14.2 ms and the lock holders sit idle in transaction
while the replica's event loop gets round to the next statement. The
critical section, three round trips long, is what more replicas
lengthen.
The design now records the choice: one CTE that locks the chat with
the membership check, stamps the message no earlier than the last one,
claims the idempotency key and inserts the message, committed by the
statement itself, so the lock never waits for the application. A taken
key is read back by a second statement outside the lock. The price is
that the chat aggregate no longer guards posting; the rules move into
the adapter's SQL and are held by integration tests. The rejected
alternatives and the before/after plan are listed, and the spec gains
the observable requirement of a short critical section.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
One to eight replicas of 1 CPU, 512 MiB and one worker each, three
repeats per rate; the 4x1 point comes from series A. Peak successful
RPS: 1 720 for one replica, 2 936 for two, 4 724 for four and only
4 260 for eight, lower than four. Up to two replicas the API is the
limit (101 % and 201 % of its CPU). At eight replicas the API uses 473
of its 800 % and the balancer 40 % of a core, while PostgreSQL is at
178 of 200 %, 47.5 of the 64 pool connections wait for a chat row lock
and the balancer queue holds 744 requests. The locking SELECT ... FOR
UPDATE of the chat takes 14.2 ms on average, 97 % of all statement
time of the step, and about 7 connections sit idle in transaction:
the lock holders wait for the application between round trips.
A universal scalability law fit over 1, 2, 4 and 8 CPUs gives
sigma 0.014 and kappa 0.038, a peak near five replicas and a decline
after it. No step of this series was rejected as disturbed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
1x4, 2x2 and 4x1 at 4 CPU and 2 GiB of API, workers equal to CPUs,
32 pool connections and 128 balancer slots in every layout, three
repeats from 600 to 6 830 rps. All three keep up with the offered
load up to 4 560 rps; the peaks are 5 375 rps for 1x4 (5 332..5 384),
4 964 for 2x2 (4 894..5 104) and 4 724 for 4x1 (4 686..4 794). At the
peak the API uses 95 % of its four CPUs in every layout, PostgreSQL
82..89 % of its two, the balancer 40 % of one, and about half of the
32 database connections wait for the lock on a chat row.
Smaller replicas cost more: at 3 040 rps the API burns 215 % CPU as
1x4, 243 % as 2x2 and about 280 % as 4x1, and p95 at 4 560 rps grows
from 9 to 69 and 150 ms.
The first attempt was aborted when a neighbour's image build froze the
Docker API; its summary is kept under aborted/. Twelve steps of the
rerun were rejected as disturbed and repeated. A config's SLO capacity
now needs a majority of saturated repeats, since a median is robust to
one disturbed repeat and "any repeat" is not. Series B starts at 450
rps per CPU.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
tools/plot_track1.py reads any set of summary.csv files and draws, per
series, successful RPS, p95 and p99 against the offered rate and CPU
per component against its limit, plus two cross-series charts: peak
successful RPS against total API CPU with the linear growth line, and
the utilisation of API, PostgreSQL, balancer and the share of the pool
waiting on a lock at each config's peak. A point is the median of the
repeats, the whiskers are min and max. capacity.csv and capacity.md
hold the SLO capacity, the peak and the resource figures at the peak.
Colours follow the config across charts, every line also has its own
marker and a direct label. matplotlib comes through uv run --with, so
the tool lives in tools/ and stays out of the 3.9 scripts; make plots
runs it, make lint covers it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
scripts/scaling.py applies a replicas x CPU layout through compose
variables: workers equal to the replica CPUs, a pool of 8 per worker,
a balancer queue of 32 per worker so every layout of a series offers
the same concurrency to the API, and the replicas pinned to CPUs 7 and
up, next to PostgreSQL on 0-1, the balancer on 2, Valkey on 3 and k6
on 4-6 with 3 CPU and 1 GiB. For every offered rate it runs k6 three
times and stops a config after two steps saturated in all repeats.
Each step keeps its raw data: k6 summary, docker stats, HAProxy
statistics and pg_stat_activity sampled every second, and the top of
pg_stat_statements. The summary row carries CPU per component,
PostgreSQL active, idle, idle in transaction and lock waiting backends,
the balancer queue and the database size. Several variants alternate
per repeat for before and after runs. host-snapshot.sh records every
container on the host before a series.
make track1-a and track1-b run the fixed budget and the growth series
from an empty database.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Track 1 compares replica layouts that must get exactly the same CPUs,
and the load generator, balancer, Valkey and PostgreSQL must not steal
cycles from the replicas or from each other. Every service now takes
an optional cpuset from LB_CPUSET, API_CPUSET, PG_CPUSET,
VALKEY_CPUSET and K6_CPUSET. They are empty by default and compose
drops an empty cpuset, so the ordinary stand is unchanged and make
limits still passes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Track 1 of grade 5 asks to compare one, two and four API replicas at a
fixed total CPU and memory budget, to grow resources until a shared
dependency saturates, to show with database, balancer and pool metrics
why more replicas stop helping, and to confirm one improvement with a
before and after measurement.
The hypotheses are stated before any run: at 4 CPU / 2 GiB the 1x4,
2x2 and 4x1 layouts reach the same ceiling within 15 percent; with one
CPU per replica throughput grows almost linearly up to four replicas
and by less than 1.5 times from four to eight, because PostgreSQL
saturates on the chat row lock held across three round trips and on
CPU, while the balancer stays far from its limit.
The new lab3-scaling-cost capability fixes the method: open-model k6
with the level 4 profile, three repeats per point, equal workers, pool
and balancer concurrency across layouts, CPU pinning per component,
raw samples per step, plots from summary files, and an improvement
chosen only after the bottleneck is found.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Grade 4 of lab 3 is closed. A replica on SIGTERM first answers 503 on
readiness for a two second window, longer than the 1.5 s worst
detection, and only then closes its listener and finishes accepted
requests within five seconds; the balancer lets a replica back only
after two passed checks. A test pins the timeout chain to compose,
HAProxy and the stop hook. make detection measured 0.71 s for SIGTERM
(before the process exited at 2.49 s), 0.67 s for a killed process,
2.76 s for a hung one and 1.23 s until no replica was left with
PostgreSQL down. make failover-each stopped and restored each replica
under k6 at 100 rps with no errors and no balancer retries, make
lb-check ran 20 same-key races of 8, make limits checks limits and
isolation against a policy, and make rolling-update replaced the
level 3 image with the current one under load with capacity kept at
two and no errors, migrate running once. The new
lab3-failover-operations spec is synced, and the lab 1 operability
and lab 2 dependency failure specs carry their deltas.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Evidence from a freshly recreated stand at the current code: the four
detection cases with their declared maxima, failover-each for both
replicas under k6 at 100 rps (no errors, no balancer retries, the
stopped replica never served during its outage), the 20 round
same-key race, the limits and isolation check with a run against
deliberate violations, and two rolling updates under load: from the
level 3 image, where the old replicas without a drain window caused
54 redispatches and 2 response errors that the balancer retried, and
between two current builds with all counters at zero. Level 3 runs
are refreshed on the new code: smoke, stack view, both failovers with
the drain sequence in the replica log, hooks with the longer stop,
persistence, stack fault tests and own tests with the shutdown and
timeout chain output.
The report gains the shutdown order, the check period and worst
detection per failure with measurements, the timeout chain, rows 4.1
to 4.4, level 4 rows of the failure table, the k6 role, the new
variables and the updated limitations; AGENTS.md lists the commands.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The repository ignores *.log, so the action log of a load run (with
the rolling update replacement lines) and the k6 output would never
reach evidence/. The probe now writes action.txt and k6/k6.txt.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
make limits, make lb-check (now 20 rounds of 8 parallel posts with one
key), make detection, make failover-each and make rolling-update each
write their file in evidence/ and fail on a violated check; make
level4 runs them in order. RATE sets the k6 arrival rate (100 rps by
default). make rolling-update builds the image before the load starts
so the build does not skew the measurement; with FROM_REV=<commit> it
first builds the image tag from git archive of that commit and
recreates the replicas on it, so the update is a real change of
version, and it writes to evidence/rolling-from-<commit>. Lint covers
scripts/limits.py. CI acceptance runs make limits and make detection.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The k6 profile from the prepared kit runs an open model against the
balancer from the edge network: constant arrival rate, 80 % keyed
message posts and 20 % history reads over 20 synthetic users and 10
chats, metrics only for the measure phase, and no setup data (tokens)
in summary.json. scripts/k6.sh runs it with the host uid so the
summary is writable on Linux too.
scripts/probe.py gains three things. check --rounds repeats the
same-key race with a new key per round and counts rounds with
identical bodies, one stored copy and 409 on another body. load runs
an action under k6 while a probe posts every 0.1 s and resends the
same key after a failure and a poller reads the number of UP servers;
it writes the action log, the probe and LB traces, the delta of the
backend retry, redispatch and error counters, and a verdict: dip for
a replica that leaves and returns, replace for a rolling update that
must keep capacity and end on new replicas only. detect runs the
action in the background and polls the balancer at the same time, so
a drain that ends after the exclusion is visible, and checks the
declared maximum, the LB answer after detection and the time to
restore.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The first failover-each run had no errors, yet the balancer counted
19 and 49 redispatches. The HAProxy log showed why: when a stopped
container comes back into Docker DNS, its server-template slot leaves
MAINT (resolution) as UP/READY at once, before any health check, so
requests went to a gunicorn that was not listening yet and were
redispatched to the other replica.
default-server now sets init-state fully-down (HAProxy 3.1+): at start,
on discovery through DNS and on leaving maintenance a server stays DOWN
until rise 2 checks pass. A rerun under 100 rps shows zero retries and
redispatches for both replicas; rejoining takes about a second longer.
The design and the failover spec of the change record the finding.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The API service now names every timeout that the chain depends on:
DRAIN_SECONDS, DB_ACQUIRE_TIMEOUT_SECONDS and DB_COMMAND_TIMEOUT_SECONDS
join REQUEST_TIMEOUT_SECONDS and SHUTDOWN_TIMEOUT_SECONDS in compose,
and the image gets the drain default. A unit test reads the compose
defaults, the HAProxy check and timeout settings and the stop hook
timeout, and checks that the chain shrinks inwards: LB server 10 s >
request 8 s > pool wait plus statement 3 s, readiness 0.8 s < check
1 s, drain 2 s > worst detection 1.5 s, shutdown 5 s leaves 3 s for
accepted requests, and shutdown < hook 6 s < stop_grace_period 8 s
< 10 s. It also pins the declared detection maxima: 1.5 s for a 503
or a refused connection and 3.5 s for a hung replica.
The k6 load generator joins the stand under the bench profile: pinned
by digest, edge network only, 2 CPU and 512 MiB, no secrets.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On SIGTERM gunicorn closed the listener at once, so a request already
queued on the closing socket reached the balancer as a dropped
connection after sending: a retry for GET and keyed messages, a 502
for user, chat and login posts. Now each worker first raises a
draining flag and keeps serving for DRAIN_SECONDS (2 s by default,
above the 1.5 s worst detection of a 503): readiness answers 503
unavailable, liveness stays 200, and the balancer takes the replica
out while it is still healthy. Only then uvicorn closes the listener
and finishes accepted requests within SHUTDOWN_TIMEOUT_SECONDS minus
the window, so gunicorn graceful_timeout stays 5 s.
The worker runs a uvicorn Server subclass whose main loop adds the
window after the exit signal; UvicornWorker._serve is overridden to
use it. Tests check readiness under the flag and run real gunicorn on
a free port: during the window ready is 503 and live 200, the port
closes after the window, a registration accepted before SIGTERM and
finished after the port closed still gets 201, and the master exits
with code 0 before the shutdown limit.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Level 4 of lab 3 asks for health checks, timeouts and SIGTERM handling
that agree with each other, a stated check period and worst detection
time, stopping every replica in turn under load with RPS, errors and
recovery time, a parallel race on one idempotency key, limits and
port and secret isolation, and a rolling update under load with
migrations run once.
The new lab3-failover-operations capability covers those guarantees.
The lab 1 operability delta makes a replica leave the rotation first
on SIGTERM: readiness answers 503 for a drain window longer than the
balancer detection time, and only then the listener closes. The lab 2
dependency failure delta states that the balancer drops every replica
and answers 503 itself while PostgreSQL is down. The design adds the
measurement tooling from the prepared kit: k6 under a bench profile,
probe load and detect modes, detection, failover-each, limits and
rolling update scripts with one make target each.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Grade 3 of lab 3 is closed: the stand runs two API replicas of one
compose service behind HAProxy as the only published entry point,
one token reaches both replicas by X-Instance-Id, stopping either
replica with scripts/stop-one.sh keeps writes going within 1.2-1.7 s
without errors or duplicates, eight concurrent posts with one key give
one message, and make test, the persistence scenario with
scripts/restart.sh, the failover scenario, lint, own tests, check,
validate and the lab 2 stack fault tests pass. The new
lab3-load-balancing and lab3-idempotent-messages specs are synced, and
the lab 1 history and operability specs and the lab 2 storage spec
carry their deltas.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Save sanitized outputs from a stand started from a clean state: the
clean clone run, smoke through the balancer, the stack view with the
replica hostnames, ports, networks, limits and the balancer server
states, the probe with two replicas behind one token and the eight
way race on one key, a failover run for each replica with the
container left exited and its graceful shutdown log, the hook
negative cases and the balancer answer without replicas, the
persistence run with StartedAt of every container before and after,
schema, migration rerun, KDF prefix, ignore rules, own tests and the
lab 2 stack fault tests through the balancer, plus the environment.
The report covers the clean clone start, the architecture, the
X-Instance-Id and idempotency decisions, the balancer settings, the
role to service table, the mapping of .env.example to the real
variables, one evidence row per level 3 criterion and the carried
lab 2 minimum, and the failure table for level 3.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
scripts/failover-evidence.sh lists the api containers by hostname, the
X-Instance-Id they answer with, before and after make test-failover,
names the container left exited, keeps its last log lines with the
gunicorn graceful shutdown, then restores it and lists the replicas
again. scripts/evidence-stack.sh records the compose config check,
published ports, the missing port of a replica, hostnames, networks,
limits, process users, the balancer view of every server and the
alternating X-Instance-Id through the balancer, plus the environment
and image versions. The persistence and storage evidence scripts now
list every replica and the idempotency key table.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
scripts/stop-one.sh maps TARGET_INSTANCE_ID to the api container whose
hostname matches, stops only it with docker stop -t 6 and leaves it
stopped; an unknown id exits 3 and touches nothing. scripts/restart.sh
replaces the lab 2 hook: it stops lb and the replicas, restarts
PostgreSQL and Valkey with their volumes and brings the stand back
with up --wait, which also reruns the idempotent migrate job.
scripts/restore.sh starts stopped replicas without recreating them
and scripts/lb-wait.sh waits until the balancer statistics show every
replica UP again, so the next failover run sees two replicas.
scripts/probe.py check (stdlib, Python 3.9 syntax) counts X-Instance-Id
over 40 calls with one token and races 8 posts with one key through
the balancer. make restore and make lb-check run them, ruff covers the
probe, and the CI acceptance job runs the failover test and the probe.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
compose.yaml now runs one api service with API_REPLICAS replicas and
no published ports, container name or hostname, behind lb, HAProxy
3.2.26 LTS pinned by digest and the only service with ports, bound to
127.0.0.1 (LB_HTTP_PORT for HTTP, LB_STATS_PORT for read-only stats).
The balancer finds replicas through Docker DNS with a server template,
checks GET /health/ready every second (fall 2, rise 2), balances round
robin without stickiness, retries connection failures on another
replica and allows a retry after a sent request only for GET and for
message posts carrying Idempotency-Key, never for users, chats or
login. Its own 502/503/504 answers use the contract error body with
the forwarded X-Request-Id.
The network is split into edge and an internal backend, every service
gets CPU and memory limits, the API runs read-only with a tmpfs /tmp,
two workers per replica, an 8 s request limit and a 5 s graceful stop
inside an 8 s stop_grace_period. Only api builds the image, migrate
reuses it once before the replicas start. PostgreSQL loads
pg_stat_statements and allows 200 connections. The short TTL instance
of lab 2 published the API around the balancer and is removed; its
expiry check stays as an in-process test.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
POST /api/v1/chats/{chat_id}/messages now requires the header, checked
at the HTTP boundary (1-128 of [A-Za-z0-9._:-], else 400). The handler
keeps the lab 2 order under the chat row lock: membership, then text,
then the receipt for (chat, sender, key) younger than
IDEMPOTENCY_TTL_SECONDS. A receipt with the same fingerprint replays
the stored message, a different one is 409 conflict; without a
receipt one statement inserts the message, the key row and the chat
mark, so both commit or neither does. Concurrent retries wait on the
same row lock and then read the committed receipt. The fingerprint is
SHA-256 of the canonical JSON of the parsed body, so layout does not
matter, and only 201 answers are ever stored.
A lock timeout now counts as a dependency failure and answers 503
instead of 500, so a retry with the same key is safe.
Own tests cover replay, a different layout of the same JSON, conflict,
missing and malformed keys, rejected attempts that leave the key free,
scope by user and chat, revalidation after logout and for outsiders,
expiry, and eight concurrent retries through two app instances
sharing one PostgreSQL. Every earlier message post sends a fresh key.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Migration 0002 stores one row per accepted message key: the chat, the
sender, the key, the SHA-256 fingerprint of the request and the
message. The primary key (chat_id, user_id, key) is the key scope of
the contract, the composite foreign key keeps the row inside the
messaging tables, as messages already do, and CHECK constraints repeat
the key format and the fingerprint length. Nothing existing changes,
so the lab 2 code keeps running on the new schema.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The platform settings default the instance id to the container
hostname, computed once per process, so every gunicorn worker of a
replica reports the same value and it matches docker ps. The request
pipeline stamps X-Instance-Id next to X-Request-Id on every response
it sends, and the unhandled error handler adds it itself, because the
500 answer is produced outside the pipeline.
Own tests cover 200, 204, 400, 401, 404, the hidden 405, 413 and 500
with a fixed instance id, and the hostname default. The application
version follows the contract version 1.3.0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Plan the grade 3 cycle of lab 3 on top of the carried-over lab 2
solution: run the API as several replicas of one compose service
behind HAProxy as the only published entry point, stamp every API
response with X-Instance-Id taken from the container hostname, require
Idempotency-Key on message posts and keep the key and the message in
one PostgreSQL transaction, and add explicit stop-one, restart and
restore hooks.
The design fixes the balancer settings and retry policy, the
idempotency table, the lookup order under the chat row lock, the
request fingerprint and the shutdown budget. Delta specs add the
lab3-load-balancing and lab3-idempotent-messages capabilities and
modify message posting, request identification, shutdown, container
delivery, migrations, restart and the reproducible stand.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Bring the Claude Code settings, the lint hook, the opsx commands and
skills, the architecture notes, the lab 1 and lab 2 specs and the
archived lab 1 and lab 2 changes into the lab 3 skeleton, so lab 3
continues the same spec history instead of starting from scratch.
Rewrite AGENTS.md and the openspec context and rules for lab 3:
HAProxy in front of several stateless API replicas, X-Instance-Id on
every response, a mandatory Idempotency-Key for messages shared by all
replicas through PostgreSQL, explicit stop-one and restart hooks, Out
of scope moved to labs 4-7 plus Swarm and HA storage, and rules on
retry policy, timeout chains and replica-local state.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>