make track1-compare builds the image of BEFORE_REV, the last commit
without the one-statement write, builds the current one, and runs 8x1
at 3 600, 5 400 and 8 100 rps with the two images alternating per
repeat, so drift of the shared VM affects both alike. make track1 runs
the host snapshot, both series, the comparison and the plots.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Sending a message used to be BEGIN, a SELECT ... FOR UPDATE of the
chat with its members, a lookup of the idempotency key, the insert CTE
and COMMIT. Three of those round trips ran under the chat row lock, and
each one waited for the replica's event loop, so under load the lock
holders sat idle in transaction while everyone else queued behind
them.
The new MessageLog port has one PostgreSQL adapter that does it all in
a single statement with an implicit transaction: lock the chat row
joined with the sender's membership, stamp the message with the later
of the replica clock and the last message plus one microsecond, claim
the key (overwriting it only if it has expired), insert the message
only when the key was claimed, and move last_message_at. The foreign
key from the key to the message is checked at the end of the statement,
so the key may be written first. When the key is already taken the
adapter reads the stored result with a second statement outside the
lock, which also covers a concurrent duplicate that waited for the
first one; the handler then replays it or answers 409 as before.
The chat aggregate no longer takes part in posting, so its post,
membership and last_message_at members and their unit tests are gone;
Message.posted() builds the event. An invalid text is still answered
404 for an outsider: only then the handler reads the membership. A new
API test pins the clock rule against a real database: a lagging
replica clock gets the last stamp plus one microsecond, a clock ahead
is kept. The concurrency, same-key race, expiry and access tests pass
unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
Series B located the bottleneck. At eight replicas the API has 40 % of
its CPU spare and the balancer 60 % of a core, yet three quarters of
the pool connections wait for a chat row lock: the locking SELECT of
the chat averages 14.2 ms and the lock holders sit idle in transaction
while the replica's event loop gets round to the next statement. The
critical section, three round trips long, is what more replicas
lengthen.
The design now records the choice: one CTE that locks the chat with
the membership check, stamps the message no earlier than the last one,
claims the idempotency key and inserts the message, committed by the
statement itself, so the lock never waits for the application. A taken
key is read back by a second statement outside the lock. The price is
that the chat aggregate no longer guards posting; the rules move into
the adapter's SQL and are held by integration tests. The rejected
alternatives and the before/after plan are listed, and the spec gains
the observable requirement of a short critical section.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
One to eight replicas of 1 CPU, 512 MiB and one worker each, three
repeats per rate; the 4x1 point comes from series A. Peak successful
RPS: 1 720 for one replica, 2 936 for two, 4 724 for four and only
4 260 for eight, lower than four. Up to two replicas the API is the
limit (101 % and 201 % of its CPU). At eight replicas the API uses 473
of its 800 % and the balancer 40 % of a core, while PostgreSQL is at
178 of 200 %, 47.5 of the 64 pool connections wait for a chat row lock
and the balancer queue holds 744 requests. The locking SELECT ... FOR
UPDATE of the chat takes 14.2 ms on average, 97 % of all statement
time of the step, and about 7 connections sit idle in transaction:
the lock holders wait for the application between round trips.
A universal scalability law fit over 1, 2, 4 and 8 CPUs gives
sigma 0.014 and kappa 0.038, a peak near five replicas and a decline
after it. No step of this series was rejected as disturbed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
1x4, 2x2 and 4x1 at 4 CPU and 2 GiB of API, workers equal to CPUs,
32 pool connections and 128 balancer slots in every layout, three
repeats from 600 to 6 830 rps. All three keep up with the offered
load up to 4 560 rps; the peaks are 5 375 rps for 1x4 (5 332..5 384),
4 964 for 2x2 (4 894..5 104) and 4 724 for 4x1 (4 686..4 794). At the
peak the API uses 95 % of its four CPUs in every layout, PostgreSQL
82..89 % of its two, the balancer 40 % of one, and about half of the
32 database connections wait for the lock on a chat row.
Smaller replicas cost more: at 3 040 rps the API burns 215 % CPU as
1x4, 243 % as 2x2 and about 280 % as 4x1, and p95 at 4 560 rps grows
from 9 to 69 and 150 ms.
The first attempt was aborted when a neighbour's image build froze the
Docker API; its summary is kept under aborted/. Twelve steps of the
rerun were rejected as disturbed and repeated. A config's SLO capacity
now needs a majority of saturated repeats, since a median is robust to
one disturbed repeat and "any repeat" is not. Series B starts at 450
rps per CPU.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
The first run of series A was spoiled by other stands on the same
Docker VM: an image build of another lab is invisible to docker stats,
yet it took CPU from the pinned replicas, doubled their CPU per request
and stalled the Docker API for fifteen minutes in the middle of two
steps.
docker-stats.sh now also samples the VM-wide /proc/stat counters, so a
step records the CPU of the whole VM, the part not used by this
project, the largest gap between samples and its wall time. A step
with more than one foreign core, a gap over 10 s or a k6 run 90 s
longer than its load is written to disturbed.csv and repeated after
the VM has been quiet for 15 s, up to three times. On a quiet VM the
foreign part is 2 % of a core at 600 rps and 25 % at 4 560 rps, which
is Docker networking and the samplers themselves.
The growth series starts at 300 rps per CPU and searches up to 9 000,
so its knees are found with fewer idle steps. The plots fit the
universal scalability law to the growth series.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
tools/plot_track1.py reads any set of summary.csv files and draws, per
series, successful RPS, p95 and p99 against the offered rate and CPU
per component against its limit, plus two cross-series charts: peak
successful RPS against total API CPU with the linear growth line, and
the utilisation of API, PostgreSQL, balancer and the share of the pool
waiting on a lock at each config's peak. A point is the median of the
repeats, the whiskers are min and max. capacity.csv and capacity.md
hold the SLO capacity, the peak and the resource figures at the peak.
Colours follow the config across charts, every line also has its own
marker and a direct label. matplotlib comes through uv run --with, so
the tool lives in tools/ and stays out of the 3.9 scripts; make plots
runs it, make lint covers it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
scripts/scaling.py applies a replicas x CPU layout through compose
variables: workers equal to the replica CPUs, a pool of 8 per worker,
a balancer queue of 32 per worker so every layout of a series offers
the same concurrency to the API, and the replicas pinned to CPUs 7 and
up, next to PostgreSQL on 0-1, the balancer on 2, Valkey on 3 and k6
on 4-6 with 3 CPU and 1 GiB. For every offered rate it runs k6 three
times and stops a config after two steps saturated in all repeats.
Each step keeps its raw data: k6 summary, docker stats, HAProxy
statistics and pg_stat_activity sampled every second, and the top of
pg_stat_statements. The summary row carries CPU per component,
PostgreSQL active, idle, idle in transaction and lock waiting backends,
the balancer queue and the database size. Several variants alternate
per repeat for before and after runs. host-snapshot.sh records every
container on the host before a series.
make track1-a and track1-b run the fixed budget and the growth series
from an empty database.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Track 1 compares replica layouts that must get exactly the same CPUs,
and the load generator, balancer, Valkey and PostgreSQL must not steal
cycles from the replicas or from each other. Every service now takes
an optional cpuset from LB_CPUSET, API_CPUSET, PG_CPUSET,
VALKEY_CPUSET and K6_CPUSET. They are empty by default and compose
drops an empty cpuset, so the ordinary stand is unchanged and make
limits still passes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Track 1 of grade 5 asks to compare one, two and four API replicas at a
fixed total CPU and memory budget, to grow resources until a shared
dependency saturates, to show with database, balancer and pool metrics
why more replicas stop helping, and to confirm one improvement with a
before and after measurement.
The hypotheses are stated before any run: at 4 CPU / 2 GiB the 1x4,
2x2 and 4x1 layouts reach the same ceiling within 15 percent; with one
CPU per replica throughput grows almost linearly up to four replicas
and by less than 1.5 times from four to eight, because PostgreSQL
saturates on the chat row lock held across three round trips and on
CPU, while the balancer stays far from its limit.
The new lab3-scaling-cost capability fixes the method: open-model k6
with the level 4 profile, three repeats per point, equal workers, pool
and balancer concurrency across layouts, CPU pinning per component,
raw samples per step, plots from summary files, and an improvement
chosen only after the bottleneck is found.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Grade 4 of lab 3 is closed. A replica on SIGTERM first answers 503 on
readiness for a two second window, longer than the 1.5 s worst
detection, and only then closes its listener and finishes accepted
requests within five seconds; the balancer lets a replica back only
after two passed checks. A test pins the timeout chain to compose,
HAProxy and the stop hook. make detection measured 0.71 s for SIGTERM
(before the process exited at 2.49 s), 0.67 s for a killed process,
2.76 s for a hung one and 1.23 s until no replica was left with
PostgreSQL down. make failover-each stopped and restored each replica
under k6 at 100 rps with no errors and no balancer retries, make
lb-check ran 20 same-key races of 8, make limits checks limits and
isolation against a policy, and make rolling-update replaced the
level 3 image with the current one under load with capacity kept at
two and no errors, migrate running once. The new
lab3-failover-operations spec is synced, and the lab 1 operability
and lab 2 dependency failure specs carry their deltas.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Evidence from a freshly recreated stand at the current code: the four
detection cases with their declared maxima, failover-each for both
replicas under k6 at 100 rps (no errors, no balancer retries, the
stopped replica never served during its outage), the 20 round
same-key race, the limits and isolation check with a run against
deliberate violations, and two rolling updates under load: from the
level 3 image, where the old replicas without a drain window caused
54 redispatches and 2 response errors that the balancer retried, and
between two current builds with all counters at zero. Level 3 runs
are refreshed on the new code: smoke, stack view, both failovers with
the drain sequence in the replica log, hooks with the longer stop,
persistence, stack fault tests and own tests with the shutdown and
timeout chain output.
The report gains the shutdown order, the check period and worst
detection per failure with measurements, the timeout chain, rows 4.1
to 4.4, level 4 rows of the failure table, the k6 role, the new
variables and the updated limitations; AGENTS.md lists the commands.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The failover evidence printed the last six log lines of the stopped
replica, which no longer reach back to the signal now that shutdown
starts with a two second drain window. It now filters the replica log
since the start of the run for the signal, the drain message, the 503
readiness answers, the listener shutdown and the process exit.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The repository ignores *.log, so the action log of a load run (with
the rolling update replacement lines) and the k6 output would never
reach evidence/. The probe now writes action.txt and k6/k6.txt.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
make limits, make lb-check (now 20 rounds of 8 parallel posts with one
key), make detection, make failover-each and make rolling-update each
write their file in evidence/ and fail on a violated check; make
level4 runs them in order. RATE sets the k6 arrival rate (100 rps by
default). make rolling-update builds the image before the load starts
so the build does not skew the measurement; with FROM_REV=<commit> it
first builds the image tag from git archive of that commit and
recreates the replicas on it, so the update is a real change of
version, and it writes to evidence/rolling-from-<commit>. Lint covers
scripts/limits.py. CI acceptance runs make limits and make detection.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The stack test for a stopped PostgreSQL asked for liveness 200 through
the balancer, which only held while the balancer had not noticed the
outage yet. Now the protected endpoints and readiness through the LB
must answer 503 unavailable within 2 s, liveness is checked inside a
replica with docker exec, the balancer must report zero UP servers
within 2 s and answer liveness itself with 503 in the contract error
format, the request id echoed and no X-Instance-Id. After PostgreSQL
starts again every replica must be back in rotation without a restart.
Stand helpers read the number of UP servers from the HAProxy stats,
excluding the backend summary row, and reach a replica by container
id, which survives the renumbering after a rolling update.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
scripts/rolling-update.sh runs migrate once before touching any
replica, then for each old replica scales the service to N+1 without
recreating the others, waits until the new container is healthy and
the balancer sees N+1 UP servers, and only then stops the old one
with docker stop -t 6 (so it drains) and removes it. The log names
the hostname and image of the old and the new replica of every step,
which is what the report quotes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
scripts/limits.py states which secrets, volumes and networks each
service needs and that only lb publishes ports, on loopback. It
compares that policy with docker compose config (with the bench
profile, so k6 is covered) and with docker inspect of the running
containers: CPU and memory limits declared and applied, no
container_name, no published port on any API replica, a read-only
root filesystem for the API, and which networks are internal. It
prints both tables and every violation and exits non-zero on any.
The stack evidence script no longer reaches replicas through
docker compose exec or port, which assume container number 1 and
break after a rolling update renumbers the replicas.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
scripts/detection.sh runs four failures one after another and brings
the stand back between them: SIGTERM through docker stop -t 6 (the
balancer must exclude the replica within 1.5 s and before the process
exits after its drain window), docker kill (1.5 s), docker pause for a
hung process (3.5 s, the check timeout counts twice) and stopping
PostgreSQL (all replicas out within 1.5 s, the balancer answers 503
itself). A 0.25 s slack covers the docker CLI and the 50 ms polling.
Each run writes evidence/detection/<case>.json and the script fails
if any case exceeds its declared maximum.
scripts/failover-each.sh stops every replica in turn with stop-one.sh,
keeps it down for 10 s and restores it, all inside one k6 load window,
and collects a summary with RPS, errors, latency, detection, rejoin,
probe recovery and balancer retry counters per replica.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The k6 profile from the prepared kit runs an open model against the
balancer from the edge network: constant arrival rate, 80 % keyed
message posts and 20 % history reads over 20 synthetic users and 10
chats, metrics only for the measure phase, and no setup data (tokens)
in summary.json. scripts/k6.sh runs it with the host uid so the
summary is writable on Linux too.
scripts/probe.py gains three things. check --rounds repeats the
same-key race with a new key per round and counts rounds with
identical bodies, one stored copy and 409 on another body. load runs
an action under k6 while a probe posts every 0.1 s and resends the
same key after a failure and a poller reads the number of UP servers;
it writes the action log, the probe and LB traces, the delta of the
backend retry, redispatch and error counters, and a verdict: dip for
a replica that leaves and returns, replace for a rolling update that
must keep capacity and end on new replicas only. detect runs the
action in the background and polls the balancer at the same time, so
a drain that ends after the exclusion is visible, and checks the
declared maximum, the LB answer after detection and the time to
restore.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The first failover-each run had no errors, yet the balancer counted
19 and 49 redispatches. The HAProxy log showed why: when a stopped
container comes back into Docker DNS, its server-template slot leaves
MAINT (resolution) as UP/READY at once, before any health check, so
requests went to a gunicorn that was not listening yet and were
redispatched to the other replica.
default-server now sets init-state fully-down (HAProxy 3.1+): at start,
on discovery through DNS and on leaving maintenance a server stays DOWN
until rise 2 checks pass. A rerun under 100 rps shows zero retries and
redispatches for both replicas; rejoining takes about a second longer.
The design and the failover spec of the change record the finding.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The API service now names every timeout that the chain depends on:
DRAIN_SECONDS, DB_ACQUIRE_TIMEOUT_SECONDS and DB_COMMAND_TIMEOUT_SECONDS
join REQUEST_TIMEOUT_SECONDS and SHUTDOWN_TIMEOUT_SECONDS in compose,
and the image gets the drain default. A unit test reads the compose
defaults, the HAProxy check and timeout settings and the stop hook
timeout, and checks that the chain shrinks inwards: LB server 10 s >
request 8 s > pool wait plus statement 3 s, readiness 0.8 s < check
1 s, drain 2 s > worst detection 1.5 s, shutdown 5 s leaves 3 s for
accepted requests, and shutdown < hook 6 s < stop_grace_period 8 s
< 10 s. It also pins the declared detection maxima: 1.5 s for a 503
or a refused connection and 3.5 s for a hung replica.
The k6 load generator joins the stand under the bench profile: pinned
by digest, edge network only, 2 CPU and 512 MiB, no secrets.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On SIGTERM gunicorn closed the listener at once, so a request already
queued on the closing socket reached the balancer as a dropped
connection after sending: a retry for GET and keyed messages, a 502
for user, chat and login posts. Now each worker first raises a
draining flag and keeps serving for DRAIN_SECONDS (2 s by default,
above the 1.5 s worst detection of a 503): readiness answers 503
unavailable, liveness stays 200, and the balancer takes the replica
out while it is still healthy. Only then uvicorn closes the listener
and finishes accepted requests within SHUTDOWN_TIMEOUT_SECONDS minus
the window, so gunicorn graceful_timeout stays 5 s.
The worker runs a uvicorn Server subclass whose main loop adds the
window after the exit signal; UvicornWorker._serve is overridden to
use it. Tests check readiness under the flag and run real gunicorn on
a free port: during the window ready is 503 and live 200, the port
closes after the window, a registration accepted before SIGTERM and
finished after the port closed still gets 201, and the master exits
with code 0 before the shutdown limit.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The readiness ping waited up to one second for a pooled connection and
one more second for SELECT 1, while HAProxy gives a check one second.
With a busy pool the balancer saw a check timeout instead of an
explicit 503. The ping now waits 0.4 s for a connection and 0.4 s for
the query, so the worst answer, 0.8 s, still arrives inside the check.
Other queries keep the pool acquire and command timeouts.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Level 4 of lab 3 asks for health checks, timeouts and SIGTERM handling
that agree with each other, a stated check period and worst detection
time, stopping every replica in turn under load with RPS, errors and
recovery time, a parallel race on one idempotency key, limits and
port and secret isolation, and a rolling update under load with
migrations run once.
The new lab3-failover-operations capability covers those guarantees.
The lab 1 operability delta makes a replica leave the rotation first
on SIGTERM: readiness answers 503 for a drain window longer than the
balancer detection time, and only then the listener closes. The lab 2
dependency failure delta states that the balancer drops every replica
and answers 503 itself while PostgreSQL is down. The design adds the
measurement tooling from the prepared kit: k6 under a bench profile,
probe load and detect modes, detection, failover-each, limits and
rolling update scripts with one make target each.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Grade 3 of lab 3 is closed: the stand runs two API replicas of one
compose service behind HAProxy as the only published entry point,
one token reaches both replicas by X-Instance-Id, stopping either
replica with scripts/stop-one.sh keeps writes going within 1.2-1.7 s
without errors or duplicates, eight concurrent posts with one key give
one message, and make test, the persistence scenario with
scripts/restart.sh, the failover scenario, lint, own tests, check,
validate and the lab 2 stack fault tests pass. The new
lab3-load-balancing and lab3-idempotent-messages specs are synced, and
the lab 1 history and operability specs and the lab 2 storage spec
carry their deltas.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Save sanitized outputs from a stand started from a clean state: the
clean clone run, smoke through the balancer, the stack view with the
replica hostnames, ports, networks, limits and the balancer server
states, the probe with two replicas behind one token and the eight
way race on one key, a failover run for each replica with the
container left exited and its graceful shutdown log, the hook
negative cases and the balancer answer without replicas, the
persistence run with StartedAt of every container before and after,
schema, migration rerun, KDF prefix, ignore rules, own tests and the
lab 2 stack fault tests through the balancer, plus the environment.
The report covers the clean clone start, the architecture, the
X-Instance-Id and idempotency decisions, the balancer settings, the
role to service table, the mapping of .env.example to the real
variables, one evidence row per level 3 criterion and the carried
lab 2 minimum, and the failure table for level 3.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
After make up, which builds with --build, a plain up in the restart
hook saw a different service hash and recreated migrate and the API
replicas instead of starting the stopped ones, so the replicas came
back with new hostnames. The hook now passes --no-recreate: lb and the
replicas are the same containers restarted, migrate reruns in its old
container, and only volumes and data carry state across the restart.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
scripts/failover-evidence.sh lists the api containers by hostname, the
X-Instance-Id they answer with, before and after make test-failover,
names the container left exited, keeps its last log lines with the
gunicorn graceful shutdown, then restores it and lists the replicas
again. scripts/evidence-stack.sh records the compose config check,
published ports, the missing port of a replica, hostnames, networks,
limits, process users, the balancer view of every server and the
alternating X-Instance-Id through the balancer, plus the environment
and image versions. The persistence and storage evidence scripts now
list every replica and the idempotency key table.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
scripts/stop-one.sh maps TARGET_INSTANCE_ID to the api container whose
hostname matches, stops only it with docker stop -t 6 and leaves it
stopped; an unknown id exits 3 and touches nothing. scripts/restart.sh
replaces the lab 2 hook: it stops lb and the replicas, restarts
PostgreSQL and Valkey with their volumes and brings the stand back
with up --wait, which also reruns the idempotent migrate job.
scripts/restore.sh starts stopped replicas without recreating them
and scripts/lb-wait.sh waits until the balancer statistics show every
replica UP again, so the next failover run sees two replicas.
scripts/probe.py check (stdlib, Python 3.9 syntax) counts X-Instance-Id
over 40 calls with one token and races 8 posts with one key through
the balancer. make restore and make lb-check run them, ruff covers the
probe, and the CI acceptance job runs the failover test and the probe.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
compose.yaml now runs one api service with API_REPLICAS replicas and
no published ports, container name or hostname, behind lb, HAProxy
3.2.26 LTS pinned by digest and the only service with ports, bound to
127.0.0.1 (LB_HTTP_PORT for HTTP, LB_STATS_PORT for read-only stats).
The balancer finds replicas through Docker DNS with a server template,
checks GET /health/ready every second (fall 2, rise 2), balances round
robin without stickiness, retries connection failures on another
replica and allows a retry after a sent request only for GET and for
message posts carrying Idempotency-Key, never for users, chats or
login. Its own 502/503/504 answers use the contract error body with
the forwarded X-Request-Id.
The network is split into edge and an internal backend, every service
gets CPU and memory limits, the API runs read-only with a tmpfs /tmp,
two workers per replica, an 8 s request limit and a 5 s graceful stop
inside an 8 s stop_grace_period. Only api builds the image, migrate
reuses it once before the replicas start. PostgreSQL loads
pg_stat_statements and allows 200 connections. The short TTL instance
of lab 2 published the API around the balancer and is removed; its
expiry check stays as an in-process test.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
POST /api/v1/chats/{chat_id}/messages now requires the header, checked
at the HTTP boundary (1-128 of [A-Za-z0-9._:-], else 400). The handler
keeps the lab 2 order under the chat row lock: membership, then text,
then the receipt for (chat, sender, key) younger than
IDEMPOTENCY_TTL_SECONDS. A receipt with the same fingerprint replays
the stored message, a different one is 409 conflict; without a
receipt one statement inserts the message, the key row and the chat
mark, so both commit or neither does. Concurrent retries wait on the
same row lock and then read the committed receipt. The fingerprint is
SHA-256 of the canonical JSON of the parsed body, so layout does not
matter, and only 201 answers are ever stored.
A lock timeout now counts as a dependency failure and answers 503
instead of 500, so a retry with the same key is safe.
Own tests cover replay, a different layout of the same JSON, conflict,
missing and malformed keys, rejected attempts that leave the key free,
scope by user and chat, revalidation after logout and for outsiders,
expiry, and eight concurrent retries through two app instances
sharing one PostgreSQL. Every earlier message post sends a fresh key.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Migration 0002 stores one row per accepted message key: the chat, the
sender, the key, the SHA-256 fingerprint of the request and the
message. The primary key (chat_id, user_id, key) is the key scope of
the contract, the composite foreign key keeps the row inside the
messaging tables, as messages already do, and CHECK constraints repeat
the key format and the fingerprint length. Nothing existing changes,
so the lab 2 code keeps running on the new schema.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The platform settings default the instance id to the container
hostname, computed once per process, so every gunicorn worker of a
replica reports the same value and it matches docker ps. The request
pipeline stamps X-Instance-Id next to X-Request-Id on every response
it sends, and the unhandled error handler adds it itself, because the
500 answer is produced outside the pipeline.
Own tests cover 200, 204, 400, 401, 404, the hidden 405, 413 and 500
with a fixed instance id, and the hostname default. The application
version follows the contract version 1.3.0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Plan the grade 3 cycle of lab 3 on top of the carried-over lab 2
solution: run the API as several replicas of one compose service
behind HAProxy as the only published entry point, stamp every API
response with X-Instance-Id taken from the container hostname, require
Idempotency-Key on message posts and keep the key and the message in
one PostgreSQL transaction, and add explicit stop-one, restart and
restore hooks.
The design fixes the balancer settings and retry policy, the
idempotency table, the lookup order under the chat row lock, the
request fingerprint and the shutdown budget. Delta specs add the
lab3-load-balancing and lab3-idempotent-messages capabilities and
modify message posting, request identification, shutdown, container
delivery, migrations, restart and the reproducible stand.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Bring the application, own tests, Dockerfile, toolchain, compose
stack, migrations, secret generator, restart hook, evidence scripts and
CI job from the lab 2 solution commit 8cddb09 unchanged, merge the lab
2 Makefile targets into the lab 3 skeleton Makefile without touching
its targets, and append the lab 2 solution variables to the skeleton
.env.example. The package version follows the contract version 1.3.0.
This is the starting point of the lab 3 cycle: lint, own tests and the
offline skeleton check pass before any lab 3 change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Bring the Claude Code settings, the lint hook, the opsx commands and
skills, the architecture notes, the lab 1 and lab 2 specs and the
archived lab 1 and lab 2 changes into the lab 3 skeleton, so lab 3
continues the same spec history instead of starting from scratch.
Rewrite AGENTS.md and the openspec context and rules for lab 3:
HAProxy in front of several stateless API replicas, X-Instance-Id on
every response, a mandatory Idempotency-Key for messages shared by all
replicas through PostgreSQL, explicit stop-one and restart hooks, Out
of scope moved to labs 4-7 plus Swarm and HA storage, and rules on
retry policy, timeout chains and replica-local state.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>