Run bench/sampler.sh in a new scan mode (cgroup walk, no network, no
psql or LB polling) for the whole load of the failover and rollout
profiles. bench/resources.py turns the samples into per-window CPU,
throttling and memory of every component and the VM, with
samples.csv, resources.csv and summary sections; CPU per successful
request only in steady windows. Check the peak CPU budget before the
stand starts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sync load-benchmark (series comparison table, track 1 before/after
rules, db4 series) and service-configuration (default pool 10 + 0)
into the main specs, move the change to the archive and drop it from
the transitional state in AGENTS.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fix the two-step threshold (x1.118), CPU-per-request spread and
PostgreSQL ranges in REPORT.md and design, record the LB side check
(lb_chkdown, error_codes) in evidence, add the db4 series to the
load-benchmark delta, refresh AGENTS.md, bench.md, PLAN.md and
proposal, and log the swapouts_delta passport defect.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Configs may carry a fifth element, the PostgreSQL CPU limit, which goes
into the config key (-pg<CPU>), the budget and the cell stand. Series db:
1, 2, 4 replicas at the tight limit and a 4 x 1 control at 2 CPU; label
summaries show PostgreSQL CPU at R* against its limit and throttling.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Close the 53300 known issue, restate H1 and saturation conclusions
against the pre-registered criteria and grid resolution, name the
unresolved host mechanism, attribute overload 503 to the pool timeout,
and plan a fresh A/A for scaling-improvement.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SQLSTATE 53300 (too_many_connections) joins UNAVAILABLE_SQLSTATES, so a
refused pool connection gives 503 unavailable instead of 500. New fault
check fills max_connections through the failure stand hook and expects
503 in the contract format.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Series a/b/pool/ab/quick in bench/run.py: per-cell fresh stand with the
configuration's replicas and max_connections, open-model mixed profile
(k6/mixed.js), fixed levels and R* search by SLO, a sampler container
for cgroup, VM, HAProxy and PostgreSQL metrics, resume and a stand
lock. Pure helpers in bench/scaling.py, SVG plots in bench/plots.py,
series mode for bench_compare and bench_ab. Hypotheses H1-H3 go to
REPORT.md before the first series.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Track 1 "cost of scaling": open-model series A (fixed 4 CPU budget),
B (1-6 replicas), pool border (max_connections 100 vs 50) and A/A,
sampler of cgroup/VM/LB/PostgreSQL metrics, generator limits, series
mode for bench_compare and bench_ab. Hypotheses H1-H3 are recorded in
the design before any series run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sync the rolling-update delta specs into db-migrations,
graceful-shutdown, load-balancing and load-benchmark, move the change to
the archive, mark it in PLAN.md and update the transitional state and
code map in AGENTS.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re-run the compatibility walk-through, the script failure cases with a
real Ctrl+C through a pseudo-terminal, the stand lifecycle after
replacement and the quick rollout and failover self-checks on 84161c0
with a clean tree; update the report rows and limitations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With lock_timeout equal to statement_timeout (5 s) every request queued
behind a migration waiting for its lock failed with 503 together with
the migration. A 2 s lock timeout lets the queue proceed once the
migration gives up.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reset the new replica ID before entering the starting state and let the
cleanup find a replica created by an interrupted compose scale; refuse
to start while LB slots keep a manual drain or maint state; log the exit
code of each stopped old replica and print full LB commands for manual
cleanup. Document the migrate step and the rollback tag limitation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A one-off run --rm left the migrate container of the previous image in
the stand; scripts/restart.sh then started it as a dependency of api and
the old migrations failed on the newer schema, so test-persistence broke.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Old image passes smoke and student tests on the new schema, rolling
update, rollback and roll forward under tests/load.py give zero failed
requests; the run_docker known issue is not reproduced and is closed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scripts/rolling-update.sh (make rolling_update) replaces API replicas one
by one: build, migrations once, new replica UP in HAProxy, old replica
drained and put into maintenance through the LB runtime API on loopback,
then stopped and removed; the slot is reset once DNS drops the address.
Expand migration adds messages.instance_id with a lock timeout. The bench
rollout profile runs the same script under constant load, starting from
the image of an older commit built with git archive.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Delta specs synced into graceful-shutdown, load-balancing, load-benchmark
and service-configuration; the change moves to
openspec/changes/archive/2026-10-11-graceful-failover/. PLAN.md marks the
change archived and AGENTS.md notes what was handed to rolling-update.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The verify fix in bench/compare.py refuses a label measured without the
load balancer; the load-benchmark delta now modifies the label comparison
requirement accordingly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rejected LB pool variant is re-measured on committed code with a
temporary second HAProxy differing in http-reuse only: the replica, not
the LB, closed the last idle connection after 60.4 s. Exploratory
never/safe SIGTERM numbers are separated from evidence. hang-key cleanup
is shown under Ctrl+C and SIGTERM, bench_compare refuses a label without
the LB. Drain before SIGTERM and the run_docker recreate issue are handed
to rolling-update in PLAN.md and KNOWN_ISSUES.md; design gains decision 10.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Failover profile label baseline from 459f84c on a checked host: stop and
kill of each replica under 200 req/s give 0 errors and LB exclusion in
0.79-1.81 s (bound 2.5 s), pause gives LB exclusion in 5.83-6.00 s
(bound 6.5 s) with 504s for half the requests and recovery in 6 s;
reconciliation of retried writes matches. Evidence for detection,
keep-alive (LB closes idle replica connections at 10.1 s) and the hung
replica key lock, regression checks, timeout chain and level 4 rows in
REPORT.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HAProxy checks get fastinter 500ms and timeout check 2.5s (hung replica
detected within 6.5 s), http-reuse never with uvicorn keep-alive 60 s so
the LB closes idle replica connections, and app connections set
idle_in_transaction_session_timeout so a frozen replica releases its
idempotency key. The bench stand runs behind HAProxy with N replicas and
gains the failover profile (make bench_failover / bench_failover_quick):
constant-rate writes with a key and reads while each replica is stopped,
killed or paused, with per-second timeline, detection and recovery time
and write reconciliation. Driver commands idle-conns and hang-key check
the keep-alive and hung-transaction decisions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main specs were already in sync with the deltas (sync re-run, no
changes). PLAN.md marks stage 2 archived and points graceful-failover
at the corrected detection formula; AGENTS.md notes what of the timeout
chain is measured so far and the new replicas evidence.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The stand was recreated from a4ca9ef (make down, make run_docker) and
all evidence of the change was taken again on it: stand, distribution,
LB errors, test-persistence, test-failover for each replica (stop and
kill), stop-one refusals, make test and student_test through the LB,
test_isolated, failure_checks and bench self-checks.
New evidence closes scenarios verify found without proof: a real git
clone on non-default ports, a hung replica (docker pause: DOWN after
7.7 s, in-flight request 504), PostgreSQL outage behind the LB, replica
tests failing against one process, aborted responses via
driver.py kill-inflight. REPORT.md gets the corrected detection
formula, the measured cost of the 1 CPU limit for Argon2id and the new
rows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- stop-one.sh: an empty TARGET_INSTANCE_ID now exits 2 in any shell,
as the design states (${VAR:?} gave 1 under bash).
- driver.py kill-inflight: reproducible "aborted response is not
retried" scenario (logins in flight, replica killed via stop-one.sh).
- Detection time of a hung replica is fall x (inter + timeout check) =
8 s, not inter x fall + timeout check = 5 s (docker pause: 7.3 s);
fixed in haproxy.cfg, balancer.md and the design.
- The 1 CPU replica limit is kept on purpose; the compose comment and
the design now say that Argon2id uses more and logins are slower.
- load-balancing spec: API_REPLICAS is read by compose and Makefile.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Evidence from 4e0cc4e: stand ports, limits and per-service env,
X-Instance-Id distribution and scaling, LB-generated errors,
test-persistence through the LB, test-failover for each replica in stop
and kill modes, stop-one refusals, test runs and bench self-checks.
REPORT.md gains the LB architecture, timeout chain, limits, criteria
rows and defence answers. Coverage exclusions for restart now point to
the lab 3 persistence evidence; stale lab 2 pointers go to
KNOWN_ISSUES.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without init-state down HAProxy marked a slot UP as soon as Docker DNS
returned the container address, before uvicorn listened, so requests
after LB restart or scale-up hit a replica that was not ready.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>