The rejected LB pool variant is re-measured on committed code with a
temporary second HAProxy differing in http-reuse only: the replica, not
the LB, closed the last idle connection after 60.4 s. Exploratory
never/safe SIGTERM numbers are separated from evidence. hang-key cleanup
is shown under Ctrl+C and SIGTERM, bench_compare refuses a label without
the LB. Drain before SIGTERM and the run_docker recreate issue are handed
to rolling-update in PLAN.md and KNOWN_ISSUES.md; design gains decision 10.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Failover profile label baseline from 459f84c on a checked host: stop and
kill of each replica under 200 req/s give 0 errors and LB exclusion in
0.79-1.81 s (bound 2.5 s), pause gives LB exclusion in 5.83-6.00 s
(bound 6.5 s) with 504s for half the requests and recovery in 6 s;
reconciliation of retried writes matches. Evidence for detection,
keep-alive (LB closes idle replica connections at 10.1 s) and the hung
replica key lock, regression checks, timeout chain and level 4 rows in
REPORT.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The stand was recreated from a4ca9ef (make down, make run_docker) and
all evidence of the change was taken again on it: stand, distribution,
LB errors, test-persistence, test-failover for each replica (stop and
kill), stop-one refusals, make test and student_test through the LB,
test_isolated, failure_checks and bench self-checks.
New evidence closes scenarios verify found without proof: a real git
clone on non-default ports, a hung replica (docker pause: DOWN after
7.7 s, in-flight request 504), PostgreSQL outage behind the LB, replica
tests failing against one process, aborted responses via
driver.py kill-inflight. REPORT.md gets the corrected detection
formula, the measured cost of the 1 CPU limit for Argon2id and the new
rows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Evidence from 4e0cc4e: stand ports, limits and per-service env,
X-Instance-Id distribution and scaling, LB-generated errors,
test-persistence through the LB, test-failover for each replica in stop
and kill modes, stop-one refusals, test runs and bench self-checks.
REPORT.md gains the LB architecture, timeout chain, limits, criteria
rows and defence answers. Coverage exclusions for restart now point to
the lab 3 persistence evidence; stale lab 2 pointers go to
KNOWN_ISSUES.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- evidence for replay, conflict, replay through a new API process and
after restarting API and storages, keys table, key race,
X-Instance-Id in responses and logs, explicit and invalid
instance_id and key TTL, test runs
- stand evidence: make wipe_data leaves the lab 2 volume intact, build
from a fresh git clone passes smoke, bench restore stage sends with
Idempotency-Key
- REPORT.md: X-Instance-Id, idempotency design, IDEMPOTENCY_TTL_SECONDS
mapping, limitations and evidence rows
- tasks.md: mark the change tasks done, add follow-ups from verify
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>