Commit Graph
42 Commits
Author SHA1 Message Date
ricoromaandClaude Opus 5.5 1694b1d0ae docs: add scaling-improvement comparisons, connection cost and checks
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 15:46:13 +03:00
ricoromaandClaude Opus 5.5 1bb8ea67ad feat: add two-cell db4 series for the scaling-improvement measurements
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 14:06:57 +03:00
ricoromaandClaude Opus 5.5 0b0421ad2f docs: record track 1 improvement hypothesis H4
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 13:23:49 +03:00
ricoromaandClaude Opus 5.5 1e50a4a3fc feat: compare PostgreSQL sessions and connections in track 1 series
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 13:23:12 +03:00
ricoromaandClaude Opus 5.5 029aa3be82 docs: propose scaling-improvement change
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 13:19:56 +03:00
ricoromaandClaude Opus 5.5 bbd6f65316 chore: archive scaling-bench change
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 12:25:18 +03:00
ricoromaandClaude Opus 5.5 dedb8ba0fa docs: report PostgreSQL saturation series db and rubric answer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 12:24:17 +03:00
ricoromaandClaude Opus 5.5 46e8f75a2c docs: record series db calibration in scaling-bench design
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 11:31:39 +03:00
ricoromaandClaude Opus 5.5 95539a1883 feat: add track 1 series db with a tight PostgreSQL CPU limit
Configs may carry a fifth element, the PostgreSQL CPU limit, which goes
into the config key (-pg<CPU>), the budget and the cell stand. Series db:
1, 2, 4 replicas at the tight limit and a 4 x 1 control at 2 CPU; label
summaries show PostgreSQL CPU at R* against its limit and throttling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 11:27:30 +03:00
ricoromaandClaude Opus 5.5 a3279374f7 docs: fix scaling-bench verify findings in report, design and rules
Close the 53300 known issue, restate H1 and saturation conclusions
against the pre-registered criteria and grid resolution, name the
unresolved host mechanism, attribute overload 503 to the pool timeout,
and plan a fresh A/A for scaling-improvement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 07:36:33 +03:00
ricoromaandClaude Opus 5.5 68edc66742 fix: answer 503 when PostgreSQL refuses connections with too many clients
SQLSTATE 53300 (too_many_connections) joins UNAVAILABLE_SQLSTATES, so a
refused pool connection gives 503 unavailable instead of 500. New fault
check fills max_connections through the failure stand hook and expects
503 in the contract format.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 07:26:36 +03:00
ricoromaandClaude Opus 5.5 ca0817390f docs: report track 1 scaling series
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 06:49:49 +03:00
ricoromaandClaude Opus 5.5 cbe3b4a43b docs: add scaling-bench self-checks
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 04:10:02 +03:00
ricoromaandClaude Opus 5.5 712d2450e4 fix: preallocate all k6 VUs in track 1 steps so the generator does not drop iterations near R*
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 03:55:21 +03:00
ricoromaandClaude Opus 5.5 02fd3d8a36 feat: add track 1 scaling series to the bench stand
Series a/b/pool/ab/quick in bench/run.py: per-cell fresh stand with the
configuration's replicas and max_connections, open-model mixed profile
(k6/mixed.js), fixed levels and R* search by SLO, a sampler container
for cgroup, VM, HAProxy and PostgreSQL metrics, resume and a stand
lock. Pure helpers in bench/scaling.py, SVG plots in bench/plots.py,
series mode for bench_compare and bench_ab. Hypotheses H1-H3 go to
REPORT.md before the first series.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 03:28:36 +03:00
ricoromaandClaude Opus 5.5 525ace7c9b docs: propose scaling-bench change
Track 1 "cost of scaling": open-model series A (fixed 4 CPU budget),
B (1-6 replicas), pool border (max_connections 100 vs 50) and A/A,
sampler of cgroup/VM/LB/PostgreSQL metrics, generator limits, series
mode for bench_compare and bench_ab. Hypotheses H1-H3 are recorded in
the design before any series run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 02:48:40 +03:00
ricoromaandClaude Opus 5.5 84ee13ca5a chore: archive rolling-update change
Sync the rolling-update delta specs into db-migrations,
graceful-shutdown, load-balancing and load-benchmark, move the change to
the archive, mark it in PLAN.md and update the transitional state and
code map in AGENTS.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 01:59:19 +03:00
ricoromaandClaude Opus 5.5 6f85d21f3b docs: retake rolling-update evidence on the final verify commit
Re-run the compatibility walk-through, the script failure cases with a
real Ctrl+C through a pseudo-terminal, the stand lifecycle after
replacement and the quick rollout and failover self-checks on 84161c0
with a clean tree; update the report rows and limitations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 01:56:30 +03:00
ricoromaandClaude Opus 5.5 84161c033a fix: keep migration lock timeout below the app statement timeout
With lock_timeout equal to statement_timeout (5 s) every request queued
behind a migration waiting for its lock failed with 503 together with
the migration. A 2 s lock timeout lets the queue proceed once the
migration gives up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 01:38:10 +03:00
ricoromaandClaude Opus 5.5 5e102e6d0c fix: harden rolling update cleanup and preconditions found in verify
Reset the new replica ID before entering the starting state and let the
cleanup find a replica created by an interrupted compose scale; refuse
to start while LB slots keep a manual drain or maint state; log the exit
code of each stopped old replica and print full LB commands for manual
cleanup. Document the migrate step and the rollback tag limitation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 01:33:33 +03:00
ricoromaandClaude Opus 5.5 69ebf5cd38 docs: add rolling-update baseline measurement and report
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 01:21:34 +03:00
ricoromaandClaude Opus 5.5 9dcea0b2b8 docs: add rolling-update regression evidence and report sections
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 01:17:34 +03:00
ricoromaandClaude Opus 5.5 de654a1c83 fix: recreate the migrate service container during rolling update
A one-off run --rm left the migrate container of the previous image in
the stand; scripts/restart.sh then started it as a dependency of api and
the old migrations failed on the newer schema, so test-persistence broke.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 01:04:50 +03:00
ricoromaandClaude Opus 5.5 5403464cfb docs: add rolling-update compatibility and script failure evidence
Old image passes smoke and student tests on the new schema, rolling
update, rollback and roll forward under tests/load.py give zero failed
requests; the run_docker known issue is not reproduced and is closed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 01:01:33 +03:00
ricoromaandClaude Opus 5.5 d2148ac566 feat: add rolling update with LB drain and rollout load profile
scripts/rolling-update.sh (make rolling_update) replaces API replicas one
by one: build, migrations once, new replica UP in HAProxy, old replica
drained and put into maintenance through the LB runtime API on loopback,
then stopped and removed; the slot is reset once DNS drops the address.
Expand migration adds messages.instance_id with a lock timeout. The bench
rollout profile runs the same script under constant load, starting from
the image of an older commit built with git archive.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 00:49:50 +03:00
ricoromaandClaude Opus 5.5 9a00795de1 docs: propose rolling-update change
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 00:26:36 +03:00
ricoromaandClaude Opus 5.5 c50b7211b8 chore: archive graceful-failover change
Delta specs synced into graceful-shutdown, load-balancing, load-benchmark
and service-configuration; the change moves to
openspec/changes/archive/2026-10-11-graceful-failover/. PLAN.md marks the
change archived and AGENTS.md notes what was handed to rolling-update.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 00:10:35 +03:00
ricoromaandClaude Opus 5.5 fc715b74d4 docs: add LB rule of bench_compare to graceful-failover delta
The verify fix in bench/compare.py refuses a label measured without the
load balancer; the load-benchmark delta now modifies the label comparison
requirement accordingly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 00:08:13 +03:00
ricoromaandClaude Opus 5.5 f38a56c4ea docs: retake graceful-failover checks after verify
The rejected LB pool variant is re-measured on committed code with a
temporary second HAProxy differing in http-reuse only: the replica, not
the LB, closed the last idle connection after 60.4 s. Exploratory
never/safe SIGTERM numbers are separated from evidence. hang-key cleanup
is shown under Ctrl+C and SIGTERM, bench_compare refuses a label without
the LB. Drain before SIGTERM and the run_docker recreate issue are handed
to rolling-update in PLAN.md and KNOWN_ISSUES.md; design gains decision 10.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 00:07:21 +03:00
ricoromaandClaude Opus 5.5 4234df5767 docs: add graceful-failover evidence and report
Failover profile label baseline from 459f84c on a checked host: stop and
kill of each replica under 200 req/s give 0 errors and LB exclusion in
0.79-1.81 s (bound 2.5 s), pause gives LB exclusion in 5.83-6.00 s
(bound 6.5 s) with 504s for half the requests and recovery in 6 s;
reconciliation of retried writes matches. Evidence for detection,
keep-alive (LB closes idle replica connections at 10.1 s) and the hung
replica key lock, regression checks, timeout chain and level 4 rows in
REPORT.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 23:52:47 +03:00
ricoromaandClaude Opus 5.5 459f84c75c feat: coordinate timeout chain and add failover load profile
HAProxy checks get fastinter 500ms and timeout check 2.5s (hung replica
detected within 6.5 s), http-reuse never with uvicorn keep-alive 60 s so
the LB closes idle replica connections, and app connections set
idle_in_transaction_session_timeout so a frozen replica releases its
idempotency key. The bench stand runs behind HAProxy with N replicas and
gains the failover profile (make bench_failover / bench_failover_quick):
constant-rate writes with a key and reads while each replica is stopped,
killed or paused, with per-second timeline, detection and recovery time
and write reconciliation. Driver commands idle-conns and hang-key check
the keep-alive and hung-transaction decisions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 23:21:12 +03:00
ricoromaandClaude Opus 5.5 b76a666621 docs: propose graceful-failover change
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 22:32:12 +03:00
ricoromaandClaude Opus 5.5 aec6f23cb0 chore: archive replicas-lb change
Main specs were already in sync with the deltas (sync re-run, no
changes). PLAN.md marks stage 2 archived and points graceful-failover
at the corrected detection formula; AGENTS.md notes what of the timeout
chain is measured so far and the new replicas evidence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 22:10:31 +03:00
ricoromaandClaude Opus 5.5 cb14586a19 docs: retake replicas-lb evidence from a4ca9ef after verify
The stand was recreated from a4ca9ef (make down, make run_docker) and
all evidence of the change was taken again on it: stand, distribution,
LB errors, test-persistence, test-failover for each replica (stop and
kill), stop-one refusals, make test and student_test through the LB,
test_isolated, failure_checks and bench self-checks.

New evidence closes scenarios verify found without proof: a real git
clone on non-default ports, a hung replica (docker pause: DOWN after
7.7 s, in-flight request 504), PostgreSQL outage behind the LB, replica
tests failing against one process, aborted responses via
driver.py kill-inflight. REPORT.md gets the corrected detection
formula, the measured cost of the 1 CPU limit for Argon2id and the new
rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 22:09:15 +03:00
ricoromaandClaude Opus 5.5 a4ca9ef3a7 fix: correct replicas-lb details found in verify
- stop-one.sh: an empty TARGET_INSTANCE_ID now exits 2 in any shell,
  as the design states (${VAR:?} gave 1 under bash).
- driver.py kill-inflight: reproducible "aborted response is not
  retried" scenario (logins in flight, replica killed via stop-one.sh).
- Detection time of a hung replica is fall x (inter + timeout check) =
  8 s, not inter x fall + timeout check = 5 s (docker pause: 7.3 s);
  fixed in haproxy.cfg, balancer.md and the design.
- The 1 CPU replica limit is kept on purpose; the compose comment and
  the design now say that Argon2id uses more and logins are slower.
- load-balancing spec: API_REPLICAS is read by compose and Makefile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 18:44:57 +03:00
ricoromaandClaude Opus 5.5 de93056eab docs: add replicas-lb evidence and report
Evidence from 4e0cc4e: stand ports, limits and per-service env,
X-Instance-Id distribution and scaling, LB-generated errors,
test-persistence through the LB, test-failover for each replica in stop
and kill modes, stop-one refusals, test runs and bench self-checks.
REPORT.md gains the LB architecture, timeout chain, limits, criteria
rows and defence answers. Coverage exclusions for restart now point to
the lab 3 persistence evidence; stale lab 2 pointers go to
KNOWN_ISSUES.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 18:31:18 +03:00
ricoromaandClaude Opus 5.5 4e0cc4e965 fix: start DNS-resolved replicas down until checks pass
Without init-state down HAProxy marked a slot UP as soon as Docker DNS
returned the container address, before uvicorn listened, so requests
after LB restart or scale-up hit a replica that was not ready.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 18:08:50 +03:00
ricoromaandClaude Opus 5.5 ab71c3b51e feat: put API replicas behind HAProxy
HAProxy 3.4 is the only published entry point; api runs as
API_REPLICAS replicas without ports, found through Docker DNS and
checked on /health/ready. Retries only on connection failure, LB errors
in the contract error format, resource limits and per-service secrets.
Adds scripts/stop-one.sh, LB-aware restart.sh, make replicas /
instances / lb_stats and replica tests driven by EXPECT_REPLICAS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 18:05:05 +03:00
ricoromaandClaude Opus 5.5 8e15e02898 docs: propose replicas-lb change
HAProxy as the single entry point, API replicas without published
ports, resource limits and per-service secrets, restart and stop-one
hooks, replica tests in student-tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 17:40:04 +03:00
ricoromaandClaude Opus 5.5 b5ae252254 chore: archive instance-id-idempotency change
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 17:23:41 +03:00
ricoromaandClaude Opus 5.5 c5840bfaa8 docs: add idempotency and X-Instance-Id evidence to the report
- evidence for replay, conflict, replay through a new API process and
  after restarting API and storages, keys table, key race,
  X-Instance-Id in responses and logs, explicit and invalid
  instance_id and key TTL, test runs
- stand evidence: make wipe_data leaves the lab 2 volume intact, build
  from a fresh git clone passes smoke, bench restore stage sends with
  Idempotency-Key
- REPORT.md: X-Instance-Id, idempotency design, IDEMPOTENCY_TTL_SECONDS
  mapping, limitations and evidence rows
- tasks.md: mark the change tasks done, add follow-ups from verify

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 17:08:47 +03:00
ricoromaandClaude Opus 5.5 af4bca08c9 feat: carry lab 2 over and add X-Instance-Id and idempotency keys
- carry the lab 2 solution over and plan lab 3 (PLAN.md, OpenSpec
  change instance-id-idempotency)
- rename the compose project to mtusi-lab3 and the image to :lab3
- add X-Instance-Id to every response and instance_id to every log
  record ([app] instance_id, hostname by default)
- require Idempotency-Key on POST messages: keys table in Postgres,
  key and message in one transaction, replay returns the same 201,
  another body gets 409, TTL from [idempotency] ttl_seconds
- send the key from student-tests, k6 scenarios and the skill driver;
  add idempotency, X-Instance-Id, key race and fault tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 19:42:36 +03:00