Commit Graph
64 Commits
Author SHA1 Message Date
ricoromaandClaude Opus 5.5 3df111cd80 docs: add scaling db A/A label for scaling-improvement
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 14:37:29 +03:00
ricoromaandClaude Opus 5.5 62fa57535e revert: measure scaling-improvement baseline without the pool change
This reverts commits 9c3959d, 623a973 and 80e64cd.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 14:10:43 +03:00
ricoromaandClaude Opus 5.5 1bb8ea67ad feat: add two-cell db4 series for the scaling-improvement measurements
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 14:06:57 +03:00
ricoromaandClaude Opus 5.5 80e64cda4a test: count PostgreSQL sessions without healthcheck and flush races
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 13:46:39 +03:00
ricoromaandClaude Opus 5.5 623a973dac test: wait for PostgreSQL session stats before counting pool reconnections
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 13:30:38 +03:00
ricoromaandClaude Opus 5.5 9c3959d768 perf: keep a fixed PostgreSQL pool without overflow connections
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 13:25:26 +03:00
ricoromaandClaude Opus 5.5 0b0421ad2f docs: record track 1 improvement hypothesis H4
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 13:23:49 +03:00
ricoromaandClaude Opus 5.5 1e50a4a3fc feat: compare PostgreSQL sessions and connections in track 1 series
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 13:23:12 +03:00
ricoromaandClaude Opus 5.5 029aa3be82 docs: propose scaling-improvement change
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 13:19:56 +03:00
ricoromaandClaude Opus 5.5 bbd6f65316 chore: archive scaling-bench change
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 12:25:18 +03:00
ricoromaandClaude Opus 5.5 dedb8ba0fa docs: report PostgreSQL saturation series db and rubric answer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 12:24:17 +03:00
ricoromaandClaude Opus 5.5 195a3f8ad9 docs: add series db plots and host check
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 12:22:45 +03:00
ricoromaandClaude Opus 5.5 9fbe2ea503 feat: draw series db in R* plot and summarize PostgreSQL saturation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 12:22:32 +03:00
ricoromaandClaude Opus 5.5 48d10dad37 docs: add scaling series db
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 12:21:04 +03:00
ricoromaandClaude Opus 5.5 46e8f75a2c docs: record series db calibration in scaling-bench design
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 11:31:39 +03:00
ricoromaandClaude Opus 5.5 95539a1883 feat: add track 1 series db with a tight PostgreSQL CPU limit
Configs may carry a fifth element, the PostgreSQL CPU limit, which goes
into the config key (-pg<CPU>), the budget and the cell stand. Series db:
1, 2, 4 replicas at the tight limit and a 4 x 1 control at 2 CPU; label
summaries show PostgreSQL CPU at R* against its limit and throttling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 11:27:30 +03:00
ricoromaandClaude Opus 5.5 a3279374f7 docs: fix scaling-bench verify findings in report, design and rules
Close the 53300 known issue, restate H1 and saturation conclusions
against the pre-registered criteria and grid resolution, name the
unresolved host mechanism, attribute overload 503 to the pool timeout,
and plan a fresh A/A for scaling-improvement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 07:36:33 +03:00
ricoromaandClaude Opus 5.5 4abedaed78 docs: add 53300 evidence, regression and regenerated track 1 summaries
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 07:35:49 +03:00
ricoromaandClaude Opus 5.5 dcea85942a feat: add cost and memory tables, passports and sampler limits to track 1 outputs
plots.py summary gains per-request CPU by component, PostgreSQL waits and
pool, H1b throttling and memory peaks; compare.py and plots.py headers
name the spec, change and host; errors logs match 5xx by status field and
keep the exception tail; series labels record sampler limits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 07:26:36 +03:00
ricoromaandClaude Opus 5.5 68edc66742 fix: answer 503 when PostgreSQL refuses connections with too many clients
SQLSTATE 53300 (too_many_connections) joins UNAVAILABLE_SQLSTATES, so a
refused pool connection gives 503 unavailable instead of 500. New fault
check fills max_connections through the failure stand hook and expects
503 in the contract format.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 07:26:36 +03:00
ricoromaandClaude Opus 5.5 ca0817390f docs: report track 1 scaling series
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 06:49:49 +03:00
ricoromaandClaude Opus 5.5 0ea787ccae docs: add scaling-bench plots and comparisons
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 06:49:03 +03:00
ricoromaandClaude Opus 5.5 bfc5eb8ede docs: add scaling A/A label
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 06:48:19 +03:00
ricoromaandClaude Opus 5.5 0479789aa3 docs: add scaling series plots for A and B
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 06:26:38 +03:00
ricoromaandClaude Opus 5.5 7b6a31b62a docs: record 500 instead of 503 on too many clients found by the pool series
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 06:26:30 +03:00
ricoromaandClaude Opus 5.5 b7e787a658 fix: move reference line labels off the bars in the scaling plots
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 06:26:03 +03:00
ricoromaandClaude Opus 5.5 f65a8e86d1 docs: add scaling pool series
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 06:26:03 +03:00
ricoromaandClaude Opus 5.5 1320584d9f docs: add scaling series B
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 06:01:52 +03:00
ricoromaandClaude Opus 5.5 3401388b5e docs: describe track 1 series in AGENTS.md
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 04:47:04 +03:00
ricoromaandClaude Opus 5.5 71359be0fb docs: add scaling series A
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 04:47:04 +03:00
ricoromaandClaude Opus 5.5 cbe3b4a43b docs: add scaling-bench self-checks
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 04:10:02 +03:00
ricoromaandClaude Opus 5.5 712d2450e4 fix: preallocate all k6 VUs in track 1 steps so the generator does not drop iterations near R*
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 03:55:21 +03:00
ricoromaandClaude Opus 5.5 68eef77119 fix: put the growth series end label above its whisker in the R* plot
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 03:45:46 +03:00
ricoromaandClaude Opus 5.5 5de4da997c fix: drop the resume image warning, the containerd image id changes on every build
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 03:44:41 +03:00
ricoromaandClaude Opus 5.5 02fd3d8a36 feat: add track 1 scaling series to the bench stand
Series a/b/pool/ab/quick in bench/run.py: per-cell fresh stand with the
configuration's replicas and max_connections, open-model mixed profile
(k6/mixed.js), fixed levels and R* search by SLO, a sampler container
for cgroup, VM, HAProxy and PostgreSQL metrics, resume and a stand
lock. Pure helpers in bench/scaling.py, SVG plots in bench/plots.py,
series mode for bench_compare and bench_ab. Hypotheses H1-H3 go to
REPORT.md before the first series.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 03:28:36 +03:00
ricoromaandClaude Opus 5.5 525ace7c9b docs: propose scaling-bench change
Track 1 "cost of scaling": open-model series A (fixed 4 CPU budget),
B (1-6 replicas), pool border (max_connections 100 vs 50) and A/A,
sampler of cgroup/VM/LB/PostgreSQL metrics, generator limits, series
mode for bench_compare and bench_ab. Hypotheses H1-H3 are recorded in
the design before any series run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 02:48:40 +03:00
ricoromaandClaude Opus 5.5 84ee13ca5a chore: archive rolling-update change
Sync the rolling-update delta specs into db-migrations,
graceful-shutdown, load-balancing and load-benchmark, move the change to
the archive, mark it in PLAN.md and update the transitional state and
code map in AGENTS.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 01:59:19 +03:00
ricoromaandClaude Opus 5.5 6f85d21f3b docs: retake rolling-update evidence on the final verify commit
Re-run the compatibility walk-through, the script failure cases with a
real Ctrl+C through a pseudo-terminal, the stand lifecycle after
replacement and the quick rollout and failover self-checks on 84161c0
with a clean tree; update the report rows and limitations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 01:56:30 +03:00
ricoromaandClaude Opus 5.5 84161c033a fix: keep migration lock timeout below the app statement timeout
With lock_timeout equal to statement_timeout (5 s) every request queued
behind a migration waiting for its lock failed with 503 together with
the migration. A 2 s lock timeout lets the queue proceed once the
migration gives up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 01:38:10 +03:00
ricoromaandClaude Opus 5.5 5e102e6d0c fix: harden rolling update cleanup and preconditions found in verify
Reset the new replica ID before entering the starting state and let the
cleanup find a replica created by an interrupted compose scale; refuse
to start while LB slots keep a manual drain or maint state; log the exit
code of each stopped old replica and print full LB commands for manual
cleanup. Document the migrate step and the rollback tag limitation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 01:33:33 +03:00
ricoromaandClaude Opus 5.5 69ebf5cd38 docs: add rolling-update baseline measurement and report
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 01:21:34 +03:00
ricoromaandClaude Opus 5.5 9dcea0b2b8 docs: add rolling-update regression evidence and report sections
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 01:17:34 +03:00
ricoromaandClaude Opus 5.5 de654a1c83 fix: recreate the migrate service container during rolling update
A one-off run --rm left the migrate container of the previous image in
the stand; scripts/restart.sh then started it as a dependency of api and
the old migrations failed on the newer schema, so test-persistence broke.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 01:04:50 +03:00
ricoromaandClaude Opus 5.5 5403464cfb docs: add rolling-update compatibility and script failure evidence
Old image passes smoke and student tests on the new schema, rolling
update, rollback and roll forward under tests/load.py give zero failed
requests; the run_docker known issue is not reproduced and is closed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 01:01:33 +03:00
ricoromaandClaude Opus 5.5 d2148ac566 feat: add rolling update with LB drain and rollout load profile
scripts/rolling-update.sh (make rolling_update) replaces API replicas one
by one: build, migrations once, new replica UP in HAProxy, old replica
drained and put into maintenance through the LB runtime API on loopback,
then stopped and removed; the slot is reset once DNS drops the address.
Expand migration adds messages.instance_id with a lock timeout. The bench
rollout profile runs the same script under constant load, starting from
the image of an older commit built with git archive.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 00:49:50 +03:00
ricoromaandClaude Opus 5.5 9a00795de1 docs: propose rolling-update change
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 00:26:36 +03:00
ricoromaandClaude Opus 5.5 c50b7211b8 chore: archive graceful-failover change
Delta specs synced into graceful-shutdown, load-balancing, load-benchmark
and service-configuration; the change moves to
openspec/changes/archive/2026-10-11-graceful-failover/. PLAN.md marks the
change archived and AGENTS.md notes what was handed to rolling-update.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 00:10:35 +03:00
ricoromaandClaude Opus 5.5 fc715b74d4 docs: add LB rule of bench_compare to graceful-failover delta
The verify fix in bench/compare.py refuses a label measured without the
load balancer; the load-benchmark delta now modifies the label comparison
requirement accordingly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 00:08:13 +03:00
ricoromaandClaude Opus 5.5 f38a56c4ea docs: retake graceful-failover checks after verify
The rejected LB pool variant is re-measured on committed code with a
temporary second HAProxy differing in http-reuse only: the replica, not
the LB, closed the last idle connection after 60.4 s. Exploratory
never/safe SIGTERM numbers are separated from evidence. hang-key cleanup
is shown under Ctrl+C and SIGTERM, bench_compare refuses a label without
the LB. Drain before SIGTERM and the run_docker recreate issue are handed
to rolling-update in PLAN.md and KNOWN_ISSUES.md; design gains decision 10.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 00:07:21 +03:00
ricoromaandClaude Opus 5.5 12783aaff1 fix: harden graceful-failover checks found in verify
hang-key names its trigger per run, turns SIGTERM and SIGHUP into an
interrupt so the trigger is always dropped and the replica unpaused, and
prints the cleanup SQL if dropping fails. idle-conns reports who closed
the idle connections and can read the pool of another LB container, so a
variant LB config can be checked against the same replicas. bench_compare
refuses a label measured without the load balancer. Stale numbers removed
from the haproxy.cfg comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 00:00:54 +03:00