Commit Graph
24 Commits
Author SHA1 Message Date
ricoromaandClaude Opus 5.5 8be58ce046 fix: bound docker events shutdown in failover and rollout profiles
stop_stream terminates the events CLI, waits 10 s and kills it after
that. The rollout profile starts k6 inside the try, so a k6 start
failure no longer leaves the project events stream running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 18:23:23 +03:00
ricoromaandClaude Opus 5.5 b37c45cbb8 revert: drop the failover events cross-check
This reverts commit a24832c.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 16:39:44 +03:00
ricoromaandClaude Opus 5.5 a24832c72b test: cross-check failover events against the daemon buffer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 16:16:31 +03:00
ricoromaandClaude Opus 5.5 fe75fcf8a9 fix: stream docker events during failover phases
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 16:16:11 +03:00
ricoromaandClaude Opus 5.5 18238588b4 fix: record a zero swap-out delta in bench passports
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 16:15:25 +03:00
ricoromaandClaude Opus 5.5 f34cb9af5c fix: label per-second rows around pause by their unfrozen part
The samples.csv row of an interval that touches the start of a freeze
now gets the window of its unfrozen part, as the window aggregates do
since the thaw correction. CPU per successful request is no longer
computed for a container absent from the window. The definitions in
the summary describe which intervals the window maximum covers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 19:23:29 +03:00
ricoromaandClaude Opus 5.5 2d267bfbe6 fix: attribute replica CPU around pause to the unfrozen part
A sample interval that crosses docker pause or unpause carried the CPU
spent before the pause and the burst after unpause into the failover
window by proportional split. Use the pause and unpause moments from
Docker events so that interval CPU lands outside the frozen span.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 18:09:05 +03:00
ricoromaandClaude Opus 5.5 6505b5e929 feat: sample CPU and memory in failover and rollout profiles
Run bench/sampler.sh in a new scan mode (cgroup walk, no network, no
psql or LB polling) for the whole load of the failover and rollout
profiles. bench/resources.py turns the samples into per-window CPU,
throttling and memory of every component and the VM, with
samples.csv, resources.csv and summary sections; CPU per successful
request only in steady windows. Check the peak CPU budget before the
stand starts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 17:42:04 +03:00
ricoromaandClaude Opus 5.5 1bb8ea67ad feat: add two-cell db4 series for the scaling-improvement measurements
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 14:06:57 +03:00
ricoromaandClaude Opus 5.5 1e50a4a3fc feat: compare PostgreSQL sessions and connections in track 1 series
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 13:23:12 +03:00
ricoromaandClaude Opus 5.5 9fbe2ea503 feat: draw series db in R* plot and summarize PostgreSQL saturation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 12:22:32 +03:00
ricoromaandClaude Opus 5.5 95539a1883 feat: add track 1 series db with a tight PostgreSQL CPU limit
Configs may carry a fifth element, the PostgreSQL CPU limit, which goes
into the config key (-pg<CPU>), the budget and the cell stand. Series db:
1, 2, 4 replicas at the tight limit and a 4 x 1 control at 2 CPU; label
summaries show PostgreSQL CPU at R* against its limit and throttling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 11:27:30 +03:00
ricoromaandClaude Opus 5.5 dcea85942a feat: add cost and memory tables, passports and sampler limits to track 1 outputs
plots.py summary gains per-request CPU by component, PostgreSQL waits and
pool, H1b throttling and memory peaks; compare.py and plots.py headers
name the spec, change and host; errors logs match 5xx by status field and
keep the exception tail; series labels record sampler limits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 07:26:36 +03:00
ricoromaandClaude Opus 5.5 b7e787a658 fix: move reference line labels off the bars in the scaling plots
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 06:26:03 +03:00
ricoromaandClaude Opus 5.5 712d2450e4 fix: preallocate all k6 VUs in track 1 steps so the generator does not drop iterations near R*
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 03:55:21 +03:00
ricoromaandClaude Opus 5.5 68eef77119 fix: put the growth series end label above its whisker in the R* plot
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 03:45:46 +03:00
ricoromaandClaude Opus 5.5 5de4da997c fix: drop the resume image warning, the containerd image id changes on every build
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 03:44:41 +03:00
ricoromaandClaude Opus 5.5 02fd3d8a36 feat: add track 1 scaling series to the bench stand
Series a/b/pool/ab/quick in bench/run.py: per-cell fresh stand with the
configuration's replicas and max_connections, open-model mixed profile
(k6/mixed.js), fixed levels and R* search by SLO, a sampler container
for cgroup, VM, HAProxy and PostgreSQL metrics, resume and a stand
lock. Pure helpers in bench/scaling.py, SVG plots in bench/plots.py,
series mode for bench_compare and bench_ab. Hypotheses H1-H3 go to
REPORT.md before the first series.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 03:28:36 +03:00
ricoromaandClaude Opus 5.5 de654a1c83 fix: recreate the migrate service container during rolling update
A one-off run --rm left the migrate container of the previous image in
the stand; scripts/restart.sh then started it as a dependency of api and
the old migrations failed on the newer schema, so test-persistence broke.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 01:04:50 +03:00
ricoromaandClaude Opus 5.5 d2148ac566 feat: add rolling update with LB drain and rollout load profile
scripts/rolling-update.sh (make rolling_update) replaces API replicas one
by one: build, migrations once, new replica UP in HAProxy, old replica
drained and put into maintenance through the LB runtime API on loopback,
then stopped and removed; the slot is reset once DNS drops the address.
Expand migration adds messages.instance_id with a lock timeout. The bench
rollout profile runs the same script under constant load, starting from
the image of an older commit built with git archive.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 00:49:50 +03:00
ricoromaandClaude Opus 5.5 12783aaff1 fix: harden graceful-failover checks found in verify
hang-key names its trigger per run, turns SIGTERM and SIGHUP into an
interrupt so the trigger is always dropped and the replica unpaused, and
prints the cleanup SQL if dropping fails. idle-conns reports who closed
the idle connections and can read the pool of another LB container, so a
variant LB config can be checked against the same replicas. bench_compare
refuses a label measured without the load balancer. Stale numbers removed
from the haproxy.cfg comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 00:00:54 +03:00
ricoromaandClaude Opus 5.5 459f84c75c feat: coordinate timeout chain and add failover load profile
HAProxy checks get fastinter 500ms and timeout check 2.5s (hung replica
detected within 6.5 s), http-reuse never with uvicorn keep-alive 60 s so
the LB closes idle replica connections, and app connections set
idle_in_transaction_session_timeout so a frozen replica releases its
idempotency key. The bench stand runs behind HAProxy with N replicas and
gains the failover profile (make bench_failover / bench_failover_quick):
constant-rate writes with a key and reads while each replica is stopped,
killed or paused, with per-second timeline, detection and recovery time
and write reconciliation. Driver commands idle-conns and hang-key check
the keep-alive and hung-transaction decisions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 23:21:12 +03:00
ricoromaandClaude Opus 5.5 ab71c3b51e feat: put API replicas behind HAProxy
HAProxy 3.4 is the only published entry point; api runs as
API_REPLICAS replicas without ports, found through Docker DNS and
checked on /health/ready. Retries only on connection failure, LB errors
in the contract error format, resource limits and per-service secrets.
Adds scripts/stop-one.sh, LB-aware restart.sh, make replicas /
instances / lb_stats and replica tests driven by EXPECT_REPLICAS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 18:05:05 +03:00
ricoromaandClaude Opus 5.5 af4bca08c9 feat: carry lab 2 over and add X-Instance-Id and idempotency keys
- carry the lab 2 solution over and plan lab 3 (PLAN.md, OpenSpec
  change instance-id-idempotency)
- rename the compose project to mtusi-lab3 and the image to :lab3
- add X-Instance-Id to every response and instance_id to every log
  record ([app] instance_id, hostname by default)
- require Idempotency-Key on POST messages: keys table in Postgres,
  key and message in one transaction, replay returns the same 201,
  another body gets 409, TTL from [idempotency] ttl_seconds
- send the key from student-tests, k6 scenarios and the skill driver;
  add idempotency, X-Instance-Id, key race and fault tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 19:42:36 +03:00