The failover evidence printed the last six log lines of the stopped
replica, which no longer reach back to the signal now that shutdown
starts with a two second drain window. It now filters the replica log
since the start of the run for the signal, the drain message, the 503
readiness answers, the listener shutdown and the process exit.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The repository ignores *.log, so the action log of a load run (with
the rolling update replacement lines) and the k6 output would never
reach evidence/. The probe now writes action.txt and k6/k6.txt.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
scripts/rolling-update.sh runs migrate once before touching any
replica, then for each old replica scales the service to N+1 without
recreating the others, waits until the new container is healthy and
the balancer sees N+1 UP servers, and only then stops the old one
with docker stop -t 6 (so it drains) and removes it. The log names
the hostname and image of the old and the new replica of every step,
which is what the report quotes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
scripts/limits.py states which secrets, volumes and networks each
service needs and that only lb publishes ports, on loopback. It
compares that policy with docker compose config (with the bench
profile, so k6 is covered) and with docker inspect of the running
containers: CPU and memory limits declared and applied, no
container_name, no published port on any API replica, a read-only
root filesystem for the API, and which networks are internal. It
prints both tables and every violation and exits non-zero on any.
The stack evidence script no longer reaches replicas through
docker compose exec or port, which assume container number 1 and
break after a rolling update renumbers the replicas.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
scripts/detection.sh runs four failures one after another and brings
the stand back between them: SIGTERM through docker stop -t 6 (the
balancer must exclude the replica within 1.5 s and before the process
exits after its drain window), docker kill (1.5 s), docker pause for a
hung process (3.5 s, the check timeout counts twice) and stopping
PostgreSQL (all replicas out within 1.5 s, the balancer answers 503
itself). A 0.25 s slack covers the docker CLI and the 50 ms polling.
Each run writes evidence/detection/<case>.json and the script fails
if any case exceeds its declared maximum.
scripts/failover-each.sh stops every replica in turn with stop-one.sh,
keeps it down for 10 s and restores it, all inside one k6 load window,
and collects a summary with RPS, errors, latency, detection, rejoin,
probe recovery and balancer retry counters per replica.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The k6 profile from the prepared kit runs an open model against the
balancer from the edge network: constant arrival rate, 80 % keyed
message posts and 20 % history reads over 20 synthetic users and 10
chats, metrics only for the measure phase, and no setup data (tokens)
in summary.json. scripts/k6.sh runs it with the host uid so the
summary is writable on Linux too.
scripts/probe.py gains three things. check --rounds repeats the
same-key race with a new key per round and counts rounds with
identical bodies, one stored copy and 409 on another body. load runs
an action under k6 while a probe posts every 0.1 s and resends the
same key after a failure and a poller reads the number of UP servers;
it writes the action log, the probe and LB traces, the delta of the
backend retry, redispatch and error counters, and a verdict: dip for
a replica that leaves and returns, replace for a rolling update that
must keep capacity and end on new replicas only. detect runs the
action in the background and polls the balancer at the same time, so
a drain that ends after the exclusion is visible, and checks the
declared maximum, the LB answer after detection and the time to
restore.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
After make up, which builds with --build, a plain up in the restart
hook saw a different service hash and recreated migrate and the API
replicas instead of starting the stopped ones, so the replicas came
back with new hostnames. The hook now passes --no-recreate: lb and the
replicas are the same containers restarted, migrate reruns in its old
container, and only volumes and data carry state across the restart.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
scripts/failover-evidence.sh lists the api containers by hostname, the
X-Instance-Id they answer with, before and after make test-failover,
names the container left exited, keeps its last log lines with the
gunicorn graceful shutdown, then restores it and lists the replicas
again. scripts/evidence-stack.sh records the compose config check,
published ports, the missing port of a replica, hostnames, networks,
limits, process users, the balancer view of every server and the
alternating X-Instance-Id through the balancer, plus the environment
and image versions. The persistence and storage evidence scripts now
list every replica and the idempotency key table.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
scripts/stop-one.sh maps TARGET_INSTANCE_ID to the api container whose
hostname matches, stops only it with docker stop -t 6 and leaves it
stopped; an unknown id exits 3 and touches nothing. scripts/restart.sh
replaces the lab 2 hook: it stops lb and the replicas, restarts
PostgreSQL and Valkey with their volumes and brings the stand back
with up --wait, which also reruns the idempotent migrate job.
scripts/restore.sh starts stopped replicas without recreating them
and scripts/lb-wait.sh waits until the balancer statistics show every
replica UP again, so the next failover run sees two replicas.
scripts/probe.py check (stdlib, Python 3.9 syntax) counts X-Instance-Id
over 40 calls with one token and races 8 posts with one key through
the balancer. make restore and make lb-check run them, ruff covers the
probe, and the CI acceptance job runs the failover test and the probe.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Bring the application, own tests, Dockerfile, toolchain, compose
stack, migrations, secret generator, restart hook, evidence scripts and
CI job from the lab 2 solution commit 8cddb09 unchanged, merge the lab
2 Makefile targets into the lab 3 skeleton Makefile without touching
its targets, and append the lab 2 solution variables to the skeleton
.env.example. The package version follows the contract version 1.3.0.
This is the starting point of the lab 3 cycle: lint, own tests and the
offline skeleton check pass before any lab 3 change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>