The k6 profile from the prepared kit runs an open model against the
balancer from the edge network: constant arrival rate, 80 % keyed
message posts and 20 % history reads over 20 synthetic users and 10
chats, metrics only for the measure phase, and no setup data (tokens)
in summary.json. scripts/k6.sh runs it with the host uid so the
summary is writable on Linux too.
scripts/probe.py gains three things. check --rounds repeats the
same-key race with a new key per round and counts rounds with
identical bodies, one stored copy and 409 on another body. load runs
an action under k6 while a probe posts every 0.1 s and resends the
same key after a failure and a poller reads the number of UP servers;
it writes the action log, the probe and LB traces, the delta of the
backend retry, redispatch and error counters, and a verdict: dip for
a replica that leaves and returns, replace for a rolling update that
must keep capacity and end on new replicas only. detect runs the
action in the background and polls the balancer at the same time, so
a drain that ends after the exclusion is visible, and checks the
declared maximum, the LB answer after detection and the time to
restore.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The first failover-each run had no errors, yet the balancer counted
19 and 49 redispatches. The HAProxy log showed why: when a stopped
container comes back into Docker DNS, its server-template slot leaves
MAINT (resolution) as UP/READY at once, before any health check, so
requests went to a gunicorn that was not listening yet and were
redispatched to the other replica.
default-server now sets init-state fully-down (HAProxy 3.1+): at start,
on discovery through DNS and on leaving maintenance a server stays DOWN
until rise 2 checks pass. A rerun under 100 rps shows zero retries and
redispatches for both replicas; rejoining takes about a second longer.
The design and the failover spec of the change record the finding.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
compose.yaml now runs one api service with API_REPLICAS replicas and
no published ports, container name or hostname, behind lb, HAProxy
3.2.26 LTS pinned by digest and the only service with ports, bound to
127.0.0.1 (LB_HTTP_PORT for HTTP, LB_STATS_PORT for read-only stats).
The balancer finds replicas through Docker DNS with a server template,
checks GET /health/ready every second (fall 2, rise 2), balances round
robin without stickiness, retries connection failures on another
replica and allows a retry after a sent request only for GET and for
message posts carrying Idempotency-Key, never for users, chats or
login. Its own 502/503/504 answers use the contract error body with
the forwarded X-Request-Id.
The network is split into edge and an internal backend, every service
gets CPU and memory limits, the API runs read-only with a tmpfs /tmp,
two workers per replica, an 8 s request limit and a 5 s graceful stop
inside an 8 s stop_grace_period. Only api builds the image, migrate
reuses it once before the replicas start. PostgreSQL loads
pg_stat_statements and allows 200 connections. The short TTL instance
of lab 2 published the API around the balancer and is removed; its
expiry check stays as an in-process test.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>