1x4, 2x2 and 4x1 at 4 CPU and 2 GiB of API, workers equal to CPUs,
32 pool connections and 128 balancer slots in every layout, three
repeats from 600 to 6 830 rps. All three keep up with the offered
load up to 4 560 rps; the peaks are 5 375 rps for 1x4 (5 332..5 384),
4 964 for 2x2 (4 894..5 104) and 4 724 for 4x1 (4 686..4 794). At the
peak the API uses 95 % of its four CPUs in every layout, PostgreSQL
82..89 % of its two, the balancer 40 % of one, and about half of the
32 database connections wait for the lock on a chat row.
Smaller replicas cost more: at 3 040 rps the API burns 215 % CPU as
1x4, 243 % as 2x2 and about 280 % as 4x1, and p95 at 4 560 rps grows
from 9 to 69 and 150 ms.
The first attempt was aborted when a neighbour's image build froze the
Docker API; its summary is kept under aborted/. Twelve steps of the
rerun were rejected as disturbed and repeated. A config's SLO capacity
now needs a majority of saturated repeats, since a median is robust to
one disturbed repeat and "any repeat" is not. Series B starts at 450
rps per CPU.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
The first run of series A was spoiled by other stands on the same
Docker VM: an image build of another lab is invisible to docker stats,
yet it took CPU from the pinned replicas, doubled their CPU per request
and stalled the Docker API for fifteen minutes in the middle of two
steps.
docker-stats.sh now also samples the VM-wide /proc/stat counters, so a
step records the CPU of the whole VM, the part not used by this
project, the largest gap between samples and its wall time. A step
with more than one foreign core, a gap over 10 s or a k6 run 90 s
longer than its load is written to disturbed.csv and repeated after
the VM has been quiet for 15 s, up to three times. On a quiet VM the
foreign part is 2 % of a core at 600 rps and 25 % at 4 560 rps, which
is Docker networking and the samplers themselves.
The growth series starts at 300 rps per CPU and searches up to 9 000,
so its knees are found with fewer idle steps. The plots fit the
universal scalability law to the growth series.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
tools/plot_track1.py reads any set of summary.csv files and draws, per
series, successful RPS, p95 and p99 against the offered rate and CPU
per component against its limit, plus two cross-series charts: peak
successful RPS against total API CPU with the linear growth line, and
the utilisation of API, PostgreSQL, balancer and the share of the pool
waiting on a lock at each config's peak. A point is the median of the
repeats, the whiskers are min and max. capacity.csv and capacity.md
hold the SLO capacity, the peak and the resource figures at the peak.
Colours follow the config across charts, every line also has its own
marker and a direct label. matplotlib comes through uv run --with, so
the tool lives in tools/ and stays out of the 3.9 scripts; make plots
runs it, make lint covers it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>