Commit Graph
63 Commits
Author SHA1 Message Date
ArtyomandClaude Opus 5.5 13e856ebd4 Propose closing the second lab 3 review gaps as an openspec change
The second review found that a request cancelled at the end of the
graceful shutdown gets uvicorn's plain 500 without the contract headers,
that readiness reaches the PostgreSQL adapter and a global flag of the
HTTP layer past the ports, that the drain window is logged past the
Logger port, that make up returns with one replica in rotation and that
the CI non-root check can never fail. The retry of an unconfirmed write
was never exercised on the stand either.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 05:37:31 +03:00
ArtyomandClaude Opus 5.5 5a3e372b08 Name the load of each column in the placement control table
The CPU per request and the lock waiters of the placement control are
taken at 4560 rps; the header now says so instead of leaving the reader
to guess between the two offered rates.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 04:47:39 +03:00
ArtyomandClaude Opus 5.5 d3ee257df1 Archive the lab 3 review gaps change
The deltas are synced to openspec/specs: the order of checks on posting
a message (401, then 400 for the request shape without touching the
chat, then 404 for access, then 400 for the text); three detection runs
with a 0.25 s measurement slack and the replica states after each stop
in failover-each; the upper bound of the track 1 grid with a ceiling on
the first grid point stated as a lower bound; control runs of the pool,
the LB queue and the placement.

Before archiving, on the running stand: make test passes, failover
recovers the first write in 2.78 s with 0 failed attempts, persistence
passes; make check, make lint and 243 own tests pass. All level 3 and 4
records were taken on b1c80ee, after the lab 2 carry over.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 04:47:14 +03:00
ArtyomandClaude Opus 5.5 fa09339388 Report the control runs of track 1
The LB queue does not move the peak of 8x1 (4526, 4677 and 4525 rps for
maxconn 16, 32 and 64, overlapping ranges), and the pool moves it the
other way: 4 connections per worker give 5358 rps, 16 give 3789, while
the lock waiters grow from 14.7 to 102. So the pool is not short, extra
connections only add writers to the hot chat row, which confirms the
coordination cost of the USL fit. On the same CPUs 8x0.5 peaks 7 % below
4x1 and costs 15 % more CPU per request, so the retrograde 8x1 comes from
the number of processes and connections, not from slower vCPUs.

The report also names the cause of the flaky compose up, the WalSync
stall of the dropped attempt, the disturbed control runs, and the
evidence index gets a commit column for track 1. The scenario of the
open change no longer presumes its own outcome.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 04:45:55 +03:00
ArtyomandClaude Opus 5.5 f4833c40c0 Record the control runs of track 1
make track1-control on b1c80ee: 8x1 at 5400 rps with a pool of 4, 8 and
16 connections per worker and an LB maxconn of 16, 32 and 64 per
replica, then 4x1 against 8x0.5 on the same CPUs 7-10, three repeats
with the variants interleaved.

The first placement attempt is kept in aborted/ only as a summary: four
4x1 runs had the write command at 7-34 ms instead of 0.2-0.6 ms, waiting
on WalSync, which the CPU based rejection does not see. The rerun right
after it is the one in control-cpus. The full output of the two failed
compose up attempts names the cause of the flaky switch: all replicas
are recreated at once and lb turns unhealthy without a ready server.

Plots and capacity.md now include both control series.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 04:45:55 +03:00
ArtyomandClaude Opus 5.5 a67bc6526a Report levels 3 and 4 on the code after the lab 2 carry over
The report now names a59e3a3 as the last change of the application code
and b1c80ee as the commit of every level 3 and 4 record, with the new
replica IDs and numbers: three detection runs within the formula without
the measurement slack, failover-each and both rolling updates without a
failed request, 12 stack tests and 243 own tests. The shutdown section
and the timeout chain follow the carried over budget: drain 2 s, then
3 s for accepted requests and a 500 for a stuck one, graceful timeout
6 s, stop hooks 7 s. make validate runs from .venv, faults evidence has
its own target, AGENTS.md lists the control runs of track 1.

The clean clone is taken again on f5fea5c without an explicit BASE_URL.
The track 1 part keeps the review fixes of the H2 verdict, the grid
bounds and the rounded numbers; the control runs follow.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 03:17:56 +03:00
ArtyomandClaude Opus 5.5 f5fea5c59c Record levels 3 and 4 again after the lab 2 carry over
The carry over of the lab 2 fixes changed the request pipeline, logout,
the worker shutdown budget and the stop hooks, so every level 3 and 4
record is taken again on b1c80ee: checks, own tests, smoke, stack,
storage, failover of both replicas, persistence, hooks, the lab 2 stack
tests, limits and the injected violations, lb-check, three detection
runs, failover-each and both rolling updates under k6.

All of them pass: 243 own tests, 12 stack tests, no failed request under
k6 in failover-each and in both rolling updates.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 02:53:12 +03:00
ArtyomandClaude Opus 5.5 b1c80ee544 Anonymize the repository path in the faults evidence
make faults-evidence wrote the pytest rootdir with the local home
directory, so the committed faults.txt was cleaned by hand and could not
be reproduced by the command from the report. The target now replaces
the repository path with <repo> and still fails when a stack test fails,
since the pipe through sed hides the pytest exit code.

The open review change also records that the lab 2 carry over moved the
default image tag to the compose project.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 02:42:45 +03:00
ArtyomandClaude Opus 5.5 ac15b2ff62 Archive the carry over of the lab 2 fixes change
Lab 3 now stands on the finished lab 2. The deltas are synced to
openspec/specs: the zones meet only in the schema and Valkey is needed
only to confirm a logout; an outsider does not wait for a chat lock;
an empty cursor is 400 on history and on the chat list; a stuck request
at SIGTERM gets 500 from the server itself within the 6 s budget, which
still leaves 3 s after the drain window for the database timeouts; a
chunked body is limited by the bytes received; a logout rejected by
PostgreSQL withdraws its revocation mark. The event publication purpose
follows lab 2.

On a stand rebuilt from empty volumes: make test passes, failover
recovers the first write in 3.1 s with 0 failed attempts, persistence
passes, test-faults 12 passed including the full restart with logout
and the shortened session after an API restart, detection passes all
four scenarios with SIGTERM detected in 0.82 s and the stop done in
2.44 s, and the cache evidence finds a tombstone and a key that does
not outlive its session.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 02:38:45 +03:00
ArtyomandClaude Opus 5.5 a59e3a39a8 Carry over the lab 2 fixes made after 8cddb09
Brings the system changes of lab 2 from 8cddb09..102579e onto the lab 3
code, resolving each conflict by meaning:

- the request pipeline counts the body bytes actually received, so a
  chunked body over the limit gets 413 instead of reaching the app;
- logout revokes through a SessionRevocations port and withdraws the
  mark when PostgreSQL rejects the delete, so 503 means the session
  still works; PostgresSessionDirectory loses its no-op invalidate;
- an empty cursor is 400 on history and on the chat list;
- EventBus.project and the test that allowed importing an event across
  zones are gone, the config rules speak of the schema foreign key;
- the worker waits one second less than the rest of the shutdown
  budget, so it cancels a stuck request with 500 and exits 0 instead
  of being killed by the gunicorn master. Lab 3 keeps 3 s after the
  drain window for the database timeouts, so SHUTDOWN_TIMEOUT_SECONDS
  becomes 6 s and the stop hooks wait 7 s, still under the 8 s grace;
- the lab 3 single statement post already locks the chat only for a
  member, so lab 2's LOCK_CHAT edit is not needed; its test is kept;
- lab 2 tests come along with Idempotency-Key where lab 3 needs it;
  the short TTL after restart runs without api-shortttl by shortening
  the session in the database; test_drain adds a stuck request;
- make validate uses .venv with openapi-spec-validator, PYTEST_ARGS,
  faults-evidence, down with the bench profile, the image is tagged by
  the compose project, start probes every 200 ms, cache evidence fails
  without a tombstone and checks the key expiry, kdf.txt shows the
  seed stub apart.

The track 2 tooling of lab 2 (track.py, postgres-restore and the
database argument of migrate) and its evidence scripts stay out.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 02:32:49 +03:00
ArtyomandClaude Opus 5.5 79a52bab58 Propose carrying over the lab 2 fixes as an openspec change
Lab 3 was carried over from lab 2 at 8cddb09, and lab 2 changed up to
102579e after two reviews. The change brings those system fixes here:
the chunked body limit, a worker that cancels a stuck request before
the master kills it, a logout that withdraws its revocation mark when
PostgreSQL rejects it, the empty cursor, the zone rule kept to the
schema and the toolchain and stand tests. To keep the lab 3 rule that
draining leaves the database timeouts their full 3 s, the shutdown
budget grows from 5 to 6 s and the stop hooks from 6 to 7 s.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 02:32:12 +03:00
ArtyomandClaude Opus 5.5 6fafc8a784 Record a clean clone run without an explicit BASE_URL
The launch block of the report now works verbatim with a changed
LB_HTTP_PORT: in a fresh clone with port 18083 make test, both
failovers, persistence and storage-evidence pass without BASE_URL, the
second failover stops the other replica after one shifting request, and
storage-evidence lists only the volumes of the clone project.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 23:26:05 +03:00
ArtyomandClaude Opus 5.5 e722cb357d Record levels 3 and 4 again on the final code
Every level 3 and level 4 file in evidence was taken on e00eb6c, before
track 1 rewrote the message write into one statement under the chat
lock. The runs are repeated on a02d442: smoke, stack, storage, both
failovers (the second after one shifting request), persistence, hooks,
the lab 2 fault tests, limits and injected violations, 20 same-key races,
three detection runs, failover-each under k6 and both rolling updates.
All pass. checks.txt adds the real output of make check, make validate
and make lint, and student-tests.txt includes the new access order test.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 23:24:11 +03:00
ArtyomandClaude Opus 5.5 a02d442085 Trim the scaling cost proposal to the word limit
The openspec rules keep a proposal within 400 words; the archived track 1
proposal had 405. Three phrasings outside the hypotheses are shortened,
the hypotheses stay as they were written before the runs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 23:13:01 +03:00
ArtyomandClaude Opus 5.5 956d24a701 Repeat detection, add track 1 control runs and isolate image tags
BASE_URL defaults to the published port of lb from docker compose, so
the targets work with a changed LB_HTTP_PORT. rolling-update FROM_REV
and track1-compare tag the old revision by its commit instead of the
shared mtusi-messenger tag, which a parallel clone could overwrite.
detection runs three times to show the spread, failover-each records
docker ps -a of the replicas right after the stop, and failover-evidence
takes SHIFT to move round robin to the other replica. track1-control
varies the pool and the balancer queue at 8x1 and compares 8x0.5 with
4x1 on the same CPUs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 23:12:55 +03:00
ArtyomandClaude Opus 5.5 16b6b28ea9 Keep the full output of a failed compose up in track 1 runs
The orchestrator printed only the last 500 characters of stderr before
retrying, which hid the cause of a flaky up on eight replicas. The whole
output of each failed attempt now goes to compose-up-failed.log next to
summary.csv.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 23:12:55 +03:00
ArtyomandClaude Opus 5.5 0ce7d6ecf0 Make storage and limits evidence reproducible from any clone
storage-evidence listed volumes by the name mtusi2026lab3, so a clone in
another directory recorded the volumes of the main stand; it now takes
the project name from docker compose config and filters by its label.
make limits wrote only the date, while every evidence file promises the
date and the commit in its first line; the commit is added.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 23:12:55 +03:00
ArtyomandClaude Opus 5.5 d4a3124a32 List gunicorn_conf in the module map of the openspec context
Since level 4 the module holds platform behaviour, the drain window on
SIGTERM, so the architecture map names it next to main and migrate.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 23:12:45 +03:00
ArtyomandClaude Opus 5.5 e9242816a5 Drop the type ignore comment from error classification
ty accepts the lookup without it, and the project keeps code free of
comments.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 23:12:45 +03:00
ArtyomandClaude Opus 5.5 a761b718b8 Check the request shape before chat access on posting a message
FastAPI rejects a missing Idempotency-Key, extra fields and wrong types
with 400 before the handler runs, so an outsider gets the same 400 for
an existing and a missing chat; only a bad text, checked by the domain,
makes the handler look up membership and answer 404. The new test pins
that order so it is a recorded rule rather than an accident.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 23:12:45 +03:00
ArtyomandClaude Opus 5.5 be17939ddb Propose closing the lab 3 review gaps as an openspec change
An independent review of the grade 5 submission found that the level 4
measurements under load and the fault tests were taken on the code
before track 1 rewrote the message write, that the claim that the pool
and the balancer are not the bottleneck rests on reasoning alone, and
that the retrograde 8x1 result is not separated from running on other
vCPUs. Some evidence tooling also writes data of another compose
project or cannot reproduce a committed file.

The change records the order of checks on posting a message (401,
request shape 400, access 404, text 400), requires three detection runs
with the 0.25 s measurement tolerance, a docker ps of replicas after
each stop in failover-each, an upper bound of the rate grid, and control
runs that vary the pool, the balancer queue and the CPU placement one at
a time.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 23:12:16 +03:00
ArtyomandClaude Opus 5.5 e425a9d236 Finalize the lab 3 report against recorded evidence
Rerun the clean clone pass on the final commit in its own compose project:
secrets, config, up with migrations on empty volumes, readiness, smoke,
replica spread and same key races, failover, persistence, repeated
migrations, down keeping volumes and reset-data removing them. Record the
student tests by name and the limits policy on the final code.

Check every figure of the report against evidence files and fix the ones
that drifted: data set sizes, the share of the chat lock statement, the
WAL waits, the aborted series, the disturbed steps, the restart timing
and the image ID of the rolling update. State which commit each evidence
file was taken on, answer the defense questions, index evidence/ and
bring AGENTS.md and the openspec context up to date for lab 3.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 03:13:24 +03:00
ArtyomandClaude Opus 5.5 e88673dfec Archive the scaling cost change
Track 1 of grade 5 is closed. At a fixed 4 CPU / 2 GiB the 1x4, 2x2
and 4x1 layouts peak at 5 375, 4 964 and 4 724 rps: within the
hypothesised 15 percent, yet smaller replicas cost a third more CPU
per request and reach a long tail earlier. Growing one-CPU replicas
gives 1 720, 2 936, 4 724 and only 4 260 rps at eight, a retrograde
curve with USL sigma 0.014 and kappa 0.038. The saturation point is
the chat row lock in PostgreSQL: at eight replicas 47.5 of 64
connections wait for it while the API, the balancer and the database
CPU still have room. Writing a message with one statement removed the
application round trips from under the lock: p95 at 3 600 rps fell
from 64.5 to 3.7 ms and the peak rose by 15 % with the same resources.

The new lab3-scaling-cost spec is synced to openspec/specs, with the
compare scenario stated as measured: below the knee fewer lock waits
and a lower p95, past it a higher peak. AGENTS.md lists make track1
and make plots. The neighbour CPU sampler is stopped and its file
closed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 01:42:38 +03:00
ArtyomandClaude Opus 5.5 3e63b16ca6 Report track 1 with plots and the checks after the improvement
make plots draws throughput and latency against offered load for both
series and the comparison, CPU per component, peak RPS against API
CPU with the USL fit and the improved 8x1 point, and the utilisation of
every link at each peak; capacity.md tabulates the ceilings.

after-checks.txt holds the level 3 and 4 checks on the new image:
student tests 240 passed, smoke PASS, 20 same-key races of 8 through
the balancer with 160 x 201 and one copy each, failover with 131
writes and no failure in 2.73 s, persistence PASS.

REPORT.md gains the track section: hypotheses as stated before the
runs, the method and its equal-resource rules, both series in tables,
the saturation point with database, balancer and pool figures, the
USL reading, the before and after table, conclusions per hypothesis
and the limits of a shared single host. The idempotent send section
now describes the one-statement write, the evidence table has the
track rows and the commands are in the clean clone section.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 01:42:08 +03:00
ArtyomandClaude Opus 5.5 dde113b4ed Record the before and after comparison of track 1
8x1 with the image of edc091d (before) and the one-statement write
(after), alternating per repeat, three repeats at 3 600, 5 400 and
8 100 rps. At 3 600 rps, below the knee, p95 falls from 64.5 ms
(35.7..142.5) to 3.7 ms (2.8..5.4) and p99 from 122 to 13 ms; the API
spends 391 instead of 457 % CPU and PostgreSQL 102 instead of 137 %;
connections waiting for a chat lock drop from 12.8 to 0.15 and none is
idle in transaction any more. Past the knee the peak rises from 4 252
(4 054..4 396) to 4 894 rps (4 494..5 104), +15 %, and at 8 100 rps
from 4 104 to 4 841 (+18 %); the ranges do not overlap. Saturated, the
lock holders now wait for the WAL flush (LWLock WALWrite) instead of
the application.

The switch to the after image for the third repeat failed in compose
while another lab was running tests on the shared VM, so that repeat
was run alone with --first-repeat 3 on the same database; apply now
retries compose up twice and prints its stderr. Two after rows were
still marked disturbed after three reruns and are kept; they are the
lower ones.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 01:37:34 +03:00
ArtyomandClaude Opus 5.5 296e070eba Add the before and after comparison of track 1
make track1-compare builds the image of BEFORE_REV, the last commit
without the one-statement write, builds the current one, and runs 8x1
at 3 600, 5 400 and 8 100 rps with the two images alternating per
repeat, so drift of the shared VM affects both alike. make track1 runs
the host snapshot, both series, the comparison and the plots.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 01:06:49 +03:00
ArtyomandClaude Opus 5.5 22f0e6244d Post a message with one statement under the chat lock
Sending a message used to be BEGIN, a SELECT ... FOR UPDATE of the
chat with its members, a lookup of the idempotency key, the insert CTE
and COMMIT. Three of those round trips ran under the chat row lock, and
each one waited for the replica's event loop, so under load the lock
holders sat idle in transaction while everyone else queued behind
them.

The new MessageLog port has one PostgreSQL adapter that does it all in
a single statement with an implicit transaction: lock the chat row
joined with the sender's membership, stamp the message with the later
of the replica clock and the last message plus one microsecond, claim
the key (overwriting it only if it has expired), insert the message
only when the key was claimed, and move last_message_at. The foreign
key from the key to the message is checked at the end of the statement,
so the key may be written first. When the key is already taken the
adapter reads the stored result with a second statement outside the
lock, which also covers a concurrent duplicate that waited for the
first one; the handler then replays it or answers 409 as before.

The chat aggregate no longer takes part in posting, so its post,
membership and last_message_at members and their unit tests are gone;
Message.posted() builds the event. An invalid text is still answered
404 for an outsider: only then the handler reads the membership. A new
API test pins the clock rule against a real database: a lagging
replica clock gets the last stamp plus one microsecond, a clock ahead
is kept. The concurrency, same-key race, expiry and access tests pass
unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 01:06:49 +03:00
ArtyomandClaude Opus 5.5 edc091dbc4 Choose a one-statement message write as the track 1 improvement
Series B located the bottleneck. At eight replicas the API has 40 % of
its CPU spare and the balancer 60 % of a core, yet three quarters of
the pool connections wait for a chat row lock: the locking SELECT of
the chat averages 14.2 ms and the lock holders sit idle in transaction
while the replica's event loop gets round to the next statement. The
critical section, three round trips long, is what more replicas
lengthen.

The design now records the choice: one CTE that locks the chat with
the membership check, stamps the message no earlier than the last one,
claims the idempotency key and inserts the message, committed by the
statement itself, so the lock never waits for the application. A taken
key is read back by a second statement outside the lock. The price is
that the chat aggregate no longer guards posting; the rules move into
the adapter's SQL and are held by integration tests. The rejected
alternatives and the before/after plan are listed, and the spec gains
the observable requirement of a short critical section.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 01:06:10 +03:00
ArtyomandClaude Opus 5.5 3bdc8fad23 Record the growth series of track 1
One to eight replicas of 1 CPU, 512 MiB and one worker each, three
repeats per rate; the 4x1 point comes from series A. Peak successful
RPS: 1 720 for one replica, 2 936 for two, 4 724 for four and only
4 260 for eight, lower than four. Up to two replicas the API is the
limit (101 % and 201 % of its CPU). At eight replicas the API uses 473
of its 800 % and the balancer 40 % of a core, while PostgreSQL is at
178 of 200 %, 47.5 of the 64 pool connections wait for a chat row lock
and the balancer queue holds 744 requests. The locking SELECT ... FOR
UPDATE of the chat takes 14.2 ms on average, 97 % of all statement
time of the step, and about 7 connections sit idle in transaction:
the lock holders wait for the application between round trips.

A universal scalability law fit over 1, 2, 4 and 8 CPUs gives
sigma 0.014 and kappa 0.038, a peak near five replicas and a decline
after it. No step of this series was rejected as disturbed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 01:05:39 +03:00
ArtyomandClaude Opus 5.5 be19abf364 Record the fixed budget series of track 1
1x4, 2x2 and 4x1 at 4 CPU and 2 GiB of API, workers equal to CPUs,
32 pool connections and 128 balancer slots in every layout, three
repeats from 600 to 6 830 rps. All three keep up with the offered
load up to 4 560 rps; the peaks are 5 375 rps for 1x4 (5 332..5 384),
4 964 for 2x2 (4 894..5 104) and 4 724 for 4x1 (4 686..4 794). At the
peak the API uses 95 % of its four CPUs in every layout, PostgreSQL
82..89 % of its two, the balancer 40 % of one, and about half of the
32 database connections wait for the lock on a chat row.

Smaller replicas cost more: at 3 040 rps the API burns 215 % CPU as
1x4, 243 % as 2x2 and about 280 % as 4x1, and p95 at 4 560 rps grows
from 9 to 69 and 150 ms.

The first attempt was aborted when a neighbour's image build froze the
Docker API; its summary is kept under aborted/. Twelve steps of the
rerun were rejected as disturbed and repeated. A config's SLO capacity
now needs a majority of saturated repeats, since a median is robust to
one disturbed repeat and "any repeat" is not. Series B starts at 450
rps per CPU.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 00:31:43 +03:00
ArtyomandClaude Opus 5.5 67f6d6866e Reject steps disturbed by foreign load on the shared VM
The first run of series A was spoiled by other stands on the same
Docker VM: an image build of another lab is invisible to docker stats,
yet it took CPU from the pinned replicas, doubled their CPU per request
and stalled the Docker API for fifteen minutes in the middle of two
steps.

docker-stats.sh now also samples the VM-wide /proc/stat counters, so a
step records the CPU of the whole VM, the part not used by this
project, the largest gap between samples and its wall time. A step
with more than one foreign core, a gap over 10 s or a k6 run 90 s
longer than its load is written to disturbed.csv and repeated after
the VM has been quiet for 15 s, up to three times. On a quiet VM the
foreign part is 2 % of a core at 600 rps and 25 % at 4 560 rps, which
is Docker networking and the samplers themselves.

The growth series starts at 300 rps per CPU and searches up to 9 000,
so its knees are found with fewer idle steps. The plots fit the
universal scalability law to the growth series.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 22:18:14 +03:00
ArtyomandClaude Opus 5.5 e8804b05d0 Plot track 1 series and tabulate the ceilings
tools/plot_track1.py reads any set of summary.csv files and draws, per
series, successful RPS, p95 and p99 against the offered rate and CPU
per component against its limit, plus two cross-series charts: peak
successful RPS against total API CPU with the linear growth line, and
the utilisation of API, PostgreSQL, balancer and the share of the pool
waiting on a lock at each config's peak. A point is the median of the
repeats, the whiskers are min and max. capacity.csv and capacity.md
hold the SLO capacity, the peak and the resource figures at the peak.

Colours follow the config across charts, every line also has its own
marker and a direct label. matplotlib comes through uv run --with, so
the tool lives in tools/ and stays out of the 3.9 scripts; make plots
runs it, make lint covers it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 20:35:05 +03:00
ArtyomandClaude Opus 5.5 8ea1996063 Add the series orchestrator and samplers for track 1
scripts/scaling.py applies a replicas x CPU layout through compose
variables: workers equal to the replica CPUs, a pool of 8 per worker,
a balancer queue of 32 per worker so every layout of a series offers
the same concurrency to the API, and the replicas pinned to CPUs 7 and
up, next to PostgreSQL on 0-1, the balancer on 2, Valkey on 3 and k6
on 4-6 with 3 CPU and 1 GiB. For every offered rate it runs k6 three
times and stops a config after two steps saturated in all repeats.

Each step keeps its raw data: k6 summary, docker stats, HAProxy
statistics and pg_stat_activity sampled every second, and the top of
pg_stat_statements. The summary row carries CPU per component,
PostgreSQL active, idle, idle in transaction and lock waiting backends,
the balancer queue and the database size. Several variants alternate
per repeat for before and after runs. host-snapshot.sh records every
container on the host before a series.

make track1-a and track1-b run the fixed budget and the growth series
from an empty database.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 20:25:23 +03:00
ArtyomandClaude Opus 5.5 107fbbbe15 Let measurements pin each component to its own CPUs
Track 1 compares replica layouts that must get exactly the same CPUs,
and the load generator, balancer, Valkey and PostgreSQL must not steal
cycles from the replicas or from each other. Every service now takes
an optional cpuset from LB_CPUSET, API_CPUSET, PG_CPUSET,
VALKEY_CPUSET and K6_CPUSET. They are empty by default and compose
drops an empty cpuset, so the ordinary stand is unchanged and make
limits still passes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 20:25:23 +03:00
ArtyomandClaude Opus 5.5 d8d551aae4 Propose the scaling cost track as an openspec change
Track 1 of grade 5 asks to compare one, two and four API replicas at a
fixed total CPU and memory budget, to grow resources until a shared
dependency saturates, to show with database, balancer and pool metrics
why more replicas stop helping, and to confirm one improvement with a
before and after measurement.

The hypotheses are stated before any run: at 4 CPU / 2 GiB the 1x4,
2x2 and 4x1 layouts reach the same ceiling within 15 percent; with one
CPU per replica throughput grows almost linearly up to four replicas
and by less than 1.5 times from four to eight, because PostgreSQL
saturates on the chat row lock held across three round trips and on
CPU, while the balancer stays far from its limit.

The new lab3-scaling-cost capability fixes the method: open-model k6
with the level 4 profile, three repeats per point, equal workers, pool
and balancer concurrency across layouts, CPU pinning per component,
raw samples per step, plots from summary files, and an improvement
chosen only after the bottleneck is found.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 20:16:19 +03:00
ArtyomandClaude Opus 5.5 61c0c28a43 Archive the failover operations change
Grade 4 of lab 3 is closed. A replica on SIGTERM first answers 503 on
readiness for a two second window, longer than the 1.5 s worst
detection, and only then closes its listener and finishes accepted
requests within five seconds; the balancer lets a replica back only
after two passed checks. A test pins the timeout chain to compose,
HAProxy and the stop hook. make detection measured 0.71 s for SIGTERM
(before the process exited at 2.49 s), 0.67 s for a killed process,
2.76 s for a hung one and 1.23 s until no replica was left with
PostgreSQL down. make failover-each stopped and restored each replica
under k6 at 100 rps with no errors and no balancer retries, make
lb-check ran 20 same-key races of 8, make limits checks limits and
isolation against a policy, and make rolling-update replaced the
level 3 image with the current one under load with capacity kept at
two and no errors, migrate running once. The new
lab3-failover-operations spec is synced, and the lab 1 operability
and lab 2 dependency failure specs carry their deltas.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 20:08:15 +03:00
ArtyomandClaude Opus 5.5 a9aab2727f Record level 4 evidence and extend the report
Evidence from a freshly recreated stand at the current code: the four
detection cases with their declared maxima, failover-each for both
replicas under k6 at 100 rps (no errors, no balancer retries, the
stopped replica never served during its outage), the 20 round
same-key race, the limits and isolation check with a run against
deliberate violations, and two rolling updates under load: from the
level 3 image, where the old replicas without a drain window caused
54 redispatches and 2 response errors that the balancer retried, and
between two current builds with all counters at zero. Level 3 runs
are refreshed on the new code: smoke, stack view, both failovers with
the drain sequence in the replica log, hooks with the longer stop,
persistence, stack fault tests and own tests with the shutdown and
timeout chain output.

The report gains the shutdown order, the check period and worst
detection per failure with measurements, the timeout chain, rows 4.1
to 4.4, level 4 rows of the failure table, the k6 role, the new
variables and the updated limitations; AGENTS.md lists the commands.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 20:07:51 +03:00
ArtyomandClaude Opus 5.5 6107c1aa34 Show the drain sequence of the stopped replica in failover evidence
The failover evidence printed the last six log lines of the stopped
replica, which no longer reach back to the signal now that shutdown
starts with a two second drain window. It now filters the replica log
since the start of the run for the signal, the drain message, the 503
readiness answers, the listener shutdown and the process exit.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 20:07:44 +03:00
ArtyomandClaude Opus 5.5 e00eb6c878 Keep probe logs out of the ignored log pattern
The repository ignores *.log, so the action log of a load run (with
the rolling update replacement lines) and the k6 output would never
reach evidence/. The probe now writes action.txt and k6/k6.txt.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 19:52:55 +03:00
ArtyomandClaude Opus 5.5 eb04fc10e3 Give every level 4 check one make target
make limits, make lb-check (now 20 rounds of 8 parallel posts with one
key), make detection, make failover-each and make rolling-update each
write their file in evidence/ and fail on a violated check; make
level4 runs them in order. RATE sets the k6 arrival rate (100 rps by
default). make rolling-update builds the image before the load starts
so the build does not skew the measurement; with FROM_REV=<commit> it
first builds the image tag from git archive of that commit and
recreates the replicas on it, so the update is a real change of
version, and it writes to evidence/rolling-from-<commit>. Lint covers
scripts/limits.py. CI acceptance runs make limits and make detection.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 19:52:29 +03:00
ArtyomandClaude Opus 5.5 3584501de8 Check that the balancer drops every replica while PostgreSQL is down
The stack test for a stopped PostgreSQL asked for liveness 200 through
the balancer, which only held while the balancer had not noticed the
outage yet. Now the protected endpoints and readiness through the LB
must answer 503 unavailable within 2 s, liveness is checked inside a
replica with docker exec, the balancer must report zero UP servers
within 2 s and answer liveness itself with 503 in the contract error
format, the request id echoed and no X-Instance-Id. After PostgreSQL
starts again every replica must be back in rotation without a restart.

Stand helpers read the number of UP servers from the HAProxy stats,
excluding the backend summary row, and reach a replica by container
id, which survives the renumbering after a rolling update.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 19:52:29 +03:00
ArtyomandClaude Opus 5.5 572204950a Replace replicas one by one under load
scripts/rolling-update.sh runs migrate once before touching any
replica, then for each old replica scales the service to N+1 without
recreating the others, waits until the new container is healthy and
the balancer sees N+1 UP servers, and only then stops the old one
with docker stop -t 6 (so it drains) and removes it. The log names
the hostname and image of the old and the new replica of every step,
which is what the report quotes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 19:52:19 +03:00
ArtyomandClaude Opus 5.5 5adb8719d4 Check limits and isolation against a declared policy
scripts/limits.py states which secrets, volumes and networks each
service needs and that only lb publishes ports, on loopback. It
compares that policy with docker compose config (with the bench
profile, so k6 is covered) and with docker inspect of the running
containers: CPU and memory limits declared and applied, no
container_name, no published port on any API replica, a read-only
root filesystem for the API, and which networks are internal. It
prints both tables and every violation and exits non-zero on any.

The stack evidence script no longer reaches replicas through
docker compose exec or port, which assume container number 1 and
break after a rolling update renumbers the replicas.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 19:52:09 +03:00
ArtyomandClaude Opus 5.5 d5f8a7c25c Measure detection and fail every replica in turn under load
scripts/detection.sh runs four failures one after another and brings
the stand back between them: SIGTERM through docker stop -t 6 (the
balancer must exclude the replica within 1.5 s and before the process
exits after its drain window), docker kill (1.5 s), docker pause for a
hung process (3.5 s, the check timeout counts twice) and stopping
PostgreSQL (all replicas out within 1.5 s, the balancer answers 503
itself). A 0.25 s slack covers the docker CLI and the 50 ms polling.
Each run writes evidence/detection/<case>.json and the script fails
if any case exceeds its declared maximum.

scripts/failover-each.sh stops every replica in turn with stop-one.sh,
keeps it down for 10 s and restores it, all inside one k6 load window,
and collects a summary with RPS, errors, latency, detection, rejoin,
probe recovery and balancer retry counters per replica.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 19:52:09 +03:00
ArtyomandClaude Opus 5.5 711c9c98dc Add the k6 load profile and the load and detect probe modes
The k6 profile from the prepared kit runs an open model against the
balancer from the edge network: constant arrival rate, 80 % keyed
message posts and 20 % history reads over 20 synthetic users and 10
chats, metrics only for the measure phase, and no setup data (tokens)
in summary.json. scripts/k6.sh runs it with the host uid so the
summary is writable on Linux too.

scripts/probe.py gains three things. check --rounds repeats the
same-key race with a new key per round and counts rounds with
identical bodies, one stored copy and 409 on another body. load runs
an action under k6 while a probe posts every 0.1 s and resends the
same key after a failure and a poller reads the number of UP servers;
it writes the action log, the probe and LB traces, the delta of the
backend retry, redispatch and error counters, and a verdict: dip for
a replica that leaves and returns, replace for a rolling update that
must keep capacity and end on new replicas only. detect runs the
action in the background and polls the balancer at the same time, so
a drain that ends after the exclusion is visible, and checks the
declared maximum, the LB answer after detection and the time to
restore.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 19:51:59 +03:00
ArtyomandClaude Opus 5.5 502c7d8ace Keep a returning replica out of rotation until its checks pass
The first failover-each run had no errors, yet the balancer counted
19 and 49 redispatches. The HAProxy log showed why: when a stopped
container comes back into Docker DNS, its server-template slot leaves
MAINT (resolution) as UP/READY at once, before any health check, so
requests went to a gunicorn that was not listening yet and were
redispatched to the other replica.

default-server now sets init-state fully-down (HAProxy 3.1+): at start,
on discovery through DNS and on leaving maintenance a server stays DOWN
until rise 2 checks pass. A rerun under 100 rps shows zero retries and
redispatches for both replicas; rejoining takes about a second longer.
The design and the failover spec of the change record the finding.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 19:51:49 +03:00
ArtyomandClaude Opus 5.5 85ed37914d Pin the timeout chain in the stand and test it
The API service now names every timeout that the chain depends on:
DRAIN_SECONDS, DB_ACQUIRE_TIMEOUT_SECONDS and DB_COMMAND_TIMEOUT_SECONDS
join REQUEST_TIMEOUT_SECONDS and SHUTDOWN_TIMEOUT_SECONDS in compose,
and the image gets the drain default. A unit test reads the compose
defaults, the HAProxy check and timeout settings and the stop hook
timeout, and checks that the chain shrinks inwards: LB server 10 s >
request 8 s > pool wait plus statement 3 s, readiness 0.8 s < check
1 s, drain 2 s > worst detection 1.5 s, shutdown 5 s leaves 3 s for
accepted requests, and shutdown < hook 6 s < stop_grace_period 8 s
< 10 s. It also pins the declared detection maxima: 1.5 s for a 503
or a refused connection and 3.5 s for a hung replica.

The k6 load generator joins the stand under the bench profile: pinned
by digest, edge network only, 2 CPU and 512 MiB, no secrets.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 19:33:55 +03:00
ArtyomandClaude Opus 5.5 6fb4587ef6 Leave the rotation before closing the listener on SIGTERM
On SIGTERM gunicorn closed the listener at once, so a request already
queued on the closing socket reached the balancer as a dropped
connection after sending: a retry for GET and keyed messages, a 502
for user, chat and login posts. Now each worker first raises a
draining flag and keeps serving for DRAIN_SECONDS (2 s by default,
above the 1.5 s worst detection of a 503): readiness answers 503
unavailable, liveness stays 200, and the balancer takes the replica
out while it is still healthy. Only then uvicorn closes the listener
and finishes accepted requests within SHUTDOWN_TIMEOUT_SECONDS minus
the window, so gunicorn graceful_timeout stays 5 s.

The worker runs a uvicorn Server subclass whose main loop adds the
window after the exit signal; UvicornWorker._serve is overridden to
use it. Tests check readiness under the flag and run real gunicorn on
a free port: during the window ready is 503 and live 200, the port
closes after the window, a registration accepted before SIGTERM and
finished after the port closed still gets 201, and the master exits
with code 0 before the shutdown limit.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 19:33:02 +03:00
ArtyomandClaude Opus 5.5 6e4c387e50 Answer readiness within the balancer check timeout
The readiness ping waited up to one second for a pooled connection and
one more second for SELECT 1, while HAProxy gives a check one second.
With a busy pool the balancer saw a check timeout instead of an
explicit 503. The ping now waits 0.4 s for a connection and 0.4 s for
the query, so the worst answer, 0.8 s, still arrives inside the check.
Other queries keep the pool acquire and command timeouts.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 19:33:01 +03:00
ArtyomandClaude Opus 5.5 1ae4b770ae Propose failover operations as an openspec change
Level 4 of lab 3 asks for health checks, timeouts and SIGTERM handling
that agree with each other, a stated check period and worst detection
time, stopping every replica in turn under load with RPS, errors and
recovery time, a parallel race on one idempotency key, limits and
port and secret isolation, and a rolling update under load with
migrations run once.

The new lab3-failover-operations capability covers those guarantees.
The lab 1 operability delta makes a replica leave the rotation first
on SIGTERM: readiness answers 503 for a drain window longer than the
balancer detection time, and only then the listener closes. The lab 2
dependency failure delta states that the balancer drops every replica
and answers 503 itself while PostgreSQL is down. The design adds the
measurement tooling from the prepared kit: k6 under a bench profile,
probe load and detect modes, detection, failover-each, limits and
rolling update scripts with one make target each.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 19:30:27 +03:00