Commit Graph
75 Commits
Author SHA1 Message Date
Artyom 7e19d4e33e Reject non-ASCII Basic payloads before decoding 2026-11-06 02:18:38 +03:00
Artyom f7bfd1d473 Propose non-ASCII Basic rejection as an openspec change 2026-11-06 02:17:03 +03:00
Artyom d60e29221a Archive the reject NUL text inputs change 2026-11-06 02:07:29 +03:00
Artyom e66e2d7d68 Reject NUL input before authentication and text persistence 2026-11-06 02:05:09 +03:00
Artyom 5c561eac26 Propose rejecting NUL text inputs as an openspec change 2026-11-06 02:02:16 +03:00
Artyom 82641cbd3d Archive the lab 3 session cache and proxy review fixes 2026-11-06 01:51:42 +03:00
Artyom 2af24d7762 Prevent revoked session refills and escape proxy error identifiers 2026-11-06 01:49:41 +03:00
Artyom da2f392442 Propose session cache and load balancer review fixes 2026-11-06 01:46:17 +03:00
Artyom f591fa64e5 Document SQLAlchemy scaling evidence and archive completed changes 2026-11-06 01:21:26 +03:00
Artyom 57744c532c Cache the immutable message statement and validate scaling evidence 2026-11-05 23:59:54 +03:00
Artyom 05d69ba6ac Preserve scaling data and reject contaminated benchmark attempts 2026-11-05 22:42:51 +03:00
Artyom 74a39fa93e Propose remeasuring SQLAlchemy scaling as an OpenSpec change 2026-11-05 22:40:04 +03:00
Artyom 5c8f605cd3 Archive grade 4 verification and record SQLAlchemy failure evidence 2026-11-05 22:33:29 +03:00
Artyom 252e3e3ffe Measure successful recovery and verify history under load 2026-11-05 22:15:18 +03:00
Artyom 8a7e027925 Propose verifying grade 4 recovery as an OpenSpec change 2026-11-05 22:13:16 +03:00
Artyom ef0b9a385e Archive pool recovery and record current grade 3 evidence 2026-11-05 22:08:14 +03:00
Artyom 0a1f48bd73 Recover stale PostgreSQL connections before starting requests 2026-11-05 21:55:14 +03:00
Artyom 18de595740 Propose restoring PostgreSQL pool recovery as an OpenSpec change 2026-11-05 21:51:20 +03:00
Artyom 7fc911c332 Document SQLAlchemy migration checks and remaining evidence work 2026-11-05 21:41:47 +03:00
ArtyomandClaude Code c376882ce8 Match the tampered migration scenario name in the Alembic delta
Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-11-05 00:47:10 +03:00
ArtyomandClaude Code c5f39dea25 Propose adopting SQLAlchemy and Alembic for the lab 3 storage
User requirement (2026-11-05, labs 3+): all PostgreSQL access goes through
SQLAlchemy 2.x async and migrations run through Alembic. The proposal keeps
every port, zone boundary and observable API behavior in place, moves the
checksum-based migration journal scenario to Alembic revision semantics and
pins the evidence that depends on SQL behavior for regeneration.

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-11-05 00:46:59 +03:00
ArtyomandClaude Opus 5.5 72a1e57467 Archive the second lab 3 review gaps change
The deltas are synced to openspec/specs: a request that outlives the
graceful shutdown gets 503 unavailable in the common error format with
X-Request-Id and X-Instance-Id instead of the bare 500; a lost answer is
retried with the same key both on a live replica and after the replica
holding the request was killed, with one copy and the same id; make up
returns once every running replica is UP in the LB.

Before archiving, on e049c26 from a reset stand: make test, both
failover runs (first write after 2.858 and 2.682 s, 0 failed attempts),
persistence, retry-check, faults and make level4 pass, the clean clone
run follows the launch block verbatim; make check, make lint, 243 own
tests and openspec validate --all --strict pass.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 06:00:40 +03:00
ArtyomandClaude Opus 5.5 e049c265b8 Reproduce a lost answer on the stand and enrich the run records
No failover run ever lost an answer, so the retry of an unconfirmed
write was only covered in process. make retry-check holds the chat row
with SELECT ... FOR UPDATE for 1.5 s (below lock_timeout), sends a
message with a 0.5 s client timeout, finds the replica whose request
waits on the lock in pg_stat_activity and then retries with the same
key and text through the LB, once with that replica alive and once
after docker kill. It checks the 201, a single copy with the id of the
retry answer and, for the live replica, the 201 it logged without a
client; created_at tells which attempt wrote the message. CI runs it.

failover-each and rolling-update now record the database size and a
docker stats snapshot in the background of the action, rolling-update
names the tag and id of every image it replaces, and scaling.py writes
started and the compose-up log marks in UTC with Z; the track README
states that the recorded rows are host time (MSK) and what
host-others.csv covers.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 05:43:44 +03:00
ArtyomandClaude Opus 5.5 fb6ff87bb3 Wait for every replica in make up and check numeric uids in CI
compose up --wait returns once lb is healthy, and lb is healthy with one
replica in rotation; the second one joins after rise 2 checks, about
2.5 s later, so make test-failover right after make up could see a
single X-Instance-Id. make up now runs scripts/lb-wait.sh, which waits
for as many UP servers as there are running api containers.

The CI non-root step compared the third column of docker compose top
with 0, but the daemon prints the user name there (root), so it could
never fail. It now reads the numeric uid of every process of every
container with docker top -eo pid,uid,comm (Docker needs the PID
column); a throwaway root container makes the check fail.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 05:41:13 +03:00
ArtyomandClaude Opus 5.5 d99a48a5db Put readiness behind a port and log the drain window through Logger
The /health/ready route imported the PostgreSQL adapter and kept the
drain flag in the HTTP layer, which gunicorn_conf imported to raise it.
Readiness is now a port of the shared application layer; its adapter in
shared/infrastructure checks the per-process drain flag and pings
PostgreSQL, and PlatformProvider assembles it. The route only knows the
port, and gunicorn_conf raises the flag through the infrastructure
module. Another critical dependency now changes the adapter and the
provider, not the HTTP layer.

The drain window was logged by a uvicorn logger with a printf string;
it is now the service.draining event of a StructlogLogger with the same
fields as the application, and failover-evidence looks for it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 05:40:16 +03:00
ArtyomandClaude Opus 5.5 aa1485bd86 Answer requests cancelled at shutdown with 503 in the contract format
After the drain window uvicorn waits for accepted requests for three
seconds and then cancels their tasks. CancelledError is a BaseException,
so neither the pipeline nor ServerErrorMiddleware saw it and uvicorn
answered with its own text/plain 500 without X-Request-Id and
X-Instance-Id. The pipeline now handles the cancellation next to the
request timeout: it logs request.cancelled, sends 503 unavailable in the
common error format when the response has not started and re-raises,
so uvicorn only closes the transport after the buffered answer.

The declared Content-Length check joins the guard as a rejection to keep
the call within the statement limit; behaviour is unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 05:39:09 +03:00
ArtyomandClaude Opus 5.5 13e856ebd4 Propose closing the second lab 3 review gaps as an openspec change
The second review found that a request cancelled at the end of the
graceful shutdown gets uvicorn's plain 500 without the contract headers,
that readiness reaches the PostgreSQL adapter and a global flag of the
HTTP layer past the ports, that the drain window is logged past the
Logger port, that make up returns with one replica in rotation and that
the CI non-root check can never fail. The retry of an unconfirmed write
was never exercised on the stand either.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 05:37:31 +03:00
ArtyomandClaude Opus 5.5 d3ee257df1 Archive the lab 3 review gaps change
The deltas are synced to openspec/specs: the order of checks on posting
a message (401, then 400 for the request shape without touching the
chat, then 404 for access, then 400 for the text); three detection runs
with a 0.25 s measurement slack and the replica states after each stop
in failover-each; the upper bound of the track 1 grid with a ceiling on
the first grid point stated as a lower bound; control runs of the pool,
the LB queue and the placement.

Before archiving, on the running stand: make test passes, failover
recovers the first write in 2.78 s with 0 failed attempts, persistence
passes; make check, make lint and 243 own tests pass. All level 3 and 4
records were taken on b1c80ee, after the lab 2 carry over.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 04:47:14 +03:00
ArtyomandClaude Opus 5.5 fa09339388 Report the control runs of track 1
The LB queue does not move the peak of 8x1 (4526, 4677 and 4525 rps for
maxconn 16, 32 and 64, overlapping ranges), and the pool moves it the
other way: 4 connections per worker give 5358 rps, 16 give 3789, while
the lock waiters grow from 14.7 to 102. So the pool is not short, extra
connections only add writers to the hot chat row, which confirms the
coordination cost of the USL fit. On the same CPUs 8x0.5 peaks 7 % below
4x1 and costs 15 % more CPU per request, so the retrograde 8x1 comes from
the number of processes and connections, not from slower vCPUs.

The report also names the cause of the flaky compose up, the WalSync
stall of the dropped attempt, the disturbed control runs, and the
evidence index gets a commit column for track 1. The scenario of the
open change no longer presumes its own outcome.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 04:45:55 +03:00
ArtyomandClaude Opus 5.5 a67bc6526a Report levels 3 and 4 on the code after the lab 2 carry over
The report now names a59e3a3 as the last change of the application code
and b1c80ee as the commit of every level 3 and 4 record, with the new
replica IDs and numbers: three detection runs within the formula without
the measurement slack, failover-each and both rolling updates without a
failed request, 12 stack tests and 243 own tests. The shutdown section
and the timeout chain follow the carried over budget: drain 2 s, then
3 s for accepted requests and a 500 for a stuck one, graceful timeout
6 s, stop hooks 7 s. make validate runs from .venv, faults evidence has
its own target, AGENTS.md lists the control runs of track 1.

The clean clone is taken again on f5fea5c without an explicit BASE_URL.
The track 1 part keeps the review fixes of the H2 verdict, the grid
bounds and the rounded numbers; the control runs follow.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 03:17:56 +03:00
ArtyomandClaude Opus 5.5 b1c80ee544 Anonymize the repository path in the faults evidence
make faults-evidence wrote the pytest rootdir with the local home
directory, so the committed faults.txt was cleaned by hand and could not
be reproduced by the command from the report. The target now replaces
the repository path with <repo> and still fails when a stack test fails,
since the pipe through sed hides the pytest exit code.

The open review change also records that the lab 2 carry over moved the
default image tag to the compose project.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 02:42:45 +03:00
ArtyomandClaude Opus 5.5 ac15b2ff62 Archive the carry over of the lab 2 fixes change
Lab 3 now stands on the finished lab 2. The deltas are synced to
openspec/specs: the zones meet only in the schema and Valkey is needed
only to confirm a logout; an outsider does not wait for a chat lock;
an empty cursor is 400 on history and on the chat list; a stuck request
at SIGTERM gets 500 from the server itself within the 6 s budget, which
still leaves 3 s after the drain window for the database timeouts; a
chunked body is limited by the bytes received; a logout rejected by
PostgreSQL withdraws its revocation mark. The event publication purpose
follows lab 2.

On a stand rebuilt from empty volumes: make test passes, failover
recovers the first write in 3.1 s with 0 failed attempts, persistence
passes, test-faults 12 passed including the full restart with logout
and the shortened session after an API restart, detection passes all
four scenarios with SIGTERM detected in 0.82 s and the stop done in
2.44 s, and the cache evidence finds a tombstone and a key that does
not outlive its session.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 02:38:45 +03:00
ArtyomandClaude Opus 5.5 a59e3a39a8 Carry over the lab 2 fixes made after 8cddb09
Brings the system changes of lab 2 from 8cddb09..102579e onto the lab 3
code, resolving each conflict by meaning:

- the request pipeline counts the body bytes actually received, so a
  chunked body over the limit gets 413 instead of reaching the app;
- logout revokes through a SessionRevocations port and withdraws the
  mark when PostgreSQL rejects the delete, so 503 means the session
  still works; PostgresSessionDirectory loses its no-op invalidate;
- an empty cursor is 400 on history and on the chat list;
- EventBus.project and the test that allowed importing an event across
  zones are gone, the config rules speak of the schema foreign key;
- the worker waits one second less than the rest of the shutdown
  budget, so it cancels a stuck request with 500 and exits 0 instead
  of being killed by the gunicorn master. Lab 3 keeps 3 s after the
  drain window for the database timeouts, so SHUTDOWN_TIMEOUT_SECONDS
  becomes 6 s and the stop hooks wait 7 s, still under the 8 s grace;
- the lab 3 single statement post already locks the chat only for a
  member, so lab 2's LOCK_CHAT edit is not needed; its test is kept;
- lab 2 tests come along with Idempotency-Key where lab 3 needs it;
  the short TTL after restart runs without api-shortttl by shortening
  the session in the database; test_drain adds a stuck request;
- make validate uses .venv with openapi-spec-validator, PYTEST_ARGS,
  faults-evidence, down with the bench profile, the image is tagged by
  the compose project, start probes every 200 ms, cache evidence fails
  without a tombstone and checks the key expiry, kdf.txt shows the
  seed stub apart.

The track 2 tooling of lab 2 (track.py, postgres-restore and the
database argument of migrate) and its evidence scripts stay out.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 02:32:49 +03:00
ArtyomandClaude Opus 5.5 79a52bab58 Propose carrying over the lab 2 fixes as an openspec change
Lab 3 was carried over from lab 2 at 8cddb09, and lab 2 changed up to
102579e after two reviews. The change brings those system fixes here:
the chunked body limit, a worker that cancels a stuck request before
the master kills it, a logout that withdraws its revocation mark when
PostgreSQL rejects it, the empty cursor, the zone rule kept to the
schema and the toolchain and stand tests. To keep the lab 3 rule that
draining leaves the database timeouts their full 3 s, the shutdown
budget grows from 5 to 6 s and the stop hooks from 6 to 7 s.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 02:32:12 +03:00
ArtyomandClaude Opus 5.5 6fafc8a784 Record a clean clone run without an explicit BASE_URL
The launch block of the report now works verbatim with a changed
LB_HTTP_PORT: in a fresh clone with port 18083 make test, both
failovers, persistence and storage-evidence pass without BASE_URL, the
second failover stops the other replica after one shifting request, and
storage-evidence lists only the volumes of the clone project.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 23:26:05 +03:00
ArtyomandClaude Opus 5.5 e722cb357d Record levels 3 and 4 again on the final code
Every level 3 and level 4 file in evidence was taken on e00eb6c, before
track 1 rewrote the message write into one statement under the chat
lock. The runs are repeated on a02d442: smoke, stack, storage, both
failovers (the second after one shifting request), persistence, hooks,
the lab 2 fault tests, limits and injected violations, 20 same-key races,
three detection runs, failover-each under k6 and both rolling updates.
All pass. checks.txt adds the real output of make check, make validate
and make lint, and student-tests.txt includes the new access order test.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 23:24:11 +03:00
ArtyomandClaude Opus 5.5 a02d442085 Trim the scaling cost proposal to the word limit
The openspec rules keep a proposal within 400 words; the archived track 1
proposal had 405. Three phrasings outside the hypotheses are shortened,
the hypotheses stay as they were written before the runs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 23:13:01 +03:00
ArtyomandClaude Opus 5.5 956d24a701 Repeat detection, add track 1 control runs and isolate image tags
BASE_URL defaults to the published port of lb from docker compose, so
the targets work with a changed LB_HTTP_PORT. rolling-update FROM_REV
and track1-compare tag the old revision by its commit instead of the
shared mtusi-messenger tag, which a parallel clone could overwrite.
detection runs three times to show the spread, failover-each records
docker ps -a of the replicas right after the stop, and failover-evidence
takes SHIFT to move round robin to the other replica. track1-control
varies the pool and the balancer queue at 8x1 and compares 8x0.5 with
4x1 on the same CPUs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 23:12:55 +03:00
ArtyomandClaude Opus 5.5 16b6b28ea9 Keep the full output of a failed compose up in track 1 runs
The orchestrator printed only the last 500 characters of stderr before
retrying, which hid the cause of a flaky up on eight replicas. The whole
output of each failed attempt now goes to compose-up-failed.log next to
summary.csv.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 23:12:55 +03:00
ArtyomandClaude Opus 5.5 0ce7d6ecf0 Make storage and limits evidence reproducible from any clone
storage-evidence listed volumes by the name mtusi2026lab3, so a clone in
another directory recorded the volumes of the main stand; it now takes
the project name from docker compose config and filters by its label.
make limits wrote only the date, while every evidence file promises the
date and the commit in its first line; the commit is added.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 23:12:55 +03:00
ArtyomandClaude Opus 5.5 d4a3124a32 List gunicorn_conf in the module map of the openspec context
Since level 4 the module holds platform behaviour, the drain window on
SIGTERM, so the architecture map names it next to main and migrate.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 23:12:45 +03:00
ArtyomandClaude Opus 5.5 e9242816a5 Drop the type ignore comment from error classification
ty accepts the lookup without it, and the project keeps code free of
comments.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 23:12:45 +03:00
ArtyomandClaude Opus 5.5 a761b718b8 Check the request shape before chat access on posting a message
FastAPI rejects a missing Idempotency-Key, extra fields and wrong types
with 400 before the handler runs, so an outsider gets the same 400 for
an existing and a missing chat; only a bad text, checked by the domain,
makes the handler look up membership and answer 404. The new test pins
that order so it is a recorded rule rather than an accident.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 23:12:45 +03:00
ArtyomandClaude Opus 5.5 be17939ddb Propose closing the lab 3 review gaps as an openspec change
An independent review of the grade 5 submission found that the level 4
measurements under load and the fault tests were taken on the code
before track 1 rewrote the message write, that the claim that the pool
and the balancer are not the bottleneck rests on reasoning alone, and
that the retrograde 8x1 result is not separated from running on other
vCPUs. Some evidence tooling also writes data of another compose
project or cannot reproduce a committed file.

The change records the order of checks on posting a message (401,
request shape 400, access 404, text 400), requires three detection runs
with the 0.25 s measurement tolerance, a docker ps of replicas after
each stop in failover-each, an upper bound of the rate grid, and control
runs that vary the pool, the balancer queue and the CPU placement one at
a time.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 23:12:16 +03:00
ArtyomandClaude Opus 5.5 e88673dfec Archive the scaling cost change
Track 1 of grade 5 is closed. At a fixed 4 CPU / 2 GiB the 1x4, 2x2
and 4x1 layouts peak at 5 375, 4 964 and 4 724 rps: within the
hypothesised 15 percent, yet smaller replicas cost a third more CPU
per request and reach a long tail earlier. Growing one-CPU replicas
gives 1 720, 2 936, 4 724 and only 4 260 rps at eight, a retrograde
curve with USL sigma 0.014 and kappa 0.038. The saturation point is
the chat row lock in PostgreSQL: at eight replicas 47.5 of 64
connections wait for it while the API, the balancer and the database
CPU still have room. Writing a message with one statement removed the
application round trips from under the lock: p95 at 3 600 rps fell
from 64.5 to 3.7 ms and the peak rose by 15 % with the same resources.

The new lab3-scaling-cost spec is synced to openspec/specs, with the
compare scenario stated as measured: below the knee fewer lock waits
and a lower p95, past it a higher peak. AGENTS.md lists make track1
and make plots. The neighbour CPU sampler is stopped and its file
closed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 01:42:38 +03:00
ArtyomandClaude Opus 5.5 3e63b16ca6 Report track 1 with plots and the checks after the improvement
make plots draws throughput and latency against offered load for both
series and the comparison, CPU per component, peak RPS against API
CPU with the USL fit and the improved 8x1 point, and the utilisation of
every link at each peak; capacity.md tabulates the ceilings.

after-checks.txt holds the level 3 and 4 checks on the new image:
student tests 240 passed, smoke PASS, 20 same-key races of 8 through
the balancer with 160 x 201 and one copy each, failover with 131
writes and no failure in 2.73 s, persistence PASS.

REPORT.md gains the track section: hypotheses as stated before the
runs, the method and its equal-resource rules, both series in tables,
the saturation point with database, balancer and pool figures, the
USL reading, the before and after table, conclusions per hypothesis
and the limits of a shared single host. The idempotent send section
now describes the one-statement write, the evidence table has the
track rows and the commands are in the clean clone section.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 01:42:08 +03:00
ArtyomandClaude Opus 5.5 dde113b4ed Record the before and after comparison of track 1
8x1 with the image of edc091d (before) and the one-statement write
(after), alternating per repeat, three repeats at 3 600, 5 400 and
8 100 rps. At 3 600 rps, below the knee, p95 falls from 64.5 ms
(35.7..142.5) to 3.7 ms (2.8..5.4) and p99 from 122 to 13 ms; the API
spends 391 instead of 457 % CPU and PostgreSQL 102 instead of 137 %;
connections waiting for a chat lock drop from 12.8 to 0.15 and none is
idle in transaction any more. Past the knee the peak rises from 4 252
(4 054..4 396) to 4 894 rps (4 494..5 104), +15 %, and at 8 100 rps
from 4 104 to 4 841 (+18 %); the ranges do not overlap. Saturated, the
lock holders now wait for the WAL flush (LWLock WALWrite) instead of
the application.

The switch to the after image for the third repeat failed in compose
while another lab was running tests on the shared VM, so that repeat
was run alone with --first-repeat 3 on the same database; apply now
retries compose up twice and prints its stderr. Two after rows were
still marked disturbed after three reruns and are kept; they are the
lower ones.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 01:37:34 +03:00
ArtyomandClaude Opus 5.5 296e070eba Add the before and after comparison of track 1
make track1-compare builds the image of BEFORE_REV, the last commit
without the one-statement write, builds the current one, and runs 8x1
at 3 600, 5 400 and 8 100 rps with the two images alternating per
repeat, so drift of the shared VM affects both alike. make track1 runs
the host snapshot, both series, the comparison and the plots.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 01:06:49 +03:00
ArtyomandClaude Opus 5.5 22f0e6244d Post a message with one statement under the chat lock
Sending a message used to be BEGIN, a SELECT ... FOR UPDATE of the
chat with its members, a lookup of the idempotency key, the insert CTE
and COMMIT. Three of those round trips ran under the chat row lock, and
each one waited for the replica's event loop, so under load the lock
holders sat idle in transaction while everyone else queued behind
them.

The new MessageLog port has one PostgreSQL adapter that does it all in
a single statement with an implicit transaction: lock the chat row
joined with the sender's membership, stamp the message with the later
of the replica clock and the last message plus one microsecond, claim
the key (overwriting it only if it has expired), insert the message
only when the key was claimed, and move last_message_at. The foreign
key from the key to the message is checked at the end of the statement,
so the key may be written first. When the key is already taken the
adapter reads the stored result with a second statement outside the
lock, which also covers a concurrent duplicate that waited for the
first one; the handler then replays it or answers 409 as before.

The chat aggregate no longer takes part in posting, so its post,
membership and last_message_at members and their unit tests are gone;
Message.posted() builds the event. An invalid text is still answered
404 for an outsider: only then the handler reads the membership. A new
API test pins the clock rule against a real database: a lagging
replica clock gets the last stamp plus one microsecond, a clock ahead
is kept. The concurrency, same-key race, expiry and access tests pass
unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 01:06:49 +03:00
ArtyomandClaude Opus 5.5 edc091dbc4 Choose a one-statement message write as the track 1 improvement
Series B located the bottleneck. At eight replicas the API has 40 % of
its CPU spare and the balancer 60 % of a core, yet three quarters of
the pool connections wait for a chat row lock: the locking SELECT of
the chat averages 14.2 ms and the lock holders sit idle in transaction
while the replica's event loop gets round to the next statement. The
critical section, three round trips long, is what more replicas
lengthen.

The design now records the choice: one CTE that locks the chat with
the membership check, stamps the message no earlier than the last one,
claims the idempotency key and inserts the message, committed by the
statement itself, so the lock never waits for the application. A taken
key is read back by a second statement outside the lock. The price is
that the chat aggregate no longer guards posting; the rules move into
the adapter's SQL and are held by integration tests. The rejected
alternatives and the before/after plan are listed, and the spec gains
the observable requirement of a short critical section.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 01:06:10 +03:00