Commit Graph
24 Commits
Author SHA1 Message Date
Artyom 7e19d4e33e Reject non-ASCII Basic payloads before decoding 2026-11-06 02:18:38 +03:00
Artyom d60e29221a Archive the reject NUL text inputs change 2026-11-06 02:07:29 +03:00
Artyom 82641cbd3d Archive the lab 3 session cache and proxy review fixes 2026-11-06 01:51:42 +03:00
Artyom 4513d90e48 Finalize lab 3 report and evidence provenance
Tie the grade 5 claim to the final SQLAlchemy revalidation and measured scaling results. Clarify clean-clone setup, failure timings, historical evidence, and operational constraints; record report-stage checks and evidence audits.
2026-11-06 01:29:44 +03:00
Artyom f591fa64e5 Document SQLAlchemy scaling evidence and archive completed changes 2026-11-06 01:21:26 +03:00
Artyom 5c8f605cd3 Archive grade 4 verification and record SQLAlchemy failure evidence 2026-11-05 22:33:29 +03:00
Artyom ef0b9a385e Archive pool recovery and record current grade 3 evidence 2026-11-05 22:08:14 +03:00
Artyom 0a1f48bd73 Recover stale PostgreSQL connections before starting requests 2026-11-05 21:55:14 +03:00
ArtyomandClaude Opus 5.5 7f2deec511 Record levels 3 and 4 again after the second review fixes
Every level 3 and 4 record is taken anew on e049c26 from a reset stand:
checks with the upstream diff of the contract, the public tests and the
task (empty) and of .github (only the own CI job), student tests with
the 503 answer to a request cancelled at shutdown, smoke, stack,
storage on the filled database, both failover runs with the
service.draining event, persistence, hooks, faults, limits, lb-check,
the new retry-check, three detection runs, failover-each and both
rolling updates with the database size, a docker stats snapshot and the
tag and id of every replaced image. The clean clone run executes the
launch block of the report verbatim and marks the observations made
between its steps.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 05:59:38 +03:00
ArtyomandClaude Opus 5.5 e049c265b8 Reproduce a lost answer on the stand and enrich the run records
No failover run ever lost an answer, so the retry of an unconfirmed
write was only covered in process. make retry-check holds the chat row
with SELECT ... FOR UPDATE for 1.5 s (below lock_timeout), sends a
message with a 0.5 s client timeout, finds the replica whose request
waits on the lock in pg_stat_activity and then retries with the same
key and text through the LB, once with that replica alive and once
after docker kill. It checks the 201, a single copy with the id of the
retry answer and, for the live replica, the 201 it logged without a
client; created_at tells which attempt wrote the message. CI runs it.

failover-each and rolling-update now record the database size and a
docker stats snapshot in the background of the action, rolling-update
names the tag and id of every image it replaces, and scaling.py writes
started and the compose-up log marks in UTC with Z; the track README
states that the recorded rows are host time (MSK) and what
host-others.csv covers.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 05:43:44 +03:00
ArtyomandClaude Opus 5.5 fa09339388 Report the control runs of track 1
The LB queue does not move the peak of 8x1 (4526, 4677 and 4525 rps for
maxconn 16, 32 and 64, overlapping ranges), and the pool moves it the
other way: 4 connections per worker give 5358 rps, 16 give 3789, while
the lock waiters grow from 14.7 to 102. So the pool is not short, extra
connections only add writers to the hot chat row, which confirms the
coordination cost of the USL fit. On the same CPUs 8x0.5 peaks 7 % below
4x1 and costs 15 % more CPU per request, so the retrograde 8x1 comes from
the number of processes and connections, not from slower vCPUs.

The report also names the cause of the flaky compose up, the WalSync
stall of the dropped attempt, the disturbed control runs, and the
evidence index gets a commit column for track 1. The scenario of the
open change no longer presumes its own outcome.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 04:45:55 +03:00
ArtyomandClaude Opus 5.5 f4833c40c0 Record the control runs of track 1
make track1-control on b1c80ee: 8x1 at 5400 rps with a pool of 4, 8 and
16 connections per worker and an LB maxconn of 16, 32 and 64 per
replica, then 4x1 against 8x0.5 on the same CPUs 7-10, three repeats
with the variants interleaved.

The first placement attempt is kept in aborted/ only as a summary: four
4x1 runs had the write command at 7-34 ms instead of 0.2-0.6 ms, waiting
on WalSync, which the CPU based rejection does not see. The rerun right
after it is the one in control-cpus. The full output of the two failed
compose up attempts names the cause of the flaky switch: all replicas
are recreated at once and lb turns unhealthy without a ready server.

Plots and capacity.md now include both control series.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 04:45:55 +03:00
ArtyomandClaude Opus 5.5 a67bc6526a Report levels 3 and 4 on the code after the lab 2 carry over
The report now names a59e3a3 as the last change of the application code
and b1c80ee as the commit of every level 3 and 4 record, with the new
replica IDs and numbers: three detection runs within the formula without
the measurement slack, failover-each and both rolling updates without a
failed request, 12 stack tests and 243 own tests. The shutdown section
and the timeout chain follow the carried over budget: drain 2 s, then
3 s for accepted requests and a 500 for a stuck one, graceful timeout
6 s, stop hooks 7 s. make validate runs from .venv, faults evidence has
its own target, AGENTS.md lists the control runs of track 1.

The clean clone is taken again on f5fea5c without an explicit BASE_URL.
The track 1 part keeps the review fixes of the H2 verdict, the grid
bounds and the rounded numbers; the control runs follow.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 03:17:56 +03:00
ArtyomandClaude Opus 5.5 f5fea5c59c Record levels 3 and 4 again after the lab 2 carry over
The carry over of the lab 2 fixes changed the request pipeline, logout,
the worker shutdown budget and the stop hooks, so every level 3 and 4
record is taken again on b1c80ee: checks, own tests, smoke, stack,
storage, failover of both replicas, persistence, hooks, the lab 2 stack
tests, limits and the injected violations, lb-check, three detection
runs, failover-each and both rolling updates under k6.

All of them pass: 243 own tests, 12 stack tests, no failed request under
k6 in failover-each and in both rolling updates.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 02:53:12 +03:00
ArtyomandClaude Opus 5.5 6fafc8a784 Record a clean clone run without an explicit BASE_URL
The launch block of the report now works verbatim with a changed
LB_HTTP_PORT: in a fresh clone with port 18083 make test, both
failovers, persistence and storage-evidence pass without BASE_URL, the
second failover stops the other replica after one shifting request, and
storage-evidence lists only the volumes of the clone project.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 23:26:05 +03:00
ArtyomandClaude Opus 5.5 e722cb357d Record levels 3 and 4 again on the final code
Every level 3 and level 4 file in evidence was taken on e00eb6c, before
track 1 rewrote the message write into one statement under the chat
lock. The runs are repeated on a02d442: smoke, stack, storage, both
failovers (the second after one shifting request), persistence, hooks,
the lab 2 fault tests, limits and injected violations, 20 same-key races,
three detection runs, failover-each under k6 and both rolling updates.
All pass. checks.txt adds the real output of make check, make validate
and make lint, and student-tests.txt includes the new access order test.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 23:24:11 +03:00
ArtyomandClaude Opus 5.5 e425a9d236 Finalize the lab 3 report against recorded evidence
Rerun the clean clone pass on the final commit in its own compose project:
secrets, config, up with migrations on empty volumes, readiness, smoke,
replica spread and same key races, failover, persistence, repeated
migrations, down keeping volumes and reset-data removing them. Record the
student tests by name and the limits policy on the final code.

Check every figure of the report against evidence files and fix the ones
that drifted: data set sizes, the share of the chat lock statement, the
WAL waits, the aborted series, the disturbed steps, the restart timing
and the image ID of the rolling update. State which commit each evidence
file was taken on, answer the defense questions, index evidence/ and
bring AGENTS.md and the openspec context up to date for lab 3.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 03:13:24 +03:00
ArtyomandClaude Opus 5.5 3e63b16ca6 Report track 1 with plots and the checks after the improvement
make plots draws throughput and latency against offered load for both
series and the comparison, CPU per component, peak RPS against API
CPU with the USL fit and the improved 8x1 point, and the utilisation of
every link at each peak; capacity.md tabulates the ceilings.

after-checks.txt holds the level 3 and 4 checks on the new image:
student tests 240 passed, smoke PASS, 20 same-key races of 8 through
the balancer with 160 x 201 and one copy each, failover with 131
writes and no failure in 2.73 s, persistence PASS.

REPORT.md gains the track section: hypotheses as stated before the
runs, the method and its equal-resource rules, both series in tables,
the saturation point with database, balancer and pool figures, the
USL reading, the before and after table, conclusions per hypothesis
and the limits of a shared single host. The idempotent send section
now describes the one-statement write, the evidence table has the
track rows and the commands are in the clean clone section.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 01:42:08 +03:00
ArtyomandClaude Opus 5.5 dde113b4ed Record the before and after comparison of track 1
8x1 with the image of edc091d (before) and the one-statement write
(after), alternating per repeat, three repeats at 3 600, 5 400 and
8 100 rps. At 3 600 rps, below the knee, p95 falls from 64.5 ms
(35.7..142.5) to 3.7 ms (2.8..5.4) and p99 from 122 to 13 ms; the API
spends 391 instead of 457 % CPU and PostgreSQL 102 instead of 137 %;
connections waiting for a chat lock drop from 12.8 to 0.15 and none is
idle in transaction any more. Past the knee the peak rises from 4 252
(4 054..4 396) to 4 894 rps (4 494..5 104), +15 %, and at 8 100 rps
from 4 104 to 4 841 (+18 %); the ranges do not overlap. Saturated, the
lock holders now wait for the WAL flush (LWLock WALWrite) instead of
the application.

The switch to the after image for the third repeat failed in compose
while another lab was running tests on the shared VM, so that repeat
was run alone with --first-repeat 3 on the same database; apply now
retries compose up twice and prints its stderr. Two after rows were
still marked disturbed after three reruns and are kept; they are the
lower ones.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 01:37:34 +03:00
ArtyomandClaude Opus 5.5 3bdc8fad23 Record the growth series of track 1
One to eight replicas of 1 CPU, 512 MiB and one worker each, three
repeats per rate; the 4x1 point comes from series A. Peak successful
RPS: 1 720 for one replica, 2 936 for two, 4 724 for four and only
4 260 for eight, lower than four. Up to two replicas the API is the
limit (101 % and 201 % of its CPU). At eight replicas the API uses 473
of its 800 % and the balancer 40 % of a core, while PostgreSQL is at
178 of 200 %, 47.5 of the 64 pool connections wait for a chat row lock
and the balancer queue holds 744 requests. The locking SELECT ... FOR
UPDATE of the chat takes 14.2 ms on average, 97 % of all statement
time of the step, and about 7 connections sit idle in transaction:
the lock holders wait for the application between round trips.

A universal scalability law fit over 1, 2, 4 and 8 CPUs gives
sigma 0.014 and kappa 0.038, a peak near five replicas and a decline
after it. No step of this series was rejected as disturbed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 01:05:39 +03:00
ArtyomandClaude Opus 5.5 be19abf364 Record the fixed budget series of track 1
1x4, 2x2 and 4x1 at 4 CPU and 2 GiB of API, workers equal to CPUs,
32 pool connections and 128 balancer slots in every layout, three
repeats from 600 to 6 830 rps. All three keep up with the offered
load up to 4 560 rps; the peaks are 5 375 rps for 1x4 (5 332..5 384),
4 964 for 2x2 (4 894..5 104) and 4 724 for 4x1 (4 686..4 794). At the
peak the API uses 95 % of its four CPUs in every layout, PostgreSQL
82..89 % of its two, the balancer 40 % of one, and about half of the
32 database connections wait for the lock on a chat row.

Smaller replicas cost more: at 3 040 rps the API burns 215 % CPU as
1x4, 243 % as 2x2 and about 280 % as 4x1, and p95 at 4 560 rps grows
from 9 to 69 and 150 ms.

The first attempt was aborted when a neighbour's image build froze the
Docker API; its summary is kept under aborted/. Twelve steps of the
rerun were rejected as disturbed and repeated. A config's SLO capacity
now needs a majority of saturated repeats, since a median is robust to
one disturbed repeat and "any repeat" is not. Series B starts at 450
rps per CPU.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 00:31:43 +03:00
ArtyomandClaude Opus 5.5 a9aab2727f Record level 4 evidence and extend the report
Evidence from a freshly recreated stand at the current code: the four
detection cases with their declared maxima, failover-each for both
replicas under k6 at 100 rps (no errors, no balancer retries, the
stopped replica never served during its outage), the 20 round
same-key race, the limits and isolation check with a run against
deliberate violations, and two rolling updates under load: from the
level 3 image, where the old replicas without a drain window caused
54 redispatches and 2 response errors that the balancer retried, and
between two current builds with all counters at zero. Level 3 runs
are refreshed on the new code: smoke, stack view, both failovers with
the drain sequence in the replica log, hooks with the longer stop,
persistence, stack fault tests and own tests with the shutdown and
timeout chain output.

The report gains the shutdown order, the check period and worst
detection per failure with measurements, the timeout chain, rows 4.1
to 4.4, level 4 rows of the failure table, the k6 role, the new
variables and the updated limitations; AGENTS.md lists the commands.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 20:07:51 +03:00
ArtyomandClaude Opus 5.5 957eabdfc3 Record level 3 evidence and fill in the report
Save sanitized outputs from a stand started from a clean state: the
clean clone run, smoke through the balancer, the stack view with the
replica hostnames, ports, networks, limits and the balancer server
states, the probe with two replicas behind one token and the eight
way race on one key, a failover run for each replica with the
container left exited and its graceful shutdown log, the hook
negative cases and the balancer answer without replicas, the
persistence run with StartedAt of every container before and after,
schema, migration rerun, KDF prefix, ignore rules, own tests and the
lab 2 stack fault tests through the balancer, plus the environment.

The report covers the clean clone start, the architecture, the
X-Instance-Id and idempotency decisions, the balancer settings, the
role to service table, the mapping of .env.example to the real
variables, one evidence row per level 3 criterion and the carried
lab 2 minimum, and the failure table for level 3.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 11:53:11 +03:00
Mikhail Verkovykh 0ab3a341aa Подготовить skeleton лабораторной 3 по мессенджеру 2026-09-10 18:01:35 +03:00