Commit Graph
18 Commits
Author SHA1 Message Date
Artyom 05d69ba6ac Preserve scaling data and reject contaminated benchmark attempts 2026-11-05 22:42:51 +03:00
Artyom 71c643c6d1 Adopt async SQLAlchemy storage and verified Alembic migrations
Keep the message and idempotency receipt in one Core CTE statement with chat-first locking. Preserve rollback, bounded dependency failures, and domain ports.

Replace the SQL runner with Alembic. Adopt legacy databases only after validating their journal and PostgreSQL schema, retain all data, and verify migrations on fresh and populated databases.
2026-11-05 21:38:12 +03:00
ArtyomandClaude Opus 5.5 e049c265b8 Reproduce a lost answer on the stand and enrich the run records
No failover run ever lost an answer, so the retry of an unconfirmed
write was only covered in process. make retry-check holds the chat row
with SELECT ... FOR UPDATE for 1.5 s (below lock_timeout), sends a
message with a 0.5 s client timeout, finds the replica whose request
waits on the lock in pg_stat_activity and then retries with the same
key and text through the LB, once with that replica alive and once
after docker kill. It checks the 201, a single copy with the id of the
retry answer and, for the live replica, the 201 it logged without a
client; created_at tells which attempt wrote the message. CI runs it.

failover-each and rolling-update now record the database size and a
docker stats snapshot in the background of the action, rolling-update
names the tag and id of every image it replaces, and scaling.py writes
started and the compose-up log marks in UTC with Z; the track README
states that the recorded rows are host time (MSK) and what
host-others.csv covers.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 05:43:44 +03:00
ArtyomandClaude Opus 5.5 fb6ff87bb3 Wait for every replica in make up and check numeric uids in CI
compose up --wait returns once lb is healthy, and lb is healthy with one
replica in rotation; the second one joins after rise 2 checks, about
2.5 s later, so make test-failover right after make up could see a
single X-Instance-Id. make up now runs scripts/lb-wait.sh, which waits
for as many UP servers as there are running api containers.

The CI non-root step compared the third column of docker compose top
with 0, but the daemon prints the user name there (root), so it could
never fail. It now reads the numeric uid of every process of every
container with docker top -eo pid,uid,comm (Docker needs the PID
column); a throwaway root container makes the check fail.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 05:41:13 +03:00
ArtyomandClaude Opus 5.5 b1c80ee544 Anonymize the repository path in the faults evidence
make faults-evidence wrote the pytest rootdir with the local home
directory, so the committed faults.txt was cleaned by hand and could not
be reproduced by the command from the report. The target now replaces
the repository path with <repo> and still fails when a stack test fails,
since the pipe through sed hides the pytest exit code.

The open review change also records that the lab 2 carry over moved the
default image tag to the compose project.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 02:42:45 +03:00
ArtyomandClaude Opus 5.5 a59e3a39a8 Carry over the lab 2 fixes made after 8cddb09
Brings the system changes of lab 2 from 8cddb09..102579e onto the lab 3
code, resolving each conflict by meaning:

- the request pipeline counts the body bytes actually received, so a
  chunked body over the limit gets 413 instead of reaching the app;
- logout revokes through a SessionRevocations port and withdraws the
  mark when PostgreSQL rejects the delete, so 503 means the session
  still works; PostgresSessionDirectory loses its no-op invalidate;
- an empty cursor is 400 on history and on the chat list;
- EventBus.project and the test that allowed importing an event across
  zones are gone, the config rules speak of the schema foreign key;
- the worker waits one second less than the rest of the shutdown
  budget, so it cancels a stuck request with 500 and exits 0 instead
  of being killed by the gunicorn master. Lab 3 keeps 3 s after the
  drain window for the database timeouts, so SHUTDOWN_TIMEOUT_SECONDS
  becomes 6 s and the stop hooks wait 7 s, still under the 8 s grace;
- the lab 3 single statement post already locks the chat only for a
  member, so lab 2's LOCK_CHAT edit is not needed; its test is kept;
- lab 2 tests come along with Idempotency-Key where lab 3 needs it;
  the short TTL after restart runs without api-shortttl by shortening
  the session in the database; test_drain adds a stuck request;
- make validate uses .venv with openapi-spec-validator, PYTEST_ARGS,
  faults-evidence, down with the bench profile, the image is tagged by
  the compose project, start probes every 200 ms, cache evidence fails
  without a tombstone and checks the key expiry, kdf.txt shows the
  seed stub apart.

The track 2 tooling of lab 2 (track.py, postgres-restore and the
database argument of migrate) and its evidence scripts stay out.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-30 02:32:49 +03:00
ArtyomandClaude Opus 5.5 956d24a701 Repeat detection, add track 1 control runs and isolate image tags
BASE_URL defaults to the published port of lb from docker compose, so
the targets work with a changed LB_HTTP_PORT. rolling-update FROM_REV
and track1-compare tag the old revision by its commit instead of the
shared mtusi-messenger tag, which a parallel clone could overwrite.
detection runs three times to show the spread, failover-each records
docker ps -a of the replicas right after the stop, and failover-evidence
takes SHIFT to move round robin to the other replica. track1-control
varies the pool and the balancer queue at 8x1 and compares 8x0.5 with
4x1 on the same CPUs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 23:12:55 +03:00
ArtyomandClaude Opus 5.5 296e070eba Add the before and after comparison of track 1
make track1-compare builds the image of BEFORE_REV, the last commit
without the one-statement write, builds the current one, and runs 8x1
at 3 600, 5 400 and 8 100 rps with the two images alternating per
repeat, so drift of the shared VM affects both alike. make track1 runs
the host snapshot, both series, the comparison and the plots.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 01:06:49 +03:00
ArtyomandClaude Opus 5.5 be19abf364 Record the fixed budget series of track 1
1x4, 2x2 and 4x1 at 4 CPU and 2 GiB of API, workers equal to CPUs,
32 pool connections and 128 balancer slots in every layout, three
repeats from 600 to 6 830 rps. All three keep up with the offered
load up to 4 560 rps; the peaks are 5 375 rps for 1x4 (5 332..5 384),
4 964 for 2x2 (4 894..5 104) and 4 724 for 4x1 (4 686..4 794). At the
peak the API uses 95 % of its four CPUs in every layout, PostgreSQL
82..89 % of its two, the balancer 40 % of one, and about half of the
32 database connections wait for the lock on a chat row.

Smaller replicas cost more: at 3 040 rps the API burns 215 % CPU as
1x4, 243 % as 2x2 and about 280 % as 4x1, and p95 at 4 560 rps grows
from 9 to 69 and 150 ms.

The first attempt was aborted when a neighbour's image build froze the
Docker API; its summary is kept under aborted/. Twelve steps of the
rerun were rejected as disturbed and repeated. A config's SLO capacity
now needs a majority of saturated repeats, since a median is robust to
one disturbed repeat and "any repeat" is not. Series B starts at 450
rps per CPU.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1d3RLHNBBMA6UqBLLJv27
2026-10-29 00:31:43 +03:00
ArtyomandClaude Opus 5.5 67f6d6866e Reject steps disturbed by foreign load on the shared VM
The first run of series A was spoiled by other stands on the same
Docker VM: an image build of another lab is invisible to docker stats,
yet it took CPU from the pinned replicas, doubled their CPU per request
and stalled the Docker API for fifteen minutes in the middle of two
steps.

docker-stats.sh now also samples the VM-wide /proc/stat counters, so a
step records the CPU of the whole VM, the part not used by this
project, the largest gap between samples and its wall time. A step
with more than one foreign core, a gap over 10 s or a k6 run 90 s
longer than its load is written to disturbed.csv and repeated after
the VM has been quiet for 15 s, up to three times. On a quiet VM the
foreign part is 2 % of a core at 600 rps and 25 % at 4 560 rps, which
is Docker networking and the samplers themselves.

The growth series starts at 300 rps per CPU and searches up to 9 000,
so its knees are found with fewer idle steps. The plots fit the
universal scalability law to the growth series.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 22:18:14 +03:00
ArtyomandClaude Opus 5.5 e8804b05d0 Plot track 1 series and tabulate the ceilings
tools/plot_track1.py reads any set of summary.csv files and draws, per
series, successful RPS, p95 and p99 against the offered rate and CPU
per component against its limit, plus two cross-series charts: peak
successful RPS against total API CPU with the linear growth line, and
the utilisation of API, PostgreSQL, balancer and the share of the pool
waiting on a lock at each config's peak. A point is the median of the
repeats, the whiskers are min and max. capacity.csv and capacity.md
hold the SLO capacity, the peak and the resource figures at the peak.

Colours follow the config across charts, every line also has its own
marker and a direct label. matplotlib comes through uv run --with, so
the tool lives in tools/ and stays out of the 3.9 scripts; make plots
runs it, make lint covers it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 20:35:05 +03:00
ArtyomandClaude Opus 5.5 8ea1996063 Add the series orchestrator and samplers for track 1
scripts/scaling.py applies a replicas x CPU layout through compose
variables: workers equal to the replica CPUs, a pool of 8 per worker,
a balancer queue of 32 per worker so every layout of a series offers
the same concurrency to the API, and the replicas pinned to CPUs 7 and
up, next to PostgreSQL on 0-1, the balancer on 2, Valkey on 3 and k6
on 4-6 with 3 CPU and 1 GiB. For every offered rate it runs k6 three
times and stops a config after two steps saturated in all repeats.

Each step keeps its raw data: k6 summary, docker stats, HAProxy
statistics and pg_stat_activity sampled every second, and the top of
pg_stat_statements. The summary row carries CPU per component,
PostgreSQL active, idle, idle in transaction and lock waiting backends,
the balancer queue and the database size. Several variants alternate
per repeat for before and after runs. host-snapshot.sh records every
container on the host before a series.

make track1-a and track1-b run the fixed budget and the growth series
from an empty database.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 20:25:23 +03:00
ArtyomandClaude Opus 5.5 eb04fc10e3 Give every level 4 check one make target
make limits, make lb-check (now 20 rounds of 8 parallel posts with one
key), make detection, make failover-each and make rolling-update each
write their file in evidence/ and fail on a violated check; make
level4 runs them in order. RATE sets the k6 arrival rate (100 rps by
default). make rolling-update builds the image before the load starts
so the build does not skew the measurement; with FROM_REV=<commit> it
first builds the image tag from git archive of that commit and
recreates the replicas on it, so the update is a real change of
version, and it writes to evidence/rolling-from-<commit>. Lint covers
scripts/limits.py. CI acceptance runs make limits and make detection.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 19:52:29 +03:00
ArtyomandClaude Opus 5.5 ae8fb0a48e Record which replica stops and which services restart
scripts/failover-evidence.sh lists the api containers by hostname, the
X-Instance-Id they answer with, before and after make test-failover,
names the container left exited, keeps its last log lines with the
gunicorn graceful shutdown, then restores it and lists the replicas
again. scripts/evidence-stack.sh records the compose config check,
published ports, the missing port of a replica, hostnames, networks,
limits, process users, the balancer view of every server and the
alternating X-Instance-Id through the balancer, plus the environment
and image versions. The persistence and storage evidence scripts now
list every replica and the idempotency key table.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 11:44:03 +03:00
ArtyomandClaude Opus 5.5 ceb266ed05 Add the stop-one, restart and restore hooks and an LB probe
scripts/stop-one.sh maps TARGET_INSTANCE_ID to the api container whose
hostname matches, stops only it with docker stop -t 6 and leaves it
stopped; an unknown id exits 3 and touches nothing. scripts/restart.sh
replaces the lab 2 hook: it stops lb and the replicas, restarts
PostgreSQL and Valkey with their volumes and brings the stand back
with up --wait, which also reruns the idempotent migrate job.
scripts/restore.sh starts stopped replicas without recreating them
and scripts/lb-wait.sh waits until the balancer statistics show every
replica UP again, so the next failover run sees two replicas.

scripts/probe.py check (stdlib, Python 3.9 syntax) counts X-Instance-Id
over 40 calls with one token and races 8 posts with one key through
the balancer. make restore and make lb-check run them, ruff covers the
probe, and the CI acceptance job runs the failover test and the probe.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 11:43:52 +03:00
ArtyomandClaude Opus 5.5 dc2015473f Run the API as replicas behind HAProxy
compose.yaml now runs one api service with API_REPLICAS replicas and
no published ports, container name or hostname, behind lb, HAProxy
3.2.26 LTS pinned by digest and the only service with ports, bound to
127.0.0.1 (LB_HTTP_PORT for HTTP, LB_STATS_PORT for read-only stats).
The balancer finds replicas through Docker DNS with a server template,
checks GET /health/ready every second (fall 2, rise 2), balances round
robin without stickiness, retries connection failures on another
replica and allows a retry after a sent request only for GET and for
message posts carrying Idempotency-Key, never for users, chats or
login. Its own 502/503/504 answers use the contract error body with
the forwarded X-Request-Id.

The network is split into edge and an internal backend, every service
gets CPU and memory limits, the API runs read-only with a tmpfs /tmp,
two workers per replica, an 8 s request limit and a 5 s graceful stop
inside an 8 s stop_grace_period. Only api builds the image, migrate
reuses it once before the replicas start. PostgreSQL loads
pg_stat_statements and allows 200 connections. The short TTL instance
of lab 2 published the API around the balancer and is removed; its
expiry check stays as an in-process test.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 11:43:43 +03:00
ArtyomandClaude Opus 5.5 416ce4f9db Carry over the lab 2 solution
Bring the application, own tests, Dockerfile, toolchain, compose
stack, migrations, secret generator, restart hook, evidence scripts and
CI job from the lab 2 solution commit 8cddb09 unchanged, merge the lab
2 Makefile targets into the lab 3 skeleton Makefile without touching
its targets, and append the lab 2 solution variables to the skeleton
.env.example. The package version follows the contract version 1.3.0.

This is the starting point of the lab 3 cycle: lint, own tests and the
offline skeleton check pass before any lab 3 change.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 11:26:15 +03:00
Mikhail Verkovykh 0ab3a341aa Подготовить skeleton лабораторной 3 по мессенджеру 2026-09-10 18:01:35 +03:00