628b5f2 accepted a cursor only if re-encoding the parsed key gave the
same string. That closed the format both ways: any encoding change, even
a compatible one (an optional field, key order, to_rfc3339 shared with
API responses), would turn cursors issued by a replica of the other
version into 400 during make rolling_update, which the tests cannot see.
decode_cursor now checks the fields one by one: v must be exactly the
integer 1, t and i must be strings, i must be in the server's notation
(lowercase 8-4-4-4-12). Unknown fields are ignored, so a tampered cursor
is still a 400 while the format stays open for compatible changes. The
rule records how to change the format: a new CURSOR_VERSION, with both
versions accepted while replicas are replaced.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
decode_cursor checked only for KeyError, TypeError and ValueError, so a
server-issued cursor whose item id was replaced with [], {}, 123 or
false crashed UUID() with AttributeError and answered 500. It now checks
the types of "t" and "i" before parsing and accepts only the exact
string the server would issue for the parsed key: re-encoding must give
the received cursor, so v=true, an extra field or the id in another
notation is a 400 instead of a page.
pydantic's UUID also accepts ids without hyphens, with the urn:uuid:
prefix and in braces, wider than the contract's format: uuid. chat_id in
the path of the three chat endpoints and member_ids elements now use
UuidIn, which requires the 8-4-4-4-12 form in any case. Response models
and the OpenAPI schema are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add black-box tests for the strict-id-validation change: a server-issued
cursor with one field changed (non-string or non-canonical item id,
v=true, extra field) on both lists, member_ids with values of other JSON
types, a non-array member_ids and other UUID notations, and chat_id in
the path of the three chat endpoints. Sync the change's delta specs into
api-conventions, chats and pagination so the coverage check knows the
new scenarios.
The cursor and UUID-notation tests fail on the current code by design;
the fix follows in the next commit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The change moves to openspec/changes/archive/2026-10-13-fix-known-issues.
Its delta requirements join the main specs: graceful-shutdown "Остановка
во время старта", load-benchmark "События Docker профилей отказа и
замены", student-tests "Доказательства исключений сверки в репозитории"
and "PostgreSQL стенда отказов без аварийных перезапусков". AGENTS and
REPORT now say all eight lab 3 changes are archived and none is active.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Report the failover ranges strictly over phases 1-5 with phase 06
apart, add per-window memory peaks to the failover table, describe
which intervals the window maximum covers, and note the per-second
rows and per-request zeros of the taken labels fixed in f34cb9a.
Record the design findings, the raw logs ignored by git as a known
issue, and the checks after the fixes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run bench/sampler.sh in a new scan mode (cgroup walk, no network, no
psql or LB polling) for the whole load of the failover and rollout
profiles. bench/resources.py turns the samples into per-window CPU,
throttling and memory of every component and the VM, with
samples.csv, resources.csv and summary sections; CPU per successful
request only in steady windows. Check the peak CPU budget before the
stand starts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sync load-benchmark (series comparison table, track 1 before/after
rules, db4 series) and service-configuration (default pool 10 + 0)
into the main specs, move the change to the archive and drop it from
the transitional state in AGENTS.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fix the two-step threshold (x1.118), CPU-per-request spread and
PostgreSQL ranges in REPORT.md and design, record the LB side check
(lb_chkdown, error_codes) in evidence, add the db4 series to the
load-benchmark delta, refresh AGENTS.md, bench.md, PLAN.md and
proposal, and log the swapouts_delta passport defect.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Configs may carry a fifth element, the PostgreSQL CPU limit, which goes
into the config key (-pg<CPU>), the budget and the cell stand. Series db:
1, 2, 4 replicas at the tight limit and a 4 x 1 control at 2 CPU; label
summaries show PostgreSQL CPU at R* against its limit and throttling.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Close the 53300 known issue, restate H1 and saturation conclusions
against the pre-registered criteria and grid resolution, name the
unresolved host mechanism, attribute overload 503 to the pool timeout,
and plan a fresh A/A for scaling-improvement.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SQLSTATE 53300 (too_many_connections) joins UNAVAILABLE_SQLSTATES, so a
refused pool connection gives 503 unavailable instead of 500. New fault
check fills max_connections through the failure stand hook and expects
503 in the contract format.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Series a/b/pool/ab/quick in bench/run.py: per-cell fresh stand with the
configuration's replicas and max_connections, open-model mixed profile
(k6/mixed.js), fixed levels and R* search by SLO, a sampler container
for cgroup, VM, HAProxy and PostgreSQL metrics, resume and a stand
lock. Pure helpers in bench/scaling.py, SVG plots in bench/plots.py,
series mode for bench_compare and bench_ab. Hypotheses H1-H3 go to
REPORT.md before the first series.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Track 1 "cost of scaling": open-model series A (fixed 4 CPU budget),
B (1-6 replicas), pool border (max_connections 100 vs 50) and A/A,
sampler of cgroup/VM/LB/PostgreSQL metrics, generator limits, series
mode for bench_compare and bench_ab. Hypotheses H1-H3 are recorded in
the design before any series run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sync the rolling-update delta specs into db-migrations,
graceful-shutdown, load-balancing and load-benchmark, move the change to
the archive, mark it in PLAN.md and update the transitional state and
code map in AGENTS.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re-run the compatibility walk-through, the script failure cases with a
real Ctrl+C through a pseudo-terminal, the stand lifecycle after
replacement and the quick rollout and failover self-checks on 84161c0
with a clean tree; update the report rows and limitations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With lock_timeout equal to statement_timeout (5 s) every request queued
behind a migration waiting for its lock failed with 503 together with
the migration. A 2 s lock timeout lets the queue proceed once the
migration gives up.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reset the new replica ID before entering the starting state and let the
cleanup find a replica created by an interrupted compose scale; refuse
to start while LB slots keep a manual drain or maint state; log the exit
code of each stopped old replica and print full LB commands for manual
cleanup. Document the migrate step and the rollback tag limitation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A one-off run --rm left the migrate container of the previous image in
the stand; scripts/restart.sh then started it as a dependency of api and
the old migrations failed on the newer schema, so test-persistence broke.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Old image passes smoke and student tests on the new schema, rolling
update, rollback and roll forward under tests/load.py give zero failed
requests; the run_docker known issue is not reproduced and is closed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scripts/rolling-update.sh (make rolling_update) replaces API replicas one
by one: build, migrations once, new replica UP in HAProxy, old replica
drained and put into maintenance through the LB runtime API on loopback,
then stopped and removed; the slot is reset once DNS drops the address.
Expand migration adds messages.instance_id with a lock timeout. The bench
rollout profile runs the same script under constant load, starting from
the image of an older commit built with git archive.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Delta specs synced into graceful-shutdown, load-balancing, load-benchmark
and service-configuration; the change moves to
openspec/changes/archive/2026-10-11-graceful-failover/. PLAN.md marks the
change archived and AGENTS.md notes what was handed to rolling-update.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The verify fix in bench/compare.py refuses a label measured without the
load balancer; the load-benchmark delta now modifies the label comparison
requirement accordingly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rejected LB pool variant is re-measured on committed code with a
temporary second HAProxy differing in http-reuse only: the replica, not
the LB, closed the last idle connection after 60.4 s. Exploratory
never/safe SIGTERM numbers are separated from evidence. hang-key cleanup
is shown under Ctrl+C and SIGTERM, bench_compare refuses a label without
the LB. Drain before SIGTERM and the run_docker recreate issue are handed
to rolling-update in PLAN.md and KNOWN_ISSUES.md; design gains decision 10.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Failover profile label baseline from 459f84c on a checked host: stop and
kill of each replica under 200 req/s give 0 errors and LB exclusion in
0.79-1.81 s (bound 2.5 s), pause gives LB exclusion in 5.83-6.00 s
(bound 6.5 s) with 504s for half the requests and recovery in 6 s;
reconciliation of retried writes matches. Evidence for detection,
keep-alive (LB closes idle replica connections at 10.1 s) and the hung
replica key lock, regression checks, timeout chain and level 4 rows in
REPORT.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HAProxy checks get fastinter 500ms and timeout check 2.5s (hung replica
detected within 6.5 s), http-reuse never with uvicorn keep-alive 60 s so
the LB closes idle replica connections, and app connections set
idle_in_transaction_session_timeout so a frozen replica releases its
idempotency key. The bench stand runs behind HAProxy with N replicas and
gains the failover profile (make bench_failover / bench_failover_quick):
constant-rate writes with a key and reads while each replica is stopped,
killed or paused, with per-second timeline, detection and recovery time
and write reconciliation. Driver commands idle-conns and hang-key check
the keep-alive and hung-transaction decisions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>