Commit Graph
15 Commits
Author SHA1 Message Date
ricoromaandClaude Opus 5.5 12783aaff1 fix: harden graceful-failover checks found in verify
hang-key names its trigger per run, turns SIGTERM and SIGHUP into an
interrupt so the trigger is always dropped and the replica unpaused, and
prints the cleanup SQL if dropping fails. idle-conns reports who closed
the idle connections and can read the pool of another LB container, so a
variant LB config can be checked against the same replicas. bench_compare
refuses a label measured without the load balancer. Stale numbers removed
from the haproxy.cfg comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 00:00:54 +03:00
ricoromaandClaude Opus 5.5 4234df5767 docs: add graceful-failover evidence and report
Failover profile label baseline from 459f84c on a checked host: stop and
kill of each replica under 200 req/s give 0 errors and LB exclusion in
0.79-1.81 s (bound 2.5 s), pause gives LB exclusion in 5.83-6.00 s
(bound 6.5 s) with 504s for half the requests and recovery in 6 s;
reconciliation of retried writes matches. Evidence for detection,
keep-alive (LB closes idle replica connections at 10.1 s) and the hung
replica key lock, regression checks, timeout chain and level 4 rows in
REPORT.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 23:52:47 +03:00
ricoromaandClaude Opus 5.5 459f84c75c feat: coordinate timeout chain and add failover load profile
HAProxy checks get fastinter 500ms and timeout check 2.5s (hung replica
detected within 6.5 s), http-reuse never with uvicorn keep-alive 60 s so
the LB closes idle replica connections, and app connections set
idle_in_transaction_session_timeout so a frozen replica releases its
idempotency key. The bench stand runs behind HAProxy with N replicas and
gains the failover profile (make bench_failover / bench_failover_quick):
constant-rate writes with a key and reads while each replica is stopped,
killed or paused, with per-second timeline, detection and recovery time
and write reconciliation. Driver commands idle-conns and hang-key check
the keep-alive and hung-transaction decisions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 23:21:12 +03:00
ricoromaandClaude Opus 5.5 b76a666621 docs: propose graceful-failover change
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 22:32:12 +03:00
ricoromaandClaude Opus 5.5 aec6f23cb0 chore: archive replicas-lb change
Main specs were already in sync with the deltas (sync re-run, no
changes). PLAN.md marks stage 2 archived and points graceful-failover
at the corrected detection formula; AGENTS.md notes what of the timeout
chain is measured so far and the new replicas evidence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 22:10:31 +03:00
ricoromaandClaude Opus 5.5 cb14586a19 docs: retake replicas-lb evidence from a4ca9ef after verify
The stand was recreated from a4ca9ef (make down, make run_docker) and
all evidence of the change was taken again on it: stand, distribution,
LB errors, test-persistence, test-failover for each replica (stop and
kill), stop-one refusals, make test and student_test through the LB,
test_isolated, failure_checks and bench self-checks.

New evidence closes scenarios verify found without proof: a real git
clone on non-default ports, a hung replica (docker pause: DOWN after
7.7 s, in-flight request 504), PostgreSQL outage behind the LB, replica
tests failing against one process, aborted responses via
driver.py kill-inflight. REPORT.md gets the corrected detection
formula, the measured cost of the 1 CPU limit for Argon2id and the new
rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 22:09:15 +03:00
ricoromaandClaude Opus 5.5 a4ca9ef3a7 fix: correct replicas-lb details found in verify
- stop-one.sh: an empty TARGET_INSTANCE_ID now exits 2 in any shell,
  as the design states (${VAR:?} gave 1 under bash).
- driver.py kill-inflight: reproducible "aborted response is not
  retried" scenario (logins in flight, replica killed via stop-one.sh).
- Detection time of a hung replica is fall x (inter + timeout check) =
  8 s, not inter x fall + timeout check = 5 s (docker pause: 7.3 s);
  fixed in haproxy.cfg, balancer.md and the design.
- The 1 CPU replica limit is kept on purpose; the compose comment and
  the design now say that Argon2id uses more and logins are slower.
- load-balancing spec: API_REPLICAS is read by compose and Makefile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 18:44:57 +03:00
ricoromaandClaude Opus 5.5 de93056eab docs: add replicas-lb evidence and report
Evidence from 4e0cc4e: stand ports, limits and per-service env,
X-Instance-Id distribution and scaling, LB-generated errors,
test-persistence through the LB, test-failover for each replica in stop
and kill modes, stop-one refusals, test runs and bench self-checks.
REPORT.md gains the LB architecture, timeout chain, limits, criteria
rows and defence answers. Coverage exclusions for restart now point to
the lab 3 persistence evidence; stale lab 2 pointers go to
KNOWN_ISSUES.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 18:31:18 +03:00
ricoromaandClaude Opus 5.5 4e0cc4e965 fix: start DNS-resolved replicas down until checks pass
Without init-state down HAProxy marked a slot UP as soon as Docker DNS
returned the container address, before uvicorn listened, so requests
after LB restart or scale-up hit a replica that was not ready.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 18:08:50 +03:00
ricoromaandClaude Opus 5.5 ab71c3b51e feat: put API replicas behind HAProxy
HAProxy 3.4 is the only published entry point; api runs as
API_REPLICAS replicas without ports, found through Docker DNS and
checked on /health/ready. Retries only on connection failure, LB errors
in the contract error format, resource limits and per-service secrets.
Adds scripts/stop-one.sh, LB-aware restart.sh, make replicas /
instances / lb_stats and replica tests driven by EXPECT_REPLICAS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 18:05:05 +03:00
ricoromaandClaude Opus 5.5 8e15e02898 docs: propose replicas-lb change
HAProxy as the single entry point, API replicas without published
ports, resource limits and per-service secrets, restart and stop-one
hooks, replica tests in student-tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 17:40:04 +03:00
ricoromaandClaude Opus 5.5 b5ae252254 chore: archive instance-id-idempotency change
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 17:23:41 +03:00
ricoromaandClaude Opus 5.5 c5840bfaa8 docs: add idempotency and X-Instance-Id evidence to the report
- evidence for replay, conflict, replay through a new API process and
  after restarting API and storages, keys table, key race,
  X-Instance-Id in responses and logs, explicit and invalid
  instance_id and key TTL, test runs
- stand evidence: make wipe_data leaves the lab 2 volume intact, build
  from a fresh git clone passes smoke, bench restore stage sends with
  Idempotency-Key
- REPORT.md: X-Instance-Id, idempotency design, IDEMPOTENCY_TTL_SECONDS
  mapping, limitations and evidence rows
- tasks.md: mark the change tasks done, add follow-ups from verify

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 17:08:47 +03:00
ricoromaandClaude Opus 5.5 af4bca08c9 feat: carry lab 2 over and add X-Instance-Id and idempotency keys
- carry the lab 2 solution over and plan lab 3 (PLAN.md, OpenSpec
  change instance-id-idempotency)
- rename the compose project to mtusi-lab3 and the image to :lab3
- add X-Instance-Id to every response and instance_id to every log
  record ([app] instance_id, hostname by default)
- require Idempotency-Key on POST messages: keys table in Postgres,
  key and message in one transaction, replay returns the same 201,
  another body gets 409, TTL from [idempotency] ttl_seconds
- send the key from student-tests, k6 scenarios and the skill driver;
  add idempotency, X-Instance-Id, key race and fault tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 19:42:36 +03:00
Mikhail Verkovykh 0ab3a341aa Подготовить skeleton лабораторной 3 по мессенджеру 2026-09-10 18:01:35 +03:00