Commit Graph
50 Commits
Author SHA1 Message Date
ricoromaandClaude Opus 5.5 fd03b84b68 docs: record track2-reconcile-cost verify fix evidence and numbers
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 20:26:54 +03:00
ricoromaandClaude Opus 5.5 ca0c4b2f71 fix: address track2-reconcile-cost verify findings
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 20:03:00 +03:00
ricoromaandClaude Opus 5.5 16f2cdeddb docs: report reconcile cost results and evidence
Add the baseline label results to REPORT.md: H1, H2, H3 and S4
confirmed, H4 not verified (the fast-unlimited contrast did not
saturate S3, r 1.08). Keep 2000 objects/s as a bound on the S3 share
and give head/deep limits by S3 CPU per object. Refresh failure_checks
(54/54, red run of task 1.3), test_isolated and the pace stop evidence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 19:14:08 +03:00
ricoromaandClaude Opus 5.5 d8b8ae3dab fix: check bg k6 lines from below and allow interrupted extras
An iteration interrupted by SIGINT after its line loses both the
iteration count and the bg_lines counter sample, so neither can match
the console lines exactly. line_problems now requires lines >= each
count (fewer: "lines lost") and at most MAX_VUS extra lines over each.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 18:06:43 +03:00
ricoromaandClaude Opus 5.5 a42dd27c24 fix: count bg lines in k6 so SIGINT-interrupted iterations do not fail the check
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 07:55:42 +03:00
ricoromaandClaude Opus 5.5 aaf3fa3b29 docs: add reconcile cost hypotheses before the label
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 04:56:09 +03:00
ricoromaandClaude Opus 5.5 8d4cf7eebf feat: add reconcile strategies, deep check, rate limit and cost profile
- reconcile --strategy head, --deep (checksum_mismatch), restore needs the same SHA-256
- [reconciler] max_objects_per_second (2000), pause interrupted by signal
- --apply stops rechecks once the limit is exceeded (unchecked)
- pass cost in the summary: S3 objects and requests, bytes, DB sessions, phases
- S3 client counters, cleaner exits with os._exit after main()
- fault tests, hook args, driver reconcile-signal --pace
- bench profile reconcile / reconcile_quick: fill outside the API, gateway metrics, background load
- sync spec deltas, rules, AGENTS.md, PLAN.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 04:52:56 +03:00
ricoromaandClaude Opus 5.5 dcdfe5f2e4 docs: propose track2-reconcile-cost
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 03:47:27 +03:00
ricoromaandClaude Opus 5.5 7f4e626b49 docs: archive fix-upload-put-flake and sync its specs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 02:34:07 +03:00
ricoromaandClaude Opus 5.5 1657dbcd5f docs: reshoot test_isolated and full failure_checks on 723b6c3
make test_isolated: 193 passed, 5 replica tests skipped; full
make failure_checks: 49/49 in 552 s with the unfiltered output, paths
anonymized; REPORT.md points to the reshoots.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 02:31:10 +03:00
ricoromaandClaude Opus 5.5 723b6c3140 fix: address fix-upload-put-flake verify findings
aiohttp 3.14.5 buffers 256 KiB and pauses reading above 512 KiB; name
the aiohttp versions actually checked instead of a range; fix line
references in the idle-connection evidence; the object-storage spec now
exempts the connect timeout from the single call timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 02:14:38 +03:00
ricoromaandClaude Opus 5.5 1ad34a7fd7 docs: add fix-upload-put-flake evidence and report
s3-idle before and after replacing replicas, red runs on 4cfaab0 and
green ones on b5983f0, failure_checks 49/49, eight student_test runs
without 503, PUT memory probe and bench_uploads_quick; drop the fixed
KNOWN_ISSUES entry and describe the S3 client timeouts in REPORT.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 01:51:46 +03:00
ricoromaandClaude Opus 5.5 b5983f0e35 fix: drop aiohttp read timeout of the S3 client
aiohttp 3.13-3.14.5 re-arms sock_read in resume_reading() after a
streamed response has already returned the connection to the pool; the
timer fires while the connection idles and the next call on it fails in
1 ms with ReadTimeoutError (503 on a healthy S3). The call and per-chunk
asyncio.timeout stay the only S3 timeout. Adds driver.py s3-idle and
documents the timeouts in the rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 01:13:36 +03:00
ricoromaandClaude Opus 5.5 4cfaab0063 test: add idle S3 upload fault check on a short S3 timeout API
Fourth API of the failure stand (S3__TIMEOUT_SECONDS=1), slow streamed
download in the test client, test_upload_after_idle_s3 and the
fix-upload-put-flake spec deltas synced into the main specs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 01:05:00 +03:00
ricoromaandClaude Opus 5.5 7d4b2cb37e docs: propose fix-upload-put-flake
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 22:52:59 +03:00
ricoromaandClaude Opus 5.5 b804cec625 docs: archive track2-reconciler and sync its specs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 21:43:41 +03:00
ricoromaandClaude Opus 5.5 96d2ecca3d docs: re-capture track2-reconciler evidence on 84f8c35
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 21:31:49 +03:00
ricoromaandClaude Opus 5.5 84f8c353e9 fix: address track2-reconciler verify findings
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 21:04:01 +03:00
ricoromaandClaude Opus 5.5 a6071dafa0 docs: add track2-reconciler evidence and report
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 20:23:01 +03:00
ricoromaandClaude Opus 5.5 c1648d9b83 feat: reconcile the bucket against attachment rows in the cleaner
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 19:32:08 +03:00
ricoromaandClaude Opus 5.5 c13df14f36 docs: propose track2-reconciler
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 17:06:14 +03:00
ricoromaandClaude Opus 5.5 5f4b9ff61c docs: archive upload-memory and sync its specs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 16:21:08 +03:00
ricoromaandClaude Opus 5.5 9d2ee30cd2 fix: address upload-memory verify findings
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 16:11:03 +03:00
ricoromaandClaude Opus 5.5 79cc06b1e0 docs: add upload-memory evidence and report
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 15:39:58 +03:00
ricoromaandClaude Opus 5.5 ce1ae2ccfe docs: add bench_uploads baseline label measured on 97a4be4
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 15:18:24 +03:00
ricoromaandClaude Opus 5.5 97a4be4951 fix: cap the LB client receive buffer so stalled upload bodies stay small
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 02:03:20 +03:00
ricoromaandClaude Opus 5.5 6cab9d3c41 feat: bound upload memory with per-replica slots and a body deadline
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 00:23:37 +03:00
ricoromaandClaude Opus 5.5 fc1524d0d6 docs: propose upload-memory
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 23:21:42 +03:00
ricoromaandClaude Opus 5.5 f894040b72 docs: archive upload-consistency and sync its specs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 22:20:11 +03:00
ricoromaandClaude Opus 5.5 9dde16e638 docs: re-capture upload-consistency evidence on 5889ca2
Re-run failure checks (32 of 32) and the cleaner pass under a paused S3
on the two-step cleanup, fix the storage-consistency scenario count in
the student test evidence and point the report at the new commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 22:17:53 +03:00
ricoromaandClaude Opus 5.5 5889ca2ad7 fix: delete the object and the row in one locked cleanup transaction
Keep the committed mark, then delete the object and the row under the
same row lock, so an older replica without the mark check that waits on
the row finds it gone instead of promoting it to ready after the object
delete. A mixed-version stand is no worse than before the mark; document
the upgrade order (replicas first, cleaner last) and fix stale wording.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 22:08:29 +03:00
ricoromaandClaude Opus 5.5 6551839807 docs: add upload-consistency verify evidence and report
Capture the cleaner pass under a paused S3 (row stays marked pending,
next pass removes it), re-run failure checks (32 of 32 with red runs of
the new checks), student tests and the isolated run on 87c48b9, and
describe the marked three-step cleanup and its rolling-update window in
the report.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 21:42:31 +03:00
ricoromaandClaude Opus 5.5 87c48b940b fix: address upload-consistency verify findings
Mark a stale pending row in its own committed transaction before the
cleaner deletes the object, and let mark_ready promote only unmarked
rows, so a cleaner rollback after the object delete can no longer leave
a ready attachment without its object. Add fault checks for an unknown
COMMIT outcome, an upload outliving the grace period and a cleaner
failure after the object delete; fix the cleaner entrypoint in rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 21:27:47 +03:00
ricoromaandClaude Opus 5.5 f2dc0b368a docs: add upload-consistency evidence and report
- evidence: consistency (driver upload-consistency), cleaner process,
  cleaner S3 key, pending index plan on the working database
- re-run failure_checks (29/29), student_test via LB and test_isolated
  with the cleaner and storage-consistency coverage
- REPORT: database and S3 consistency, compensation table, grace period
  rationale, "PUT succeeded, COMMIT did not" answer, level 4 rows,
  limitations; PLAN stage 3 and AGENTS evidence map

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 06:32:48 +03:00
ricoromaandClaude Opus 5.5 5b42bd2891 feat: compensate failed uploads and clean stale pending attachments
- upload: delete the stored object when ready fails before COMMIT or the
  pending row is gone (503 instead of 500); unknown COMMIT outcome keeps it
- cleaner service (python -m src.cleaner): stale pending rows older than
  grace (900 s, must exceed 17 s live-upload bound) under FOR UPDATE SKIP
  LOCKED, object before row, own S3 user with read/delete only
- shared session_scope for API and cleaner; early_stop process name
- partial index on pending attachments, CREATE INDEX CONCURRENTLY
- s3-bootstrap: cleaner user, policy and checks; make env_setup appends
  missing template variables to an existing .env
- failure checks: cleaner on the failure stand, id-list hook actions,
  five new and three updated tests; driver upload-consistency
- sync upload-consistency deltas into main specs (storage-consistency)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 06:10:37 +03:00
ricoromaandClaude Opus 5.5 450b091ae3 docs: propose upload-consistency
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 05:17:35 +03:00
ricoromaandClaude Opus 5.5 d9d44c511c docs: archive attachments and sync its specs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 02:28:30 +03:00
ricoromaandClaude Opus 5.5 6970dad871 docs: re-measure attachments rollout on a clean tree and update report
- bench_rollout label attachments from e1f8531 (dirty: false):
  replacement 8.67 s, 0 errors, 12001 messages = keys
- student_test through LB (197 passed) and failure_checks (24/24)
  after verify fixes
- REPORT: upload memory ~1.2 bodies with fixed mmap threshold,
  strict UTF-8 filename, env does not change attachment limits

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 02:26:35 +03:00
ricoromaandClaude Opus 5.5 dba40bc325 fix: address attachments verify findings
- keep one upload body in memory: BytesIO + getvalue() without copy,
  fixed glibc mmap threshold in the image (no realloc copy on growth)
- stop PNG inflate at expected size + 1 byte
- reject filename that is not valid UTF-8 after percent-decoding (400)
- failure stand: env limit variables do not change attachment limits;
  history with attachment_ids while S3 is stopped
- drop dead constants; restate "old image on new schema" scenario

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 02:16:49 +03:00
ricoromaandClaude Opus 5.5 8dc925e8ca docs: add attachments evidence and report
- re-capture coverage-exclusion evidence on the lab 4 stand: cache,
  auth storage, replay after API recreate, test-persistence,
  attachment-failover (stop, kill), attachment-storage
- green evidence/tests: student_test via LB, test_isolated,
  failure_checks; smoke, slow transfer, test-failover per replica
- bench_rollout label attachments: e1161f7 -> schema with attachments
  under load, no errors, messages = keys
- REPORT.md: attachments architecture, write order, defense answers,
  level 3/4 evidence rows, limitations; PLAN.md stage 2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 01:50:10 +03:00
ricoromaandClaude Opus 5.5 d56b8928b4 fix: wait for recreated replicas before restore checks in bench
switch_api recreated the API replica with the same IP while HAProxy
still showed the old container's slot UP, so restore_check hit 503
<NOSRV> before the new process listened. Wait until each new replica
answers /health/ready from inside its container, then for LB UP.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 01:27:51 +03:00
ricoromaandClaude Opus 5.5 116d8d3bf0 feat: add image attachments to messages
- upload, metadata and download endpoints for PNG/JPEG attachments:
  limits before reading and decompression, type by content,
  structural integrity check without decoding pixels, safe filename
- write order: pending row with object key -> PUT to S3 -> ready;
  per-step DB sessions so no pool connection is held during transfers
- attachment_ids in messages (validation, idempotency hash, history);
  whitespace-only text allowed with an attachment (teacher, 2026-10-20)
- migration: attachments and message_attachments tables, text length
  CHECK relaxed with NOT VALID
- student tests, fault checks and driver commands for attachments;
  rollout profile starts from e1161f7 and checks the commit first
- sync attachments deltas into main specs; rules, AGENTS.md, PLAN.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 01:08:56 +03:00
ricoromaandClaude Opus 5.5 73779f7acd docs: propose attachments
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 23:44:50 +03:00
ricoromaandClaude Opus 5.5 e1161f7e6b docs: archive s3-storage and sync its specs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 21:51:36 +03:00
ricoromaandClaude Opus 5.5 1471f07a0e docs: re-capture s3-storage evidence after verify fixes
- evidence/s3 and evidence/tests re-captured from f57635a: first write
  after healthy, deleted bucket, netstat-based port check, readiness
  with s3 paused, prefix cleanup check
- PLAN: HEAD on a deleted bucket is an accepted limitation
- KNOWN_ISSUES: bootstrap after a manual bucket delete

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 21:47:01 +03:00
ricoromaandClaude Opus 5.5 f57635a5fb fix: address s3-storage verify findings
- s3 healthcheck and fault hook probe wait for the volume server to
  register with master (first write after restart no longer waits ~5 s)
- s3-bootstrap mounts /data as tmpfs: no anonymous volume per container
- S3 client: NoSuchBucket and AccessDenied are 503 with their own error
  logs, not a missing object or rejected credentials; delete ignores only
  a missing key
- test_isolated verifies prefix removal (weed shell exits 0 on errors)
- s3_check derives ports from listening sockets in the s3 container
- readiness without S3 is also checked with s3 paused

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 21:23:38 +03:00
ricoromaandClaude Opus 5.5 7d340c0673 docs: add s3-storage evidence and report
- evidence/s3: bootstrap, privacy (make s3_check), gRPC without token,
  restart and SIGKILL, client failures, memory of 20 concurrent uploads
- evidence/tests: failure_checks with own S3, test_isolated prefix
  cleanup (red tests: no attachment_ids until attachments)
- REPORT.md: S3 role, services table, bootstrap, readiness
  dependencies, skeleton S3_* names, evidence rows, limitations
- refine expected-red counts in tasks, design and PLAN

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 20:31:35 +03:00
ricoromaandClaude Opus 5.5 2e997f9b42 feat: add private S3 storage (SeaweedFS) with bootstrap and client
- s3 and s3-bootstrap services: SeaweedFS 4.48 pinned by digest,
  internals on loopback, signing key, private bucket, API user policy,
  privacy checks on every bootstrap
- [s3] settings and aiobotocore client in src/infra/s3: deadline per
  call, no retries, 503 on outage or rejected credentials
- make start_s3, stop_s3, s3_check; run_dev, test_isolated, wipe_data
  and restart hook include S3
- failure_checks with own S3 and readiness check without it; bench
  generates S3 secrets and filters them out of the passport
- rules, AGENTS.md, skill; sync health delta; KNOWN_ISSUES: rollout_from

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 20:12:04 +03:00
ricoromaandClaude Opus 5.5 379ac81374 docs: propose s3-storage
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 19:26:52 +03:00
ricoromaandClaude Opus 5.5 5d5d8f9d2a chore: carry lab 3 over
- carry the lab 3 solution over from mtusi2026lab3-safronov
  (solution/bvt2301-safronov, ce89eba): src, migrations, student-tests,
  bench, openspec, .claude, infra/haproxy and infra/valkey, own scripts,
  Dockerfile, compose, pyproject and uv lock
- merge Makefile, .gitignore and .dockerignore (the lab 3 versions only
  add lines to the skeleton, which is the same in lab 4) and
  .env.example (lab 4 skeleton names listed in the header)
- rename the compose project to mtusi-lab4 and the image to :lab4 so
  the stand does not reuse the lab 3 containers and volumes
- update AGENTS.md for lab 4: current lab, origin commit, contract and
  test files of the new skeleton

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 17:32:33 +03:00