Commit Graph
21 Commits
Author SHA1 Message Date
ricoromaandClaude Opus 5.5 87c48b940b fix: address upload-consistency verify findings
Mark a stale pending row in its own committed transaction before the
cleaner deletes the object, and let mark_ready promote only unmarked
rows, so a cleaner rollback after the object delete can no longer leave
a ready attachment without its object. Add fault checks for an unknown
COMMIT outcome, an upload outliving the grace period and a cleaner
failure after the object delete; fix the cleaner entrypoint in rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 21:27:47 +03:00
ricoromaandClaude Opus 5.5 f2dc0b368a docs: add upload-consistency evidence and report
- evidence: consistency (driver upload-consistency), cleaner process,
  cleaner S3 key, pending index plan on the working database
- re-run failure_checks (29/29), student_test via LB and test_isolated
  with the cleaner and storage-consistency coverage
- REPORT: database and S3 consistency, compensation table, grace period
  rationale, "PUT succeeded, COMMIT did not" answer, level 4 rows,
  limitations; PLAN stage 3 and AGENTS evidence map

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 06:32:48 +03:00
ricoromaandClaude Opus 5.5 5b42bd2891 feat: compensate failed uploads and clean stale pending attachments
- upload: delete the stored object when ready fails before COMMIT or the
  pending row is gone (503 instead of 500); unknown COMMIT outcome keeps it
- cleaner service (python -m src.cleaner): stale pending rows older than
  grace (900 s, must exceed 17 s live-upload bound) under FOR UPDATE SKIP
  LOCKED, object before row, own S3 user with read/delete only
- shared session_scope for API and cleaner; early_stop process name
- partial index on pending attachments, CREATE INDEX CONCURRENTLY
- s3-bootstrap: cleaner user, policy and checks; make env_setup appends
  missing template variables to an existing .env
- failure checks: cleaner on the failure stand, id-list hook actions,
  five new and three updated tests; driver upload-consistency
- sync upload-consistency deltas into main specs (storage-consistency)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 06:10:37 +03:00
ricoromaandClaude Opus 5.5 450b091ae3 docs: propose upload-consistency
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 05:17:35 +03:00
ricoromaandClaude Opus 5.5 d9d44c511c docs: archive attachments and sync its specs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 02:28:30 +03:00
ricoromaandClaude Opus 5.5 6970dad871 docs: re-measure attachments rollout on a clean tree and update report
- bench_rollout label attachments from e1f8531 (dirty: false):
  replacement 8.67 s, 0 errors, 12001 messages = keys
- student_test through LB (197 passed) and failure_checks (24/24)
  after verify fixes
- REPORT: upload memory ~1.2 bodies with fixed mmap threshold,
  strict UTF-8 filename, env does not change attachment limits

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 02:26:35 +03:00
ricoromaandClaude Opus 5.5 e1f85316cb docs: drop rollout evidence measured on a dirty tree
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 02:17:26 +03:00
ricoromaandClaude Opus 5.5 dba40bc325 fix: address attachments verify findings
- keep one upload body in memory: BytesIO + getvalue() without copy,
  fixed glibc mmap threshold in the image (no realloc copy on growth)
- stop PNG inflate at expected size + 1 byte
- reject filename that is not valid UTF-8 after percent-decoding (400)
- failure stand: env limit variables do not change attachment limits;
  history with attachment_ids while S3 is stopped
- drop dead constants; restate "old image on new schema" scenario

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 02:16:49 +03:00
ricoromaandClaude Opus 5.5 8dc925e8ca docs: add attachments evidence and report
- re-capture coverage-exclusion evidence on the lab 4 stand: cache,
  auth storage, replay after API recreate, test-persistence,
  attachment-failover (stop, kill), attachment-storage
- green evidence/tests: student_test via LB, test_isolated,
  failure_checks; smoke, slow transfer, test-failover per replica
- bench_rollout label attachments: e1161f7 -> schema with attachments
  under load, no errors, messages = keys
- REPORT.md: attachments architecture, write order, defense answers,
  level 3/4 evidence rows, limitations; PLAN.md stage 2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 01:50:10 +03:00
ricoromaandClaude Opus 5.5 d56b8928b4 fix: wait for recreated replicas before restore checks in bench
switch_api recreated the API replica with the same IP while HAProxy
still showed the old container's slot UP, so restore_check hit 503
<NOSRV> before the new process listened. Wait until each new replica
answers /health/ready from inside its container, then for LB UP.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 01:27:51 +03:00
ricoromaandClaude Opus 5.5 116d8d3bf0 feat: add image attachments to messages
- upload, metadata and download endpoints for PNG/JPEG attachments:
  limits before reading and decompression, type by content,
  structural integrity check without decoding pixels, safe filename
- write order: pending row with object key -> PUT to S3 -> ready;
  per-step DB sessions so no pool connection is held during transfers
- attachment_ids in messages (validation, idempotency hash, history);
  whitespace-only text allowed with an attachment (teacher, 2026-10-20)
- migration: attachments and message_attachments tables, text length
  CHECK relaxed with NOT VALID
- student tests, fault checks and driver commands for attachments;
  rollout profile starts from e1161f7 and checks the commit first
- sync attachments deltas into main specs; rules, AGENTS.md, PLAN.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 01:08:56 +03:00
ricoromaandClaude Opus 5.5 73779f7acd docs: propose attachments
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 23:44:50 +03:00
ricoromaandClaude Opus 5.5 e1161f7e6b docs: archive s3-storage and sync its specs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 21:51:36 +03:00
ricoromaandClaude Opus 5.5 1471f07a0e docs: re-capture s3-storage evidence after verify fixes
- evidence/s3 and evidence/tests re-captured from f57635a: first write
  after healthy, deleted bucket, netstat-based port check, readiness
  with s3 paused, prefix cleanup check
- PLAN: HEAD on a deleted bucket is an accepted limitation
- KNOWN_ISSUES: bootstrap after a manual bucket delete

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 21:47:01 +03:00
ricoromaandClaude Opus 5.5 f57635a5fb fix: address s3-storage verify findings
- s3 healthcheck and fault hook probe wait for the volume server to
  register with master (first write after restart no longer waits ~5 s)
- s3-bootstrap mounts /data as tmpfs: no anonymous volume per container
- S3 client: NoSuchBucket and AccessDenied are 503 with their own error
  logs, not a missing object or rejected credentials; delete ignores only
  a missing key
- test_isolated verifies prefix removal (weed shell exits 0 on errors)
- s3_check derives ports from listening sockets in the s3 container
- readiness without S3 is also checked with s3 paused

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 21:23:38 +03:00
ricoromaandClaude Opus 5.5 7d340c0673 docs: add s3-storage evidence and report
- evidence/s3: bootstrap, privacy (make s3_check), gRPC without token,
  restart and SIGKILL, client failures, memory of 20 concurrent uploads
- evidence/tests: failure_checks with own S3, test_isolated prefix
  cleanup (red tests: no attachment_ids until attachments)
- REPORT.md: S3 role, services table, bootstrap, readiness
  dependencies, skeleton S3_* names, evidence rows, limitations
- refine expected-red counts in tasks, design and PLAN

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 20:31:35 +03:00
ricoromaandClaude Opus 5.5 2e997f9b42 feat: add private S3 storage (SeaweedFS) with bootstrap and client
- s3 and s3-bootstrap services: SeaweedFS 4.48 pinned by digest,
  internals on loopback, signing key, private bucket, API user policy,
  privacy checks on every bootstrap
- [s3] settings and aiobotocore client in src/infra/s3: deadline per
  call, no retries, 503 on outage or rejected credentials
- make start_s3, stop_s3, s3_check; run_dev, test_isolated, wipe_data
  and restart hook include S3
- failure_checks with own S3 and readiness check without it; bench
  generates S3 secrets and filters them out of the passport
- rules, AGENTS.md, skill; sync health delta; KNOWN_ISSUES: rollout_from

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 20:12:04 +03:00
ricoromaandClaude Opus 5.5 379ac81374 docs: propose s3-storage
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 19:26:52 +03:00
ricoromaandClaude Opus 5.5 305d9b5627 docs: add lab 4 plan
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 18:21:07 +03:00
ricoromaandClaude Opus 5.5 5d5d8f9d2a chore: carry lab 3 over
- carry the lab 3 solution over from mtusi2026lab3-safronov
  (solution/bvt2301-safronov, ce89eba): src, migrations, student-tests,
  bench, openspec, .claude, infra/haproxy and infra/valkey, own scripts,
  Dockerfile, compose, pyproject and uv lock
- merge Makefile, .gitignore and .dockerignore (the lab 3 versions only
  add lines to the skeleton, which is the same in lab 4) and
  .env.example (lab 4 skeleton names listed in the header)
- rename the compose project to mtusi-lab4 and the image to :lab4 so
  the stand does not reuse the lab 3 containers and volumes
- update AGENTS.md for lab 4: current lab, origin commit, contract and
  test files of the new skeleton

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 17:32:33 +03:00
Mikhail Verkovykh ec540a1a72 Подготовить skeleton лабораторной 4 по мессенджеру 2026-09-10 18:01:35 +03:00