aiohttp 3.13-3.14.5 re-arms sock_read in resume_reading() after a
streamed response has already returned the connection to the pool; the
timer fires while the connection idles and the next call on it fails in
1 ms with ReadTimeoutError (503 on a healthy S3). The call and per-chunk
asyncio.timeout stay the only S3 timeout. Adds driver.py s3-idle and
documents the timeouts in the rules.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fourth API of the failure stand (S3__TIMEOUT_SECONDS=1), slow streamed
download in the test client, test_upload_after_idle_s3 and the
fix-upload-put-flake spec deltas synced into the main specs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re-run failure checks (32 of 32) and the cleaner pass under a paused S3
on the two-step cleanup, fix the storage-consistency scenario count in
the student test evidence and point the report at the new commit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keep the committed mark, then delete the object and the row under the
same row lock, so an older replica without the mark check that waits on
the row finds it gone instead of promoting it to ready after the object
delete. A mixed-version stand is no worse than before the mark; document
the upgrade order (replicas first, cleaner last) and fix stale wording.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Capture the cleaner pass under a paused S3 (row stays marked pending,
next pass removes it), re-run failure checks (32 of 32 with red runs of
the new checks), student tests and the isolated run on 87c48b9, and
describe the marked three-step cleanup and its rolling-update window in
the report.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Mark a stale pending row in its own committed transaction before the
cleaner deletes the object, and let mark_ready promote only unmarked
rows, so a cleaner rollback after the object delete can no longer leave
a ready attachment without its object. Add fault checks for an unknown
COMMIT outcome, an upload outliving the grace period and a cleaner
failure after the object delete; fix the cleaner entrypoint in rules.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- evidence: consistency (driver upload-consistency), cleaner process,
cleaner S3 key, pending index plan on the working database
- re-run failure_checks (29/29), student_test via LB and test_isolated
with the cleaner and storage-consistency coverage
- REPORT: database and S3 consistency, compensation table, grace period
rationale, "PUT succeeded, COMMIT did not" answer, level 4 rows,
limitations; PLAN stage 3 and AGENTS evidence map
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- upload: delete the stored object when ready fails before COMMIT or the
pending row is gone (503 instead of 500); unknown COMMIT outcome keeps it
- cleaner service (python -m src.cleaner): stale pending rows older than
grace (900 s, must exceed 17 s live-upload bound) under FOR UPDATE SKIP
LOCKED, object before row, own S3 user with read/delete only
- shared session_scope for API and cleaner; early_stop process name
- partial index on pending attachments, CREATE INDEX CONCURRENTLY
- s3-bootstrap: cleaner user, policy and checks; make env_setup appends
missing template variables to an existing .env
- failure checks: cleaner on the failure stand, id-list hook actions,
five new and three updated tests; driver upload-consistency
- sync upload-consistency deltas into main specs (storage-consistency)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- keep one upload body in memory: BytesIO + getvalue() without copy,
fixed glibc mmap threshold in the image (no realloc copy on growth)
- stop PNG inflate at expected size + 1 byte
- reject filename that is not valid UTF-8 after percent-decoding (400)
- failure stand: env limit variables do not change attachment limits;
history with attachment_ids while S3 is stopped
- drop dead constants; restate "old image on new schema" scenario
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
switch_api recreated the API replica with the same IP while HAProxy
still showed the old container's slot UP, so restore_check hit 503
<NOSRV> before the new process listened. Wait until each new replica
answers /health/ready from inside its container, then for LB UP.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- upload, metadata and download endpoints for PNG/JPEG attachments:
limits before reading and decompression, type by content,
structural integrity check without decoding pixels, safe filename
- write order: pending row with object key -> PUT to S3 -> ready;
per-step DB sessions so no pool connection is held during transfers
- attachment_ids in messages (validation, idempotency hash, history);
whitespace-only text allowed with an attachment (teacher, 2026-10-20)
- migration: attachments and message_attachments tables, text length
CHECK relaxed with NOT VALID
- student tests, fault checks and driver commands for attachments;
rollout profile starts from e1161f7 and checks the commit first
- sync attachments deltas into main specs; rules, AGENTS.md, PLAN.md
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- evidence/s3 and evidence/tests re-captured from f57635a: first write
after healthy, deleted bucket, netstat-based port check, readiness
with s3 paused, prefix cleanup check
- PLAN: HEAD on a deleted bucket is an accepted limitation
- KNOWN_ISSUES: bootstrap after a manual bucket delete
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- s3 healthcheck and fault hook probe wait for the volume server to
register with master (first write after restart no longer waits ~5 s)
- s3-bootstrap mounts /data as tmpfs: no anonymous volume per container
- S3 client: NoSuchBucket and AccessDenied are 503 with their own error
logs, not a missing object or rejected credentials; delete ignores only
a missing key
- test_isolated verifies prefix removal (weed shell exits 0 on errors)
- s3_check derives ports from listening sockets in the s3 container
- readiness without S3 is also checked with s3 paused
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- s3 and s3-bootstrap services: SeaweedFS 4.48 pinned by digest,
internals on loopback, signing key, private bucket, API user policy,
privacy checks on every bootstrap
- [s3] settings and aiobotocore client in src/infra/s3: deadline per
call, no retries, 503 on outage or rejected credentials
- make start_s3, stop_s3, s3_check; run_dev, test_isolated, wipe_data
and restart hook include S3
- failure_checks with own S3 and readiness check without it; bench
generates S3 secrets and filters them out of the passport
- rules, AGENTS.md, skill; sync health delta; KNOWN_ISSUES: rollout_from
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- carry the lab 3 solution over from mtusi2026lab3-safronov
(solution/bvt2301-safronov, ce89eba): src, migrations, student-tests,
bench, openspec, .claude, infra/haproxy and infra/valkey, own scripts,
Dockerfile, compose, pyproject and uv lock
- merge Makefile, .gitignore and .dockerignore (the lab 3 versions only
add lines to the skeleton, which is the same in lab 4) and
.env.example (lab 4 skeleton names listed in the header)
- rename the compose project to mtusi-lab4 and the image to :lab4 so
the stand does not reuse the lab 3 containers and volumes
- update AGENTS.md for lab 4: current lab, origin commit, contract and
test files of the new skeleton
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>