- run 325 on 5e791a6: every step of the application job exits 0
(make check, test, validate, student_test, test-persistence,
test-failover stop and kill); link it in REPORT.md and the evidence
table, note that the runner had the stand images cached by then
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- scripts/restart.sh waits for postgres, valkey and s3 with
up --wait --no-recreate: Compose 5.4 recreated containers whose image
was pulled by the first up (index digest in the label, platform
manifest digest on the check), s3-bootstrap could not join the new s3
network namespace (exit 128) and make test-persistence failed on a
clean host; found by the CI job on the Gitea runner
- record the runner failure, the repro across Compose 5.4.0 and 5.6.1
and the fixed and control runs on a fresh Docker daemon in
evidence/tests/ci-application.md, REPORT.md and the docker rule
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- github.server_url on this Gitea is the internal http://gitea:3000,
which the job container cannot resolve: the first run failed at
checkout; take https://gitea.privcloud.tech literally
- the skeleton contracts.yml fails at actions/checkout for the same
reason, so the application job also runs make validate
- record the failed first run and the fix in
evidence/tests/ci-application.md and REPORT.md
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- fill the skeleton sections of REPORT.md: author and target grade,
launch from a clean clone, architecture and variable mapping, open
questions to the teacher, limitations, sources and AI use
- add evidence/tests/clean-clone-final.md: clean clone of de98d77
- add .github/workflows/application.yml (CONTRIBUTING.md, CI) and the
.github/ci/ override for the Gitea runner: configs instead of bind
mounts, no host ports; evidence/tests/ci-application.md
- remove PLAN.md now that all changes are archived; point AGENTS.md
and the rules at REPORT.md
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add the baseline label results to REPORT.md: H1, H2, H3 and S4
confirmed, H4 not verified (the fast-unlimited contrast did not
saturate S3, r 1.08). Keep 2000 objects/s as a bound on the S3 share
and give head/deep limits by S3 CPU per object. Refresh failure_checks
(54/54, red run of task 1.3), test_isolated and the pace stop evidence.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An iteration interrupted by SIGINT after its line loses both the
iteration count and the bg_lines counter sample, so neither can match
the console lines exactly. line_problems now requires lines >= each
count (fewer: "lines lost") and at most MAX_VUS extra lines over each.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
make test_isolated: 193 passed, 5 replica tests skipped; full
make failure_checks: 49/49 in 552 s with the unfiltered output, paths
anonymized; REPORT.md points to the reshoots.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
aiohttp 3.14.5 buffers 256 KiB and pauses reading above 512 KiB; name
the aiohttp versions actually checked instead of a range; fix line
references in the idle-connection evidence; the object-storage spec now
exempts the connect timeout from the single call timeout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
s3-idle before and after replacing replicas, red runs on 4cfaab0 and
green ones on b5983f0, failure_checks 49/49, eight student_test runs
without 503, PUT memory probe and bench_uploads_quick; drop the fixed
KNOWN_ISSUES entry and describe the S3 client timeouts in REPORT.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
aiohttp 3.13-3.14.5 re-arms sock_read in resume_reading() after a
streamed response has already returned the connection to the pool; the
timer fires while the connection idles and the next call on it fails in
1 ms with ReadTimeoutError (503 on a healthy S3). The call and per-chunk
asyncio.timeout stay the only S3 timeout. Adds driver.py s3-idle and
documents the timeouts in the rules.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fourth API of the failure stand (S3__TIMEOUT_SECONDS=1), slow streamed
download in the test client, test_upload_after_idle_s3 and the
fix-upload-put-flake spec deltas synced into the main specs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re-run failure checks (32 of 32) and the cleaner pass under a paused S3
on the two-step cleanup, fix the storage-consistency scenario count in
the student test evidence and point the report at the new commit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keep the committed mark, then delete the object and the row under the
same row lock, so an older replica without the mark check that waits on
the row finds it gone instead of promoting it to ready after the object
delete. A mixed-version stand is no worse than before the mark; document
the upgrade order (replicas first, cleaner last) and fix stale wording.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Capture the cleaner pass under a paused S3 (row stays marked pending,
next pass removes it), re-run failure checks (32 of 32 with red runs of
the new checks), student tests and the isolated run on 87c48b9, and
describe the marked three-step cleanup and its rolling-update window in
the report.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Mark a stale pending row in its own committed transaction before the
cleaner deletes the object, and let mark_ready promote only unmarked
rows, so a cleaner rollback after the object delete can no longer leave
a ready attachment without its object. Add fault checks for an unknown
COMMIT outcome, an upload outliving the grace period and a cleaner
failure after the object delete; fix the cleaner entrypoint in rules.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- evidence: consistency (driver upload-consistency), cleaner process,
cleaner S3 key, pending index plan on the working database
- re-run failure_checks (29/29), student_test via LB and test_isolated
with the cleaner and storage-consistency coverage
- REPORT: database and S3 consistency, compensation table, grace period
rationale, "PUT succeeded, COMMIT did not" answer, level 4 rows,
limitations; PLAN stage 3 and AGENTS evidence map
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- upload: delete the stored object when ready fails before COMMIT or the
pending row is gone (503 instead of 500); unknown COMMIT outcome keeps it
- cleaner service (python -m src.cleaner): stale pending rows older than
grace (900 s, must exceed 17 s live-upload bound) under FOR UPDATE SKIP
LOCKED, object before row, own S3 user with read/delete only
- shared session_scope for API and cleaner; early_stop process name
- partial index on pending attachments, CREATE INDEX CONCURRENTLY
- s3-bootstrap: cleaner user, policy and checks; make env_setup appends
missing template variables to an existing .env
- failure checks: cleaner on the failure stand, id-list hook actions,
five new and three updated tests; driver upload-consistency
- sync upload-consistency deltas into main specs (storage-consistency)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- keep one upload body in memory: BytesIO + getvalue() without copy,
fixed glibc mmap threshold in the image (no realloc copy on growth)
- stop PNG inflate at expected size + 1 byte
- reject filename that is not valid UTF-8 after percent-decoding (400)
- failure stand: env limit variables do not change attachment limits;
history with attachment_ids while S3 is stopped
- drop dead constants; restate "old image on new schema" scenario
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
switch_api recreated the API replica with the same IP while HAProxy
still showed the old container's slot UP, so restore_check hit 503
<NOSRV> before the new process listened. Wait until each new replica
answers /health/ready from inside its container, then for LB UP.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- upload, metadata and download endpoints for PNG/JPEG attachments:
limits before reading and decompression, type by content,
structural integrity check without decoding pixels, safe filename
- write order: pending row with object key -> PUT to S3 -> ready;
per-step DB sessions so no pool connection is held during transfers
- attachment_ids in messages (validation, idempotency hash, history);
whitespace-only text allowed with an attachment (teacher, 2026-10-20)
- migration: attachments and message_attachments tables, text length
CHECK relaxed with NOT VALID
- student tests, fault checks and driver commands for attachments;
rollout profile starts from e1161f7 and checks the commit first
- sync attachments deltas into main specs; rules, AGENTS.md, PLAN.md
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- evidence/s3 and evidence/tests re-captured from f57635a: first write
after healthy, deleted bucket, netstat-based port check, readiness
with s3 paused, prefix cleanup check
- PLAN: HEAD on a deleted bucket is an accepted limitation
- KNOWN_ISSUES: bootstrap after a manual bucket delete
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- s3 healthcheck and fault hook probe wait for the volume server to
register with master (first write after restart no longer waits ~5 s)
- s3-bootstrap mounts /data as tmpfs: no anonymous volume per container
- S3 client: NoSuchBucket and AccessDenied are 503 with their own error
logs, not a missing object or rejected credentials; delete ignores only
a missing key
- test_isolated verifies prefix removal (weed shell exits 0 on errors)
- s3_check derives ports from listening sockets in the s3 container
- readiness without S3 is also checked with s3 paused
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- s3 and s3-bootstrap services: SeaweedFS 4.48 pinned by digest,
internals on loopback, signing key, private bucket, API user policy,
privacy checks on every bootstrap
- [s3] settings and aiobotocore client in src/infra/s3: deadline per
call, no retries, 503 on outage or rejected credentials
- make start_s3, stop_s3, s3_check; run_dev, test_isolated, wipe_data
and restart hook include S3
- failure_checks with own S3 and readiness check without it; bench
generates S3 secrets and filters them out of the passport
- rules, AGENTS.md, skill; sync health delta; KNOWN_ISSUES: rollout_from
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- carry the lab 3 solution over from mtusi2026lab3-safronov
(solution/bvt2301-safronov, ce89eba): src, migrations, student-tests,
bench, openspec, .claude, infra/haproxy and infra/valkey, own scripts,
Dockerfile, compose, pyproject and uv lock
- merge Makefile, .gitignore and .dockerignore (the lab 3 versions only
add lines to the skeleton, which is the same in lab 4) and
.env.example (lab 4 skeleton names listed in the header)
- rename the compose project to mtusi-lab4 and the image to :lab4 so
the stand does not reuse the lab 3 containers and volumes
- update AGENTS.md for lab 4: current lab, origin commit, contract and
test files of the new skeleton
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>