- run 325 on 5e791a6: every step of the application job exits 0
(make check, test, validate, student_test, test-persistence,
test-failover stop and kill); link it in REPORT.md and the evidence
table, note that the runner had the stand images cached by then
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- scripts/restart.sh waits for postgres, valkey and s3 with
up --wait --no-recreate: Compose 5.4 recreated containers whose image
was pulled by the first up (index digest in the label, platform
manifest digest on the check), s3-bootstrap could not join the new s3
network namespace (exit 128) and make test-persistence failed on a
clean host; found by the CI job on the Gitea runner
- record the runner failure, the repro across Compose 5.4.0 and 5.6.1
and the fixed and control runs on a fresh Docker daemon in
evidence/tests/ci-application.md, REPORT.md and the docker rule
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- github.server_url on this Gitea is the internal http://gitea:3000,
which the job container cannot resolve: the first run failed at
checkout; take https://gitea.privcloud.tech literally
- the skeleton contracts.yml fails at actions/checkout for the same
reason, so the application job also runs make validate
- record the failed first run and the fix in
evidence/tests/ci-application.md and REPORT.md
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- fill the skeleton sections of REPORT.md: author and target grade,
launch from a clean clone, architecture and variable mapping, open
questions to the teacher, limitations, sources and AI use
- add evidence/tests/clean-clone-final.md: clean clone of de98d77
- add .github/workflows/application.yml (CONTRIBUTING.md, CI) and the
.github/ci/ override for the Gitea runner: configs instead of bind
mounts, no host ports; evidence/tests/ci-application.md
- remove PLAN.md now that all changes are archived; point AGENTS.md
and the rules at REPORT.md
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add the baseline label results to REPORT.md: H1, H2, H3 and S4
confirmed, H4 not verified (the fast-unlimited contrast did not
saturate S3, r 1.08). Keep 2000 objects/s as a bound on the S3 share
and give head/deep limits by S3 CPU per object. Refresh failure_checks
(54/54, red run of task 1.3), test_isolated and the pace stop evidence.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An iteration interrupted by SIGINT after its line loses both the
iteration count and the bg_lines counter sample, so neither can match
the console lines exactly. line_problems now requires lines >= each
count (fewer: "lines lost") and at most MAX_VUS extra lines over each.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
make test_isolated: 193 passed, 5 replica tests skipped; full
make failure_checks: 49/49 in 552 s with the unfiltered output, paths
anonymized; REPORT.md points to the reshoots.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
aiohttp 3.14.5 buffers 256 KiB and pauses reading above 512 KiB; name
the aiohttp versions actually checked instead of a range; fix line
references in the idle-connection evidence; the object-storage spec now
exempts the connect timeout from the single call timeout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
s3-idle before and after replacing replicas, red runs on 4cfaab0 and
green ones on b5983f0, failure_checks 49/49, eight student_test runs
without 503, PUT memory probe and bench_uploads_quick; drop the fixed
KNOWN_ISSUES entry and describe the S3 client timeouts in REPORT.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
aiohttp 3.13-3.14.5 re-arms sock_read in resume_reading() after a
streamed response has already returned the connection to the pool; the
timer fires while the connection idles and the next call on it fails in
1 ms with ReadTimeoutError (503 on a healthy S3). The call and per-chunk
asyncio.timeout stay the only S3 timeout. Adds driver.py s3-idle and
documents the timeouts in the rules.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fourth API of the failure stand (S3__TIMEOUT_SECONDS=1), slow streamed
download in the test client, test_upload_after_idle_s3 and the
fix-upload-put-flake spec deltas synced into the main specs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re-run failure checks (32 of 32) and the cleaner pass under a paused S3
on the two-step cleanup, fix the storage-consistency scenario count in
the student test evidence and point the report at the new commit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keep the committed mark, then delete the object and the row under the
same row lock, so an older replica without the mark check that waits on
the row finds it gone instead of promoting it to ready after the object
delete. A mixed-version stand is no worse than before the mark; document
the upgrade order (replicas first, cleaner last) and fix stale wording.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Capture the cleaner pass under a paused S3 (row stays marked pending,
next pass removes it), re-run failure checks (32 of 32 with red runs of
the new checks), student tests and the isolated run on 87c48b9, and
describe the marked three-step cleanup and its rolling-update window in
the report.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Mark a stale pending row in its own committed transaction before the
cleaner deletes the object, and let mark_ready promote only unmarked
rows, so a cleaner rollback after the object delete can no longer leave
a ready attachment without its object. Add fault checks for an unknown
COMMIT outcome, an upload outliving the grace period and a cleaner
failure after the object delete; fix the cleaner entrypoint in rules.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- evidence: consistency (driver upload-consistency), cleaner process,
cleaner S3 key, pending index plan on the working database
- re-run failure_checks (29/29), student_test via LB and test_isolated
with the cleaner and storage-consistency coverage
- REPORT: database and S3 consistency, compensation table, grace period
rationale, "PUT succeeded, COMMIT did not" answer, level 4 rows,
limitations; PLAN stage 3 and AGENTS evidence map
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- upload: delete the stored object when ready fails before COMMIT or the
pending row is gone (503 instead of 500); unknown COMMIT outcome keeps it
- cleaner service (python -m src.cleaner): stale pending rows older than
grace (900 s, must exceed 17 s live-upload bound) under FOR UPDATE SKIP
LOCKED, object before row, own S3 user with read/delete only
- shared session_scope for API and cleaner; early_stop process name
- partial index on pending attachments, CREATE INDEX CONCURRENTLY
- s3-bootstrap: cleaner user, policy and checks; make env_setup appends
missing template variables to an existing .env
- failure checks: cleaner on the failure stand, id-list hook actions,
five new and three updated tests; driver upload-consistency
- sync upload-consistency deltas into main specs (storage-consistency)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- keep one upload body in memory: BytesIO + getvalue() without copy,
fixed glibc mmap threshold in the image (no realloc copy on growth)
- stop PNG inflate at expected size + 1 byte
- reject filename that is not valid UTF-8 after percent-decoding (400)
- failure stand: env limit variables do not change attachment limits;
history with attachment_ids while S3 is stopped
- drop dead constants; restate "old image on new schema" scenario
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
switch_api recreated the API replica with the same IP while HAProxy
still showed the old container's slot UP, so restore_check hit 503
<NOSRV> before the new process listened. Wait until each new replica
answers /health/ready from inside its container, then for LB UP.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>