Re-run failure checks (32 of 32) and the cleaner pass under a paused S3
on the two-step cleanup, fix the storage-consistency scenario count in
the student test evidence and point the report at the new commit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keep the committed mark, then delete the object and the row under the
same row lock, so an older replica without the mark check that waits on
the row finds it gone instead of promoting it to ready after the object
delete. A mixed-version stand is no worse than before the mark; document
the upgrade order (replicas first, cleaner last) and fix stale wording.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Capture the cleaner pass under a paused S3 (row stays marked pending,
next pass removes it), re-run failure checks (32 of 32 with red runs of
the new checks), student tests and the isolated run on 87c48b9, and
describe the marked three-step cleanup and its rolling-update window in
the report.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- evidence: consistency (driver upload-consistency), cleaner process,
cleaner S3 key, pending index plan on the working database
- re-run failure_checks (29/29), student_test via LB and test_isolated
with the cleaner and storage-consistency coverage
- REPORT: database and S3 consistency, compensation table, grace period
rationale, "PUT succeeded, COMMIT did not" answer, level 4 rows,
limitations; PLAN stage 3 and AGENTS evidence map
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- evidence/s3 and evidence/tests re-captured from f57635a: first write
after healthy, deleted bucket, netstat-based port check, readiness
with s3 paused, prefix cleanup check
- PLAN: HEAD on a deleted bucket is an accepted limitation
- KNOWN_ISSUES: bootstrap after a manual bucket delete
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- s3 healthcheck and fault hook probe wait for the volume server to
register with master (first write after restart no longer waits ~5 s)
- s3-bootstrap mounts /data as tmpfs: no anonymous volume per container
- S3 client: NoSuchBucket and AccessDenied are 503 with their own error
logs, not a missing object or rejected credentials; delete ignores only
a missing key
- test_isolated verifies prefix removal (weed shell exits 0 on errors)
- s3_check derives ports from listening sockets in the s3 container
- readiness without S3 is also checked with s3 paused
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>