# consistency: lab4: aborted upload, crash between S3 and DB, janitor, orphans, S3 outage
# date: 2026-10-24 15:40:02, commit: 276dbda + uncommitted changes
# command: python.exe student-tests/test_consistency.py
# exit code: 0, 156.4 s

declared 9 000 000 bytes, sent 211817, closed the connection
declared 9 000 000 bytes, sent 3111817, closed the connection
rows in attachments: 219 -> 219; objects in bucket: 860 -> 860
temporary upload files in api containers: {'b9ad6d33d190': 0, '5aa14958c864': 0} (removed when the request ends)
api containers have no volumes: the original is never stored locally
PASS: an interrupted upload leaves no row, no object and no local file
crash after S3 PUT: row 3a083fe9 state=uploading, object staging/5bda5e392029444d915a233aef8e7af9 exists in S3
the half-written attachment is invisible: GET -> 404, attach to a message -> 404
janitor with default grace (3600 s) right away: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 0, "stale_uploads_deleted": 0, "staging_ready_deleted": 0, "staging_failed_expired": 0, "staging_kept_for_replay": 5, "outbox_rows_deleted": 0, "objects_scanned": 865, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 1, "seconds": 0.31} -> fresh row and object are kept (the upload might still be running)
row made 2 h old; janitor --dry-run: {"dry_run": true, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 0, "staging_ready_deleted": 0, "staging_failed_expired": 0, "staging_kept_for_replay": 5, "outbox_rows_deleted": 0, "objects_scanned": 865, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 1, "seconds": 0.301} -> nothing deleted
janitor: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 1, "staging_ready_deleted": 0, "staging_failed_expired": 0, "staging_kept_for_replay": 5, "outbox_rows_deleted": 0, "objects_scanned": 864, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 1, "seconds": 0.29} -> row and object deleted
janitor again: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 0, "stale_uploads_deleted": 0, "staging_ready_deleted": 0, "staging_failed_expired": 0, "staging_kept_for_replay": 5, "outbox_rows_deleted": 0, "objects_scanned": 864, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 1, "seconds": 0.278} -> idempotent
the ready attachment of the same chat is untouched
PASS: S3 written but DB not committed -> invisible, reclaimed by janitor only after the grace period
object staging/bea3e5320fc84973a5b2b762fe96ddd8 put into the bucket without a row in PostgreSQL
janitor --grace-seconds 3 immediately: {"dry_run": false, "grace_seconds": 3, "stale_uploads": 0, "stale_uploads_deleted": 0, "staging_ready_deleted": 0, "staging_failed_expired": 0, "staging_kept_for_replay": 5, "outbox_rows_deleted": 0, "objects_scanned": 869, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 1, "seconds": 0.292} -> fresh object kept
after 4 s, --dry-run: {"dry_run": true, "grace_seconds": 3, "stale_uploads": 0, "stale_uploads_deleted": 0, "staging_ready_deleted": 0, "staging_failed_expired": 0, "staging_kept_for_replay": 5, "outbox_rows_deleted": 0, "objects_scanned": 869, "orphan_objects": 1, "orphan_objects_deleted": 0, "missing_objects": 1, "seconds": 0.298} -> reported, kept
janitor: {"dry_run": false, "grace_seconds": 3, "stale_uploads": 0, "stale_uploads_deleted": 0, "staging_ready_deleted": 0, "staging_failed_expired": 0, "staging_kept_for_replay": 5, "outbox_rows_deleted": 0, "objects_scanned": 869, "orphan_objects": 1, "orphan_objects_deleted": 1, "missing_objects": 1, "seconds": 0.295} -> orphan deleted
ready attachments older than the grace period are untouched (they have rows)
object of a ready attachment deleted behind the API's back: janitor reports missing_objects=2 and keeps the row; download -> 503 (not 500)
PASS: orphans are found and deleted only after the grace period; missing objects are reported
--- S3 stopped (connection refused / name not resolved): docker compose stop s3
upload -> 503 unavailable in 19922 ms, no storage details in the body
download of an existing file -> 503
metadata, messages (also with an already stored attachment) and /health/ready keep working: S3 is needed only for file bytes
uploading rows of this chat: before 0, after the failed upload 1 (kept as a pointer for janitor if the rollback could not reach S3); not visible through the API
docker compose start s3
after S3 recovery the old file downloads byte for byte
new upload -> 201
janitor after grace period: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 1, "staging_ready_deleted": 0, "staging_failed_expired": 0, "staging_kept_for_replay": 5, "outbox_rows_deleted": 0, "objects_scanned": 879, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 2, "seconds": 0.29}
PASS: S3 stopped (connection refused / name not resolved) -> 503 for file operations only, no visible half-uploads, everything cleaned up
--- S3 frozen (no answer, timeouts): docker compose pause s3
upload -> 503 unavailable in 14860 ms, no storage details in the body
download of an existing file -> 503
metadata, messages (also with an already stored attachment) and /health/ready keep working: S3 is needed only for file bytes
uploading rows of this chat: before 0, after the failed upload 1 (kept as a pointer for janitor if the rollback could not reach S3); not visible through the API
docker compose unpause s3
after S3 recovery the old file downloads byte for byte
new upload -> 201
janitor after grace period: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 1, "staging_ready_deleted": 0, "staging_failed_expired": 0, "staging_kept_for_replay": 5, "outbox_rows_deleted": 0, "objects_scanned": 887, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 2, "seconds": 0.284}
PASS: S3 frozen (no answer, timeouts) -> 503 for file operations only, no visible half-uploads, everything cleaned up
5 passed, 0 failed
