# consistency: lab4: aborted upload, crash between S3 and DB, janitor, orphans, S3 outage
# date: 2026-11-07 19:42:46, commit: 20afc2c + uncommitted changes
# command: python.exe student-tests/test_consistency.py
# exit code: 0, 131.5 s

declared 9 000 000 bytes, sent 211817, closed the connection
declared 9 000 000 bytes, sent 3111817, closed the connection
rows in attachments: 481 -> 481; objects in bucket: 1886 -> 1886
temporary upload files in api containers: {'b7f37f051ba1': 0, '0c7893e1043d': 0} (removed when the request ends)
api containers have no volumes: the original is never stored locally
PASS: an interrupted upload leaves no row, no object and no local file
crash after S3 PUT: row aa892679 state=uploading, object staging/a6d6cacccc714e1683e0cbf58f6c3bb8 exists in S3
the half-written attachment is invisible: GET -> 404, attach to a message -> 404
janitor with default grace (3600 s) right away: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 0, "stale_uploads_deleted": 0, "staging_ready_deleted": 0, "staging_failed_expired": 0, "staging_kept_for_replay": 0, "outbox_rows_deleted": 0, "objects_scanned": 1891, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 2, "seconds": 0.698} -> fresh row and object are kept (the upload might still be running)
row made 2 h old; janitor --dry-run: {"dry_run": true, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 0, "staging_ready_deleted": 0, "staging_failed_expired": 0, "staging_kept_for_replay": 0, "outbox_rows_deleted": 0, "objects_scanned": 1891, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 2, "seconds": 0.695} -> nothing deleted
janitor: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 1, "staging_ready_deleted": 0, "staging_failed_expired": 0, "staging_kept_for_replay": 0, "outbox_rows_deleted": 0, "objects_scanned": 1890, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 2, "seconds": 0.691} -> row and object deleted
janitor again: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 0, "stale_uploads_deleted": 0, "staging_ready_deleted": 0, "staging_failed_expired": 0, "staging_kept_for_replay": 0, "outbox_rows_deleted": 0, "objects_scanned": 1890, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 2, "seconds": 0.599} -> idempotent
the ready attachment of the same chat is untouched
PASS: S3 written but DB not committed -> invisible, reclaimed by janitor only after the grace period
object staging/71f7fcf571ca43e18de2aa82aac94d8e put into the bucket without a row in PostgreSQL
janitor --grace-seconds 3 immediately: {"dry_run": false, "grace_seconds": 3, "stale_uploads": 0, "stale_uploads_deleted": 0, "staging_ready_deleted": 0, "staging_failed_expired": 0, "staging_kept_for_replay": 0, "outbox_rows_deleted": 0, "objects_scanned": 1895, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 2, "seconds": 0.689} -> fresh object kept
after 4 s, --dry-run: {"dry_run": true, "grace_seconds": 3, "stale_uploads": 0, "stale_uploads_deleted": 0, "staging_ready_deleted": 0, "staging_failed_expired": 0, "staging_kept_for_replay": 0, "outbox_rows_deleted": 0, "objects_scanned": 1895, "orphan_objects": 1, "orphan_objects_deleted": 0, "missing_objects": 2, "seconds": 0.793} -> reported, kept
janitor: {"dry_run": false, "grace_seconds": 3, "stale_uploads": 0, "stale_uploads_deleted": 0, "staging_ready_deleted": 0, "staging_failed_expired": 0, "staging_kept_for_replay": 0, "outbox_rows_deleted": 0, "objects_scanned": 1895, "orphan_objects": 1, "orphan_objects_deleted": 1, "missing_objects": 2, "seconds": 0.678} -> orphan deleted
ready attachments older than the grace period are untouched (they have rows)
object of a ready attachment deleted behind the API's back: janitor reports missing_objects=3 and keeps the row; download -> 503 (not 500)
PASS: orphans are found and deleted only after the grace period; missing objects are reported
--- S3 stopped (connection refused / name not resolved): docker compose stop s3
upload -> 503 unavailable in 781 ms, no storage details in the body
download of an existing file -> 503
metadata, messages (also with an already stored attachment) and /health/ready keep working: S3 is needed only for file bytes
uploading rows of this chat: before 0, after the failed upload 1 (kept as a pointer for janitor if the rollback could not reach S3); not visible through the API
docker compose start s3
after S3 recovery the old file downloads byte for byte
new upload -> 202 (14.7 s after the old file became readable)
janitor after grace period: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 1, "staging_ready_deleted": 0, "staging_failed_expired": 0, "staging_kept_for_replay": 0, "outbox_rows_deleted": 0, "objects_scanned": 1905, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 3, "seconds": 0.695}
PASS: S3 stopped (connection refused / name not resolved) -> 503 for file operations only, no visible half-uploads, everything cleaned up
--- S3 frozen (no answer, timeouts): docker compose pause s3
upload -> 503 unavailable in 8703 ms, no storage details in the body
download of an existing file -> 503
metadata, messages (also with an already stored attachment) and /health/ready keep working: S3 is needed only for file bytes
uploading rows of this chat: before 0, after the failed upload 1 (kept as a pointer for janitor if the rollback could not reach S3); not visible through the API
docker compose unpause s3
after S3 recovery the old file downloads byte for byte
new upload -> 202 (0.3 s after the old file became readable)
janitor after grace period: {"dry_run": false, "grace_seconds": 3600, "stale_uploads": 1, "stale_uploads_deleted": 1, "staging_ready_deleted": 0, "staging_failed_expired": 0, "staging_kept_for_replay": 0, "outbox_rows_deleted": 0, "objects_scanned": 1914, "orphan_objects": 0, "orphan_objects_deleted": 0, "missing_objects": 3, "seconds": 0.685}
PASS: S3 frozen (no answer, timeouts) -> 503 for file operations only, no visible half-uploads, everything cleaned up
5 passed, 0 failed
