# 2026-11-13T18:13:19Z python3 student-tests/s3_outage.py --output evidence/s3-outage.json
[   2.03s] worker stopped, job accepted before outage {"attachment_id": "6cdcfa17-a4bd-4764-8d3a-8a0fd6067d01"}
[  24.40s] S3 `stop` for 5 s {"upload": {"status": 503, "ms": 5, "code": "unavailable", "retry_after": "1"}, "download_ready": {"status": 503, "ms": 3, "code": "unavailable", "retry_after": "1"}, "metadata_queued": {"status": 200, "ms": 2, "code": null, "retry_after": null}, "text_message": {"status": 201, "ms": 4, "code": null, "retry_after": null}, "ready": {"status": 200, "ms": 1, "code": null, "retry_after": null}}
[  25.71s] worker meets S3 outage -> retry with backoff {"events": ["enqueued", "claimed", "retry_scheduled"], "retries": [{"error_code": "storage_unavailable", "next_attempt": 2, "delay_seconds": 1.798}]}
[  45.24s] S3 `start` -> healthy {"seconds": 19.3}
[  50.39s] S3 `pause` for 5 s {"upload": {"status": 503, "ms": 14, "code": "unavailable", "retry_after": "1"}, "download_ready": {"status": 503, "ms": 10, "code": "unavailable", "retry_after": "1"}, "metadata_queued": {"status": 200, "ms": 7, "code": null, "retry_after": null}, "text_message": {"status": 201, "ms": 10, "code": null, "retry_after": null}, "ready": {"status": 200, "ms": 2, "code": null, "retry_after": null}}
[  56.29s] S3 `unpause` -> healthy {"seconds": 5.8}
[  57.12s] job accepted before outage completed {"final": "ready", "replay": null, "attempts": 3, "events": ["enqueued", "claimed", "retry_scheduled", "claimed", "retry_scheduled", "claimed", "ready", "staging_deleted"]}
[  57.47s] upload after recovery {"status": "ready"}
PASS: S3 outage handled by API (503) and worker (retry), accepted work completed
exit=0
