START stack 2026-11-11T20:32:07.954371+00:00

PASS stack: separate durable broker/worker/dispatcher, private internal ports, bounded worker memory

START durability 2026-11-11T20:32:08.262127+00:00

PASS durability: worker down -> 202+linked message+409 -> API/broker restart with volumes -> same job ready; job_id=1d457f73-a915-41c9-913c-5a0014df0511

START outbox 2026-11-11T20:32:38.208630+00:00

PASS outbox: COMMIT persisted while dispatcher stopped -> kill API before publish -> restart -> same job ready; job_id=6b472c6b-52ec-432f-8472-b27fb56598c1

START broker_outage 2026-11-11T20:33:07.567493+00:00

PASS broker outage: API remains ready, upload 202 and messages commit, durable outbox later reaches ready; job_id=a363aa46-6f18-45cb-9fe2-109f83086db7

START redelivery 2026-11-11T20:33:25.177987+00:00

PASS redelivery: same ready job acked without mutation, same 3 variants/checksums; job_id=f951285c-bd7a-4763-b3c5-3a6b649d9a22

START crash 2026-11-11T20:33:34.557181+00:00

PASS crash after 4 S3 writes before DB result/ack: same job recovered; no duplicate variants; job_id=ea420ed9-8c7c-47e7-9d58-a5ccc923b782 attempts=2 window={"final": true, "staging": true, "staging_deleted": false, "status": "processing", "variant_objects": 3}

START lease 2026-11-11T20:34:42.684279+00:00

PASS expired lease: killed stalled processing worker -> automatic recovery, same job; job_id=766e3884-a0df-426c-94c5-f1ba947b6e2a old_lease_until=2026-11-11T20:35:39.564619+00:00 attempts=2

PASS fencing: old lease cannot finish or fail after recovered ready commit

START retries 2026-11-11T20:35:52.377383+00:00

PASS transient errors retry with durable backoff/jitter; job_id=cb2a9208-6580-4003-a015-93873c7191bc audit=[{"action": "retry", "actor": "f7611f3a576d-fb830a49-0", "created_at": "2026-11-11T20:36:05.032253+00:00", "details": {"attempt": 1, "available_at": "2026-11-11T20:36:05.607447Z", "delay_seconds": 0.5774477733124553, "run_attempts": 1}, "reason": "injected_transient"}, {"action": "retry", "actor": "f7611f3a576d-fb830a49-0", "created_at": "2026-11-11T20:36:05.777829+00:00", "details": {"attempt": 2, "available_at": "2026-11-11T20:36:06.849152Z", "delay_seconds": 1.0739815038117153, "run_attempts": 2}, "reason": "injected_transient"}]

PASS retry exhaustion -> failed+409+retained staging -> explicit audited replay -> ready -> ready replay no-op; job_id=6a9cc04b-cc23-4e90-9788-f185773a53be max_attempts=5 error_code=retry_exhausted replay={"attachment_id": "0640b252-52f0-42d6-af9d-6a9bc1743b64", "attempt": 6, "error_code": null, "job_id": "6a9cc04b-cc23-4e90-9788-f185773a53be", "run_attempts": 1, "status": "ready"}

START permanent 2026-11-11T20:37:01.785708+00:00

PASS permanent failure: failed after one attempt; content/message rejected with409; job_id=c4d6f975-751a-4208-9e49-e9e34c15f9d2 error_code=invalid_image

START staging 2026-11-11T20:37:19.184496+00:00

PASS staging cleanup: queued protected; staging deleted only after ready commit; final original retained; attachment_id=6425f731-feab-466c-bfa1-31596c1cacb3

PASS selected lab5 grade 3/4 checks; no grade 5 track claimed

