# fault injection suite (levels 3-4, track 5.2)
# date: 2026-11-10T13:42:20Z, FAULT_INJECTION=1, WORKER_REPLICAS=1, WORKER_CONCURRENCY=2, LEASE_SECONDS=20
# command: python3 student-tests/faults_test.py --output evidence/faults.json

--- worker_down: worker stopped -> upload 202 -> message -> API + broker restart -> worker start -> ready
    worker: messenger-lab5-worker-1 exited Exited (0) Less than a second ago
    queue before broker restart: {"jobs": {"superseded": 1, "failed": 6, "done": 33, "queued": 1}, "outbox_unpublished": 0, "queue_ready_messages": 1, "queue_consumers": 0, "dlq_messages": 1}
    queue after broker restart (volume kept): {"jobs": {"superseded": 1, "failed": 6, "done": 33, "queued": 1}, "outbox_unpublished": 0, "queue_ready_messages": 1, "queue_consumers": 0, "dlq_messages": 1}
    attachment 1f804983-862e-4fcc-b3ca-ee2af8564aba ready after worker start; thumbnail 256x160
PASS worker_down (31.1s)
--- outbox_gap: COMMIT done, relay down (= crash between COMMIT and publish) -> relay restart publishes -> ready
    relay down: outbox [{'id': 60, 'attempt': '1', 'published': False, 'publish_count': 0}], attachment queued
    after relay start: ready, events ['accepted', 'claimed', 'ready']
PASS outbox_gap (15.8s)
--- relay_crash_after_confirm: relay dies after the publisher confirm but before marking the row -> duplicate publish -> worker no-op
    events: ['accepted', 'fault_injected', 'claimed', 'ready', 'duplicate_noop']
PASS relay_crash_after_confirm (7.6s)
--- crash_after_claim: crash before any side effect: lease expires, reaper re-enqueues attempt 2
    worker.after_claim: ready in 23.9s; events ['accepted', 'claimed', 'fault_injected', 'lease_held_noop', 'lease_expired', 'retry_scheduled', 'claimed', 'ready']; variant attempts [2]
PASS crash_after_claim (28.5s)
--- crash_after_s3: crash after writing S3 objects, before COMMIT: retry overwrites the same keys, one set of variants
    worker.after_s3_before_commit: ready in 24.2s; events ['accepted', 'claimed', 'fault_injected', 'lease_held_noop', 'lease_expired', 'retry_scheduled', 'claimed', 'ready']; variant attempts [2]
PASS crash_after_s3 (29.0s)
--- crash_after_commit: crash after COMMIT, before ack: redelivery sees done -> ack as no-op, no second attempt
    worker.after_commit_before_ack: ready in 1.7s; events ['accepted', 'claimed', 'ready', 'fault_injected', 'duplicate_noop']; variant attempts [1]
PASS crash_after_commit (6.6s)
--- transient_backoff: two transient failures -> retries with exponential backoff + jitter -> ready on attempt 3
    retry delays [1.12, 3.15] (windows [1,2] and [2,4] s)
PASS transient_backoff (12.8s)
--- exhausted_then_replay: 5 transient failures -> failed retry_exhausted (registry) -> explicit replay -> ready; replay of ready refused
    failed after 5 attempts, registry entry present
    replayed job 5ff64075-edfa-4472-afed-a003bdbb601d (attempt 6 next)
    replay of ready: refused: job is done (attachment ready); replay applies only to failed jobs
PASS exhausted_then_replay (38.3s)
--- long_running_not_hung: a 30 s step longer than the 20 s lease: heartbeat renews the lease -> no second attempt
    30 s processing with 20 s lease: attempt=1, no lease_expired
PASS long_running_not_hung (37.4s)
--- duplicate: 3 raw duplicates of an already completed job: ack as no-op, nothing rewritten
    3 duplicates -> 3 duplicate_noop, updated_at unchanged
PASS duplicate (12.4s)
--- reorder: relay stopped; deliveries arrive out of order: attempt 1, then 1 again and 4, then the outbox backlog
    events: ['accepted', 'claimed', 'fault_injected', 'retry_scheduled', 'stale_attempt_noop', 'stale_attempt_noop', 'stale_attempt_noop', 'claimed', 'ready']
PASS reorder (20.2s)
--- new_version: v1 ready -> reprocess v2 -> v2 published; replaying a v1 message afterwards changes nothing
    v1 optimized 1090d84ed841.. -> v2 f2878c4a9d89..; stale v1 message: duplicate_noop
PASS new_version (11.9s)
--- stale_version_race: v1 attempt is slow; v2 commits first; v1 then tries to commit -> superseded, v2 result kept
    commit order: [('ready', 'v2: thumbnail,crop,optimized'), ('superseded', 'published v2 is newer than v1')]; v1 objects are unreferenced leftovers for the sweeper
PASS stale_version_race (16.3s)
--- poison_message: a message that violates jobs.schema.json goes to the DLQ instead of looping
    DLQ 1 -> 2
PASS poison_message (6.4s)
PASS: 14 fault scenarios; final {"jobs": {"superseded": 2, "failed": 6, "done": 47}, "outbox_unpublished": 0, "queue_ready_messages": 0, "queue_consumers": 1, "dlq_messages": 2}
