# 2026-11-14T16:31:40Z python3 student-tests/incident_worker_stop.py --output evidence/incident-worker-stop.json
[   0.48s] traffic started {"threads": 4, "think_s": 0.2, "upload_every": 4}
[  45.49s] baseline {"requests": 1584, "ok": 1584, "codes": {"200": 704, "201": 704, "202": 176}, "error_ratio_5xx": 0.0, "ok_rps": 39.7, "p50_ms": 8.4, "p95_ms": 20.2, "p99_ms": 37.0}
[  46.95s] FAULT: docker compose stop worker {"running": []}
[  46.98s] alert MessengerQueueLagGrowing -> inactive 
[  92.09s] alert MessengerQueueLagGrowing -> pending 
[ 152.29s] alert MessengerQueueLagGrowing -> firing 
[ 152.31s] alert firing {"detect_s": 106.8, "ready_lag_s": 105.109317, "queued_jobs_db": 456.0, "rabbitmq_ready": 455.0}
[ 172.32s] API during worker outage {"requests": 4952, "ok": 4952, "codes": {"200": 2200, "201": 2200, "202": 552}, "error_ratio_5xx": 0.0, "ok_rps": 39.1, "p50_ms": 9.9, "p95_ms": 20.1, "p99_ms": 39.8}
[ 178.39s] RECOVERY: docker compose up worker {"outage_s": 126.8}
[ 253.82s] alert resolved {"after_worker_start_s": 81.5, "ready_lag_s": 62.089227}
[ 332.12s] backlog drained: ready lag < 5 s {"after_worker_start_s": 159.8, "queued_jobs_db": 20.0}
[ 350.14s] after recovery {"requests": 743, "ok": 743, "codes": {"200": 330, "201": 330, "202": 83}, "error_ratio_5xx": 0.0, "ok_rps": 41.4, "p50_ms": 5.6, "p95_ms": 14.7, "p99_ms": 29.8}
[ 350.56s] trace of an upload accepted during outage {"trace_id": "0892d3393587abe986ca6166ea70f698", "publish_to_consume_gap_s": 97.5, "process_ms": 238.965, "queue_wait_attr_s": 97.473}
[ 350.58s] worker lifecycle logs (Loki) {"events": [[0.3, "worker.draining", "ff3fb8ef2677"], [0.3, "worker.draining", "ff6ca1cafae0"], [1.1, "worker.stopped", "ff3fb8ef2677"], [1.1, "worker.stopped", "ff6ca1cafae0"], [127.8, "worker.started", "ff3fb8ef2677"], [127.8, "worker.started", "ff6ca1cafae0"]]}
PASS: worker outage detected in 107s, alert resolved 81s after restart, drained 160s after restart; API 5xx during outage 0.00%
exit=0
