# incident-worker: lab6 incident 1: worker stopped -> queue lag + no consumers firing -> resolved
# date: 2026-11-01 00:28:44, commit: unknown
# command: python student-tests/incidents.py worker-stop
# exit code: 0, 128.6 s

=== worker-stop
[   0.1s] baseline: consumers=2.0, oldest job age=0.0 s
[   1.3s] docker compose stop worker
[   1.4s] uploaded 3 attachments: 202 queued (API accepts without worker)
[   1.4s]   consumers=2.0 backlog queued=0.0 oldest age=0.0 s, alerts: {}
[   6.4s]   consumers=0.0 backlog queued=3.0 oldest age=2.1 s, alerts: {'MessengerWorkerNoConsumers': 'pending'}
[  11.4s]   consumers=0.0 backlog queued=3.0 oldest age=2.1 s, alerts: {'MessengerWorkerNoConsumers': 'pending'}
[  16.4s]   consumers=0.0 backlog queued=3.0 oldest age=12.1 s, alerts: {'MessengerWorkerNoConsumers': 'pending'}
[  21.4s]   consumers=0.0 backlog queued=3.0 oldest age=12.1 s, alerts: {'MessengerWorkerNoConsumers': 'pending'}
[  26.4s]   consumers=0.0 backlog queued=3.0 oldest age=22.1 s, alerts: {'MessengerWorkerNoConsumers': 'pending'}
[  31.4s]   consumers=0.0 backlog queued=3.0 oldest age=22.1 s, alerts: {'MessengerWorkerNoConsumers': 'pending'}
[  36.4s]   consumers=0.0 backlog queued=3.0 oldest age=27.1 s, alerts: {'MessengerWorkerNoConsumers': 'pending'}
[  41.4s]   consumers=0.0 backlog queued=3.0 oldest age=27.1 s, alerts: {'MessengerWorkerNoConsumers': 'pending'}
[  46.4s]   consumers=0.0 backlog queued=3.0 oldest age=37.1 s, alerts: {'MessengerWorkerNoConsumers': 'pending'}
[  51.4s]   consumers=0.0 backlog queued=3.0 oldest age=37.1 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'pending'}
[  56.4s]   consumers=0.0 backlog queued=3.0 oldest age=47.2 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'pending'}
[  61.5s]   consumers=0.0 backlog queued=3.0 oldest age=47.2 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'pending'}
[  66.5s] ALERT MessengerWorkerNoConsumers firing (66.4 s after stop)
[  66.5s]   consumers=0.0 backlog queued=3.0 oldest age=57.2 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'firing'}
[  71.5s]   consumers=0.0 backlog queued=3.0 oldest age=57.2 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'firing'}
[  76.5s]   consumers=0.0 backlog queued=3.0 oldest age=67.2 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'firing'}
[  81.5s]   consumers=0.0 backlog queued=3.0 oldest age=67.2 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'firing'}
[  86.5s]   consumers=0.0 backlog queued=3.0 oldest age=77.2 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'firing'}
[  91.5s]   consumers=0.0 backlog queued=3.0 oldest age=77.2 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'firing'}
[  96.5s]   consumers=0.0 backlog queued=3.0 oldest age=87.2 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'firing'}
[ 101.5s]   consumers=0.0 backlog queued=3.0 oldest age=87.2 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'firing'}
[ 106.5s]   consumers=0.0 backlog queued=3.0 oldest age=97.3 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'firing'}
[ 111.5s] ALERT MessengerQueueLagGrowing firing (111.4 s after stop)
[ 111.5s] diagnosis: {"queue_consumers": 0.0, "broker_queue_depth": 3.0, "jobs_backlog_queued": 3.0, "oldest_job_age_s": 97.3, "outbox_oldest_age_s": 0.0, "published_ok_last_5m": 3.102218085931441, "worker_metrics_last_seen_s": 110.0, "last_worker_log": {"event": "service.stopped", "message": "stopped"}, "upload_trace_span_names": ["INSERT", "NOTIFY", "POST /api/v1/chats/{chat_id}/attachments", "S3 PutObject", "SELECT", "UPDATE", "create attachments", "inspect image", "publish attachments", "redis GET"], "upload_trace_has_consumer_span": false}
[ 111.6s] docker compose start worker
[ 113.5s] all attachments ready 2.0 s after worker start; backlog now 3.0
[ 113.5s]   ... MessengerWorkerNoConsumers: firing (0 s)
[ 118.5s]   ... MessengerWorkerNoConsumers: firing (5 s)
[ 123.5s]   ... MessengerWorkerNoConsumers: firing (10 s)
[ 128.5s] alerts resolved: {'MessengerWorkerNoConsumers': 17.0, 'MessengerQueueLagGrowing': 17.0}; now {'jobs_backlog_queued': 0.0, 'oldest_job_age_s': 0.0, 'queue_consumers': 2.0}
[ 128.5s] saved /home/claude/work/mtusi2026lab6/evidence/native-sandbox/incident-worker-stop.json
PASS: worker stop -> no consumers + growing queue lag -> firing; cause from metrics/logs/trace; start -> backlog drained -> resolved
1 passed, 0 failed
