# incident-worker: lab6 incident 1: worker stopped -> queue lag + no consumers firing -> resolved
# date: 2026-11-01 20:12:40, commit: 470d9d6 + uncommitted changes
# command: python.exe student-tests/incidents.py worker-stop
# exit code: 0, 161.8 s

=== worker-stop
[   0.1s] baseline: consumers=2.0, oldest job age=0.0 s
[   1.2s] docker compose stop worker
[   1.3s] uploaded 3 attachments: 202 queued (API accepts without worker)
[   1.4s]   consumers=2.0 backlog queued=0.0 oldest age=0.0 s, alerts: {}
[   6.4s]   consumers=2.0 backlog queued=0.0 oldest age=0.0 s, alerts: {}
[  11.4s]   consumers=0.0 backlog queued=3.0 oldest age=7.1 s, alerts: {}
[  16.5s]   consumers=0.0 backlog queued=3.0 oldest age=7.1 s, alerts: {}
[  21.5s]   consumers=0.0 backlog queued=3.0 oldest age=17.1 s, alerts: {}
[  26.5s]   consumers=0.0 backlog queued=3.0 oldest age=17.1 s, alerts: {}
[  31.5s]   consumers=0.0 backlog queued=3.0 oldest age=27.1 s, alerts: {}
[  36.5s]   consumers=0.0 backlog queued=3.0 oldest age=27.1 s, alerts: {}
[  41.6s]   consumers=0.0 backlog queued=3.0 oldest age=37.1 s, alerts: {'MessengerWorkerNoConsumers': 'pending'}
[  46.6s]   consumers=0.0 backlog queued=3.0 oldest age=37.1 s, alerts: {'MessengerWorkerNoConsumers': 'pending'}
[  51.6s]   consumers=0.0 backlog queued=3.0 oldest age=47.1 s, alerts: {'MessengerWorkerNoConsumers': 'pending'}
[  56.6s]   consumers=0.0 backlog queued=3.0 oldest age=47.1 s, alerts: {'MessengerWorkerNoConsumers': 'pending'}
[  61.6s]   consumers=0.0 backlog queued=3.0 oldest age=57.1 s, alerts: {'MessengerWorkerNoConsumers': 'pending'}
[  66.6s]   consumers=0.0 backlog queued=3.0 oldest age=57.1 s, alerts: {'MessengerWorkerNoConsumers': 'pending'}
[  71.7s]   consumers=0.0 backlog queued=3.0 oldest age=67.2 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'pending'}
[  76.7s]   consumers=0.0 backlog queued=3.0 oldest age=67.2 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'pending'}
[  81.7s]   consumers=0.0 backlog queued=3.0 oldest age=77.2 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'pending'}
[  86.7s]   consumers=0.0 backlog queued=3.0 oldest age=77.2 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'pending'}
[  91.8s]   consumers=0.0 backlog queued=3.0 oldest age=87.2 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'pending'}
[  96.8s]   consumers=0.0 backlog queued=3.0 oldest age=87.2 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'pending'}
[ 101.8s] ALERT MessengerWorkerNoConsumers firing (101.7 s after stop)
[ 101.8s]   consumers=0.0 backlog queued=3.0 oldest age=97.2 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'firing'}
[ 106.8s]   consumers=0.0 backlog queued=3.0 oldest age=97.2 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'firing'}
[ 111.8s]   consumers=0.0 backlog queued=3.0 oldest age=107.2 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'firing'}
[ 116.8s]   consumers=0.0 backlog queued=3.0 oldest age=107.2 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'firing'}
[ 121.8s]   consumers=0.0 backlog queued=3.0 oldest age=117.3 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'firing'}
[ 126.9s]   consumers=0.0 backlog queued=3.0 oldest age=117.3 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'firing'}
[ 131.9s] ALERT MessengerQueueLagGrowing firing (131.8 s after stop)
[ 132.0s] diagnosis: {"queue_consumers": 0.0, "broker_queue_depth": 6.0, "jobs_backlog_queued": 3.0, "oldest_job_age_s": 127.3, "outbox_oldest_age_s": 0.0, "published_ok_last_5m": 16.41239671427261, "worker_metrics_last_seen_s": 131.0, "last_worker_log": {"event": "service.stopped", "message": "stopped"}, "upload_trace_span_names": ["INSERT", "NOTIFY", "POST /api/v1/chats/{chat_id}/attachments", "S3 PutObject", "SELECT", "UPDATE", "create attachments", "inspect image", "publish attachments", "redis GET", "republish lost message"], "upload_trace_has_consumer_span": false}
[ 135.3s] docker compose start worker
[ 136.6s] all attachments ready 4.7 s after worker start; backlog now 3.0
[ 136.6s]   ... MessengerWorkerNoConsumers: firing (0 s)
[ 141.6s]   ... MessengerWorkerNoConsumers: firing (5 s)
[ 146.6s]   ... MessengerWorkerNoConsumers: firing (10 s)
[ 151.7s]   ... MessengerWorkerNoConsumers: firing (15 s)
[ 156.7s]   ... MessengerWorkerNoConsumers: firing (20 s)
[ 161.7s] alerts resolved: {'MessengerWorkerNoConsumers': 29.7, 'MessengerQueueLagGrowing': 29.7}; now {'jobs_backlog_queued': 0.0, 'oldest_job_age_s': 0.0, 'queue_consumers': 2.0}
[ 161.7s] saved C:\Users\Gigabyte\Desktop\lab6\mtusi2026lab6\evidence\incident-worker-stop.json
PASS: worker stop -> no consumers + growing queue lag -> firing; cause from metrics/logs/trace; start -> backlog drained -> resolved
1 passed, 0 failed
