# incident-worker: lab6 incident 1: worker stopped -> queue lag + no consumers firing -> resolved
# date: 2026-11-07 19:17:54, commit: 20afc2c + uncommitted changes
# command: python.exe student-tests/incidents.py worker-stop
# exit code: 0, 170.7 s

=== worker-stop
[   0.2s] baseline: consumers=2.0, oldest job age=0.0 s
[   2.6s] docker compose stop worker
[   2.7s] uploaded 3 attachments: 202 queued (API accepts without worker)
[   2.8s]   consumers=2.0 backlog queued=0.0 oldest age=0.0 s, alerts: {}
[   7.9s]   consumers=2.0 backlog queued=0.0 oldest age=0.0 s, alerts: {}
[  13.0s]   consumers=0.0 backlog queued=3.0 oldest age=5.1 s, alerts: {}
[  18.1s]   consumers=0.0 backlog queued=3.0 oldest age=5.1 s, alerts: {}
[  23.1s]   consumers=0.0 backlog queued=3.0 oldest age=15.1 s, alerts: {}
[  28.2s]   consumers=0.0 backlog queued=3.0 oldest age=15.1 s, alerts: {}
[  33.3s]   consumers=0.0 backlog queued=3.0 oldest age=25.2 s, alerts: {}
[  38.4s]   consumers=0.0 backlog queued=3.0 oldest age=25.2 s, alerts: {'MessengerWorkerNoConsumers': 'pending'}
[  43.5s]   consumers=0.0 backlog queued=3.0 oldest age=35.2 s, alerts: {'MessengerWorkerNoConsumers': 'pending'}
[  48.5s]   consumers=0.0 backlog queued=3.0 oldest age=35.2 s, alerts: {'MessengerWorkerNoConsumers': 'pending'}
[  53.7s]   consumers=0.0 backlog queued=3.0 oldest age=45.2 s, alerts: {'MessengerWorkerNoConsumers': 'pending'}
[  58.7s]   consumers=0.0 backlog queued=3.0 oldest age=45.2 s, alerts: {'MessengerWorkerNoConsumers': 'pending'}
[  63.8s]   consumers=0.0 backlog queued=3.0 oldest age=55.2 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'pending'}
[  68.8s]   consumers=0.0 backlog queued=3.0 oldest age=55.2 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'pending'}
[  73.9s]   consumers=0.0 backlog queued=3.0 oldest age=65.2 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'pending'}
[  79.0s]   consumers=0.0 backlog queued=3.0 oldest age=65.2 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'pending'}
[  84.1s]   consumers=0.0 backlog queued=3.0 oldest age=75.3 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'pending'}
[  89.2s]   consumers=0.0 backlog queued=3.0 oldest age=75.3 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'pending'}
[  94.2s] ALERT MessengerWorkerNoConsumers firing (94.0 s after stop)
[  94.2s]   consumers=0.0 backlog queued=3.0 oldest age=85.3 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'firing'}
[  99.3s]   consumers=0.0 backlog queued=3.0 oldest age=85.3 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'firing'}
[ 104.4s]   consumers=0.0 backlog queued=3.0 oldest age=95.3 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'firing'}
[ 109.5s]   consumers=0.0 backlog queued=3.0 oldest age=95.3 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'firing'}
[ 114.6s]   consumers=0.0 backlog queued=3.0 oldest age=105.4 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'firing'}
[ 119.7s]   consumers=0.0 backlog queued=3.0 oldest age=105.4 s, alerts: {'MessengerQueueLagGrowing': 'pending', 'MessengerWorkerNoConsumers': 'firing'}
[ 124.7s] ALERT MessengerQueueLagGrowing firing (124.5 s after stop)
[ 124.8s] diagnosis: {"queue_consumers": 0.0, "broker_queue_depth": 3.0, "jobs_backlog_queued": 3.0, "oldest_job_age_s": 115.4, "outbox_oldest_age_s": 0.0, "published_ok_last_5m": 12.39609660691289, "worker_metrics_last_seen_s": 123.0, "last_worker_log": {"event": "service.stopped", "message": "stopped"}, "upload_trace_span_names": ["INSERT", "NOTIFY", "POST /api/v1/chats/{chat_id}/attachments", "S3 PutObject", "SELECT", "UPDATE", "create attachments", "inspect image", "publish attachments", "redis GET"], "upload_trace_has_consumer_span": false}
[ 128.9s] docker compose start worker
[ 130.3s] all attachments ready 5.4 s after worker start; backlog now 3.0
[ 130.3s]   ... MessengerWorkerNoConsumers: firing (0 s)
[ 135.3s]   ... MessengerWorkerNoConsumers: firing (5 s)
[ 140.3s]   ... MessengerWorkerNoConsumers: firing (10 s)
[ 145.4s]   ... MessengerWorkerNoConsumers: firing (15 s)
[ 150.4s]   ... MessengerWorkerNoConsumers: firing (20 s)
[ 155.4s]   ... MessengerWorkerNoConsumers: firing (25 s)
[ 160.4s]   ... MessengerWorkerNoConsumers: firing (30 s)
[ 165.5s]   ... MessengerWorkerNoConsumers: firing (35 s)
[ 170.5s] alerts resolved: {'MessengerWorkerNoConsumers': 45.6, 'MessengerQueueLagGrowing': 45.6}; now {'jobs_backlog_queued': 0.0, 'oldest_job_age_s': 0.0, 'queue_consumers': 2.0}
[ 170.6s] saved C:\Users\Gigabyte\Desktop\lab7\mtusi2026lab7\evidence\incident-worker-stop.json
PASS: worker stop -> no consumers + growing queue lag -> firing; cause from metrics/logs/trace; start -> backlog drained -> resolved
1 passed, 0 failed
