17:32:13 background traffic: 10 req/s open-loop (75% POST message, 25% history, 2% X-Fault: slow)
17:32:43 === A. worker stopped (queue lag)
17:32:44 worker stopped at 17:32:44
17:32:44 3 uploads accepted (202) while worker is down: ['62859e73', '99403faf', '4ad2b534']
17:33:54   MessengerQueueLagGrowing pending   (+71 s)
17:34:54   MessengerQueueLagGrowing firing (+131 s)
17:34:54 diagnosis from telemetry:
17:34:54   metrics: oldest job age=126 s, depth={'processing': 0.0, 'queued': 3.0}, worker jobs/s over 1m=0.000
17:34:54   logs: worker job_* events after stop: 0; last worker events: ['worker_stopped', 'worker_stopped']
17:34:54   trace of upload 62859e73: span kinds ['PRODUCER', 'SERVER'] -> published, no CONSUMER: the job waits in the queue
17:34:57 worker started (+131 s)
17:34:58 backlog drained: all 3 ready 3.6 s after worker start
17:35:08   MessengerQueueLagGrowing resolved (+14 s)
17:35:08 === B. DB lock on messages (slow/unavailable DB for writes and history reads)
17:35:08 ACCESS EXCLUSIVE lock taken on table messages at 17:35:08
17:35:53   MessengerApiHighErrorRate pending   (+45 s)
17:36:53   MessengerApiHighErrorRate firing (+105 s)
17:36:53 diagnosis from telemetry:
17:36:53   metrics: 5xx ratio=70.7%, p95=6.92 s, pool used(max replica)=10 waiting=19
17:36:53     5xx/s /api/v1/chats                            500 0.00
17:36:53     5xx/s /api/v1/chats/{chat_id}/messages         503 9.59
17:36:53   trace 23982edb8bd1246afc6bd6522faaa524 (failed request):
      api      SERVER    POST /api/v1/chats/{chat_id}/messages          5004.2 ms ERROR {"messenger.request_id": "0f42d95e28624ecb9e443dc88c491177", "messenger.instance_id": "api-1", "http.response.status_code": "503"}
        api      CLIENT    SELECT                                            0.1 ms UNSET {"db.system": "postgresql"}
        api      CLIENT    SELECT chat_members                               0.1 ms UNSET {"db.system": "postgresql"}
        api      CLIENT    SELECT idempotency_keys                           0.1 ms UNSET {"db.system": "postgresql"}
        api      CLIENT    INSERT messages                                5001.0 ms ERROR {"db.system": "postgresql"}
17:36:53   => DB span INSERT messages took 5001 ms of 5004 ms request, status ERROR, exception QueryCanceled: the time is spent in PostgreSQL
17:36:53   contrast: slow HANDLER trace 47ee6988aff06d8a0f5592a97fb9a967: request 615 ms, longest DB span 1.1 ms, all DB 8.3 ms
17:36:53   logs: 29 API warning/error records in 5 min, events=['not_ready', 'outbox_relay_paused']
17:36:53   operator step (pg_stat_activity) finds the blocker: 4246|active|Timeout|BEGIN; LOCK TABLE messages IN ACCESS EXCLUSIVE MODE; SELECT
17:36:53 lock released (+106 s)
17:39:09   MessengerApiHighErrorRate resolved (+135 s)
17:39:09 === C1. Tempo and Loki stopped (telemetry backends down)
17:39:22   + 14 s collector mem=190.5MiB cpu=0.05% queue tempo=6 loki=0 (capacity 100 batches)
17:39:35   + 26 s collector mem=190.5MiB cpu=0.07% queue tempo=6 loki=0 (capacity 100 batches)
17:39:47   + 38 s collector mem=190.5MiB cpu=0.04% queue tempo=14 loki=0 (capacity 100 batches)
17:39:59   + 50 s collector mem=191.5MiB cpu=0.58% queue tempo=21 loki=0 (capacity 100 batches)
17:40:10   + 62 s collector mem=190.8MiB cpu=0.31% queue tempo=29 loki=0 (capacity 100 batches)
17:40:23   + 74 s collector mem=190.7MiB cpu=0.02% queue tempo=36 loki=0 (capacity 100 batches)
17:40:35   + 86 s collector mem=190.3MiB cpu=0.13% queue tempo=36 loki=0 (capacity 100 batches)
17:40:47   + 98 s collector mem=190.3MiB cpu=0.01% queue tempo=44 loki=0 (capacity 100 batches)
17:40:59   +110 s collector mem=191.7MiB cpu=0.02% queue tempo=51 loki=0 (capacity 100 batches)
17:41:10   +122 s collector mem=192.7MiB cpu=0.37% queue tempo=57 loki=0 (capacity 100 batches)
17:41:23   +134 s collector mem=191.9MiB cpu=0.10% queue tempo=64 loki=0 (capacity 100 batches)
17:41:35   +146 s collector mem=192.4MiB cpu=0.38% queue tempo=64 loki=0 (capacity 100 batches)
17:41:47   +158 s collector mem=192.4MiB cpu=0.22% queue tempo=72 loki=0 (capacity 100 batches)
17:41:59   +170 s collector mem=192MiB cpu=0.03% queue tempo=79 loki=0 (capacity 100 batches)
17:41:59   alert TelemetryExportFailing during outage: firing
17:41:59   API during the outage: 1703 requests, unexpected errors=0; alert TelemetryExportFailing=firing
17:42:20   tempo/loki started and ready (+192 s)
17:42:30   new traces after restart visible: 20 (export resumed)
17:42:56   TelemetryExportFailing resolved (+35 s)
17:43:16   telemetry lost and counted: enqueue_failed(queue full)=0 items, send_failed(retries exhausted)=387 items
17:43:16 === C2. Collector stopped (SDK side buffering)
17:44:30   api-1 RSS before=108.6MiB during=['108.8MiB', '109.3MiB', '109.3MiB', '109.1MiB', '109.3MiB', '119.3MiB']; 740 requests, unexpected errors=0
17:44:30 summary written to /Users/alenchha/mtusi2026lab6/evidence/incidents.json
PASS: incidents A,B,C diagnosed from telemetry, alerts fired and resolved
