# 2026-11-14T16:37:30Z python3 student-tests/incident_db_slow.py --output evidence/incident-db-slow.json
[  16.03s] api recreated with FAULT_INJECTION=1 (slow-handler only by header) 
[  23.35s] reference: slow handler trace {"trace_id": "c54c8f2d29db3b326c2730d9ff66d84a", "server_ms": 1515.951, "handler_compute_ms": 1505.253, "db_spans_total_ms": 7.3}
[  23.35s] alerts before experiment {"states": {}}
[  63.37s] baseline {"db_p95_s": 0.004413773502610202, "requests": 1218, "ok": 1218, "codes": {"200": 609, "201": 609}, "error_ratio_5xx": 0.0, "ok_rps": 35.1, "p50_ms": 10.9, "p95_ms": 23.0, "p99_ms": 36.7}
[  63.37s] FAULT: LOCK TABLE sessions IN ACCESS EXCLUSIVE MODE held by a separate session 
[  83.55s] pg_stat_activity during incident {"active_by_wait": [["Lock", "relation", 4], ["Timeout", "PgSleep", 1]]}
[ 158.86s] alert MessengerHighErrorRate{source=api} -> pending 
[ 158.86s] alert MessengerHighErrorRate{source=lb} -> pending 
[ 173.92s] alert MessengerDbSlow -> pending 
[ 219.08s] alert MessengerHighErrorRate{source=api} -> firing 
[ 219.08s] alert MessengerHighErrorRate{source=lb} -> firing 
[ 234.13s] alert MessengerDbSlow -> firing 
[ 239.17s] alerts firing {"detect_s": {"MessengerHighErrorRate{source=api}": 155.7, "MessengerHighErrorRate{source=lb}": 155.7, "MessengerDbSlow": 170.8}, "api_5xx_ratio": 1.0, "db_p95_s": 5.475, "db_p95_by_op_s": {"get_session_user": 5.475}, "pool_used": 11.0, "pool_pending": 0.0, "error_types_rps": {"db_unavailable": 0.8}, "lb_up_servers": 2.0}
[ 259.18s] client view during lock {"requests": 160, "ok": 8, "codes": {"200": 4, "201": 4, "503": 152}, "error_ratio_5xx": 0.95, "ok_rps": 0.0, "p50_ms": 11.3, "p95_ms": 11.8, "p99_ms": 11.8}
[ 259.35s] RECOVERY: lock holder terminated {"terminated": 1, "lock_s": 195.8}
[ 294.45s] alert MessengerDbSlow resolved {"after_release_s": 35.3}
[ 309.58s] alert MessengerHighErrorRate{source=api} resolved {"after_release_s": 50.4}
[ 309.58s] alert MessengerHighErrorRate{source=lb} resolved {"after_release_s": 50.4}
[ 324.58s] after recovery {"requests": 352, "ok": 352, "codes": {"200": 176, "201": 176}, "error_ratio_5xx": 0.0, "ok_rps": 35.8, "p50_ms": 10.9, "p95_ms": 24.3, "p99_ms": 46.4}
[ 324.85s] error trace during lock {"trace_id": "186875d9b7250b947e79e9e2e2d7da4c", "server_ms": 5006.021, "slowest_db_span": "db get_session_user", "slowest_db_ms": 5004.691, "db_share": 1.0, "error_type": ["db_unavailable"]}
[ 324.89s] API logs event=db.unavailable (Loki) {"count": 152, "sample_trace_id": "4003216644d91fa911f914f164ba06e6", "sample_error_type": "db_unavailable", "sample": "database unavailable request_id=0009c50ab423433eb2ad84d3acba99ec (QueryCanceled)"}
[ 335.18s] api restored with FAULT_INJECTION=0 
PASS: DB lock detected {'MessengerHighErrorRate{source=api}': 155.7, 'MessengerHighErrorRate{source=lb}': 155.7, 'MessengerDbSlow': 170.8}; resolved after release {'MessengerDbSlow': 35.3, 'MessengerHighErrorRate{source=api}': 50.4, 'MessengerHighErrorRate{source=lb}': 50.4}
exit=0
