# incident-db: lab6 incident 2: slow PostgreSQL (table lock) -> high error rate firing, cause from traces -> resolved
# date: 2026-11-01 00:47:15, commit: unknown
# command: python student-tests/incidents.py db-slow
# exit code: 0, 265.4 s

=== db-slow
[   0.1s] load: 8 threads POST /messages
[  25.1s] baseline: {'requests': 2648, 'codes': {'201': 2648}, 'success_ratio': 1.0, 'success_rps': 132.4, 'p95_ms_successful': 18.5, 'server_p95_s': 0.0214}
[  25.1s] LOCK TABLE messages IN ACCESS EXCLUSIVE MODE for 210 s (PostgreSQL alive, writes wait)
[  25.1s]   5xx ratio=0.0 p95=0.021 s pool used=0.0 waiting=0.0
[  30.1s]   5xx ratio=0.0 p95=0.022 s pool used=8.0 waiting=0.0
[  35.1s]   5xx ratio=0.0 p95=0.022 s pool used=8.0 waiting=0.0
[  40.1s]   5xx ratio=0.0 p95=0.022 s pool used=8.0 waiting=0.0
[  45.1s]   5xx ratio=0.005 p95=0.022 s pool used=8.0 waiting=0.0
[  50.1s]   5xx ratio=0.005 p95=0.023 s pool used=8.0 waiting=0.0
[  55.1s]   5xx ratio=0.005 p95=0.023 s pool used=8.0 waiting=0.0
[  60.1s]   5xx ratio=0.01 p95=0.024 s pool used=8.0 waiting=0.0
[  65.1s]   5xx ratio=0.01 p95=0.024 s pool used=8.0 waiting=0.0
[  70.1s]   5xx ratio=0.01 p95=5.688 s pool used=8.0 waiting=0.0
[  75.2s]   5xx ratio=0.069 p95=5.688 s pool used=8.0 waiting=0.0
[  80.2s]   5xx ratio=0.069 p95=7.375 s pool used=8.0 waiting=0.0
[  85.2s]   5xx ratio=0.069 p95=7.375 s pool used=8.0 waiting=0.0
[  90.2s]   5xx ratio=1.0 p95=7.375 s pool used=8.0 waiting=0.0
[  95.2s]   5xx ratio=1.0 p95=7.375 s pool used=8.0 waiting=0.0
[ 100.2s]   5xx ratio=1.0 p95=7.375 s pool used=8.0 waiting=0.0
[ 105.2s]   5xx ratio=1.0 p95=7.375 s pool used=8.0 waiting=0.0
[ 110.2s]   5xx ratio=1.0 p95=7.375 s pool used=8.0 waiting=0.0
[ 115.2s]   5xx ratio=1.0 p95=7.375 s pool used=8.0 waiting=0.0
[ 120.2s]   5xx ratio=1.0 p95=7.375 s pool used=8.0 waiting=0.0
[ 125.2s]   5xx ratio=1.0 p95=7.375 s pool used=8.0 waiting=0.0
[ 130.2s]   5xx ratio=1.0 p95=7.375 s pool used=8.0 waiting=0.0
[ 135.2s]   5xx ratio=1.0 p95=7.375 s pool used=8.0 waiting=0.0
[ 140.2s]   5xx ratio=1.0 p95=7.375 s pool used=8.0 waiting=0.0
[ 145.2s] ALERT MessengerHighErrorRate firing (120.1 s after lock)
[ 151.5s] diagnosis: {"http_5xx_ratio_1m": 1.0, "http_p95_messages_s": 7.375, "pool_used": 8.0, "pool_wait_p95_s": 0.001, "status_codes_1m": {"201": 0.0, "200": 0.0, "404": 0.0, "400": 0.0, "409": 0.0, "503": 1.6}, "reads_during_lock_ms": {"GET /users/me + GET /chats, avg per request": 4.7, "max": 5.4}, "api_log_db_unavailable_by_error_type": {"QueryCanceled": 184}}
[ 151.5s] error trace (DB): {"trace_id": "1c7f8dd06427574a5f155558aa4cd975", "server_ms": 5006.7, "slowest_db_span": "INSERT", "slowest_db_ms": 5000.6, "db_span_status": "ERROR", "db_share": 0.999}
      - [api] POST /api/v1/chats/{chat_id}/messages (SERVER, 5006.7 ms) ERROR http.response.status_code=503 error.type=503
        - [api] redis GET (CLIENT, 0.2 ms)
        - [api] SELECT (CLIENT, 0.4 ms)
        - [api] INSERT (CLIENT, 5000.6 ms) ERROR
[ 151.5s] contrast, slow handler (CPU): {"trace_id": "a6da2e3fe7b0c229773f36fa507293ca", "server_ms": 34.0, "argon2id_ms": 29.0, "db_ms_total": 0.6}
      - [api] POST /api/v1/users (SERVER, 34.0 ms) http.response.status_code=201
        - [api] argon2id hash (INTERNAL, 29.0 ms)
        - [api] INSERT (CLIENT, 0.6 ms)
[ 235.4s] lock released (psql exit 0), 210.3 s after LOCK
[ 235.4s]   ... MessengerHighErrorRate: firing (0 s)
[ 240.4s]   ... MessengerHighErrorRate: firing (5 s)
[ 245.4s]   ... MessengerHighErrorRate: firing (10 s)
[ 250.4s]   ... MessengerHighErrorRate: firing (15 s)
[ 255.4s]   ... MessengerHighErrorRate: firing (20 s)
[ 260.4s]   ... MessengerHighErrorRate: firing (25 s)
[ 265.4s] ALERT MessengerHighErrorRate resolved 30.0 s after unlock
[ 265.4s] load: during lock {'requests': 326, 'codes': {'503': 320, '201': 6}, 'success_ratio': 0.0184, 'success_rps': 0.0, 'p95_ms_successful': 2718.4}; after {'requests': 3298, 'codes': {'201': 3298}, 'success_ratio': 1.0, 'success_rps': 131.9, 'p95_ms_successful': 21.0}
[ 265.4s] saved /home/claude/work/mtusi2026lab6/evidence/native-sandbox/incident-db-slow.json
PASS: slow DB -> 503 -> MessengerHighErrorRate firing; traces show the time in the INSERT span (not the handler); unlock -> resolved
1 passed, 0 failed
