# incident-db: lab6 incident 2: slow PostgreSQL (table lock) -> high error rate firing, cause from traces -> resolved
# date: 2026-11-07 19:22:25, commit: 20afc2c + uncommitted changes
# command: python.exe student-tests/incidents.py db-slow
# exit code: 0, 271.0 s

=== db-slow
[   0.1s] load: 8 threads POST /messages
[  25.2s] baseline: {'requests': 2042, 'codes': {'201': 2042}, 'success_ratio': 1.0, 'success_rps': 102.1, 'p95_ms_successful': 32.0, 'server_p95_s': 0.0236}
[  25.2s] LOCK TABLE messages IN ACCESS EXCLUSIVE MODE for 210 s (PostgreSQL alive, writes wait)
[  25.3s]   5xx ratio=0.0 p95=0.024 s pool used=0.0 waiting=0.0
[  30.4s]   5xx ratio=0.0 p95=0.024 s pool used=0.0 waiting=0.0
[  35.5s]   5xx ratio=0.0 p95=0.024 s pool used=8.0 waiting=0.0
[  40.6s]   5xx ratio=0.0 p95=0.024 s pool used=8.0 waiting=0.0
[  45.7s]   5xx ratio=0.0 p95=0.024 s pool used=8.0 waiting=0.0
[  50.7s]   5xx ratio=0.0 p95=0.024 s pool used=8.0 waiting=0.0
[  55.8s]   5xx ratio=0.0 p95=0.024 s pool used=8.0 waiting=0.0
[  60.9s]   5xx ratio=0.008 p95=0.024 s pool used=8.0 waiting=0.0
[  66.0s]   5xx ratio=0.008 p95=0.025 s pool used=8.0 waiting=0.0
[  71.1s]   5xx ratio=0.008 p95=0.025 s pool used=8.0 waiting=0.0
[  76.3s]   5xx ratio=0.016 p95=6.67 s pool used=8.0 waiting=0.0
[  81.4s]   5xx ratio=0.016 p95=6.737 s pool used=8.0 waiting=0.0
[  86.5s]   5xx ratio=0.016 p95=7.375 s pool used=8.0 waiting=0.0
[  91.6s]   5xx ratio=0.14 p95=7.375 s pool used=8.0 waiting=0.0
[  96.7s]   5xx ratio=0.14 p95=7.375 s pool used=8.0 waiting=0.0
[ 101.8s]   5xx ratio=0.14 p95=7.375 s pool used=8.0 waiting=0.0
[ 106.9s]   5xx ratio=1.0 p95=7.375 s pool used=8.0 waiting=0.0
[ 112.0s]   5xx ratio=1.0 p95=7.375 s pool used=8.0 waiting=0.0
[ 117.1s]   5xx ratio=1.0 p95=7.375 s pool used=8.0 waiting=0.0
[ 122.2s]   5xx ratio=1.0 p95=7.375 s pool used=8.0 waiting=0.0
[ 127.4s]   5xx ratio=1.0 p95=7.375 s pool used=8.0 waiting=0.0
[ 132.5s]   5xx ratio=1.0 p95=7.375 s pool used=8.0 waiting=0.0
[ 137.6s]   5xx ratio=1.0 p95=7.375 s pool used=2.0 waiting=0.0
[ 142.7s]   5xx ratio=1.0 p95=7.375 s pool used=2.0 waiting=0.0
[ 147.7s] ALERT MessengerHighErrorRate firing (122.5 s after lock)
[ 152.2s] diagnosis: {"http_5xx_ratio_1m": 1.0, "http_p95_messages_s": 7.375, "pool_used": 8.0, "pool_wait_p95_s": 0.001, "status_codes_1m": {"201": 0.0, "200": 0.0, "400": 0.0, "409": 0.0, "404": 0.0, "503": 1.44}, "reads_during_lock_ms": {"GET /users/me + GET /chats, avg per request": 15.6, "max": 31.5}, "api_log_db_unavailable_by_error_type": {"QueryCanceled": 184}}
[ 152.2s] error trace (DB): {"trace_id": "50418f127c336af7f2e4dc0190adf7ea", "server_ms": 5010.0, "slowest_db_span": "INSERT", "slowest_db_ms": 5001.6, "db_span_status": "ERROR", "db_share": 0.998}
      - [api] POST /api/v1/chats/{chat_id}/messages (SERVER, 5010.0 ms) ERROR http.response.status_code=503 error.type=503
        - [api] redis GET (CLIENT, 1.5 ms)
        - [api] SELECT (CLIENT, 0.6 ms)
        - [api] INSERT (CLIENT, 5001.6 ms) ERROR
[ 152.2s] contrast, slow handler (CPU): {"trace_id": "2e774b9d5912ba768f3e9881f3c41e80", "server_ms": 47.0, "argon2id_ms": 32.2, "db_ms_total": 6.9}
      - [api] POST /api/v1/users (SERVER, 47.0 ms) http.response.status_code=201
        - [api] redis INCR (CLIENT, 6.2 ms)
        - [api] argon2id hash (INTERNAL, 32.2 ms)
        - [api] INSERT (CLIENT, 0.7 ms)
[ 235.7s] lock released (psql exit 0), 210.5 s after LOCK
[ 235.7s]   ... MessengerHighErrorRate: firing (0 s)
[ 240.7s]   ... MessengerHighErrorRate: firing (5 s)
[ 245.8s]   ... MessengerHighErrorRate: firing (10 s)
[ 250.8s]   ... MessengerHighErrorRate: firing (15 s)
[ 255.8s]   ... MessengerHighErrorRate: firing (20 s)
[ 260.8s]   ... MessengerHighErrorRate: firing (25 s)
[ 265.8s]   ... MessengerHighErrorRate: firing (30 s)
[ 270.8s] ALERT MessengerHighErrorRate resolved 35.1 s after unlock
[ 270.8s] load: during lock {'requests': 328, 'codes': {'503': 320, '201': 8}, 'success_ratio': 0.0244, 'success_rps': 0.0, 'p95_ms_successful': 2031.0}; after {'requests': 3086, 'codes': {'201': 3086}, 'success_ratio': 1.0, 'success_rps': 102.4, 'p95_ms_successful': 32.0}
[ 270.9s] saved C:\Users\Gigabyte\Desktop\lab7\mtusi2026lab7\evidence\incident-db-slow.json
PASS: slow DB -> 503 -> MessengerHighErrorRate firing; traces show the time in the INSERT span (not the handler); unlock -> resolved
1 passed, 0 failed
