# incident-db: lab6 incident 2: slow PostgreSQL (table lock) -> high error rate firing, cause from traces -> resolved
# date: 2026-11-01 20:18:39, commit: 470d9d6 + uncommitted changes
# command: python.exe student-tests/incidents.py db-slow
# exit code: 0, 285.7 s

=== db-slow
[   0.0s] load: 8 threads POST /messages
[  25.0s] baseline: {'requests': 2310, 'codes': {'201': 2310}, 'success_ratio': 1.0, 'success_rps': 115.5, 'p95_ms_successful': 32.0, 'server_p95_s': 0.019}
[  25.0s] LOCK TABLE messages IN ACCESS EXCLUSIVE MODE for 210 s (PostgreSQL alive, writes wait)
[  25.1s]   5xx ratio=None p95=0.019 s pool used=0.0 waiting=0.0
[  30.1s]   5xx ratio=0.0 p95=0.02 s pool used=1.0 waiting=0.0
[  35.1s]   5xx ratio=0.0 p95=0.02 s pool used=1.0 waiting=0.0
[  40.1s]   5xx ratio=0.0 p95=0.02 s pool used=8.0 waiting=0.0
[  45.2s]   5xx ratio=0.0 p95=0.02 s pool used=8.0 waiting=0.0
[  50.2s]   5xx ratio=0.0 p95=0.021 s pool used=8.0 waiting=0.0
[  55.3s]   5xx ratio=0.0 p95=0.021 s pool used=8.0 waiting=0.0
[  60.3s]   5xx ratio=0.0 p95=0.022 s pool used=8.0 waiting=0.0
[  65.3s]   5xx ratio=0.0 p95=0.024 s pool used=8.0 waiting=0.0
[  70.4s]   5xx ratio=0.0 p95=0.045 s pool used=8.0 waiting=0.0
[  75.4s]   5xx ratio=0.015 p95=7.18 s pool used=8.0 waiting=0.0
[  80.4s]   5xx ratio=0.015 p95=7.234 s pool used=8.0 waiting=0.0
[  85.4s]   5xx ratio=0.015 p95=7.375 s pool used=8.0 waiting=0.0
[  90.4s]   5xx ratio=0.044 p95=7.375 s pool used=8.0 waiting=0.0
[  95.5s]   5xx ratio=0.044 p95=7.375 s pool used=8.0 waiting=0.0
[ 100.5s]   5xx ratio=0.044 p95=7.375 s pool used=8.0 waiting=0.0
[ 105.5s]   5xx ratio=1.0 p95=7.375 s pool used=8.0 waiting=0.0
[ 110.6s]   5xx ratio=1.0 p95=7.375 s pool used=8.0 waiting=0.0
[ 115.6s]   5xx ratio=1.0 p95=7.375 s pool used=8.0 waiting=0.0
[ 120.7s]   5xx ratio=1.0 p95=7.375 s pool used=8.0 waiting=0.0
[ 125.7s]   5xx ratio=1.0 p95=7.375 s pool used=8.0 waiting=0.0
[ 130.7s]   5xx ratio=1.0 p95=7.375 s pool used=8.0 waiting=0.0
[ 135.7s]   5xx ratio=1.0 p95=7.375 s pool used=8.0 waiting=0.0
[ 140.8s]   5xx ratio=1.0 p95=7.375 s pool used=8.0 waiting=0.0
[ 145.8s]   5xx ratio=1.0 p95=7.375 s pool used=8.0 waiting=0.0
[ 150.8s]   5xx ratio=1.0 p95=7.375 s pool used=8.0 waiting=0.0
[ 155.8s]   5xx ratio=1.0 p95=7.375 s pool used=8.0 waiting=0.0
[ 160.8s]   5xx ratio=1.0 p95=7.375 s pool used=8.0 waiting=0.0
[ 165.8s] ALERT MessengerHighErrorRate firing (140.8 s after lock)
[ 172.2s] diagnosis: {"http_5xx_ratio_1m": 1.0, "http_p95_messages_s": 7.375, "pool_used": 8.0, "pool_wait_p95_s": 0.001, "status_codes_1m": {"201": 0.0, "200": 0.0, "404": 0.0, "400": 0.0, "409": 0.0, "503": 1.6}, "reads_during_lock_ms": {"GET /users/me + GET /chats, avg per request": 17.2, "max": 23.5}, "api_log_db_unavailable_by_error_type": {"QueryCanceled": 216}}
[ 172.2s] error trace (DB): {"trace_id": "237e2d6b1c7db22adbb7d4752fa47f2", "server_ms": 5013.3, "slowest_db_span": "INSERT", "slowest_db_ms": 5001.3, "db_span_status": "ERROR", "db_share": 0.998}
      - [api] POST /api/v1/chats/{chat_id}/messages (SERVER, 5013.3 ms) ERROR http.response.status_code=503 error.type=503
        - [api] redis GET (CLIENT, 1.1 ms)
        - [api] SELECT (CLIENT, 1.6 ms)
        - [api] INSERT (CLIENT, 5001.3 ms) ERROR
[ 172.2s] contrast, slow handler (CPU): {"trace_id": "4a222e478c2d51006538a7bfeb485eb1", "server_ms": 21.2, "argon2id_ms": 16.4, "db_ms_total": 0.7}
      - [api] POST /api/v1/users (SERVER, 21.2 ms) http.response.status_code=201
        - [api] argon2id hash (INTERNAL, 16.4 ms)
        - [api] INSERT (CLIENT, 0.7 ms)
[ 235.5s] lock released (psql exit 0), 210.5 s after LOCK
[ 235.5s]   ... MessengerHighErrorRate: firing (0 s)
[ 240.5s]   ... MessengerHighErrorRate: firing (5 s)
[ 245.5s]   ... MessengerHighErrorRate: firing (10 s)
[ 250.5s]   ... MessengerHighErrorRate: firing (15 s)
[ 255.5s]   ... MessengerHighErrorRate: firing (20 s)
[ 260.6s]   ... MessengerHighErrorRate: firing (25 s)
[ 265.6s]   ... MessengerHighErrorRate: firing (30 s)
[ 270.6s]   ... MessengerHighErrorRate: firing (35 s)
[ 275.6s]   ... MessengerHighErrorRate: firing (40 s)
[ 280.6s]   ... MessengerHighErrorRate: firing (45 s)
[ 285.6s] ALERT MessengerHighErrorRate resolved 50.1 s after unlock
[ 285.6s] load: during lock {'requests': 328, 'codes': {'503': 320, '201': 8}, 'success_ratio': 0.0244, 'success_rps': 0.0, 'p95_ms_successful': 2156.0}; after {'requests': 5269, 'codes': {'201': 5269}, 'success_ratio': 1.0, 'success_rps': 116.8, 'p95_ms_successful': 31.0}
[ 285.6s] saved C:\Users\Gigabyte\Desktop\lab6\mtusi2026lab6\evidence\incident-db-slow.json
PASS: slow DB -> 503 -> MessengerHighErrorRate firing; traces show the time in the INSERT span (not the handler); unlock -> resolved
1 passed, 0 failed
