forked from mtusi2026/mtusi2026lab6
16 lines
891 B
Markdown
16 lines
891 B
Markdown
# Slow or unavailable database
|
|
|
|
`messenger_db_up` is 0 when the process cannot run `SELECT 1`. Business calls then return 503 `unavailable`. That increases the API error ratio. Health `/health/live` stays 200. `/health/ready` becomes 503.
|
|
|
|
A slow database is different from a slow handler:
|
|
|
|
- slow database: the `db.transaction` span is long and the server span is about the same length
|
|
- slow handler: `handler_slow` sits in the server span, and `db.transaction` stays short
|
|
|
|
`DatabaseUnready` fires when `min(messenger_db_up) == 0` for 30 seconds.
|
|
|
|
1. Compare one server span with its `db.transaction` child in Tempo.
|
|
2. If the database span failed, check PostgreSQL. Grafana being down does not cause this alert.
|
|
3. Restore PostgreSQL. `messenger_db_up` returns to 1 and the error-budget alerts resolve after the windows slide.
|
|
4. The collector and Grafana are not on the readiness path.
|