forked from mtusi2026/mtusi2026lab6
891 B
891 B
Slow or unavailable database
messenger_db_up is 0 when the process cannot run SELECT 1. Business calls then return 503 unavailable. That increases the API error ratio. Health /health/live stays 200. /health/ready becomes 503.
A slow database is different from a slow handler:
- slow database: the
db.transactionspan is long and the server span is about the same length - slow handler:
handler_slowsits in the server span, anddb.transactionstays short
DatabaseUnready fires when min(messenger_db_up) == 0 for 30 seconds.
- Compare one server span with its
db.transactionchild in Tempo. - If the database span failed, check PostgreSQL. Grafana being down does not cause this alert.
- Restore PostgreSQL.
messenger_db_upreturns to 1 and the error-budget alerts resolve after the windows slide. - The collector and Grafana are not on the readiness path.