forked from mtusi2026/mtusi2026lab6
1.2 KiB
1.2 KiB
API error budget
SLO: 99% of business HTTP responses are not 5xx, measured on routes other than /health/*.
4xx is a client outcome and is outside the numerator. The ratio is 5xx / all business requests.
Windows used on this lab stand:
- fast burn: 1 minute and 5 minutes, both above 14.4% (14.4 times a 1% budget),
for30s, and at least 0.2 requests/s - slow burn: 5 minutes and 15 minutes, both above 6%,
for1 minute - simple page: 5 minute ratio above 5% for 1 minute
A short spike can move the 1 minute window without moving the 5 minute window enough, so the fast burn stays inactive. A sustained burst moves both.
- Open Prometheus
ALERTSor the Messenger dashboard panel "Prometheus alert state". - If
ApiErrorBudgetFastBurnorApiHighErrorRateis firing, open a trace for a 5xx request. - If the
db.transactionspan is the long or erroring child, followdocs/runbooks/db-latency.md. - If the server span is long and the database span is short, the handler itself is slow (
handler_slow_msfault or application code). - Remove the fault file and wait for the next evaluations. The alert leaves
firingwhen the expression is false;fordoes not delay resolution.