Files
mtusi2026lab6/docs/runbooks/api-error-budget.md
T

19 lines
1.2 KiB
Markdown

# API error budget
SLO: 99% of business HTTP responses are not 5xx, measured on routes other than `/health/*`.
4xx is a client outcome and is outside the numerator. The ratio is `5xx / all business requests`.
Windows used on this lab stand:
- fast burn: 1 minute and 5 minutes, both above 14.4% (14.4 times a 1% budget), `for` 30s, and at least 0.2 requests/s
- slow burn: 5 minutes and 15 minutes, both above 6%, `for` 1 minute
- simple page: 5 minute ratio above 5% for 1 minute
A short spike can move the 1 minute window without moving the 5 minute window enough, so the fast burn stays inactive. A sustained burst moves both.
1. Open Prometheus `ALERTS` or the Messenger dashboard panel "Prometheus alert state".
2. If `ApiErrorBudgetFastBurn` or `ApiHighErrorRate` is firing, open a trace for a 5xx request.
3. If the `db.transaction` span is the long or erroring child, follow `docs/runbooks/db-latency.md`.
4. If the server span is long and the database span is short, the handler itself is slow (`handler_slow_ms` fault or application code).
5. Remove the fault file and wait for the next evaluations. The alert leaves `firing` when the expression is false; `for` does not delay resolution.