19 lines
1.2 KiB
Markdown
19 lines
1.2 KiB
Markdown
# API error budget
|
|
|
|
SLO: 99% of business HTTP responses are not 5xx, measured on routes other than `/health/*`.
|
|
4xx is a client outcome and is outside the numerator. The ratio is `5xx / all business requests`.
|
|
|
|
Windows used on this lab stand:
|
|
|
|
- fast burn: 1 minute and 5 minutes, both above 14.4% (14.4 times a 1% budget), `for` 30s, and at least 0.2 requests/s
|
|
- slow burn: 5 minutes and 15 minutes, both above 6%, `for` 1 minute
|
|
- simple page: 5 minute ratio above 5% for 1 minute
|
|
|
|
A short spike can move the 1 minute window without moving the 5 minute window enough, so the fast burn stays inactive. A sustained burst moves both.
|
|
|
|
1. Open Prometheus `ALERTS` or the Messenger dashboard panel "Prometheus alert state".
|
|
2. If `ApiErrorBudgetFastBurn` or `ApiHighErrorRate` is firing, open a trace for a 5xx request.
|
|
3. If the `db.transaction` span is the long or erroring child, follow `docs/runbooks/db-latency.md`.
|
|
4. If the server span is long and the database span is short, the handler itself is slow (`handler_slow_ms` fault or application code).
|
|
5. Remove the fault file and wait for the next evaluations. The alert leaves `firing` when the expression is false; `for` does not delay resolution.
|