Files
mtusi2026lab6/docs/runbooks/api-error-budget.md
T

1.2 KiB

API error budget

SLO: 99% of business HTTP responses are not 5xx, measured on routes other than /health/*. 4xx is a client outcome and is outside the numerator. The ratio is 5xx / all business requests.

Windows used on this lab stand:

  • fast burn: 1 minute and 5 minutes, both above 14.4% (14.4 times a 1% budget), for 30s, and at least 0.2 requests/s
  • slow burn: 5 minutes and 15 minutes, both above 6%, for 1 minute
  • simple page: 5 minute ratio above 5% for 1 minute

A short spike can move the 1 minute window without moving the 5 minute window enough, so the fast burn stays inactive. A sustained burst moves both.

  1. Open Prometheus ALERTS or the Messenger dashboard panel "Prometheus alert state".
  2. If ApiErrorBudgetFastBurn or ApiHighErrorRate is firing, open a trace for a 5xx request.
  3. If the db.transaction span is the long or erroring child, follow docs/runbooks/db-latency.md.
  4. If the server span is long and the database span is short, the handler itself is slow (handler_slow_ms fault or application code).
  5. Remove the fault file and wait for the next evaluations. The alert leaves firing when the expression is false; for does not delay resolution.