forked from mtusi2026/mtusi2026lab6
9 lines
655 B
Markdown
9 lines
655 B
Markdown
# Observability runbook
|
|
|
|
SLO: 99% successful API requests per rolling 30 minutes and 95% of accepted attachments reaching `ready` within 60 seconds. `ApiHighErrorRate` detects a two-minute burn; `AttachmentQueueLag` detects a sustained queue-age breach.
|
|
|
|
1. Open the Grafana dashboard and compare API p95 with queue age.
|
|
2. Follow the trace from the API request into the worker span using the trace id in structured logs.
|
|
3. Check Prometheus `messenger_http_requests_total`, `messenger_http_request_duration_seconds_bucket`, and worker/job metrics.
|
|
4. If the backend is unavailable, keep API/worker running; restore the backend and verify alerts resolve.
|