Files
mtusi2026lab6/docs/runbook.md

655 B

Observability runbook

SLO: 99% successful API requests per rolling 30 minutes and 95% of accepted attachments reaching ready within 60 seconds. ApiHighErrorRate detects a two-minute burn; AttachmentQueueLag detects a sustained queue-age breach.

  1. Open the Grafana dashboard and compare API p95 with queue age.
  2. Follow the trace from the API request into the worker span using the trace id in structured logs.
  3. Check Prometheus messenger_http_requests_total, messenger_http_request_duration_seconds_bucket, and worker/job metrics.
  4. If the backend is unavailable, keep API/worker running; restore the backend and verify alerts resolve.