Complete lab 6 OpenTelemetry observability stack
This commit is contained in:
@@ -0,0 +1,8 @@
|
||||
# Observability runbook
|
||||
|
||||
SLO: 99% successful API requests per rolling 30 minutes and 95% of accepted attachments reaching `ready` within 60 seconds. `ApiHighErrorRate` detects a two-minute burn; `AttachmentQueueLag` detects a sustained queue-age breach.
|
||||
|
||||
1. Open the Grafana dashboard and compare API p95 with queue age.
|
||||
2. Follow the trace from the API request into the worker span using the trace id in structured logs.
|
||||
3. Check Prometheus `messenger_http_requests_total`, `messenger_http_request_duration_seconds_bucket`, and worker/job metrics.
|
||||
4. If the backend is unavailable, keep API/worker running; restore the backend and verify alerts resolve.
|
||||
Reference in New Issue
Block a user