Complete lab 6 OpenTelemetry observability stack

This commit is contained in:
2026-11-20 21:09:24 +03:00
parent ac2d74b74a
commit 8c59734bd2
13 changed files with 188 additions and 25 deletions
+8
View File
@@ -0,0 +1,8 @@
# Observability runbook
SLO: 99% successful API requests per rolling 30 minutes and 95% of accepted attachments reaching `ready` within 60 seconds. `ApiHighErrorRate` detects a two-minute burn; `AttachmentQueueLag` detects a sustained queue-age breach.
1. Open the Grafana dashboard and compare API p95 with queue age.
2. Follow the trace from the API request into the worker span using the trace id in structured logs.
3. Check Prometheus `messenger_http_requests_total`, `messenger_http_request_duration_seconds_bucket`, and worker/job metrics.
4. If the backend is unavailable, keep API/worker running; restore the backend and verify alerts resolve.