forked from mtusi2026/mtusi2026lab6
Собрать наблюдаемый стенд, чтобы причину деградации API и worker было видно по метрикам, логам и трассам.
This commit is contained in:
@@ -0,0 +1,18 @@
|
||||
# API error budget
|
||||
|
||||
SLO: 99% of business HTTP responses are not 5xx, measured on routes other than `/health/*`.
|
||||
4xx is a client outcome and is outside the numerator. The ratio is `5xx / all business requests`.
|
||||
|
||||
Windows used on this lab stand:
|
||||
|
||||
- fast burn: 1 minute and 5 minutes, both above 14.4% (14.4 times a 1% budget), `for` 30s, and at least 0.2 requests/s
|
||||
- slow burn: 5 minutes and 15 minutes, both above 6%, `for` 1 minute
|
||||
- simple page: 5 minute ratio above 5% for 1 minute
|
||||
|
||||
A short spike can move the 1 minute window without moving the 5 minute window enough, so the fast burn stays inactive. A sustained burst moves both.
|
||||
|
||||
1. Open Prometheus `ALERTS` or the Messenger dashboard panel "Prometheus alert state".
|
||||
2. If `ApiErrorBudgetFastBurn` or `ApiHighErrorRate` is firing, open a trace for a 5xx request.
|
||||
3. If the `db.transaction` span is the long or erroring child, follow `docs/runbooks/db-latency.md`.
|
||||
4. If the server span is long and the database span is short, the handler itself is slow (`handler_slow_ms` fault or application code).
|
||||
5. Remove the fault file and wait for the next evaluations. The alert leaves `firing` when the expression is false; `for` does not delay resolution.
|
||||
@@ -0,0 +1,15 @@
|
||||
# Slow or unavailable database
|
||||
|
||||
`messenger_db_up` is 0 when the process cannot run `SELECT 1`. Business calls then return 503 `unavailable`. That increases the API error ratio. Health `/health/live` stays 200. `/health/ready` becomes 503.
|
||||
|
||||
A slow database is different from a slow handler:
|
||||
|
||||
- slow database: the `db.transaction` span is long and the server span is about the same length
|
||||
- slow handler: `handler_slow` sits in the server span, and `db.transaction` stays short
|
||||
|
||||
`DatabaseUnready` fires when `min(messenger_db_up) == 0` for 30 seconds.
|
||||
|
||||
1. Compare one server span with its `db.transaction` child in Tempo.
|
||||
2. If the database span failed, check PostgreSQL. Grafana being down does not cause this alert.
|
||||
3. Restore PostgreSQL. `messenger_db_up` returns to 1 and the error-budget alerts resolve after the windows slide.
|
||||
4. The collector and Grafana are not on the readiness path.
|
||||
@@ -0,0 +1,14 @@
|
||||
# Queue lag and worker loss
|
||||
|
||||
`messenger_queue_oldest_age_seconds` is the age of the oldest attachment still `queued` or `processing`. It is not the duration of a completed job. `messenger_queue_depth` is how many such rows exist. Both are read from PostgreSQL by the API, so they keep moving when the worker process is stopped.
|
||||
|
||||
`messenger_worker_consumers` is the RabbitMQ consumer count. The API polls the management API, so the series becomes 0 instead of going stale.
|
||||
|
||||
Alert `QueueLagHigh`: max age above 30 seconds for 1 minute.
|
||||
Alert `WorkerConsumersZero`: max consumers equals 0 for 1 minute.
|
||||
|
||||
1. Confirm consumers are 0 or age is climbing on the Messenger dashboard.
|
||||
2. Check worker logs in Loki for `job.failed` or `broker.reconnect`. A clean stop has neither new `job.consumed` nor a database error.
|
||||
3. Start the worker again: `docker compose --env-file .env start worker`.
|
||||
4. Age returns to 0 after the backlog is acknowledged. Alerts resolve on the next evaluation.
|
||||
5. Do not use p95 time-to-ready alone: it only observes jobs that reached `ready`.
|
||||
@@ -0,0 +1,8 @@
|
||||
# Telemetry backend unavailable
|
||||
|
||||
The API and worker export OTLP on a background queue (256 spans or log records, 5 second timeout). The collector limits itself to 192 MiB, batches, and keeps a sending queue of 200 items with retry capped at 30 seconds. When Tempo, Loki, or Prometheus is down, exports fail and `messenger_telemetry_dropped_total` increases. Message acceptance does not wait on those exports.
|
||||
|
||||
1. Stop one backend, for example `docker compose --env-file .env stop tempo`.
|
||||
2. Send a few API requests. `/health/ready` stays 200 and messages still return 201.
|
||||
3. Watch collector memory. It should stay near its limit, not grow without bound.
|
||||
4. Start the backend again. New spans appear. Spans dropped during the outage are not replayed from the in-memory queue once it has overflowed.
|
||||
@@ -0,0 +1,11 @@
|
||||
# Time to ready
|
||||
|
||||
SLO: 95% of attachments that reach `ready` do so within 10 seconds of durable acceptance.
|
||||
The series is `messenger_attachment_time_to_ready_seconds`, a histogram in seconds.
|
||||
`AttachmentReadyLatencyHigh` uses `histogram_quantile(0.95, ...)` over 15 minutes and pages after 2 minutes above 10 seconds.
|
||||
|
||||
This SLO does not see jobs that never finish. Pair it with queue age.
|
||||
|
||||
1. If p95 is high and queue age is low, open a worker trace and compare `s3.put`, `s3.get`, and `db.transaction`.
|
||||
2. A retry span with `attempt > 1` and event `retry` means the first attempt was transient. The successful attempt should still contain S3 and the ready update.
|
||||
3. If p95 is quiet but users wait, read queue age. The worker is likely stopped or stuck before completion.
|
||||
Reference in New Issue
Block a user