Собрать наблюдаемый стенд, чтобы причину деградации API и worker было видно по метрикам, логам и трассам.

This commit is contained in:
2026-11-07 01:36:19 +03:00
parent 470d9d6f1e
commit 7d4e1237a3
56 changed files with 4419 additions and 53 deletions
+18
View File
@@ -0,0 +1,18 @@
# API error budget
SLO: 99% of business HTTP responses are not 5xx, measured on routes other than `/health/*`.
4xx is a client outcome and is outside the numerator. The ratio is `5xx / all business requests`.
Windows used on this lab stand:
- fast burn: 1 minute and 5 minutes, both above 14.4% (14.4 times a 1% budget), `for` 30s, and at least 0.2 requests/s
- slow burn: 5 minutes and 15 minutes, both above 6%, `for` 1 minute
- simple page: 5 minute ratio above 5% for 1 minute
A short spike can move the 1 minute window without moving the 5 minute window enough, so the fast burn stays inactive. A sustained burst moves both.
1. Open Prometheus `ALERTS` or the Messenger dashboard panel "Prometheus alert state".
2. If `ApiErrorBudgetFastBurn` or `ApiHighErrorRate` is firing, open a trace for a 5xx request.
3. If the `db.transaction` span is the long or erroring child, follow `docs/runbooks/db-latency.md`.
4. If the server span is long and the database span is short, the handler itself is slow (`handler_slow_ms` fault or application code).
5. Remove the fault file and wait for the next evaluations. The alert leaves `firing` when the expression is false; `for` does not delay resolution.
+15
View File
@@ -0,0 +1,15 @@
# Slow or unavailable database
`messenger_db_up` is 0 when the process cannot run `SELECT 1`. Business calls then return 503 `unavailable`. That increases the API error ratio. Health `/health/live` stays 200. `/health/ready` becomes 503.
A slow database is different from a slow handler:
- slow database: the `db.transaction` span is long and the server span is about the same length
- slow handler: `handler_slow` sits in the server span, and `db.transaction` stays short
`DatabaseUnready` fires when `min(messenger_db_up) == 0` for 30 seconds.
1. Compare one server span with its `db.transaction` child in Tempo.
2. If the database span failed, check PostgreSQL. Grafana being down does not cause this alert.
3. Restore PostgreSQL. `messenger_db_up` returns to 1 and the error-budget alerts resolve after the windows slide.
4. The collector and Grafana are not on the readiness path.
+14
View File
@@ -0,0 +1,14 @@
# Queue lag and worker loss
`messenger_queue_oldest_age_seconds` is the age of the oldest attachment still `queued` or `processing`. It is not the duration of a completed job. `messenger_queue_depth` is how many such rows exist. Both are read from PostgreSQL by the API, so they keep moving when the worker process is stopped.
`messenger_worker_consumers` is the RabbitMQ consumer count. The API polls the management API, so the series becomes 0 instead of going stale.
Alert `QueueLagHigh`: max age above 30 seconds for 1 minute.
Alert `WorkerConsumersZero`: max consumers equals 0 for 1 minute.
1. Confirm consumers are 0 or age is climbing on the Messenger dashboard.
2. Check worker logs in Loki for `job.failed` or `broker.reconnect`. A clean stop has neither new `job.consumed` nor a database error.
3. Start the worker again: `docker compose --env-file .env start worker`.
4. Age returns to 0 after the backlog is acknowledged. Alerts resolve on the next evaluation.
5. Do not use p95 time-to-ready alone: it only observes jobs that reached `ready`.
+8
View File
@@ -0,0 +1,8 @@
# Telemetry backend unavailable
The API and worker export OTLP on a background queue (256 spans or log records, 5 second timeout). The collector limits itself to 192 MiB, batches, and keeps a sending queue of 200 items with retry capped at 30 seconds. When Tempo, Loki, or Prometheus is down, exports fail and `messenger_telemetry_dropped_total` increases. Message acceptance does not wait on those exports.
1. Stop one backend, for example `docker compose --env-file .env stop tempo`.
2. Send a few API requests. `/health/ready` stays 200 and messages still return 201.
3. Watch collector memory. It should stay near its limit, not grow without bound.
4. Start the backend again. New spans appear. Spans dropped during the outage are not replayed from the in-memory queue once it has overflowed.
+11
View File
@@ -0,0 +1,11 @@
# Time to ready
SLO: 95% of attachments that reach `ready` do so within 10 seconds of durable acceptance.
The series is `messenger_attachment_time_to_ready_seconds`, a histogram in seconds.
`AttachmentReadyLatencyHigh` uses `histogram_quantile(0.95, ...)` over 15 minutes and pages after 2 minutes above 10 seconds.
This SLO does not see jobs that never finish. Pair it with queue age.
1. If p95 is high and queue age is low, open a worker trace and compare `s3.put`, `s3.get`, and `db.transaction`.
2. A retry span with `attempt > 1` and event `retry` means the first attempt was transient. The successful attempt should still contain S3 and the ready update.
3. If p95 is quiet but users wait, read queue age. The worker is likely stopped or stuck before completion.