Собрать наблюдаемый стенд, чтобы причину деградации API и worker было видно по метрикам, логам и трассам.

This commit is contained in:
2026-11-07 01:36:19 +03:00
parent 470d9d6f1e
commit 7d4e1237a3
56 changed files with 4419 additions and 53 deletions
+14
View File
@@ -0,0 +1,14 @@
# Queue lag and worker loss
`messenger_queue_oldest_age_seconds` is the age of the oldest attachment still `queued` or `processing`. It is not the duration of a completed job. `messenger_queue_depth` is how many such rows exist. Both are read from PostgreSQL by the API, so they keep moving when the worker process is stopped.
`messenger_worker_consumers` is the RabbitMQ consumer count. The API polls the management API, so the series becomes 0 instead of going stale.
Alert `QueueLagHigh`: max age above 30 seconds for 1 minute.
Alert `WorkerConsumersZero`: max consumers equals 0 for 1 minute.
1. Confirm consumers are 0 or age is climbing on the Messenger dashboard.
2. Check worker logs in Loki for `job.failed` or `broker.reconnect`. A clean stop has neither new `job.consumed` nor a database error.
3. Start the worker again: `docker compose --env-file .env start worker`.
4. Age returns to 0 after the backlog is acknowledged. Alerts resolve on the next evaluation.
5. Do not use p95 time-to-ready alone: it only observes jobs that reached `ready`.