forked from mtusi2026/mtusi2026lab6
Собрать наблюдаемый стенд, чтобы причину деградации API и worker было видно по метрикам, логам и трассам.
This commit is contained in:
@@ -0,0 +1,14 @@
|
||||
# Queue lag and worker loss
|
||||
|
||||
`messenger_queue_oldest_age_seconds` is the age of the oldest attachment still `queued` or `processing`. It is not the duration of a completed job. `messenger_queue_depth` is how many such rows exist. Both are read from PostgreSQL by the API, so they keep moving when the worker process is stopped.
|
||||
|
||||
`messenger_worker_consumers` is the RabbitMQ consumer count. The API polls the management API, so the series becomes 0 instead of going stale.
|
||||
|
||||
Alert `QueueLagHigh`: max age above 30 seconds for 1 minute.
|
||||
Alert `WorkerConsumersZero`: max consumers equals 0 for 1 minute.
|
||||
|
||||
1. Confirm consumers are 0 or age is climbing on the Messenger dashboard.
|
||||
2. Check worker logs in Loki for `job.failed` or `broker.reconnect`. A clean stop has neither new `job.consumed` nor a database error.
|
||||
3. Start the worker again: `docker compose --env-file .env start worker`.
|
||||
4. Age returns to 0 after the backlog is acknowledged. Alerts resolve on the next evaluation.
|
||||
5. Do not use p95 time-to-ready alone: it only observes jobs that reached `ready`.
|
||||
Reference in New Issue
Block a user