forked from mtusi2026/mtusi2026lab6
15 lines
1.0 KiB
Markdown
15 lines
1.0 KiB
Markdown
# Queue lag and worker loss
|
|
|
|
`messenger_queue_oldest_age_seconds` is the age of the oldest attachment still `queued` or `processing`. It is not the duration of a completed job. `messenger_queue_depth` is how many such rows exist. Both are read from PostgreSQL by the API, so they keep moving when the worker process is stopped.
|
|
|
|
`messenger_worker_consumers` is the RabbitMQ consumer count. The API polls the management API, so the series becomes 0 instead of going stale.
|
|
|
|
Alert `QueueLagHigh`: max age above 30 seconds for 1 minute.
|
|
Alert `WorkerConsumersZero`: max consumers equals 0 for 1 minute.
|
|
|
|
1. Confirm consumers are 0 or age is climbing on the Messenger dashboard.
|
|
2. Check worker logs in Loki for `job.failed` or `broker.reconnect`. A clean stop has neither new `job.consumed` nor a database error.
|
|
3. Start the worker again: `docker compose --env-file .env start worker`.
|
|
4. Age returns to 0 after the backlog is acknowledged. Alerts resolve on the next evaluation.
|
|
5. Do not use p95 time-to-ready alone: it only observes jobs that reached `ready`.
|