1.0 KiB
1.0 KiB
Queue lag and worker loss
messenger_queue_oldest_age_seconds is the age of the oldest attachment still queued or processing. It is not the duration of a completed job. messenger_queue_depth is how many such rows exist. Both are read from PostgreSQL by the API, so they keep moving when the worker process is stopped.
messenger_worker_consumers is the RabbitMQ consumer count. The API polls the management API, so the series becomes 0 instead of going stale.
Alert QueueLagHigh: max age above 30 seconds for 1 minute.
Alert WorkerConsumersZero: max consumers equals 0 for 1 minute.
- Confirm consumers are 0 or age is climbing on the Messenger dashboard.
- Check worker logs in Loki for
job.failedorbroker.reconnect. A clean stop has neither newjob.consumednor a database error. - Start the worker again:
docker compose --env-file .env start worker. - Age returns to 0 after the backlog is acknowledged. Alerts resolve on the next evaluation.
- Do not use p95 time-to-ready alone: it only observes jobs that reached
ready.