Files
mtusi2026lab6/docs/runbooks/queue-lag.md
T

1.0 KiB

Queue lag and worker loss

messenger_queue_oldest_age_seconds is the age of the oldest attachment still queued or processing. It is not the duration of a completed job. messenger_queue_depth is how many such rows exist. Both are read from PostgreSQL by the API, so they keep moving when the worker process is stopped.

messenger_worker_consumers is the RabbitMQ consumer count. The API polls the management API, so the series becomes 0 instead of going stale.

Alert QueueLagHigh: max age above 30 seconds for 1 minute. Alert WorkerConsumersZero: max consumers equals 0 for 1 minute.

  1. Confirm consumers are 0 or age is climbing on the Messenger dashboard.
  2. Check worker logs in Loki for job.failed or broker.reconnect. A clean stop has neither new job.consumed nor a database error.
  3. Start the worker again: docker compose --env-file .env start worker.
  4. Age returns to 0 after the backlog is acknowledged. Alerts resolve on the next evaluation.
  5. Do not use p95 time-to-ready alone: it only observes jobs that reached ready.