Files
mtusi2026lab6/docs/runbooks/telemetry-backend.md
T

799 B

Telemetry backend unavailable

The API and worker export OTLP on a background queue (256 spans or log records, 5 second timeout). The collector limits itself to 192 MiB, batches, and keeps a sending queue of 200 items with retry capped at 30 seconds. When Tempo, Loki, or Prometheus is down, exports fail and messenger_telemetry_dropped_total increases. Message acceptance does not wait on those exports.

  1. Stop one backend, for example docker compose --env-file .env stop tempo.
  2. Send a few API requests. /health/ready stays 200 and messages still return 201.
  3. Watch collector memory. It should stay near its limit, not grow without bound.
  4. Start the backend again. New spans appear. Spans dropped during the outage are not replayed from the in-memory queue once it has overflowed.