forked from mtusi2026/mtusi2026lab6
799 B
799 B
Telemetry backend unavailable
The API and worker export OTLP on a background queue (256 spans or log records, 5 second timeout). The collector limits itself to 192 MiB, batches, and keeps a sending queue of 200 items with retry capped at 30 seconds. When Tempo, Loki, or Prometheus is down, exports fail and messenger_telemetry_dropped_total increases. Message acceptance does not wait on those exports.
- Stop one backend, for example
docker compose --env-file .env stop tempo. - Send a few API requests.
/health/readystays 200 and messages still return 201. - Watch collector memory. It should stay near its limit, not grow without bound.
- Start the backend again. New spans appear. Spans dropped during the outage are not replayed from the in-memory queue once it has overflowed.