forked from mtusi2026/mtusi2026lab6
9 lines
799 B
Markdown
9 lines
799 B
Markdown
# Telemetry backend unavailable
|
|
|
|
The API and worker export OTLP on a background queue (256 spans or log records, 5 second timeout). The collector limits itself to 192 MiB, batches, and keeps a sending queue of 200 items with retry capped at 30 seconds. When Tempo, Loki, or Prometheus is down, exports fail and `messenger_telemetry_dropped_total` increases. Message acceptance does not wait on those exports.
|
|
|
|
1. Stop one backend, for example `docker compose --env-file .env stop tempo`.
|
|
2. Send a few API requests. `/health/ready` stays 200 and messages still return 201.
|
|
3. Watch collector memory. It should stay near its limit, not grow without bound.
|
|
4. Start the backend again. New spans appear. Spans dropped during the outage are not replayed from the in-memory queue once it has overflowed.
|