Files
mtusi2026lab7/docs/runbooks/telemetry-backend.md
T
vika baafe2a3ae Добавить стенд лабораторной 7 с mTLS, OpenBao и least privilege worker.
Закрывает обязательный минимум и трек 1: ротация leaf и секрета, sealed OpenBao и сохранность queued-задачи.
2026-11-07 20:46:02 +03:00

799 B

Telemetry backend unavailable

The API and worker export OTLP on a background queue (256 spans or log records, 5 second timeout). The collector limits itself to 192 MiB, batches, and keeps a sending queue of 200 items with retry capped at 30 seconds. When Tempo, Loki, or Prometheus is down, exports fail and messenger_telemetry_dropped_total increases. Message acceptance does not wait on those exports.

  1. Stop one backend, for example docker compose --env-file .env stop tempo.
  2. Send a few API requests. /health/ready stays 200 and messages still return 201.
  3. Watch collector memory. It should stay near its limit, not grow without bound.
  4. Start the backend again. New spans appear. Spans dropped during the outage are not replayed from the in-memory queue once it has overflowed.