Files
mtusi2026lab6/evidence/telemetry/trace-chain.md
T
2026-11-14 02:36:17 +03:00

15 KiB
Raw Blame History

driver.py --port 8080 trace-chain (change job-trace-propagation, задачи 12.1 и 13.11; change slo-alerts, задача 11.4)

  • Основа: commit 15ea156 (feat: add SLOs, Prometheus alert rules and telemetry loss accounting), рабочее дерево чистое (вне evidence/ изменений нет). Файлы src/ и migrations/ (151) в контейнерах api (2), relay, worker и cleaner побайтно совпадают с деревом commit (SHA-256 каждого файла); ревизия схемы e4b8c2d17f90; Grafana запущена после последней правки dashboard JSON (рестарт в telemetry-outage, 23:26Z). Прежние прогоны этого файла — на 72cfa9a и на 72e2598 (до change slo-alerts: dashboard без новых строк, SDK без внутренних метрик).
  • Дата: 2026-11-13, 23:27:25Z – 23:30:16Z (~2,9 мин), сразу после telemetry-outage; рабочий стенд mtusi-lab6 (две реплики API, relay, worker, уборщик, Collector, Prometheus, Loki, Tempo, Grafana), команда из корня репозитория: python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 trace-chain, код выхода 0.
  • Всё — через HTTP API Grafana на 127.0.0.1:3000 с паролем из .env (источники-прокси uid prometheus, loki, tempo); PostgreSQL — только чтение, кроме одного UPDATE контекста своего синтетического вложения (шаг 4). Пользователь, пароль, токен и файлы — синтетические, в вывод не попадают; в выводе только instance_id контейнеров.
  • Сценарии observability: «Очередь при остановленном worker» (шаг 1), «Цепочка одной загрузки» и «Повторная попытка в той же трассе» (шаг 2), «Replay в той же трассе» (шаг 3), «Задание без контекста» (шаг 4), «Один источник очереди», «Пулы процессов стенда», «Пределы как в compose», «Без сокета Docker», «Панели уровня 4 после трафика», «Время до готовности в метрике» (шаг 5), «Проверка цепочки» (команда целиком); service-logging — «Записи задания ведут к трассе» (записи worker по F несут trace_id T записи «Uploaded attachment» запроса R).
  • С change slo-alerts: «Новые панели после трафика» — обход панелей с id от 14 теперь включает и строки «SLO and alerts», «Load balancer», «Telemetry pipeline» (id от 25): 50 запросов к Prometheus (было 19) без ошибок и с данными (level 4 panel queries (50)), макрос $__range подменён на 1h (design change slo-alerts, «Уточнения apply», п. 5). Цепочка, пулы, пределы и время до ready — как на 72e2598: внутренние метрики SDK и новый порядок сборки провайдеров трассы не меняют.
  • С правками verify change job-trace-propagation («Уточнения apply», п. 19, 22, 23): предусловие — глубина очереди 0 с ожиданием до 120 с (здесь дождалась сразу, сразу после telemetry-outage; не дождалась бы — код 2 «stand is not quiet»); CPU каждого контейнера api, worker, relay вырос за прогон (increase(container_cpu_time_seconds_total[156s]) > 0).
  • Шаги 2 и 3: Tempo 3.1 собрал span producer и consumer разных попыток (повтор после pause s3, replay) в одной трассе загрузки. Время до ready: worker перезапускался дважды; с change slo-alerts счёт по-прежнему складывает «тихие» значения двух процессов: перезапуск через ~30 с быстрее срока рядов экспортёра (1 мин), это ограничение design change slo-alerts, решение 6 (полный перезапуск дольше 2 мин проверяет alerts-check, шаг 1).
  • Расхождений с design нет.

Ход прогона:

queue depth 0 waited: 0 s
queue depth before reaches 0 within 120 s (no unfinished jobs): yes
docker compose stop worker
upload queue0: 202
upload queue1: 202
upload queue2: 202
outbox + published with worker stopped: 3
oldest age not less than time since first upload - 35 s: yes
oldest age grows: yes
docker start worker
all queued attachments ready after worker start: yes
queue depth and age 0 after ready: yes
waiting for the time-to-ready histogram to settle
docker compose stop worker
upload retry: 202
job of F published with worker stopped: yes
docker compose pause s3
docker start worker
docker compose unpause s3
worker stdout: Retrying job of F (storage_unavailable): storage_unavailable
F becomes ready after S3 is back: ready
Loki: Uploaded attachment of R has trace_id T: yes
worker records of F carry trace_id T: yes
root: server span of the upload: POST /api/v1/chats/{chat_id}/attachments messenger-api
producer spans: 2
consumer spans: 2
producers: messenger-relay, children of the root: yes
producer service.instance.id = relay instance_id: b90816dcb1cd
producer attempts: [1, 2]
retry reason of the second producer: storage_unavailable
consumers: messenger-worker, parent and link = own producer: yes
consumer outcomes and attempts: [('retry', 1), ('ready', 2)]
retry consumer reason: storage_unavailable
retry consumer: S3 span with error: yes
ready consumer: S3.PutObject >= 4, render, COMMIT: yes
undecodable JPEG generated: yes
upload replay: 202
undecodable JPEG becomes failed: failed
replay --apply exit code: 0
replayed job failed again: yes
producer span with messenger.job.replayed: 1
consumer span of the replayed attempt (failed): failed
two consumer spans failed in the upload trace: 2
docker compose stop relay
upload plain: 202
context of one job removed (UPDATE rows): 1
docker start relay
job without context becomes ready: ready
worker trace_id differs from the upload trace: yes
its root is the consumer span: SPAN_KIND_CONSUMER messenger-worker
producer spans in the worker trace: 0
producer and consumer spans in the upload trace: 0
pool max: API replicas, worker, relay: ['10', '10'] ['2'] ['1']
pools with used above max: 0
worker capacity: 2
queue series only from relay: ['messenger-relay']
api 224ebf0e32ba limits = docker inspect: 1 536870912
api 224ebf0e32ba memory below the limit: yes
api 224ebf0e32ba CPU grows during the run (increase over 156 s > 0): yes
api 4b9782f7240b limits = docker inspect: 1 536870912
api 4b9782f7240b memory below the limit: yes
api 4b9782f7240b CPU grows during the run (increase over 156 s > 0): yes
worker dd039bfb3324 limits = docker inspect: 1 1073741824
worker dd039bfb3324 memory below the limit: yes
worker dd039bfb3324 CPU grows during the run (increase over 156 s > 0): yes
relay b90816dcb1cd limits = docker inspect: 0.25 268435456
relay b90816dcb1cd memory below the limit: yes
relay b90816dcb1cd CPU grows during the run (increase over 156 s > 0): yes
Docker socket mounted by any service: no
dashboard loaded: 200
level 4 panel queries (50) without error: []
level 4 panel queries with data: []
waiting for the time-to-ready histogram to settle
time to ready count grew by the uploads without replay: 5
time to ready sum grew by at least 3 x 32 s (worker stopped): yes

Итог:

queue depth before reaches 0 within 120 s (no unfinished jobs)         expected yes             got yes
upload queue0                                                          expected 202             got 202
upload queue1                                                          expected 202             got 202
upload queue2                                                          expected 202             got 202
outbox + published with worker stopped                                 expected 3               got 3
oldest age not less than time since first upload - 35 s                expected yes             got yes
oldest age grows                                                       expected yes             got yes
all queued attachments ready after worker start                        expected yes             got yes
queue depth and age 0 after ready                                      expected yes             got yes
upload retry                                                           expected 202             got 202
job of F published with worker stopped                                 expected yes             got yes
worker stdout: Retrying job of F (storage_unavailable)                 expected storage_unavailable got storage_unavailable
F becomes ready after S3 is back                                       expected ready           got ready
Loki: Uploaded attachment of R has trace_id T                          expected yes             got yes
worker records of F carry trace_id T                                   expected yes             got yes
root: server span of the upload                                        expected POST /api/v1/chats/{chat_id}/attachments messenger-api got POST /api/v1/chats/{chat_id}/attachments messenger-api
producer spans                                                         expected 2               got 2
consumer spans                                                         expected 2               got 2
producers: messenger-relay, children of the root                       expected yes             got yes
producer service.instance.id = relay instance_id                       expected b90816dcb1cd    got b90816dcb1cd
producer attempts                                                      expected [1, 2]          got [1, 2]
retry reason of the second producer                                    expected storage_unavailable got storage_unavailable
consumers: messenger-worker, parent and link = own producer            expected yes             got yes
consumer outcomes and attempts                                         expected [('retry', 1), ('ready', 2)] got [('retry', 1), ('ready', 2)]
retry consumer reason                                                  expected storage_unavailable got storage_unavailable
retry consumer: S3 span with error                                     expected yes             got yes
ready consumer: S3.PutObject >= 4, render, COMMIT                      expected yes             got yes
undecodable JPEG generated                                             expected yes             got yes
upload replay                                                          expected 202             got 202
undecodable JPEG becomes failed                                        expected failed          got failed
replay --apply exit code                                               expected 0               got 0
replayed job failed again                                              expected yes             got yes
producer span with messenger.job.replayed                              expected 1               got 1
consumer span of the replayed attempt (failed)                         expected failed          got failed
two consumer spans failed in the upload trace                          expected 2               got 2
upload plain                                                           expected 202             got 202
context of one job removed (UPDATE rows)                               expected 1               got 1
job without context becomes ready                                      expected ready           got ready
worker trace_id differs from the upload trace                          expected yes             got yes
its root is the consumer span                                          expected SPAN_KIND_CONSUMER messenger-worker got SPAN_KIND_CONSUMER messenger-worker
producer spans in the worker trace                                     expected 0               got 0
producer and consumer spans in the upload trace                        expected 0               got 0
pool max: API replicas, worker, relay                                  expected ['10', '10'] ['2'] ['1'] got ['10', '10'] ['2'] ['1']
pools with used above max                                              expected 0               got 0
worker capacity                                                        expected 2               got 2
queue series only from relay                                           expected ['messenger-relay'] got ['messenger-relay']
api 224ebf0e32ba limits = docker inspect                               expected 1 536870912     got 1 536870912
api 224ebf0e32ba memory below the limit                                expected yes             got yes
api 224ebf0e32ba CPU grows during the run (increase over 156 s > 0)    expected yes             got yes
api 4b9782f7240b limits = docker inspect                               expected 1 536870912     got 1 536870912
api 4b9782f7240b memory below the limit                                expected yes             got yes
api 4b9782f7240b CPU grows during the run (increase over 156 s > 0)    expected yes             got yes
worker dd039bfb3324 limits = docker inspect                            expected 1 1073741824    got 1 1073741824
worker dd039bfb3324 memory below the limit                             expected yes             got yes
worker dd039bfb3324 CPU grows during the run (increase over 156 s > 0) expected yes             got yes
relay b90816dcb1cd limits = docker inspect                             expected 0.25 268435456  got 0.25 268435456
relay b90816dcb1cd memory below the limit                              expected yes             got yes
relay b90816dcb1cd CPU grows during the run (increase over 156 s > 0)  expected yes             got yes
Docker socket mounted by any service                                   expected no              got no
dashboard loaded                                                       expected 200             got 200
level 4 panel queries (50) without error                               expected []              got []
level 4 panel queries with data                                        expected []              got []
time to ready count grew by the uploads without replay                 expected 5               got 5
time to ready sum grew by at least 3 x 32 s (worker stopped)           expected yes             got yes
PASS: one upload is one trace: producer and consumer per attempt, retry and replay in it, level 4 signals found
restored: s3 unpaused, worker and relay running: yes

Код выхода — 0.