15 KiB
15 KiB
driver.py --port 8080 trace-chain (change job-trace-propagation, задачи 12.1 и 13.11; change slo-alerts, задача 11.4)
- Основа: commit
15ea156(feat: add SLOs, Prometheus alert rules and telemetry loss accounting), рабочее дерево чистое (внеevidence/изменений нет). Файлыsrc/иmigrations/(151) в контейнерахapi(2),relay,workerиcleanerпобайтно совпадают с деревом commit (SHA-256 каждого файла); ревизия схемыe4b8c2d17f90; Grafana запущена после последней правки dashboard JSON (рестарт вtelemetry-outage, 23:26Z). Прежние прогоны этого файла — на72cfa9aи на72e2598(до changeslo-alerts: dashboard без новых строк, SDK без внутренних метрик). - Дата: 2026-11-13, 23:27:25Z – 23:30:16Z (~2,9 мин), сразу после
telemetry-outage; рабочий стендmtusi-lab6(две реплики API, relay, worker, уборщик, Collector, Prometheus, Loki, Tempo, Grafana), команда из корня репозитория:python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 trace-chain, код выхода 0. - Всё — через HTTP API Grafana на
127.0.0.1:3000с паролем из.env(источники-прокси uidprometheus,loki,tempo); PostgreSQL — только чтение, кроме одногоUPDATEконтекста своего синтетического вложения (шаг 4). Пользователь, пароль, токен и файлы — синтетические, в вывод не попадают; в выводе толькоinstance_idконтейнеров. - Сценарии
observability: «Очередь при остановленном worker» (шаг 1), «Цепочка одной загрузки» и «Повторная попытка в той же трассе» (шаг 2), «Replay в той же трассе» (шаг 3), «Задание без контекста» (шаг 4), «Один источник очереди», «Пулы процессов стенда», «Пределы как в compose», «Без сокета Docker», «Панели уровня 4 после трафика», «Время до готовности в метрике» (шаг 5), «Проверка цепочки» (команда целиком);service-logging— «Записи задания ведут к трассе» (записи worker по F несутtrace_idT записи «Uploaded attachment» запроса R). - С change
slo-alerts: «Новые панели после трафика» — обход панелей с id от 14 теперь включает и строки «SLO and alerts», «Load balancer», «Telemetry pipeline» (id от 25): 50 запросов к Prometheus (было 19) без ошибок и с данными (level 4 panel queries (50)), макрос$__rangeподменён на1h(design changeslo-alerts, «Уточнения apply», п. 5). Цепочка, пулы, пределы и время доready— как на72e2598: внутренние метрики SDK и новый порядок сборки провайдеров трассы не меняют. - С правками verify change
job-trace-propagation(«Уточнения apply», п. 19, 22, 23): предусловие — глубина очереди 0 с ожиданием до 120 с (здесь дождалась сразу, сразу послеtelemetry-outage; не дождалась бы — код 2 «stand is not quiet»); CPU каждого контейнераapi,worker,relayвырос за прогон (increase(container_cpu_time_seconds_total[156s])> 0). - Шаги 2 и 3: Tempo 3.1 собрал span producer и consumer разных попыток (повтор после
pause s3, replay) в одной трассе загрузки. Время доready: worker перезапускался дважды; с changeslo-alertsсчёт по-прежнему складывает «тихие» значения двух процессов: перезапуск через ~30 с быстрее срока рядов экспортёра (1 мин), это ограничение design changeslo-alerts, решение 6 (полный перезапуск дольше 2 мин проверяетalerts-check, шаг 1). - Расхождений с design нет.
Ход прогона:
queue depth 0 waited: 0 s
queue depth before reaches 0 within 120 s (no unfinished jobs): yes
docker compose stop worker
upload queue0: 202
upload queue1: 202
upload queue2: 202
outbox + published with worker stopped: 3
oldest age not less than time since first upload - 35 s: yes
oldest age grows: yes
docker start worker
all queued attachments ready after worker start: yes
queue depth and age 0 after ready: yes
waiting for the time-to-ready histogram to settle
docker compose stop worker
upload retry: 202
job of F published with worker stopped: yes
docker compose pause s3
docker start worker
docker compose unpause s3
worker stdout: Retrying job of F (storage_unavailable): storage_unavailable
F becomes ready after S3 is back: ready
Loki: Uploaded attachment of R has trace_id T: yes
worker records of F carry trace_id T: yes
root: server span of the upload: POST /api/v1/chats/{chat_id}/attachments messenger-api
producer spans: 2
consumer spans: 2
producers: messenger-relay, children of the root: yes
producer service.instance.id = relay instance_id: b90816dcb1cd
producer attempts: [1, 2]
retry reason of the second producer: storage_unavailable
consumers: messenger-worker, parent and link = own producer: yes
consumer outcomes and attempts: [('retry', 1), ('ready', 2)]
retry consumer reason: storage_unavailable
retry consumer: S3 span with error: yes
ready consumer: S3.PutObject >= 4, render, COMMIT: yes
undecodable JPEG generated: yes
upload replay: 202
undecodable JPEG becomes failed: failed
replay --apply exit code: 0
replayed job failed again: yes
producer span with messenger.job.replayed: 1
consumer span of the replayed attempt (failed): failed
two consumer spans failed in the upload trace: 2
docker compose stop relay
upload plain: 202
context of one job removed (UPDATE rows): 1
docker start relay
job without context becomes ready: ready
worker trace_id differs from the upload trace: yes
its root is the consumer span: SPAN_KIND_CONSUMER messenger-worker
producer spans in the worker trace: 0
producer and consumer spans in the upload trace: 0
pool max: API replicas, worker, relay: ['10', '10'] ['2'] ['1']
pools with used above max: 0
worker capacity: 2
queue series only from relay: ['messenger-relay']
api 224ebf0e32ba limits = docker inspect: 1 536870912
api 224ebf0e32ba memory below the limit: yes
api 224ebf0e32ba CPU grows during the run (increase over 156 s > 0): yes
api 4b9782f7240b limits = docker inspect: 1 536870912
api 4b9782f7240b memory below the limit: yes
api 4b9782f7240b CPU grows during the run (increase over 156 s > 0): yes
worker dd039bfb3324 limits = docker inspect: 1 1073741824
worker dd039bfb3324 memory below the limit: yes
worker dd039bfb3324 CPU grows during the run (increase over 156 s > 0): yes
relay b90816dcb1cd limits = docker inspect: 0.25 268435456
relay b90816dcb1cd memory below the limit: yes
relay b90816dcb1cd CPU grows during the run (increase over 156 s > 0): yes
Docker socket mounted by any service: no
dashboard loaded: 200
level 4 panel queries (50) without error: []
level 4 panel queries with data: []
waiting for the time-to-ready histogram to settle
time to ready count grew by the uploads without replay: 5
time to ready sum grew by at least 3 x 32 s (worker stopped): yes
Итог:
queue depth before reaches 0 within 120 s (no unfinished jobs) expected yes got yes
upload queue0 expected 202 got 202
upload queue1 expected 202 got 202
upload queue2 expected 202 got 202
outbox + published with worker stopped expected 3 got 3
oldest age not less than time since first upload - 35 s expected yes got yes
oldest age grows expected yes got yes
all queued attachments ready after worker start expected yes got yes
queue depth and age 0 after ready expected yes got yes
upload retry expected 202 got 202
job of F published with worker stopped expected yes got yes
worker stdout: Retrying job of F (storage_unavailable) expected storage_unavailable got storage_unavailable
F becomes ready after S3 is back expected ready got ready
Loki: Uploaded attachment of R has trace_id T expected yes got yes
worker records of F carry trace_id T expected yes got yes
root: server span of the upload expected POST /api/v1/chats/{chat_id}/attachments messenger-api got POST /api/v1/chats/{chat_id}/attachments messenger-api
producer spans expected 2 got 2
consumer spans expected 2 got 2
producers: messenger-relay, children of the root expected yes got yes
producer service.instance.id = relay instance_id expected b90816dcb1cd got b90816dcb1cd
producer attempts expected [1, 2] got [1, 2]
retry reason of the second producer expected storage_unavailable got storage_unavailable
consumers: messenger-worker, parent and link = own producer expected yes got yes
consumer outcomes and attempts expected [('retry', 1), ('ready', 2)] got [('retry', 1), ('ready', 2)]
retry consumer reason expected storage_unavailable got storage_unavailable
retry consumer: S3 span with error expected yes got yes
ready consumer: S3.PutObject >= 4, render, COMMIT expected yes got yes
undecodable JPEG generated expected yes got yes
upload replay expected 202 got 202
undecodable JPEG becomes failed expected failed got failed
replay --apply exit code expected 0 got 0
replayed job failed again expected yes got yes
producer span with messenger.job.replayed expected 1 got 1
consumer span of the replayed attempt (failed) expected failed got failed
two consumer spans failed in the upload trace expected 2 got 2
upload plain expected 202 got 202
context of one job removed (UPDATE rows) expected 1 got 1
job without context becomes ready expected ready got ready
worker trace_id differs from the upload trace expected yes got yes
its root is the consumer span expected SPAN_KIND_CONSUMER messenger-worker got SPAN_KIND_CONSUMER messenger-worker
producer spans in the worker trace expected 0 got 0
producer and consumer spans in the upload trace expected 0 got 0
pool max: API replicas, worker, relay expected ['10', '10'] ['2'] ['1'] got ['10', '10'] ['2'] ['1']
pools with used above max expected 0 got 0
worker capacity expected 2 got 2
queue series only from relay expected ['messenger-relay'] got ['messenger-relay']
api 224ebf0e32ba limits = docker inspect expected 1 536870912 got 1 536870912
api 224ebf0e32ba memory below the limit expected yes got yes
api 224ebf0e32ba CPU grows during the run (increase over 156 s > 0) expected yes got yes
api 4b9782f7240b limits = docker inspect expected 1 536870912 got 1 536870912
api 4b9782f7240b memory below the limit expected yes got yes
api 4b9782f7240b CPU grows during the run (increase over 156 s > 0) expected yes got yes
worker dd039bfb3324 limits = docker inspect expected 1 1073741824 got 1 1073741824
worker dd039bfb3324 memory below the limit expected yes got yes
worker dd039bfb3324 CPU grows during the run (increase over 156 s > 0) expected yes got yes
relay b90816dcb1cd limits = docker inspect expected 0.25 268435456 got 0.25 268435456
relay b90816dcb1cd memory below the limit expected yes got yes
relay b90816dcb1cd CPU grows during the run (increase over 156 s > 0) expected yes got yes
Docker socket mounted by any service expected no got no
dashboard loaded expected 200 got 200
level 4 panel queries (50) without error expected [] got []
level 4 panel queries with data expected [] got []
time to ready count grew by the uploads without replay expected 5 got 5
time to ready sum grew by at least 3 x 32 s (worker stopped) expected yes got yes
PASS: one upload is one trace: producer and consumer per attempt, retry and replay in it, level 4 signals found
restored: s3 unpaused, worker and relay running: yes
Код выхода — 0.