docs: re-capture trace-chain and collector-outage evidence on 72e2598

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-11 21:02:20 +03:00
co-authored by Claude Opus 5.5
parent 72e2598d28
commit c1055b4d6c
5 changed files with 286 additions and 189 deletions
+5 -4
View File
@@ -32,9 +32,10 @@ relay и consumer span worker (parent и link) — одна трасса на з
(`CountingQueuePool`), насыщение worker, CPU и память контейнеров из cgroup
(без `docker.sock`); панели уровня 4 в том же dashboard; команда драйвера
`trace-chain`. Код — commit `72cfa9a`, evidence снято на нём:
`evidence/telemetry/trace-chain.md`, `check.md`, `collector-outage.md`,
`queue-snapshot.md`, раздел change в `budget.md`,
`evidence/migrations/job-trace-context.md`. Следующий — `slo-alerts` (SLO,
`evidence/telemetry/check.md`, `queue-snapshot.md`, раздел change в
`budget.md`, `evidence/migrations/job-trace-context.md`; правки verify —
commit `72e2598`, на нём пересняты `evidence/telemetry/trace-chain.md` и
`collector-outage.md`. Следующий — `slo-alerts` (SLO,
alerts, runbooks, инциденты).
Код перенесён из лабы 5 (репозиторий
`mtusi2026lab5-safronov`, ветка `solution/bvt2301-safronov`, commit
@@ -715,7 +716,7 @@ evidence/attachments/ # change async-attachments: defense.md (driver.py d
evidence/s3/ # privacy.md — make s3_check на текущем коде; change s3-storage: bootstrap.md (первый запуск, повтор, смена секрета, ошибка шага, до bootstrap), privacy.md (make s3_check, нарушение и исправление, порты, секреты по сервисам), grpc-admin.md (gRPC шлюза без токена), restart.md (restart.sh и SIGKILL S3), client.md (операции, pause/stop, неверный секрет, журнал), memory.md (20 × 10 MiB PUT); change upload-consistency: cleaner-key.md (пользователь уборщика в bootstrap, смена секрета, make s3_check с ключом уборщика, секреты по сервисам); change track2-reconciler: cleaner-list.md (bootstrap с политикой списка по префиксу, неверный префикс, make s3_check со строками списка уборщика, окружение s3-bootstrap)
evidence/reconcile/ # change track2-reconciler: cycle.md (driver.py reconcile-cycle), stress.md (driver.py reconcile-stress), service.md (Started cleaner с параметрами сверки, плановая сверка при старте, docker stop во время сверки при pause s3, код 1 разового прохода по сигналу — driver.py reconcile-signal, отказ старта на RECONCILER__*, коды 3 и 0; с change track2-reconcile-cost — остановка во время паузы предела скорости, driver.py reconcile-signal --pace)
evidence/reconcile/<метка>/ # результаты make bench_reconcile (change track2-reconcile-cost): meta.json (раздел reconcile: план, этапы и модели, commit и образы «до», аргументы S3), passes.csv, windows.csv, summary.md (H1–H4 и S4), cost-by-n.svg, requests-by-n.svg, background-p95.svg, raw/ (записи уборщика по проходам, итог k6 блока H4)
evidence/telemetry/ # change otel-telemetry: budget.md (пики и лимиты сервисов телеметрии, реплик и worker с телеметрией и без); на commit 72cfa9a change job-trace-propagation переснято: check.md (driver.py telemetry-check), collector-outage.md (driver.py telemetry-outage, с relay); новое: trace-chain.md (driver.py trace-chain), queue-snapshot.md (EXPLAIN снимка очереди), раздел change в budget.md (relay с SDK, уборщик после ленивого импорта)
evidence/telemetry/ # change otel-telemetry: budget.md (пики и лимиты сервисов телеметрии, реплик и worker с телеметрией и без); на commit 72cfa9a change job-trace-propagation переснято: check.md (driver.py telemetry-check); на 72e2598 (правки verify) — collector-outage.md (driver.py telemetry-outage, с relay, сигналы без перезапуска) и новое trace-chain.md (driver.py trace-chain); новое на 72cfa9a: queue-snapshot.md (EXPLAIN снимка очереди), раздел change в budget.md (relay с SDK, уборщик после ленивого импорта)
evidence/migrations/ # change job-trace-propagation: job-trace-context.md — ревизия e4b8c2d17f90 (временная база: с нуля, повтор, откат, CHECK, lock_timeout; --sql; рабочий стенд: откат реплик на ff0073e с MIGRATE=0, ревизия до и после make rolling_update)
evidence/stand/ # change otel-telemetry: from-scratch.md (driver.py stand-from-scratch --runs 2)
evidence/storage/ # отказы БД, пул, statement_timeout, рестарт, порты Postgres, свежий clone; connection-cost.md — цена нового соединения и пинга; freeze-sessions.md — change unit-of-work: сессии и журнал пула после заморозки PostgreSQL (driver.py freeze-sessions и freeze-lb)
+5 -3
View File
@@ -19,7 +19,9 @@ Change `otel-telemetry` (design — `openspec/changes/archive/2026-11-11-otel-te
номера решений в скобках). Числа и проверки сняты на закоммиченном коде при чистом рабочем дереве:
`telemetry-check` и `telemetry-outage` сначала на `68f1e97` (правки verify), затем пересняты на
`72cfa9a` (код change `job-trace-propagation`: relay с SDK, шаги задания по consumer span), там
же `trace-chain`, план снимка очереди, проба ревизии `e4b8c2d17f90` и раздел change бюджета;
же план снимка очереди, проба ревизии `e4b8c2d17f90` и раздел change бюджета; `trace-chain` и
`telemetry-outage` — ещё раз на `72e2598` (правки verify: сигналы relay без перезапуска, тихая
очередь в предусловии, рост CPU каждого процесса);
бюджет сервисов телеметрии — на `4178b3d` (лимиты Tempo и Grafana по замеру), шапки файлов
`evidence/telemetry/` и `evidence/migrations/`. `evidence/stand/from-
scratch.md` ещё снят на рабочем дереве поверх `9b35c25`: пересъёмка ждёт разрешения пользователя
@@ -286,10 +288,10 @@ replay, консоль, `make test_isolated`, стенд отказов и `benc
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Уровень 3: Collector, хранилища, provisioning, свой запрос и задание найдены, приватность | `driver.py --port 8080 telemetry-check` | код 0: Loki → Tempo по `X-Request-Id`, трасса worker, рост счётчиков, история после рестарта хранилищ | `evidence/telemetry/check.md` |
| Телеметрия не меняет ответы и остановку | `driver.py --port 8080 telemetry-outage` | код 0: `make test` PASS и остановка < 10 с при остановленном и замороженном Collector | `evidence/telemetry/collector-outage.md` |
| Телеметрия не меняет ответы и остановку | `driver.py --port 8080 telemetry-outage` | код 0: `make test` PASS и остановка < 10 с при остановленном и замороженном Collector, сигналы после его возврата без перезапуска процессов | `evidence/telemetry/collector-outage.md` (commit `72e2598`) |
| Стенд с нуля одной командой | `driver.py stand-from-scratch --runs 2` | код 0 оба раза, `RestartCount` брокера и телеметрии 0 | `evidence/stand/from-scratch.md` |
| Лимиты и бюджет телеметрии | замер `docker stats`, `memory.peak` и `memory.stat` | лимит ≥ 1,5 × пика у всех сервисов телеметрии (у Grafana пик — `anon` + `kernel` + `shmem` без файлового кэша) | `evidence/telemetry/budget.md` |
| Уровень 4: контекст W3C API → relay → worker, producer и consumer span (parent и link), цепочка одной загрузки до S3 и БД с повтором и replay | `driver.py --port 8080 trace-chain` | код 0: одна трасса, две пары producer / consumer, span S3 с ошибкой и запись оригинала и вариантов, replay в той же трассе, задание без контекста — корень worker | `evidence/telemetry/trace-chain.md` (commit `72cfa9a`) |
| Уровень 4: контекст W3C API → relay → worker, producer и consumer span (parent и link), цепочка одной загрузки до S3 и БД с повтором и replay | `driver.py --port 8080 trace-chain` | код 0: одна трасса, две пары producer / consumer, span S3 с ошибкой и запись оригинала и вариантов, replay в той же трассе, задание без контекста — корень worker | `evidence/telemetry/trace-chain.md` (commit `72e2598`) |
| Уровень 4: dashboard — p50 / p95 / p99, доля ошибок, глубина и возраст очереди, пул БД, насыщение worker, контейнеры | `driver.py --port 8080 trace-chain`, `telemetry-check` | запросы панелей уровня 4 без ошибок и с данными; ряды очереди только у relay; пределы = `docker inspect`; нет сокета Docker | `evidence/telemetry/trace-chain.md`, `evidence/telemetry/check.md` |
| Уровень 4: relay с SDK не меняет остановку | `driver.py --port 8080 telemetry-outage` | `docker stop` relay < 10 с, код 0, «Stopped relay» при остановленном и замороженном Collector | `evidence/telemetry/collector-outage.md` |
| Уровень 4: схема контекста задания | миграция с нуля, повтор, откат, `--sql`, `make rolling_update`; план снимка очереди | `lock_timeout`, `NOT VALID`; без Seq Scan | `evidence/migrations/job-trace-context.md`, `evidence/telemetry/queue-snapshot.md` (commit `72cfa9a`) |
+175 -99
View File
@@ -1,118 +1,194 @@
# `driver.py --port 8080 telemetry-outage` (change `otel-telemetry`, задача 11.2; change `job-trace-propagation`, задача 12.2)
# `driver.py --port 8080 telemetry-outage` (change `otel-telemetry`, задача 11.2; change `job-trace-propagation`, задачи 12.2 и 13.11)
- Основа: commit `72cfa9a` (feat: propagate trace context through outbox and add pipeline saturation
metrics), рабочее дерево чистое (`git status --short` до прогона пуст; после — только файлы
`evidence/`). Файлы `src/` и `migrations/` в контейнерах `api` (2), `relay`, `worker` и `cleaner`
побайтно совпадают с деревом commit (SHA-256 каждого файла). Прежний прогон этого файла — на
`68f1e97` (до change `job-trace-propagation`, без relay).
- Дата: 2026-11-11, 17:01:52Z – 17:04:11Z (~2,3 мин); рабочий стенд `mtusi-lab6`, команда из корня:
- Основа: commit `72e2598` (fix: address job-trace-propagation verify findings), рабочее дерево
чистое (`git status --short` до прогона пуст; после — только файлы `evidence/`). Файлы `src/` и
`migrations/` (151) в контейнерах `api` (2), `relay`, `worker` и `cleaner` побайтно совпадают с
деревом commit (SHA-256 каждого файла). Прежние прогоны этого файла — на `68f1e97` (до change
`job-trace-propagation`, без relay) и на `72cfa9a` (до правок verify: сигналы relay после
возврата Collector приходили от перезапущенного контейнера).
- Дата: 2026-11-11, 17:53:55Z – 17:56:10Z (~2,3 мин); рабочий стенд `mtusi-lab6`, команда из корня:
`python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 telemetry-outage`, код выхода 0.
- Сценарии: `observability` «Collector остановлен», «Collector заморожен», «Collector недоступен для
relay»; `service-logging` «Отказ отправки журнала»; `container-run` «Приложение без телеметрии»;
`graceful-shutdown` «docker stop при остановленном Collector», «docker stop при замороженном
Collector» (реплика API, worker и с change `job-trace-propagation` relay).
- Ход: на фазу — `docker compose stop` (или `pause`) `otel-collector`; `make test` через LB;
`/health/ready`; свой `GET /users/me` и его запись в stdout; загрузка `sample.png` до `ready`;
окно не меньше 20 с, чтобы экспорт трёх сигналов успел отказать; записи логгеров
`opentelemetry.*` в stdout каждого контейнера `api` и `worker` с начала фазы; `docker stop
--timeout 10` одной реплики и `docker compose stop worker`; затем relay: `StartedAt` и
`RestartCount` не изменились с начала фазы (загрузка стала `ready`, relay её опубликовал при
недоступном Collector), `docker stop` relay тем же замером (время, код выхода, «Stopped relay»);
`make replicas`, `docker start` worker и relay; возврат Collector (`start` / `unpause`) — и новые
запись API в Loki, трасса API в Tempo, запись worker в Loki, span producer relay в Tempo, ряды
API, worker и relay в Prometheus с отметкой времени позже возврата. Последняя фаза — все пять
- Ход (с verify W2, design change `job-trace-propagation`, «Уточнения apply», п. 18): на фазу —
`StartedAt` и `RestartCount` всех контейнеров `api` (2), `worker` и `relay`; `docker compose stop`
(или `pause`) `otel-collector`; `make test` через LB; `/health/ready`; свой `GET /users/me` и его
запись в stdout; загрузка `sample.png` до `ready` (relay её опубликовал при недоступном
Collector); окно не меньше 20 с, чтобы экспорт трёх сигналов успел отказать; записи логгеров
`opentelemetry.*` в stdout каждого контейнера `api` и `worker` с начала фазы. Затем возврат
Collector (`start` / `unpause`) — новые запись API в Loki, трасса API в Tempo, запись worker в
Loki, span producer relay в Tempo, ряды API, worker и relay в Prometheus с отметкой времени позже
возврата — и `StartedAt` / `RestartCount` тех же четырёх контейнеров прежние: сигналы приходят от
процессов, переживших недоступность, без перезапуска. Затем второе окно того же вида: Collector
снова недоступен, `make test` (свежие данные в буферах SDK), `docker stop --timeout 10` одной
реплики, `docker compose stop worker`, `docker stop` relay (время, код выхода, «Stopped …»),
`make replicas`, `docker start` worker и relay, возврат Collector. Последняя фаза — все пять
сервисов телеметрии остановлены, `docker compose restart api worker relay`, `make test`, затем
возврат телеметрии и сигналы снова. В `finally` — `unpause` и `start` телеметрии, `make
replicas`, worker и relay.
- Время остановки при недоступном Collector: реплика API — 1,71 с (stop) и 1,50 с (pause), worker —
0,74 и 0,73 с, relay — 0,64 и 0,63 с; срок закрытия SDK — 1 с у API и окно 0,5 с у worker и relay
- Время остановки при недоступном Collector: реплика API — 1,64 с (stop) и 1,66 с (pause), worker —
0,72 и 0,71 с, relay — 0,63 и 0,64 с; срок закрытия SDK — 1 с у API и окно 0,5 с у worker и relay
(`TELEMETRY_SHUTDOWN_TIMEOUT_SECONDS`, design change `otel-telemetry`, решение 13; design change
`job-trace-propagation`, решение 15).
- Записи SDK об отказе — по одной на модуль экспортёра (трассы, метрики, логи) на контейнер `api` и
`worker` за окно: фильтр пропускает не больше одной в минуту на логгер. В фазе `pause` экспортёр
логов к моменту подсчёта ещё не дождался срока отправки — записей по две. Записи SDK relay в
этом подсчёте не участвуют.
- Ограничение проверки: relay — единственный, и `docker stop` relay (сценарии `graceful-shutdown`)
идёт до возврата Collector, поэтому сигналы relay после возврата приходят от того же контейнера
после `docker start`, а не от непрерывно живущего процесса (design, «Уточнения apply», п. 9). Что
relay переживает недоступный Collector без перезапуска, показывает строка «relay start time and
restarts unchanged».
`worker` за окно: фильтр пропускает не больше одной в минуту на логгер. В фазе `pause` к моменту
подсчёта отказать успели не все экспортёры (запрос к замороженному Collector ждёт срока), и
записей меньше — у второй реплики ни одной. Записи SDK relay в этом подсчёте не участвуют.
```
[collector stop] SDK records in 20 s: {'a4a1f203ef70 opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, 'a4a1f203ef70 opentelemetry.exporter.otlp.proto.http._log_exporter': 1, 'a4a1f203ef70 opentelemetry.exporter.otlp.proto.http.metric_exporter': 1, '48b425abbd15 opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, '48b425abbd15 opentelemetry.exporter.otlp.proto.http._log_exporter': 1, '48b425abbd15 opentelemetry.exporter.otlp.proto.http.metric_exporter': 1, 'b221c6088b4b opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, 'b221c6088b4b opentelemetry.exporter.otlp.proto.http._log_exporter': 1, 'b221c6088b4b opentelemetry.exporter.otlp.proto.http.metric_exporter': 1}
[collector stop] docker stop api a4a1f203ef70: 1.71 s
[collector stop] docker compose stop worker: 0.74 s
[collector stop] docker stop relay: 0.64 s
[collector pause] SDK records in 20 s: {'a4a1f203ef70 opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, 'a4a1f203ef70 opentelemetry.exporter.otlp.proto.http.metric_exporter': 1, '48b425abbd15 opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, '48b425abbd15 opentelemetry.exporter.otlp.proto.http.metric_exporter': 1, 'b221c6088b4b opentelemetry.exporter.otlp.proto.http.metric_exporter': 1, 'b221c6088b4b opentelemetry.exporter.otlp.proto.http.trace_exporter': 1}
[collector pause] docker stop api a4a1f203ef70: 1.50 s
[collector pause] docker compose stop worker: 0.73 s
[collector pause] docker stop relay: 0.63 s
[collector stop] docker compose stop otel-collector
[collector stop] make test: 0 PASS
[collector stop] /health/ready: 200
[collector stop] upload becomes ready: yes
[collector stop] own request record in stdout: yes
[collector stop] SDK records in 20 s: {'820012004c61 opentelemetry.exporter.otlp.proto.http._log_exporter': 1, '820012004c61 opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, '820012004c61 opentelemetry.exporter.otlp.proto.http.metric_exporter': 1, '6c86638cdbce opentelemetry.exporter.otlp.proto.http._log_exporter': 1, '6c86638cdbce opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, '6c86638cdbce opentelemetry.exporter.otlp.proto.http.metric_exporter': 1, 'd04ac10e81df opentelemetry.exporter.otlp.proto.http._log_exporter': 1, 'd04ac10e81df opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, 'd04ac10e81df opentelemetry.exporter.otlp.proto.http.metric_exporter': 1}
[collector stop] SDK records per logger at most 1: yes
[collector stop] otel-collector back
[collector stop] API record reaches Loki again: yes
[collector stop] API trace reaches Tempo again: yes
[collector stop] worker record reaches Loki again: yes
[collector stop] relay producer span reaches Tempo again: yes
[collector stop] relay metrics scraped again: yes
[collector stop] API metrics scraped again: yes
[collector stop] worker metrics scraped again: yes
[collector stop] containers (StartedAt, RestartCount): 4 before, 4 after
[collector stop] api, worker and relay start time and restarts unchanged: yes
[collector stop] docker compose stop otel-collector (stops)
[collector stop] make test before stops: 0 PASS
[collector stop] docker stop api 820012004c61: 1.64 s
[collector stop] docker stop api faster than 10 s: yes
[collector stop] api exit code: 0
[collector stop] Stopped API in stdout: yes
[collector stop] docker compose stop worker: 0.72 s
[collector stop] stop worker faster than 10 s: yes
[collector stop] worker exit code: 0
[collector stop] Stopped worker in stdout: yes
[collector stop] docker stop relay: 0.63 s
[collector stop] stop relay faster than 10 s: yes
[collector stop] relay exit code: 0
[collector stop] Stopped relay in stdout: yes
[collector stop] worker and relay started again: yes
[collector stop] LB ready: yes
[collector stop] otel-collector back (after stops)
[collector pause] docker compose pause otel-collector
[collector pause] make test: 0 PASS
[collector pause] /health/ready: 200
[collector pause] upload becomes ready: yes
[collector pause] own request record in stdout: yes
[collector pause] SDK records in 20 s: {'820012004c61 opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, '820012004c61 opentelemetry.exporter.otlp.proto.http.metric_exporter': 1, 'd04ac10e81df opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, 'd04ac10e81df opentelemetry.exporter.otlp.proto.http.metric_exporter': 1}
[collector pause] SDK records per logger at most 1: yes
[collector pause] otel-collector back
[collector pause] API record reaches Loki again: yes
[collector pause] API trace reaches Tempo again: yes
[collector pause] worker record reaches Loki again: yes
[collector pause] relay producer span reaches Tempo again: yes
[collector pause] relay metrics scraped again: yes
[collector pause] API metrics scraped again: yes
[collector pause] worker metrics scraped again: yes
[collector pause] containers (StartedAt, RestartCount): 4 before, 4 after
[collector pause] api, worker and relay start time and restarts unchanged: yes
[collector pause] docker compose pause otel-collector (stops)
[collector pause] make test before stops: 0 PASS
[collector pause] docker stop api 820012004c61: 1.66 s
[collector pause] docker stop api faster than 10 s: yes
[collector pause] api exit code: 0
[collector pause] Stopped API in stdout: yes
[collector pause] docker compose stop worker: 0.71 s
[collector pause] stop worker faster than 10 s: yes
[collector pause] worker exit code: 0
[collector pause] Stopped worker in stdout: yes
[collector pause] docker stop relay: 0.64 s
[collector pause] stop relay faster than 10 s: yes
[collector pause] relay exit code: 0
[collector pause] Stopped relay in stdout: yes
[collector pause] worker and relay started again: yes
[collector pause] LB ready: yes
[collector pause] otel-collector back (after stops)
[telemetry down] app services depend on telemetry: []
[telemetry down] docker compose stop otel-collector prometheus loki tempo grafana
[telemetry down] docker compose restart api worker relay: 0
[telemetry down] LB ready: yes
[telemetry down] replicas started: 2
[telemetry down] worker started: yes
[telemetry down] relay started: yes
[telemetry down] make test: 0 PASS
[telemetry down] Grafana back: yes
[telemetry down] API record reaches Loki again: yes
[telemetry down] API trace reaches Tempo again: yes
[telemetry down] worker record reaches Loki again: yes
[telemetry down] relay producer span reaches Tempo again: yes
[telemetry down] relay metrics scraped again: yes
[telemetry down] API metrics scraped again: yes
[telemetry down] worker metrics scraped again: yes
[collector stop] make test expected 0 PASS got 0 PASS
[collector stop] /health/ready expected 200 got 200
[collector stop] upload becomes ready expected yes got yes
[collector stop] own request record in stdout expected yes got yes
[collector stop] SDK records per logger at most 1 expected yes got yes
[collector stop] docker stop api faster than 10 s expected yes got yes
[collector stop] api exit code expected 0 got 0
[collector stop] Stopped API in stdout expected yes got yes
[collector stop] stop worker faster than 10 s expected yes got yes
[collector stop] worker exit code expected 0 got 0
[collector stop] Stopped worker in stdout expected yes got yes
[collector stop] relay start time and restarts unchanged expected yes got yes
[collector stop] stop relay faster than 10 s expected yes got yes
[collector stop] relay exit code expected 0 got 0
[collector stop] Stopped relay in stdout expected yes got yes
[collector stop] worker and relay started again expected yes got yes
[collector stop] LB ready expected yes got yes
[collector stop] API record reaches Loki again expected yes got yes
[collector stop] API trace reaches Tempo again expected yes got yes
[collector stop] worker record reaches Loki again expected yes got yes
[collector stop] relay producer span reaches Tempo again expected yes got yes
[collector stop] relay metrics scraped again expected yes got yes
[collector stop] API metrics scraped again expected yes got yes
[collector stop] worker metrics scraped again expected yes got yes
[collector pause] make test expected 0 PASS got 0 PASS
[collector pause] /health/ready expected 200 got 200
[collector pause] upload becomes ready expected yes got yes
[collector pause] own request record in stdout expected yes got yes
[collector pause] SDK records per logger at most 1 expected yes got yes
[collector pause] docker stop api faster than 10 s expected yes got yes
[collector pause] api exit code expected 0 got 0
[collector pause] Stopped API in stdout expected yes got yes
[collector pause] stop worker faster than 10 s expected yes got yes
[collector pause] worker exit code expected 0 got 0
[collector pause] Stopped worker in stdout expected yes got yes
[collector pause] relay start time and restarts unchanged expected yes got yes
[collector pause] stop relay faster than 10 s expected yes got yes
[collector pause] relay exit code expected 0 got 0
[collector pause] Stopped relay in stdout expected yes got yes
[collector pause] worker and relay started again expected yes got yes
[collector pause] LB ready expected yes got yes
[collector pause] API record reaches Loki again expected yes got yes
[collector pause] API trace reaches Tempo again expected yes got yes
[collector pause] worker record reaches Loki again expected yes got yes
[collector pause] relay producer span reaches Tempo again expected yes got yes
[collector pause] relay metrics scraped again expected yes got yes
[collector pause] API metrics scraped again expected yes got yes
[collector pause] worker metrics scraped again expected yes got yes
[telemetry down] app services depend on telemetry expected [] got []
[telemetry down] docker compose restart api worker relay expected 0 got 0
[telemetry down] LB ready expected yes got yes
[telemetry down] replicas started expected 2 got 2
[telemetry down] worker started expected yes got yes
[telemetry down] relay started expected yes got yes
[telemetry down] make test expected 0 PASS got 0 PASS
[telemetry down] Grafana back expected yes got yes
[telemetry down] API record reaches Loki again expected yes got yes
[telemetry down] API trace reaches Tempo again expected yes got yes
[telemetry down] worker record reaches Loki again expected yes got yes
[telemetry down] relay producer span reaches Tempo again expected yes got yes
[telemetry down] relay metrics scraped again expected yes got yes
[telemetry down] API metrics scraped again expected yes got yes
[telemetry down] worker metrics scraped again expected yes got yes
PASS: API, worker and relay work and stop in time without the Collector, signals come back without restarts
[collector stop] make test expected 0 PASS got 0 PASS
[collector stop] /health/ready expected 200 got 200
[collector stop] upload becomes ready expected yes got yes
[collector stop] own request record in stdout expected yes got yes
[collector stop] SDK records per logger at most 1 expected yes got yes
[collector stop] API record reaches Loki again expected yes got yes
[collector stop] API trace reaches Tempo again expected yes got yes
[collector stop] worker record reaches Loki again expected yes got yes
[collector stop] relay producer span reaches Tempo again expected yes got yes
[collector stop] relay metrics scraped again expected yes got yes
[collector stop] API metrics scraped again expected yes got yes
[collector stop] worker metrics scraped again expected yes got yes
[collector stop] api, worker and relay start time and restarts unchanged expected yes got yes
[collector stop] make test before stops expected 0 PASS got 0 PASS
[collector stop] docker stop api faster than 10 s expected yes got yes
[collector stop] api exit code expected 0 got 0
[collector stop] Stopped API in stdout expected yes got yes
[collector stop] stop worker faster than 10 s expected yes got yes
[collector stop] worker exit code expected 0 got 0
[collector stop] Stopped worker in stdout expected yes got yes
[collector stop] stop relay faster than 10 s expected yes got yes
[collector stop] relay exit code expected 0 got 0
[collector stop] Stopped relay in stdout expected yes got yes
[collector stop] worker and relay started again expected yes got yes
[collector stop] LB ready expected yes got yes
[collector pause] make test expected 0 PASS got 0 PASS
[collector pause] /health/ready expected 200 got 200
[collector pause] upload becomes ready expected yes got yes
[collector pause] own request record in stdout expected yes got yes
[collector pause] SDK records per logger at most 1 expected yes got yes
[collector pause] API record reaches Loki again expected yes got yes
[collector pause] API trace reaches Tempo again expected yes got yes
[collector pause] worker record reaches Loki again expected yes got yes
[collector pause] relay producer span reaches Tempo again expected yes got yes
[collector pause] relay metrics scraped again expected yes got yes
[collector pause] API metrics scraped again expected yes got yes
[collector pause] worker metrics scraped again expected yes got yes
[collector pause] api, worker and relay start time and restarts unchanged expected yes got yes
[collector pause] make test before stops expected 0 PASS got 0 PASS
[collector pause] docker stop api faster than 10 s expected yes got yes
[collector pause] api exit code expected 0 got 0
[collector pause] Stopped API in stdout expected yes got yes
[collector pause] stop worker faster than 10 s expected yes got yes
[collector pause] worker exit code expected 0 got 0
[collector pause] Stopped worker in stdout expected yes got yes
[collector pause] stop relay faster than 10 s expected yes got yes
[collector pause] relay exit code expected 0 got 0
[collector pause] Stopped relay in stdout expected yes got yes
[collector pause] worker and relay started again expected yes got yes
[collector pause] LB ready expected yes got yes
[telemetry down] app services depend on telemetry expected [] got []
[telemetry down] docker compose restart api worker relay expected 0 got 0
[telemetry down] LB ready expected yes got yes
[telemetry down] replicas started expected 2 got 2
[telemetry down] worker started expected yes got yes
[telemetry down] relay started expected yes got yes
[telemetry down] make test expected 0 PASS got 0 PASS
[telemetry down] Grafana back expected yes got yes
[telemetry down] API record reaches Loki again expected yes got yes
[telemetry down] API trace reaches Tempo again expected yes got yes
[telemetry down] worker record reaches Loki again expected yes got yes
[telemetry down] relay producer span reaches Tempo again expected yes got yes
[telemetry down] relay metrics scraped again expected yes got yes
[telemetry down] API metrics scraped again expected yes got yes
[telemetry down] worker metrics scraped again expected yes got yes
PASS: API, worker and relay work and stop in time without the Collector; after it returns their signals come back without restarts; restarted without telemetry services, they start and work
restored: telemetry services started, replicas and worker running: yes
```
+100 -82
View File
@@ -1,12 +1,14 @@
# `driver.py --port 8080 trace-chain` (change `job-trace-propagation`, задача 12.1)
# `driver.py --port 8080 trace-chain` (change `job-trace-propagation`, задачи 12.1 и 13.11)
- Основа: commit `72cfa9a` (feat: propagate trace context through outbox and add pipeline saturation
metrics), рабочее дерево чистое (`git status --short` пуст до прогона; после — только новый файл
`evidence/`). Файлы `src/` и `migrations/` в контейнерах `api` (2), `relay`, `worker` и `cleaner`
побайтно совпадают с деревом commit (SHA-256 каждого файла); ревизия схемы `e4b8c2d17f90`;
Grafana запущена после последней правки dashboard JSON.
- Дата: 2026-11-11, 16:55:56Z – 16:59:00Z (~3 мин); рабочий стенд `mtusi-lab6` (две реплики API,
relay, worker, уборщик, Collector, Prometheus, Loki, Tempo, Grafana), команда из корня репозитория:
- Основа: commit `72e2598` (fix: address job-trace-propagation verify findings), рабочее дерево
чистое (`git status --short` пуст до прогона; после — только файлы `evidence/`). Файлы `src/` и
`migrations/` (151) в контейнерах `api` (2), `relay`, `worker` и `cleaner` побайтно совпадают с
деревом commit (SHA-256 каждого файла); ревизия схемы `e4b8c2d17f90`; Grafana запущена
(17:55:46Z, рестарт в `telemetry-outage`) после последней правки dashboard JSON. Прежний прогон
этого файла — на `72cfa9a`, до правок verify.
- Дата: 2026-11-11, 17:56:20Z – 17:59:11Z (~3 мин), сразу после `telemetry-outage`; рабочий стенд
`mtusi-lab6` (две реплики API, relay, worker, уборщик, Collector, Prometheus, Loki, Tempo,
Grafana), команда из корня репозитория:
`python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 trace-chain`, код выхода 0.
- Всё — через HTTP API Grafana на `127.0.0.1:3000` с паролем из `.env` (источники-прокси uid
`prometheus`, `loki`, `tempo`); PostgreSQL — только чтение, кроме одного `UPDATE` контекста своего
@@ -18,16 +20,22 @@
сокета Docker», «Панели уровня 4 после трафика», «Время до готовности в метрике» (шаг 5), «Проверка
цепочки» (команда целиком); `service-logging` — «Записи задания ведут к трассе» (записи worker по F
несут `trace_id` T записи «Uploaded attachment» запроса R).
- С правками verify (design, «Уточнения apply», п. 19, 22, 23): предусловие — глубина очереди 0
с ожиданием до 120 с (здесь дождалась сразу; не дождалась бы — код 2 «stand is not quiet»); CPU
каждого контейнера `api`, `worker`, `relay` вырос за прогон (`increase(container_cpu_time_seconds_total[161s])`
> 0 — сценарий «Пределы как в compose», «CPU растёт под нагрузкой»); панель пула с двумя
единицами (ожидающие и отказы в секунду) — среди 19 запросов панелей уровня 4 без ошибок и с
данными.
- Шаги 2 и 3: Tempo 3.1 собрал span producer и consumer разных попыток (повтор после `pause s3`,
replay) в одной трассе загрузки (design, «Уточнения apply», п. 7). Время до `ready`: worker перезапускался дважды, проверка складывает «тихие» значения двух
процессов (п. 8).
replay) в одной трассе загрузки (п. 7). Время до `ready`: worker перезапускался дважды, проверка
складывает «тихие» значения двух процессов (п. 8).
- Расхождений с design нет.
Ход прогона:
```text
2026-11-11T16:55:56Z
queue depth before (no unfinished jobs): 0
queue depth 0 waited: 0 s
queue depth before reaches 0 within 120 s (no unfinished jobs): yes
docker compose stop worker
upload queue0: 202
upload queue1: 202
@@ -53,7 +61,7 @@ root: server span of the upload: POST /api/v1/chats/{chat_id}/attachments messen
producer spans: 2
consumer spans: 2
producers: messenger-relay, children of the root: yes
producer service.instance.id = relay instance_id: f1a1d8fef0bc
producer service.instance.id = relay instance_id: 0ac09c58168f
producer attempts: [1, 2]
retry reason of the second producer: storage_unavailable
consumers: messenger-worker, parent and link = own producer: yes
@@ -82,86 +90,96 @@ pool max: API replicas, worker, relay: ['10', '10'] ['2'] ['1']
pools with used above max: 0
worker capacity: 2
queue series only from relay: ['messenger-relay']
api a4a1f203ef70 limits = docker inspect: 1 536870912
api a4a1f203ef70 memory below the limit: yes
api 48b425abbd15 limits = docker inspect: 1 536870912
api 48b425abbd15 memory below the limit: yes
worker b221c6088b4b limits = docker inspect: 1 1073741824
worker b221c6088b4b memory below the limit: yes
relay f1a1d8fef0bc limits = docker inspect: 0.25 268435456
relay f1a1d8fef0bc memory below the limit: yes
api 820012004c61 limits = docker inspect: 1 536870912
api 820012004c61 memory below the limit: yes
api 820012004c61 CPU grows during the run (increase over 161 s > 0): yes
api 6c86638cdbce limits = docker inspect: 1 536870912
api 6c86638cdbce memory below the limit: yes
api 6c86638cdbce CPU grows during the run (increase over 161 s > 0): yes
worker d04ac10e81df limits = docker inspect: 1 1073741824
worker d04ac10e81df memory below the limit: yes
worker d04ac10e81df CPU grows during the run (increase over 161 s > 0): yes
relay 0ac09c58168f limits = docker inspect: 0.25 268435456
relay 0ac09c58168f memory below the limit: yes
relay 0ac09c58168f CPU grows during the run (increase over 161 s > 0): yes
Docker socket mounted by any service: no
dashboard loaded: 200
level 4 panel queries (19) without error: []
level 4 panel queries with data: []
waiting for the time-to-ready histogram to settle
time to ready count grew by the uploads without replay: 5
time to ready sum grew by at least 3 x 41 s (worker stopped): yes
time to ready sum grew by at least 3 x 32 s (worker stopped): yes
```
Итог («with data: []» — список панелей без данных пуст):
```text
queue depth before (no unfinished jobs) expected 0 got 0
upload queue0 expected 202 got 202
upload queue1 expected 202 got 202
upload queue2 expected 202 got 202
outbox + published with worker stopped expected 3 got 3
oldest age not less than time since first upload - 35 s expected yes got yes
oldest age grows expected yes got yes
all queued attachments ready after worker start expected yes got yes
queue depth and age 0 after ready expected yes got yes
upload retry expected 202 got 202
job of F published with worker stopped expected yes got yes
worker stdout: Retrying job of F (storage_unavailable) expected storage_unavailable got storage_unavailable
F becomes ready after S3 is back expected ready got ready
Loki: Uploaded attachment of R has trace_id T expected yes got yes
worker records of F carry trace_id T expected yes got yes
root: server span of the upload expected POST /api/v1/chats/{chat_id}/attachments messenger-api got POST /api/v1/chats/{chat_id}/attachments messenger-api
producer spans expected 2 got 2
consumer spans expected 2 got 2
producers: messenger-relay, children of the root expected yes got yes
producer service.instance.id = relay instance_id expected f1a1d8fef0bc got f1a1d8fef0bc
producer attempts expected [1, 2] got [1, 2]
retry reason of the second producer expected storage_unavailable got storage_unavailable
consumers: messenger-worker, parent and link = own producer expected yes got yes
consumer outcomes and attempts expected [('retry', 1), ('ready', 2)] got [('retry', 1), ('ready', 2)]
retry consumer reason expected storage_unavailable got storage_unavailable
retry consumer: S3 span with error expected yes got yes
ready consumer: S3.PutObject >= 4, render, COMMIT expected yes got yes
undecodable JPEG generated expected yes got yes
upload replay expected 202 got 202
undecodable JPEG becomes failed expected failed got failed
replay --apply exit code expected 0 got 0
replayed job failed again expected yes got yes
producer span with messenger.job.replayed expected 1 got 1
consumer span of the replayed attempt (failed) expected failed got failed
two consumer spans failed in the upload trace expected 2 got 2
upload plain expected 202 got 202
context of one job removed (UPDATE rows) expected 1 got 1
job without context becomes ready expected ready got ready
worker trace_id differs from the upload trace expected yes got yes
its root is the consumer span expected SPAN_KIND_CONSUMER messenger-worker got SPAN_KIND_CONSUMER messenger-worker
producer spans in the worker trace expected 0 got 0
producer and consumer spans in the upload trace expected 0 got 0
pool max: API replicas, worker, relay expected ['10', '10'] ['2'] ['1'] got ['10', '10'] ['2'] ['1']
pools with used above max expected 0 got 0
worker capacity expected 2 got 2
queue series only from relay expected ['messenger-relay'] got ['messenger-relay']
api a4a1f203ef70 limits = docker inspect expected 1 536870912 got 1 536870912
api a4a1f203ef70 memory below the limit expected yes got yes
api 48b425abbd15 limits = docker inspect expected 1 536870912 got 1 536870912
api 48b425abbd15 memory below the limit expected yes got yes
worker b221c6088b4b limits = docker inspect expected 1 1073741824 got 1 1073741824
worker b221c6088b4b memory below the limit expected yes got yes
relay f1a1d8fef0bc limits = docker inspect expected 0.25 268435456 got 0.25 268435456
relay f1a1d8fef0bc memory below the limit expected yes got yes
Docker socket mounted by any service expected no got no
dashboard loaded expected 200 got 200
level 4 panel queries (19) without error expected [] got []
level 4 panel queries with data expected [] got []
time to ready count grew by the uploads without replay expected 5 got 5
time to ready sum grew by at least 3 x 41 s (worker stopped) expected yes got yes
queue depth before reaches 0 within 120 s (no unfinished jobs) expected yes got yes
upload queue0 expected 202 got 202
upload queue1 expected 202 got 202
upload queue2 expected 202 got 202
outbox + published with worker stopped expected 3 got 3
oldest age not less than time since first upload - 35 s expected yes got yes
oldest age grows expected yes got yes
all queued attachments ready after worker start expected yes got yes
queue depth and age 0 after ready expected yes got yes
upload retry expected 202 got 202
job of F published with worker stopped expected yes got yes
worker stdout: Retrying job of F (storage_unavailable) expected storage_unavailable got storage_unavailable
F becomes ready after S3 is back expected ready got ready
Loki: Uploaded attachment of R has trace_id T expected yes got yes
worker records of F carry trace_id T expected yes got yes
root: server span of the upload expected POST /api/v1/chats/{chat_id}/attachments messenger-api got POST /api/v1/chats/{chat_id}/attachments messenger-api
producer spans expected 2 got 2
consumer spans expected 2 got 2
producers: messenger-relay, children of the root expected yes got yes
producer service.instance.id = relay instance_id expected 0ac09c58168f got 0ac09c58168f
producer attempts expected [1, 2] got [1, 2]
retry reason of the second producer expected storage_unavailable got storage_unavailable
consumers: messenger-worker, parent and link = own producer expected yes got yes
consumer outcomes and attempts expected [('retry', 1), ('ready', 2)] got [('retry', 1), ('ready', 2)]
retry consumer reason expected storage_unavailable got storage_unavailable
retry consumer: S3 span with error expected yes got yes
ready consumer: S3.PutObject >= 4, render, COMMIT expected yes got yes
undecodable JPEG generated expected yes got yes
upload replay expected 202 got 202
undecodable JPEG becomes failed expected failed got failed
replay --apply exit code expected 0 got 0
replayed job failed again expected yes got yes
producer span with messenger.job.replayed expected 1 got 1
consumer span of the replayed attempt (failed) expected failed got failed
two consumer spans failed in the upload trace expected 2 got 2
upload plain expected 202 got 202
context of one job removed (UPDATE rows) expected 1 got 1
job without context becomes ready expected ready got ready
worker trace_id differs from the upload trace expected yes got yes
its root is the consumer span expected SPAN_KIND_CONSUMER messenger-worker got SPAN_KIND_CONSUMER messenger-worker
producer spans in the worker trace expected 0 got 0
producer and consumer spans in the upload trace expected 0 got 0
pool max: API replicas, worker, relay expected ['10', '10'] ['2'] ['1'] got ['10', '10'] ['2'] ['1']
pools with used above max expected 0 got 0
worker capacity expected 2 got 2
queue series only from relay expected ['messenger-relay'] got ['messenger-relay']
api 820012004c61 limits = docker inspect expected 1 536870912 got 1 536870912
api 820012004c61 memory below the limit expected yes got yes
api 820012004c61 CPU grows during the run (increase over 161 s > 0) expected yes got yes
api 6c86638cdbce limits = docker inspect expected 1 536870912 got 1 536870912
api 6c86638cdbce memory below the limit expected yes got yes
api 6c86638cdbce CPU grows during the run (increase over 161 s > 0) expected yes got yes
worker d04ac10e81df limits = docker inspect expected 1 1073741824 got 1 1073741824
worker d04ac10e81df memory below the limit expected yes got yes
worker d04ac10e81df CPU grows during the run (increase over 161 s > 0) expected yes got yes
relay 0ac09c58168f limits = docker inspect expected 0.25 268435456 got 0.25 268435456
relay 0ac09c58168f memory below the limit expected yes got yes
relay 0ac09c58168f CPU grows during the run (increase over 161 s > 0) expected yes got yes
Docker socket mounted by any service expected no got no
dashboard loaded expected 200 got 200
level 4 panel queries (19) without error expected [] got []
level 4 panel queries with data expected [] got []
time to ready count grew by the uploads without replay expected 5 got 5
time to ready sum grew by at least 3 x 32 s (worker stopped) expected yes got yes
PASS: one upload is one trace: producer and consumer per attempt, retry and replay in it, level 4 signals found
restored: s3 unpaused, worker and relay running: yes
```
Код выхода — 0.
@@ -237,7 +237,7 @@ evidence — после коммита этих правок основной с
`make rolling_update`), `make test`, `make student_test` (красный только `test_exclusion_evidence_exists`), `docker
compose config --quiet`, пробные `trace-chain` и `telemetry-outage` (вывод в scratchpad), `openspec validate
job-trace-propagation --strict`.
- [ ] 13.11 Пересъёмка evidence (после коммита правок основной сессией): `driver.py --port 8080 telemetry-outage` →
- [x] 13.11 Пересъёмка evidence (после коммита правок основной сессией): `driver.py --port 8080 telemetry-outage` →
`evidence/telemetry/collector-outage.md` (итоговая строка без ложного «without restarts»), `driver.py --port 8080
trace-chain` → `evidence/telemetry/trace-chain.md` (затронут: предусловие, CPU, панель пула); ссылки commit в `REPORT.md`
и `AGENTS.md`.