diff --git a/AGENTS.md b/AGENTS.md index c5876fb..5a2de73 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -32,9 +32,10 @@ relay и consumer span worker (parent и link) — одна трасса на з (`CountingQueuePool`), насыщение worker, CPU и память контейнеров из cgroup (без `docker.sock`); панели уровня 4 в том же dashboard; команда драйвера `trace-chain`. Код — commit `72cfa9a`, evidence снято на нём: -`evidence/telemetry/trace-chain.md`, `check.md`, `collector-outage.md`, -`queue-snapshot.md`, раздел change в `budget.md`, -`evidence/migrations/job-trace-context.md`. Следующий — `slo-alerts` (SLO, +`evidence/telemetry/check.md`, `queue-snapshot.md`, раздел change в +`budget.md`, `evidence/migrations/job-trace-context.md`; правки verify — +commit `72e2598`, на нём пересняты `evidence/telemetry/trace-chain.md` и +`collector-outage.md`. Следующий — `slo-alerts` (SLO, alerts, runbooks, инциденты). Код перенесён из лабы 5 (репозиторий `mtusi2026lab5-safronov`, ветка `solution/bvt2301-safronov`, commit @@ -715,7 +716,7 @@ evidence/attachments/ # change async-attachments: defense.md (driver.py d evidence/s3/ # privacy.md — make s3_check на текущем коде; change s3-storage: bootstrap.md (первый запуск, повтор, смена секрета, ошибка шага, до bootstrap), privacy.md (make s3_check, нарушение и исправление, порты, секреты по сервисам), grpc-admin.md (gRPC шлюза без токена), restart.md (restart.sh и SIGKILL S3), client.md (операции, pause/stop, неверный секрет, журнал), memory.md (20 × 10 MiB PUT); change upload-consistency: cleaner-key.md (пользователь уборщика в bootstrap, смена секрета, make s3_check с ключом уборщика, секреты по сервисам); change track2-reconciler: cleaner-list.md (bootstrap с политикой списка по префиксу, неверный префикс, make s3_check со строками списка уборщика, окружение s3-bootstrap) evidence/reconcile/ # change track2-reconciler: cycle.md (driver.py reconcile-cycle), stress.md (driver.py reconcile-stress), service.md (Started cleaner с параметрами сверки, плановая сверка при старте, docker stop во время сверки при pause s3, код 1 разового прохода по сигналу — driver.py reconcile-signal, отказ старта на RECONCILER__*, коды 3 и 0; с change track2-reconcile-cost — остановка во время паузы предела скорости, driver.py reconcile-signal --pace) evidence/reconcile/<метка>/ # результаты make bench_reconcile (change track2-reconcile-cost): meta.json (раздел reconcile: план, этапы и модели, commit и образы «до», аргументы S3), passes.csv, windows.csv, summary.md (H1–H4 и S4), cost-by-n.svg, requests-by-n.svg, background-p95.svg, raw/ (записи уборщика по проходам, итог k6 блока H4) -evidence/telemetry/ # change otel-telemetry: budget.md (пики и лимиты сервисов телеметрии, реплик и worker с телеметрией и без); на commit 72cfa9a change job-trace-propagation переснято: check.md (driver.py telemetry-check), collector-outage.md (driver.py telemetry-outage, с relay); новое: trace-chain.md (driver.py trace-chain), queue-snapshot.md (EXPLAIN снимка очереди), раздел change в budget.md (relay с SDK, уборщик после ленивого импорта) +evidence/telemetry/ # change otel-telemetry: budget.md (пики и лимиты сервисов телеметрии, реплик и worker с телеметрией и без); на commit 72cfa9a change job-trace-propagation переснято: check.md (driver.py telemetry-check); на 72e2598 (правки verify) — collector-outage.md (driver.py telemetry-outage, с relay, сигналы без перезапуска) и новое trace-chain.md (driver.py trace-chain); новое на 72cfa9a: queue-snapshot.md (EXPLAIN снимка очереди), раздел change в budget.md (relay с SDK, уборщик после ленивого импорта) evidence/migrations/ # change job-trace-propagation: job-trace-context.md — ревизия e4b8c2d17f90 (временная база: с нуля, повтор, откат, CHECK, lock_timeout; --sql; рабочий стенд: откат реплик на ff0073e с MIGRATE=0, ревизия до и после make rolling_update) evidence/stand/ # change otel-telemetry: from-scratch.md (driver.py stand-from-scratch --runs 2) evidence/storage/ # отказы БД, пул, statement_timeout, рестарт, порты Postgres, свежий clone; connection-cost.md — цена нового соединения и пинга; freeze-sessions.md — change unit-of-work: сессии и журнал пула после заморозки PostgreSQL (driver.py freeze-sessions и freeze-lb) diff --git a/REPORT.md b/REPORT.md index 050626d..a50fdb5 100644 --- a/REPORT.md +++ b/REPORT.md @@ -19,7 +19,9 @@ Change `otel-telemetry` (design — `openspec/changes/archive/2026-11-11-otel-te номера решений в скобках). Числа и проверки сняты на закоммиченном коде при чистом рабочем дереве: `telemetry-check` и `telemetry-outage` сначала на `68f1e97` (правки verify), затем пересняты на `72cfa9a` (код change `job-trace-propagation`: relay с SDK, шаги задания по consumer span), там -же `trace-chain`, план снимка очереди, проба ревизии `e4b8c2d17f90` и раздел change бюджета; +же план снимка очереди, проба ревизии `e4b8c2d17f90` и раздел change бюджета; `trace-chain` и +`telemetry-outage` — ещё раз на `72e2598` (правки verify: сигналы relay без перезапуска, тихая +очередь в предусловии, рост CPU каждого процесса); бюджет сервисов телеметрии — на `4178b3d` (лимиты Tempo и Grafana по замеру), шапки файлов `evidence/telemetry/` и `evidence/migrations/`. `evidence/stand/from- scratch.md` ещё снят на рабочем дереве поверх `9b35c25`: пересъёмка ждёт разрешения пользователя @@ -286,10 +288,10 @@ replay, консоль, `make test_isolated`, стенд отказов и `benc | Публичный smoke | make test | TODO | TODO | | Остальные пункты уровня 3 | TODO | TODO | TODO | | Уровень 3: Collector, хранилища, provisioning, свой запрос и задание найдены, приватность | `driver.py --port 8080 telemetry-check` | код 0: Loki → Tempo по `X-Request-Id`, трасса worker, рост счётчиков, история после рестарта хранилищ | `evidence/telemetry/check.md` | -| Телеметрия не меняет ответы и остановку | `driver.py --port 8080 telemetry-outage` | код 0: `make test` PASS и остановка < 10 с при остановленном и замороженном Collector | `evidence/telemetry/collector-outage.md` | +| Телеметрия не меняет ответы и остановку | `driver.py --port 8080 telemetry-outage` | код 0: `make test` PASS и остановка < 10 с при остановленном и замороженном Collector, сигналы после его возврата без перезапуска процессов | `evidence/telemetry/collector-outage.md` (commit `72e2598`) | | Стенд с нуля одной командой | `driver.py stand-from-scratch --runs 2` | код 0 оба раза, `RestartCount` брокера и телеметрии 0 | `evidence/stand/from-scratch.md` | | Лимиты и бюджет телеметрии | замер `docker stats`, `memory.peak` и `memory.stat` | лимит ≥ 1,5 × пика у всех сервисов телеметрии (у Grafana пик — `anon` + `kernel` + `shmem` без файлового кэша) | `evidence/telemetry/budget.md` | -| Уровень 4: контекст W3C API → relay → worker, producer и consumer span (parent и link), цепочка одной загрузки до S3 и БД с повтором и replay | `driver.py --port 8080 trace-chain` | код 0: одна трасса, две пары producer / consumer, span S3 с ошибкой и запись оригинала и вариантов, replay в той же трассе, задание без контекста — корень worker | `evidence/telemetry/trace-chain.md` (commit `72cfa9a`) | +| Уровень 4: контекст W3C API → relay → worker, producer и consumer span (parent и link), цепочка одной загрузки до S3 и БД с повтором и replay | `driver.py --port 8080 trace-chain` | код 0: одна трасса, две пары producer / consumer, span S3 с ошибкой и запись оригинала и вариантов, replay в той же трассе, задание без контекста — корень worker | `evidence/telemetry/trace-chain.md` (commit `72e2598`) | | Уровень 4: dashboard — p50 / p95 / p99, доля ошибок, глубина и возраст очереди, пул БД, насыщение worker, контейнеры | `driver.py --port 8080 trace-chain`, `telemetry-check` | запросы панелей уровня 4 без ошибок и с данными; ряды очереди только у relay; пределы = `docker inspect`; нет сокета Docker | `evidence/telemetry/trace-chain.md`, `evidence/telemetry/check.md` | | Уровень 4: relay с SDK не меняет остановку | `driver.py --port 8080 telemetry-outage` | `docker stop` relay < 10 с, код 0, «Stopped relay» при остановленном и замороженном Collector | `evidence/telemetry/collector-outage.md` | | Уровень 4: схема контекста задания | миграция с нуля, повтор, откат, `--sql`, `make rolling_update`; план снимка очереди | `lock_timeout`, `NOT VALID`; без Seq Scan | `evidence/migrations/job-trace-context.md`, `evidence/telemetry/queue-snapshot.md` (commit `72cfa9a`) | diff --git a/evidence/telemetry/collector-outage.md b/evidence/telemetry/collector-outage.md index ed5907e..57fce0c 100644 --- a/evidence/telemetry/collector-outage.md +++ b/evidence/telemetry/collector-outage.md @@ -1,118 +1,194 @@ -# `driver.py --port 8080 telemetry-outage` (change `otel-telemetry`, задача 11.2; change `job-trace-propagation`, задача 12.2) +# `driver.py --port 8080 telemetry-outage` (change `otel-telemetry`, задача 11.2; change `job-trace-propagation`, задачи 12.2 и 13.11) -- Основа: commit `72cfa9a` (feat: propagate trace context through outbox and add pipeline saturation - metrics), рабочее дерево чистое (`git status --short` до прогона пуст; после — только файлы - `evidence/`). Файлы `src/` и `migrations/` в контейнерах `api` (2), `relay`, `worker` и `cleaner` - побайтно совпадают с деревом commit (SHA-256 каждого файла). Прежний прогон этого файла — на - `68f1e97` (до change `job-trace-propagation`, без relay). -- Дата: 2026-11-11, 17:01:52Z – 17:04:11Z (~2,3 мин); рабочий стенд `mtusi-lab6`, команда из корня: +- Основа: commit `72e2598` (fix: address job-trace-propagation verify findings), рабочее дерево + чистое (`git status --short` до прогона пуст; после — только файлы `evidence/`). Файлы `src/` и + `migrations/` (151) в контейнерах `api` (2), `relay`, `worker` и `cleaner` побайтно совпадают с + деревом commit (SHA-256 каждого файла). Прежние прогоны этого файла — на `68f1e97` (до change + `job-trace-propagation`, без relay) и на `72cfa9a` (до правок verify: сигналы relay после + возврата Collector приходили от перезапущенного контейнера). +- Дата: 2026-11-11, 17:53:55Z – 17:56:10Z (~2,3 мин); рабочий стенд `mtusi-lab6`, команда из корня: `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 telemetry-outage`, код выхода 0. - Сценарии: `observability` «Collector остановлен», «Collector заморожен», «Collector недоступен для relay»; `service-logging` «Отказ отправки журнала»; `container-run` «Приложение без телеметрии»; `graceful-shutdown` «docker stop при остановленном Collector», «docker stop при замороженном Collector» (реплика API, worker и с change `job-trace-propagation` relay). -- Ход: на фазу — `docker compose stop` (или `pause`) `otel-collector`; `make test` через LB; - `/health/ready`; свой `GET /users/me` и его запись в stdout; загрузка `sample.png` до `ready`; - окно не меньше 20 с, чтобы экспорт трёх сигналов успел отказать; записи логгеров - `opentelemetry.*` в stdout каждого контейнера `api` и `worker` с начала фазы; `docker stop - --timeout 10` одной реплики и `docker compose stop worker`; затем relay: `StartedAt` и - `RestartCount` не изменились с начала фазы (загрузка стала `ready`, relay её опубликовал при - недоступном Collector), `docker stop` relay тем же замером (время, код выхода, «Stopped relay»); - `make replicas`, `docker start` worker и relay; возврат Collector (`start` / `unpause`) — и новые - запись API в Loki, трасса API в Tempo, запись worker в Loki, span producer relay в Tempo, ряды - API, worker и relay в Prometheus с отметкой времени позже возврата. Последняя фаза — все пять +- Ход (с verify W2, design change `job-trace-propagation`, «Уточнения apply», п. 18): на фазу — + `StartedAt` и `RestartCount` всех контейнеров `api` (2), `worker` и `relay`; `docker compose stop` + (или `pause`) `otel-collector`; `make test` через LB; `/health/ready`; свой `GET /users/me` и его + запись в stdout; загрузка `sample.png` до `ready` (relay её опубликовал при недоступном + Collector); окно не меньше 20 с, чтобы экспорт трёх сигналов успел отказать; записи логгеров + `opentelemetry.*` в stdout каждого контейнера `api` и `worker` с начала фазы. Затем возврат + Collector (`start` / `unpause`) — новые запись API в Loki, трасса API в Tempo, запись worker в + Loki, span producer relay в Tempo, ряды API, worker и relay в Prometheus с отметкой времени позже + возврата — и `StartedAt` / `RestartCount` тех же четырёх контейнеров прежние: сигналы приходят от + процессов, переживших недоступность, без перезапуска. Затем второе окно того же вида: Collector + снова недоступен, `make test` (свежие данные в буферах SDK), `docker stop --timeout 10` одной + реплики, `docker compose stop worker`, `docker stop` relay (время, код выхода, «Stopped …»), + `make replicas`, `docker start` worker и relay, возврат Collector. Последняя фаза — все пять сервисов телеметрии остановлены, `docker compose restart api worker relay`, `make test`, затем возврат телеметрии и сигналы снова. В `finally` — `unpause` и `start` телеметрии, `make replicas`, worker и relay. -- Время остановки при недоступном Collector: реплика API — 1,71 с (stop) и 1,50 с (pause), worker — - 0,74 и 0,73 с, relay — 0,64 и 0,63 с; срок закрытия SDK — 1 с у API и окно 0,5 с у worker и relay +- Время остановки при недоступном Collector: реплика API — 1,64 с (stop) и 1,66 с (pause), worker — + 0,72 и 0,71 с, relay — 0,63 и 0,64 с; срок закрытия SDK — 1 с у API и окно 0,5 с у worker и relay (`TELEMETRY_SHUTDOWN_TIMEOUT_SECONDS`, design change `otel-telemetry`, решение 13; design change `job-trace-propagation`, решение 15). - Записи SDK об отказе — по одной на модуль экспортёра (трассы, метрики, логи) на контейнер `api` и - `worker` за окно: фильтр пропускает не больше одной в минуту на логгер. В фазе `pause` экспортёр - логов к моменту подсчёта ещё не дождался срока отправки — записей по две. Записи SDK relay в - этом подсчёте не участвуют. -- Ограничение проверки: relay — единственный, и `docker stop` relay (сценарии `graceful-shutdown`) - идёт до возврата Collector, поэтому сигналы relay после возврата приходят от того же контейнера - после `docker start`, а не от непрерывно живущего процесса (design, «Уточнения apply», п. 9). Что - relay переживает недоступный Collector без перезапуска, показывает строка «relay start time and - restarts unchanged». + `worker` за окно: фильтр пропускает не больше одной в минуту на логгер. В фазе `pause` к моменту + подсчёта отказать успели не все экспортёры (запрос к замороженному Collector ждёт срока), и + записей меньше — у второй реплики ни одной. Записи SDK relay в этом подсчёте не участвуют. ``` -[collector stop] SDK records in 20 s: {'a4a1f203ef70 opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, 'a4a1f203ef70 opentelemetry.exporter.otlp.proto.http._log_exporter': 1, 'a4a1f203ef70 opentelemetry.exporter.otlp.proto.http.metric_exporter': 1, '48b425abbd15 opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, '48b425abbd15 opentelemetry.exporter.otlp.proto.http._log_exporter': 1, '48b425abbd15 opentelemetry.exporter.otlp.proto.http.metric_exporter': 1, 'b221c6088b4b opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, 'b221c6088b4b opentelemetry.exporter.otlp.proto.http._log_exporter': 1, 'b221c6088b4b opentelemetry.exporter.otlp.proto.http.metric_exporter': 1} -[collector stop] docker stop api a4a1f203ef70: 1.71 s -[collector stop] docker compose stop worker: 0.74 s -[collector stop] docker stop relay: 0.64 s -[collector pause] SDK records in 20 s: {'a4a1f203ef70 opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, 'a4a1f203ef70 opentelemetry.exporter.otlp.proto.http.metric_exporter': 1, '48b425abbd15 opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, '48b425abbd15 opentelemetry.exporter.otlp.proto.http.metric_exporter': 1, 'b221c6088b4b opentelemetry.exporter.otlp.proto.http.metric_exporter': 1, 'b221c6088b4b opentelemetry.exporter.otlp.proto.http.trace_exporter': 1} -[collector pause] docker stop api a4a1f203ef70: 1.50 s -[collector pause] docker compose stop worker: 0.73 s -[collector pause] docker stop relay: 0.63 s +[collector stop] docker compose stop otel-collector +[collector stop] make test: 0 PASS +[collector stop] /health/ready: 200 +[collector stop] upload becomes ready: yes +[collector stop] own request record in stdout: yes +[collector stop] SDK records in 20 s: {'820012004c61 opentelemetry.exporter.otlp.proto.http._log_exporter': 1, '820012004c61 opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, '820012004c61 opentelemetry.exporter.otlp.proto.http.metric_exporter': 1, '6c86638cdbce opentelemetry.exporter.otlp.proto.http._log_exporter': 1, '6c86638cdbce opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, '6c86638cdbce opentelemetry.exporter.otlp.proto.http.metric_exporter': 1, 'd04ac10e81df opentelemetry.exporter.otlp.proto.http._log_exporter': 1, 'd04ac10e81df opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, 'd04ac10e81df opentelemetry.exporter.otlp.proto.http.metric_exporter': 1} +[collector stop] SDK records per logger at most 1: yes +[collector stop] otel-collector back +[collector stop] API record reaches Loki again: yes +[collector stop] API trace reaches Tempo again: yes +[collector stop] worker record reaches Loki again: yes +[collector stop] relay producer span reaches Tempo again: yes +[collector stop] relay metrics scraped again: yes +[collector stop] API metrics scraped again: yes +[collector stop] worker metrics scraped again: yes +[collector stop] containers (StartedAt, RestartCount): 4 before, 4 after +[collector stop] api, worker and relay start time and restarts unchanged: yes +[collector stop] docker compose stop otel-collector (stops) +[collector stop] make test before stops: 0 PASS +[collector stop] docker stop api 820012004c61: 1.64 s +[collector stop] docker stop api faster than 10 s: yes +[collector stop] api exit code: 0 +[collector stop] Stopped API in stdout: yes +[collector stop] docker compose stop worker: 0.72 s +[collector stop] stop worker faster than 10 s: yes +[collector stop] worker exit code: 0 +[collector stop] Stopped worker in stdout: yes +[collector stop] docker stop relay: 0.63 s +[collector stop] stop relay faster than 10 s: yes +[collector stop] relay exit code: 0 +[collector stop] Stopped relay in stdout: yes +[collector stop] worker and relay started again: yes +[collector stop] LB ready: yes +[collector stop] otel-collector back (after stops) +[collector pause] docker compose pause otel-collector +[collector pause] make test: 0 PASS +[collector pause] /health/ready: 200 +[collector pause] upload becomes ready: yes +[collector pause] own request record in stdout: yes +[collector pause] SDK records in 20 s: {'820012004c61 opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, '820012004c61 opentelemetry.exporter.otlp.proto.http.metric_exporter': 1, 'd04ac10e81df opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, 'd04ac10e81df opentelemetry.exporter.otlp.proto.http.metric_exporter': 1} +[collector pause] SDK records per logger at most 1: yes +[collector pause] otel-collector back +[collector pause] API record reaches Loki again: yes +[collector pause] API trace reaches Tempo again: yes +[collector pause] worker record reaches Loki again: yes +[collector pause] relay producer span reaches Tempo again: yes +[collector pause] relay metrics scraped again: yes +[collector pause] API metrics scraped again: yes +[collector pause] worker metrics scraped again: yes +[collector pause] containers (StartedAt, RestartCount): 4 before, 4 after +[collector pause] api, worker and relay start time and restarts unchanged: yes +[collector pause] docker compose pause otel-collector (stops) +[collector pause] make test before stops: 0 PASS +[collector pause] docker stop api 820012004c61: 1.66 s +[collector pause] docker stop api faster than 10 s: yes +[collector pause] api exit code: 0 +[collector pause] Stopped API in stdout: yes +[collector pause] docker compose stop worker: 0.71 s +[collector pause] stop worker faster than 10 s: yes +[collector pause] worker exit code: 0 +[collector pause] Stopped worker in stdout: yes +[collector pause] docker stop relay: 0.64 s +[collector pause] stop relay faster than 10 s: yes +[collector pause] relay exit code: 0 +[collector pause] Stopped relay in stdout: yes +[collector pause] worker and relay started again: yes +[collector pause] LB ready: yes +[collector pause] otel-collector back (after stops) +[telemetry down] app services depend on telemetry: [] [telemetry down] docker compose stop otel-collector prometheus loki tempo grafana +[telemetry down] docker compose restart api worker relay: 0 +[telemetry down] LB ready: yes +[telemetry down] replicas started: 2 +[telemetry down] worker started: yes +[telemetry down] relay started: yes +[telemetry down] make test: 0 PASS +[telemetry down] Grafana back: yes +[telemetry down] API record reaches Loki again: yes +[telemetry down] API trace reaches Tempo again: yes +[telemetry down] worker record reaches Loki again: yes +[telemetry down] relay producer span reaches Tempo again: yes +[telemetry down] relay metrics scraped again: yes +[telemetry down] API metrics scraped again: yes +[telemetry down] worker metrics scraped again: yes -[collector stop] make test expected 0 PASS got 0 PASS -[collector stop] /health/ready expected 200 got 200 -[collector stop] upload becomes ready expected yes got yes -[collector stop] own request record in stdout expected yes got yes -[collector stop] SDK records per logger at most 1 expected yes got yes -[collector stop] docker stop api faster than 10 s expected yes got yes -[collector stop] api exit code expected 0 got 0 -[collector stop] Stopped API in stdout expected yes got yes -[collector stop] stop worker faster than 10 s expected yes got yes -[collector stop] worker exit code expected 0 got 0 -[collector stop] Stopped worker in stdout expected yes got yes -[collector stop] relay start time and restarts unchanged expected yes got yes -[collector stop] stop relay faster than 10 s expected yes got yes -[collector stop] relay exit code expected 0 got 0 -[collector stop] Stopped relay in stdout expected yes got yes -[collector stop] worker and relay started again expected yes got yes -[collector stop] LB ready expected yes got yes -[collector stop] API record reaches Loki again expected yes got yes -[collector stop] API trace reaches Tempo again expected yes got yes -[collector stop] worker record reaches Loki again expected yes got yes -[collector stop] relay producer span reaches Tempo again expected yes got yes -[collector stop] relay metrics scraped again expected yes got yes -[collector stop] API metrics scraped again expected yes got yes -[collector stop] worker metrics scraped again expected yes got yes -[collector pause] make test expected 0 PASS got 0 PASS -[collector pause] /health/ready expected 200 got 200 -[collector pause] upload becomes ready expected yes got yes -[collector pause] own request record in stdout expected yes got yes -[collector pause] SDK records per logger at most 1 expected yes got yes -[collector pause] docker stop api faster than 10 s expected yes got yes -[collector pause] api exit code expected 0 got 0 -[collector pause] Stopped API in stdout expected yes got yes -[collector pause] stop worker faster than 10 s expected yes got yes -[collector pause] worker exit code expected 0 got 0 -[collector pause] Stopped worker in stdout expected yes got yes -[collector pause] relay start time and restarts unchanged expected yes got yes -[collector pause] stop relay faster than 10 s expected yes got yes -[collector pause] relay exit code expected 0 got 0 -[collector pause] Stopped relay in stdout expected yes got yes -[collector pause] worker and relay started again expected yes got yes -[collector pause] LB ready expected yes got yes -[collector pause] API record reaches Loki again expected yes got yes -[collector pause] API trace reaches Tempo again expected yes got yes -[collector pause] worker record reaches Loki again expected yes got yes -[collector pause] relay producer span reaches Tempo again expected yes got yes -[collector pause] relay metrics scraped again expected yes got yes -[collector pause] API metrics scraped again expected yes got yes -[collector pause] worker metrics scraped again expected yes got yes -[telemetry down] app services depend on telemetry expected [] got [] -[telemetry down] docker compose restart api worker relay expected 0 got 0 -[telemetry down] LB ready expected yes got yes -[telemetry down] replicas started expected 2 got 2 -[telemetry down] worker started expected yes got yes -[telemetry down] relay started expected yes got yes -[telemetry down] make test expected 0 PASS got 0 PASS -[telemetry down] Grafana back expected yes got yes -[telemetry down] API record reaches Loki again expected yes got yes -[telemetry down] API trace reaches Tempo again expected yes got yes -[telemetry down] worker record reaches Loki again expected yes got yes -[telemetry down] relay producer span reaches Tempo again expected yes got yes -[telemetry down] relay metrics scraped again expected yes got yes -[telemetry down] API metrics scraped again expected yes got yes -[telemetry down] worker metrics scraped again expected yes got yes -PASS: API, worker and relay work and stop in time without the Collector, signals come back without restarts +[collector stop] make test expected 0 PASS got 0 PASS +[collector stop] /health/ready expected 200 got 200 +[collector stop] upload becomes ready expected yes got yes +[collector stop] own request record in stdout expected yes got yes +[collector stop] SDK records per logger at most 1 expected yes got yes +[collector stop] API record reaches Loki again expected yes got yes +[collector stop] API trace reaches Tempo again expected yes got yes +[collector stop] worker record reaches Loki again expected yes got yes +[collector stop] relay producer span reaches Tempo again expected yes got yes +[collector stop] relay metrics scraped again expected yes got yes +[collector stop] API metrics scraped again expected yes got yes +[collector stop] worker metrics scraped again expected yes got yes +[collector stop] api, worker and relay start time and restarts unchanged expected yes got yes +[collector stop] make test before stops expected 0 PASS got 0 PASS +[collector stop] docker stop api faster than 10 s expected yes got yes +[collector stop] api exit code expected 0 got 0 +[collector stop] Stopped API in stdout expected yes got yes +[collector stop] stop worker faster than 10 s expected yes got yes +[collector stop] worker exit code expected 0 got 0 +[collector stop] Stopped worker in stdout expected yes got yes +[collector stop] stop relay faster than 10 s expected yes got yes +[collector stop] relay exit code expected 0 got 0 +[collector stop] Stopped relay in stdout expected yes got yes +[collector stop] worker and relay started again expected yes got yes +[collector stop] LB ready expected yes got yes +[collector pause] make test expected 0 PASS got 0 PASS +[collector pause] /health/ready expected 200 got 200 +[collector pause] upload becomes ready expected yes got yes +[collector pause] own request record in stdout expected yes got yes +[collector pause] SDK records per logger at most 1 expected yes got yes +[collector pause] API record reaches Loki again expected yes got yes +[collector pause] API trace reaches Tempo again expected yes got yes +[collector pause] worker record reaches Loki again expected yes got yes +[collector pause] relay producer span reaches Tempo again expected yes got yes +[collector pause] relay metrics scraped again expected yes got yes +[collector pause] API metrics scraped again expected yes got yes +[collector pause] worker metrics scraped again expected yes got yes +[collector pause] api, worker and relay start time and restarts unchanged expected yes got yes +[collector pause] make test before stops expected 0 PASS got 0 PASS +[collector pause] docker stop api faster than 10 s expected yes got yes +[collector pause] api exit code expected 0 got 0 +[collector pause] Stopped API in stdout expected yes got yes +[collector pause] stop worker faster than 10 s expected yes got yes +[collector pause] worker exit code expected 0 got 0 +[collector pause] Stopped worker in stdout expected yes got yes +[collector pause] stop relay faster than 10 s expected yes got yes +[collector pause] relay exit code expected 0 got 0 +[collector pause] Stopped relay in stdout expected yes got yes +[collector pause] worker and relay started again expected yes got yes +[collector pause] LB ready expected yes got yes +[telemetry down] app services depend on telemetry expected [] got [] +[telemetry down] docker compose restart api worker relay expected 0 got 0 +[telemetry down] LB ready expected yes got yes +[telemetry down] replicas started expected 2 got 2 +[telemetry down] worker started expected yes got yes +[telemetry down] relay started expected yes got yes +[telemetry down] make test expected 0 PASS got 0 PASS +[telemetry down] Grafana back expected yes got yes +[telemetry down] API record reaches Loki again expected yes got yes +[telemetry down] API trace reaches Tempo again expected yes got yes +[telemetry down] worker record reaches Loki again expected yes got yes +[telemetry down] relay producer span reaches Tempo again expected yes got yes +[telemetry down] relay metrics scraped again expected yes got yes +[telemetry down] API metrics scraped again expected yes got yes +[telemetry down] worker metrics scraped again expected yes got yes +PASS: API, worker and relay work and stop in time without the Collector; after it returns their signals come back without restarts; restarted without telemetry services, they start and work restored: telemetry services started, replicas and worker running: yes ``` diff --git a/evidence/telemetry/trace-chain.md b/evidence/telemetry/trace-chain.md index 3728a15..2b06dd0 100644 --- a/evidence/telemetry/trace-chain.md +++ b/evidence/telemetry/trace-chain.md @@ -1,12 +1,14 @@ -# `driver.py --port 8080 trace-chain` (change `job-trace-propagation`, задача 12.1) +# `driver.py --port 8080 trace-chain` (change `job-trace-propagation`, задачи 12.1 и 13.11) -- Основа: commit `72cfa9a` (feat: propagate trace context through outbox and add pipeline saturation - metrics), рабочее дерево чистое (`git status --short` пуст до прогона; после — только новый файл - `evidence/`). Файлы `src/` и `migrations/` в контейнерах `api` (2), `relay`, `worker` и `cleaner` - побайтно совпадают с деревом commit (SHA-256 каждого файла); ревизия схемы `e4b8c2d17f90`; - Grafana запущена после последней правки dashboard JSON. -- Дата: 2026-11-11, 16:55:56Z – 16:59:00Z (~3 мин); рабочий стенд `mtusi-lab6` (две реплики API, - relay, worker, уборщик, Collector, Prometheus, Loki, Tempo, Grafana), команда из корня репозитория: +- Основа: commit `72e2598` (fix: address job-trace-propagation verify findings), рабочее дерево + чистое (`git status --short` пуст до прогона; после — только файлы `evidence/`). Файлы `src/` и + `migrations/` (151) в контейнерах `api` (2), `relay`, `worker` и `cleaner` побайтно совпадают с + деревом commit (SHA-256 каждого файла); ревизия схемы `e4b8c2d17f90`; Grafana запущена + (17:55:46Z, рестарт в `telemetry-outage`) после последней правки dashboard JSON. Прежний прогон + этого файла — на `72cfa9a`, до правок verify. +- Дата: 2026-11-11, 17:56:20Z – 17:59:11Z (~3 мин), сразу после `telemetry-outage`; рабочий стенд + `mtusi-lab6` (две реплики API, relay, worker, уборщик, Collector, Prometheus, Loki, Tempo, + Grafana), команда из корня репозитория: `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 trace-chain`, код выхода 0. - Всё — через HTTP API Grafana на `127.0.0.1:3000` с паролем из `.env` (источники-прокси uid `prometheus`, `loki`, `tempo`); PostgreSQL — только чтение, кроме одного `UPDATE` контекста своего @@ -18,16 +20,22 @@ сокета Docker», «Панели уровня 4 после трафика», «Время до готовности в метрике» (шаг 5), «Проверка цепочки» (команда целиком); `service-logging` — «Записи задания ведут к трассе» (записи worker по F несут `trace_id` T записи «Uploaded attachment» запроса R). +- С правками verify (design, «Уточнения apply», п. 19, 22, 23): предусловие — глубина очереди 0 + с ожиданием до 120 с (здесь дождалась сразу; не дождалась бы — код 2 «stand is not quiet»); CPU + каждого контейнера `api`, `worker`, `relay` вырос за прогон (`increase(container_cpu_time_seconds_total[161s])` + > 0 — сценарий «Пределы как в compose», «CPU растёт под нагрузкой»); панель пула с двумя + единицами (ожидающие и отказы в секунду) — среди 19 запросов панелей уровня 4 без ошибок и с + данными. - Шаги 2 и 3: Tempo 3.1 собрал span producer и consumer разных попыток (повтор после `pause s3`, - replay) в одной трассе загрузки (design, «Уточнения apply», п. 7). Время до `ready`: worker перезапускался дважды, проверка складывает «тихие» значения двух - процессов (п. 8). + replay) в одной трассе загрузки (п. 7). Время до `ready`: worker перезапускался дважды, проверка + складывает «тихие» значения двух процессов (п. 8). - Расхождений с design нет. Ход прогона: ```text -2026-11-11T16:55:56Z -queue depth before (no unfinished jobs): 0 +queue depth 0 waited: 0 s +queue depth before reaches 0 within 120 s (no unfinished jobs): yes docker compose stop worker upload queue0: 202 upload queue1: 202 @@ -53,7 +61,7 @@ root: server span of the upload: POST /api/v1/chats/{chat_id}/attachments messen producer spans: 2 consumer spans: 2 producers: messenger-relay, children of the root: yes -producer service.instance.id = relay instance_id: f1a1d8fef0bc +producer service.instance.id = relay instance_id: 0ac09c58168f producer attempts: [1, 2] retry reason of the second producer: storage_unavailable consumers: messenger-worker, parent and link = own producer: yes @@ -82,86 +90,96 @@ pool max: API replicas, worker, relay: ['10', '10'] ['2'] ['1'] pools with used above max: 0 worker capacity: 2 queue series only from relay: ['messenger-relay'] -api a4a1f203ef70 limits = docker inspect: 1 536870912 -api a4a1f203ef70 memory below the limit: yes -api 48b425abbd15 limits = docker inspect: 1 536870912 -api 48b425abbd15 memory below the limit: yes -worker b221c6088b4b limits = docker inspect: 1 1073741824 -worker b221c6088b4b memory below the limit: yes -relay f1a1d8fef0bc limits = docker inspect: 0.25 268435456 -relay f1a1d8fef0bc memory below the limit: yes +api 820012004c61 limits = docker inspect: 1 536870912 +api 820012004c61 memory below the limit: yes +api 820012004c61 CPU grows during the run (increase over 161 s > 0): yes +api 6c86638cdbce limits = docker inspect: 1 536870912 +api 6c86638cdbce memory below the limit: yes +api 6c86638cdbce CPU grows during the run (increase over 161 s > 0): yes +worker d04ac10e81df limits = docker inspect: 1 1073741824 +worker d04ac10e81df memory below the limit: yes +worker d04ac10e81df CPU grows during the run (increase over 161 s > 0): yes +relay 0ac09c58168f limits = docker inspect: 0.25 268435456 +relay 0ac09c58168f memory below the limit: yes +relay 0ac09c58168f CPU grows during the run (increase over 161 s > 0): yes Docker socket mounted by any service: no dashboard loaded: 200 level 4 panel queries (19) without error: [] level 4 panel queries with data: [] waiting for the time-to-ready histogram to settle time to ready count grew by the uploads without replay: 5 -time to ready sum grew by at least 3 x 41 s (worker stopped): yes +time to ready sum grew by at least 3 x 32 s (worker stopped): yes ``` Итог («with data: []» — список панелей без данных пуст): ```text -queue depth before (no unfinished jobs) expected 0 got 0 -upload queue0 expected 202 got 202 -upload queue1 expected 202 got 202 -upload queue2 expected 202 got 202 -outbox + published with worker stopped expected 3 got 3 -oldest age not less than time since first upload - 35 s expected yes got yes -oldest age grows expected yes got yes -all queued attachments ready after worker start expected yes got yes -queue depth and age 0 after ready expected yes got yes -upload retry expected 202 got 202 -job of F published with worker stopped expected yes got yes -worker stdout: Retrying job of F (storage_unavailable) expected storage_unavailable got storage_unavailable -F becomes ready after S3 is back expected ready got ready -Loki: Uploaded attachment of R has trace_id T expected yes got yes -worker records of F carry trace_id T expected yes got yes -root: server span of the upload expected POST /api/v1/chats/{chat_id}/attachments messenger-api got POST /api/v1/chats/{chat_id}/attachments messenger-api -producer spans expected 2 got 2 -consumer spans expected 2 got 2 -producers: messenger-relay, children of the root expected yes got yes -producer service.instance.id = relay instance_id expected f1a1d8fef0bc got f1a1d8fef0bc -producer attempts expected [1, 2] got [1, 2] -retry reason of the second producer expected storage_unavailable got storage_unavailable -consumers: messenger-worker, parent and link = own producer expected yes got yes -consumer outcomes and attempts expected [('retry', 1), ('ready', 2)] got [('retry', 1), ('ready', 2)] -retry consumer reason expected storage_unavailable got storage_unavailable -retry consumer: S3 span with error expected yes got yes -ready consumer: S3.PutObject >= 4, render, COMMIT expected yes got yes -undecodable JPEG generated expected yes got yes -upload replay expected 202 got 202 -undecodable JPEG becomes failed expected failed got failed -replay --apply exit code expected 0 got 0 -replayed job failed again expected yes got yes -producer span with messenger.job.replayed expected 1 got 1 -consumer span of the replayed attempt (failed) expected failed got failed -two consumer spans failed in the upload trace expected 2 got 2 -upload plain expected 202 got 202 -context of one job removed (UPDATE rows) expected 1 got 1 -job without context becomes ready expected ready got ready -worker trace_id differs from the upload trace expected yes got yes -its root is the consumer span expected SPAN_KIND_CONSUMER messenger-worker got SPAN_KIND_CONSUMER messenger-worker -producer spans in the worker trace expected 0 got 0 -producer and consumer spans in the upload trace expected 0 got 0 -pool max: API replicas, worker, relay expected ['10', '10'] ['2'] ['1'] got ['10', '10'] ['2'] ['1'] -pools with used above max expected 0 got 0 -worker capacity expected 2 got 2 -queue series only from relay expected ['messenger-relay'] got ['messenger-relay'] -api a4a1f203ef70 limits = docker inspect expected 1 536870912 got 1 536870912 -api a4a1f203ef70 memory below the limit expected yes got yes -api 48b425abbd15 limits = docker inspect expected 1 536870912 got 1 536870912 -api 48b425abbd15 memory below the limit expected yes got yes -worker b221c6088b4b limits = docker inspect expected 1 1073741824 got 1 1073741824 -worker b221c6088b4b memory below the limit expected yes got yes -relay f1a1d8fef0bc limits = docker inspect expected 0.25 268435456 got 0.25 268435456 -relay f1a1d8fef0bc memory below the limit expected yes got yes -Docker socket mounted by any service expected no got no -dashboard loaded expected 200 got 200 -level 4 panel queries (19) without error expected [] got [] -level 4 panel queries with data expected [] got [] -time to ready count grew by the uploads without replay expected 5 got 5 -time to ready sum grew by at least 3 x 41 s (worker stopped) expected yes got yes +queue depth before reaches 0 within 120 s (no unfinished jobs) expected yes got yes +upload queue0 expected 202 got 202 +upload queue1 expected 202 got 202 +upload queue2 expected 202 got 202 +outbox + published with worker stopped expected 3 got 3 +oldest age not less than time since first upload - 35 s expected yes got yes +oldest age grows expected yes got yes +all queued attachments ready after worker start expected yes got yes +queue depth and age 0 after ready expected yes got yes +upload retry expected 202 got 202 +job of F published with worker stopped expected yes got yes +worker stdout: Retrying job of F (storage_unavailable) expected storage_unavailable got storage_unavailable +F becomes ready after S3 is back expected ready got ready +Loki: Uploaded attachment of R has trace_id T expected yes got yes +worker records of F carry trace_id T expected yes got yes +root: server span of the upload expected POST /api/v1/chats/{chat_id}/attachments messenger-api got POST /api/v1/chats/{chat_id}/attachments messenger-api +producer spans expected 2 got 2 +consumer spans expected 2 got 2 +producers: messenger-relay, children of the root expected yes got yes +producer service.instance.id = relay instance_id expected 0ac09c58168f got 0ac09c58168f +producer attempts expected [1, 2] got [1, 2] +retry reason of the second producer expected storage_unavailable got storage_unavailable +consumers: messenger-worker, parent and link = own producer expected yes got yes +consumer outcomes and attempts expected [('retry', 1), ('ready', 2)] got [('retry', 1), ('ready', 2)] +retry consumer reason expected storage_unavailable got storage_unavailable +retry consumer: S3 span with error expected yes got yes +ready consumer: S3.PutObject >= 4, render, COMMIT expected yes got yes +undecodable JPEG generated expected yes got yes +upload replay expected 202 got 202 +undecodable JPEG becomes failed expected failed got failed +replay --apply exit code expected 0 got 0 +replayed job failed again expected yes got yes +producer span with messenger.job.replayed expected 1 got 1 +consumer span of the replayed attempt (failed) expected failed got failed +two consumer spans failed in the upload trace expected 2 got 2 +upload plain expected 202 got 202 +context of one job removed (UPDATE rows) expected 1 got 1 +job without context becomes ready expected ready got ready +worker trace_id differs from the upload trace expected yes got yes +its root is the consumer span expected SPAN_KIND_CONSUMER messenger-worker got SPAN_KIND_CONSUMER messenger-worker +producer spans in the worker trace expected 0 got 0 +producer and consumer spans in the upload trace expected 0 got 0 +pool max: API replicas, worker, relay expected ['10', '10'] ['2'] ['1'] got ['10', '10'] ['2'] ['1'] +pools with used above max expected 0 got 0 +worker capacity expected 2 got 2 +queue series only from relay expected ['messenger-relay'] got ['messenger-relay'] +api 820012004c61 limits = docker inspect expected 1 536870912 got 1 536870912 +api 820012004c61 memory below the limit expected yes got yes +api 820012004c61 CPU grows during the run (increase over 161 s > 0) expected yes got yes +api 6c86638cdbce limits = docker inspect expected 1 536870912 got 1 536870912 +api 6c86638cdbce memory below the limit expected yes got yes +api 6c86638cdbce CPU grows during the run (increase over 161 s > 0) expected yes got yes +worker d04ac10e81df limits = docker inspect expected 1 1073741824 got 1 1073741824 +worker d04ac10e81df memory below the limit expected yes got yes +worker d04ac10e81df CPU grows during the run (increase over 161 s > 0) expected yes got yes +relay 0ac09c58168f limits = docker inspect expected 0.25 268435456 got 0.25 268435456 +relay 0ac09c58168f memory below the limit expected yes got yes +relay 0ac09c58168f CPU grows during the run (increase over 161 s > 0) expected yes got yes +Docker socket mounted by any service expected no got no +dashboard loaded expected 200 got 200 +level 4 panel queries (19) without error expected [] got [] +level 4 panel queries with data expected [] got [] +time to ready count grew by the uploads without replay expected 5 got 5 +time to ready sum grew by at least 3 x 32 s (worker stopped) expected yes got yes PASS: one upload is one trace: producer and consumer per attempt, retry and replay in it, level 4 signals found restored: s3 unpaused, worker and relay running: yes ``` + +Код выхода — 0. diff --git a/openspec/changes/job-trace-propagation/tasks.md b/openspec/changes/job-trace-propagation/tasks.md index e9d5179..c748ae4 100644 --- a/openspec/changes/job-trace-propagation/tasks.md +++ b/openspec/changes/job-trace-propagation/tasks.md @@ -237,7 +237,7 @@ evidence — после коммита этих правок основной с `make rolling_update`), `make test`, `make student_test` (красный только `test_exclusion_evidence_exists`), `docker compose config --quiet`, пробные `trace-chain` и `telemetry-outage` (вывод в scratchpad), `openspec validate job-trace-propagation --strict`. -- [ ] 13.11 Пересъёмка evidence (после коммита правок основной сессией): `driver.py --port 8080 telemetry-outage` → +- [x] 13.11 Пересъёмка evidence (после коммита правок основной сессией): `driver.py --port 8080 telemetry-outage` → `evidence/telemetry/collector-outage.md` (итоговая строка без ложного «without restarts»), `driver.py --port 8080 trace-chain` → `evidence/telemetry/trace-chain.md` (затронут: предусловие, CPU, панель пула); ссылки commit в `REPORT.md` и `AGENTS.md`.