docs: re-capture trace-chain and collector-outage evidence on 72e2598
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -32,9 +32,10 @@ relay и consumer span worker (parent и link) — одна трасса на з
|
||||
(`CountingQueuePool`), насыщение worker, CPU и память контейнеров из cgroup
|
||||
(без `docker.sock`); панели уровня 4 в том же dashboard; команда драйвера
|
||||
`trace-chain`. Код — commit `72cfa9a`, evidence снято на нём:
|
||||
`evidence/telemetry/trace-chain.md`, `check.md`, `collector-outage.md`,
|
||||
`queue-snapshot.md`, раздел change в `budget.md`,
|
||||
`evidence/migrations/job-trace-context.md`. Следующий — `slo-alerts` (SLO,
|
||||
`evidence/telemetry/check.md`, `queue-snapshot.md`, раздел change в
|
||||
`budget.md`, `evidence/migrations/job-trace-context.md`; правки verify —
|
||||
commit `72e2598`, на нём пересняты `evidence/telemetry/trace-chain.md` и
|
||||
`collector-outage.md`. Следующий — `slo-alerts` (SLO,
|
||||
alerts, runbooks, инциденты).
|
||||
Код перенесён из лабы 5 (репозиторий
|
||||
`mtusi2026lab5-safronov`, ветка `solution/bvt2301-safronov`, commit
|
||||
@@ -715,7 +716,7 @@ evidence/attachments/ # change async-attachments: defense.md (driver.py d
|
||||
evidence/s3/ # privacy.md — make s3_check на текущем коде; change s3-storage: bootstrap.md (первый запуск, повтор, смена секрета, ошибка шага, до bootstrap), privacy.md (make s3_check, нарушение и исправление, порты, секреты по сервисам), grpc-admin.md (gRPC шлюза без токена), restart.md (restart.sh и SIGKILL S3), client.md (операции, pause/stop, неверный секрет, журнал), memory.md (20 × 10 MiB PUT); change upload-consistency: cleaner-key.md (пользователь уборщика в bootstrap, смена секрета, make s3_check с ключом уборщика, секреты по сервисам); change track2-reconciler: cleaner-list.md (bootstrap с политикой списка по префиксу, неверный префикс, make s3_check со строками списка уборщика, окружение s3-bootstrap)
|
||||
evidence/reconcile/ # change track2-reconciler: cycle.md (driver.py reconcile-cycle), stress.md (driver.py reconcile-stress), service.md (Started cleaner с параметрами сверки, плановая сверка при старте, docker stop во время сверки при pause s3, код 1 разового прохода по сигналу — driver.py reconcile-signal, отказ старта на RECONCILER__*, коды 3 и 0; с change track2-reconcile-cost — остановка во время паузы предела скорости, driver.py reconcile-signal --pace)
|
||||
evidence/reconcile/<метка>/ # результаты make bench_reconcile (change track2-reconcile-cost): meta.json (раздел reconcile: план, этапы и модели, commit и образы «до», аргументы S3), passes.csv, windows.csv, summary.md (H1–H4 и S4), cost-by-n.svg, requests-by-n.svg, background-p95.svg, raw/ (записи уборщика по проходам, итог k6 блока H4)
|
||||
evidence/telemetry/ # change otel-telemetry: budget.md (пики и лимиты сервисов телеметрии, реплик и worker с телеметрией и без); на commit 72cfa9a change job-trace-propagation переснято: check.md (driver.py telemetry-check), collector-outage.md (driver.py telemetry-outage, с relay); новое: trace-chain.md (driver.py trace-chain), queue-snapshot.md (EXPLAIN снимка очереди), раздел change в budget.md (relay с SDK, уборщик после ленивого импорта)
|
||||
evidence/telemetry/ # change otel-telemetry: budget.md (пики и лимиты сервисов телеметрии, реплик и worker с телеметрией и без); на commit 72cfa9a change job-trace-propagation переснято: check.md (driver.py telemetry-check); на 72e2598 (правки verify) — collector-outage.md (driver.py telemetry-outage, с relay, сигналы без перезапуска) и новое trace-chain.md (driver.py trace-chain); новое на 72cfa9a: queue-snapshot.md (EXPLAIN снимка очереди), раздел change в budget.md (relay с SDK, уборщик после ленивого импорта)
|
||||
evidence/migrations/ # change job-trace-propagation: job-trace-context.md — ревизия e4b8c2d17f90 (временная база: с нуля, повтор, откат, CHECK, lock_timeout; --sql; рабочий стенд: откат реплик на ff0073e с MIGRATE=0, ревизия до и после make rolling_update)
|
||||
evidence/stand/ # change otel-telemetry: from-scratch.md (driver.py stand-from-scratch --runs 2)
|
||||
evidence/storage/ # отказы БД, пул, statement_timeout, рестарт, порты Postgres, свежий clone; connection-cost.md — цена нового соединения и пинга; freeze-sessions.md — change unit-of-work: сессии и журнал пула после заморозки PostgreSQL (driver.py freeze-sessions и freeze-lb)
|
||||
|
||||
@@ -19,7 +19,9 @@ Change `otel-telemetry` (design — `openspec/changes/archive/2026-11-11-otel-te
|
||||
номера решений в скобках). Числа и проверки сняты на закоммиченном коде при чистом рабочем дереве:
|
||||
`telemetry-check` и `telemetry-outage` сначала на `68f1e97` (правки verify), затем пересняты на
|
||||
`72cfa9a` (код change `job-trace-propagation`: relay с SDK, шаги задания по consumer span), там
|
||||
же `trace-chain`, план снимка очереди, проба ревизии `e4b8c2d17f90` и раздел change бюджета;
|
||||
же план снимка очереди, проба ревизии `e4b8c2d17f90` и раздел change бюджета; `trace-chain` и
|
||||
`telemetry-outage` — ещё раз на `72e2598` (правки verify: сигналы relay без перезапуска, тихая
|
||||
очередь в предусловии, рост CPU каждого процесса);
|
||||
бюджет сервисов телеметрии — на `4178b3d` (лимиты Tempo и Grafana по замеру), шапки файлов
|
||||
`evidence/telemetry/` и `evidence/migrations/`. `evidence/stand/from-
|
||||
scratch.md` ещё снят на рабочем дереве поверх `9b35c25`: пересъёмка ждёт разрешения пользователя
|
||||
@@ -286,10 +288,10 @@ replay, консоль, `make test_isolated`, стенд отказов и `benc
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Уровень 3: Collector, хранилища, provisioning, свой запрос и задание найдены, приватность | `driver.py --port 8080 telemetry-check` | код 0: Loki → Tempo по `X-Request-Id`, трасса worker, рост счётчиков, история после рестарта хранилищ | `evidence/telemetry/check.md` |
|
||||
| Телеметрия не меняет ответы и остановку | `driver.py --port 8080 telemetry-outage` | код 0: `make test` PASS и остановка < 10 с при остановленном и замороженном Collector | `evidence/telemetry/collector-outage.md` |
|
||||
| Телеметрия не меняет ответы и остановку | `driver.py --port 8080 telemetry-outage` | код 0: `make test` PASS и остановка < 10 с при остановленном и замороженном Collector, сигналы после его возврата без перезапуска процессов | `evidence/telemetry/collector-outage.md` (commit `72e2598`) |
|
||||
| Стенд с нуля одной командой | `driver.py stand-from-scratch --runs 2` | код 0 оба раза, `RestartCount` брокера и телеметрии 0 | `evidence/stand/from-scratch.md` |
|
||||
| Лимиты и бюджет телеметрии | замер `docker stats`, `memory.peak` и `memory.stat` | лимит ≥ 1,5 × пика у всех сервисов телеметрии (у Grafana пик — `anon` + `kernel` + `shmem` без файлового кэша) | `evidence/telemetry/budget.md` |
|
||||
| Уровень 4: контекст W3C API → relay → worker, producer и consumer span (parent и link), цепочка одной загрузки до S3 и БД с повтором и replay | `driver.py --port 8080 trace-chain` | код 0: одна трасса, две пары producer / consumer, span S3 с ошибкой и запись оригинала и вариантов, replay в той же трассе, задание без контекста — корень worker | `evidence/telemetry/trace-chain.md` (commit `72cfa9a`) |
|
||||
| Уровень 4: контекст W3C API → relay → worker, producer и consumer span (parent и link), цепочка одной загрузки до S3 и БД с повтором и replay | `driver.py --port 8080 trace-chain` | код 0: одна трасса, две пары producer / consumer, span S3 с ошибкой и запись оригинала и вариантов, replay в той же трассе, задание без контекста — корень worker | `evidence/telemetry/trace-chain.md` (commit `72e2598`) |
|
||||
| Уровень 4: dashboard — p50 / p95 / p99, доля ошибок, глубина и возраст очереди, пул БД, насыщение worker, контейнеры | `driver.py --port 8080 trace-chain`, `telemetry-check` | запросы панелей уровня 4 без ошибок и с данными; ряды очереди только у relay; пределы = `docker inspect`; нет сокета Docker | `evidence/telemetry/trace-chain.md`, `evidence/telemetry/check.md` |
|
||||
| Уровень 4: relay с SDK не меняет остановку | `driver.py --port 8080 telemetry-outage` | `docker stop` relay < 10 с, код 0, «Stopped relay» при остановленном и замороженном Collector | `evidence/telemetry/collector-outage.md` |
|
||||
| Уровень 4: схема контекста задания | миграция с нуля, повтор, откат, `--sql`, `make rolling_update`; план снимка очереди | `lock_timeout`, `NOT VALID`; без Seq Scan | `evidence/migrations/job-trace-context.md`, `evidence/telemetry/queue-snapshot.md` (commit `72cfa9a`) |
|
||||
|
||||
@@ -1,118 +1,194 @@
|
||||
# `driver.py --port 8080 telemetry-outage` (change `otel-telemetry`, задача 11.2; change `job-trace-propagation`, задача 12.2)
|
||||
# `driver.py --port 8080 telemetry-outage` (change `otel-telemetry`, задача 11.2; change `job-trace-propagation`, задачи 12.2 и 13.11)
|
||||
|
||||
- Основа: commit `72cfa9a` (feat: propagate trace context through outbox and add pipeline saturation
|
||||
metrics), рабочее дерево чистое (`git status --short` до прогона пуст; после — только файлы
|
||||
`evidence/`). Файлы `src/` и `migrations/` в контейнерах `api` (2), `relay`, `worker` и `cleaner`
|
||||
побайтно совпадают с деревом commit (SHA-256 каждого файла). Прежний прогон этого файла — на
|
||||
`68f1e97` (до change `job-trace-propagation`, без relay).
|
||||
- Дата: 2026-11-11, 17:01:52Z – 17:04:11Z (~2,3 мин); рабочий стенд `mtusi-lab6`, команда из корня:
|
||||
- Основа: commit `72e2598` (fix: address job-trace-propagation verify findings), рабочее дерево
|
||||
чистое (`git status --short` до прогона пуст; после — только файлы `evidence/`). Файлы `src/` и
|
||||
`migrations/` (151) в контейнерах `api` (2), `relay`, `worker` и `cleaner` побайтно совпадают с
|
||||
деревом commit (SHA-256 каждого файла). Прежние прогоны этого файла — на `68f1e97` (до change
|
||||
`job-trace-propagation`, без relay) и на `72cfa9a` (до правок verify: сигналы relay после
|
||||
возврата Collector приходили от перезапущенного контейнера).
|
||||
- Дата: 2026-11-11, 17:53:55Z – 17:56:10Z (~2,3 мин); рабочий стенд `mtusi-lab6`, команда из корня:
|
||||
`python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 telemetry-outage`, код выхода 0.
|
||||
- Сценарии: `observability` «Collector остановлен», «Collector заморожен», «Collector недоступен для
|
||||
relay»; `service-logging` «Отказ отправки журнала»; `container-run` «Приложение без телеметрии»;
|
||||
`graceful-shutdown` «docker stop при остановленном Collector», «docker stop при замороженном
|
||||
Collector» (реплика API, worker и с change `job-trace-propagation` relay).
|
||||
- Ход: на фазу — `docker compose stop` (или `pause`) `otel-collector`; `make test` через LB;
|
||||
`/health/ready`; свой `GET /users/me` и его запись в stdout; загрузка `sample.png` до `ready`;
|
||||
окно не меньше 20 с, чтобы экспорт трёх сигналов успел отказать; записи логгеров
|
||||
`opentelemetry.*` в stdout каждого контейнера `api` и `worker` с начала фазы; `docker stop
|
||||
--timeout 10` одной реплики и `docker compose stop worker`; затем relay: `StartedAt` и
|
||||
`RestartCount` не изменились с начала фазы (загрузка стала `ready`, relay её опубликовал при
|
||||
недоступном Collector), `docker stop` relay тем же замером (время, код выхода, «Stopped relay»);
|
||||
`make replicas`, `docker start` worker и relay; возврат Collector (`start` / `unpause`) — и новые
|
||||
запись API в Loki, трасса API в Tempo, запись worker в Loki, span producer relay в Tempo, ряды
|
||||
API, worker и relay в Prometheus с отметкой времени позже возврата. Последняя фаза — все пять
|
||||
- Ход (с verify W2, design change `job-trace-propagation`, «Уточнения apply», п. 18): на фазу —
|
||||
`StartedAt` и `RestartCount` всех контейнеров `api` (2), `worker` и `relay`; `docker compose stop`
|
||||
(или `pause`) `otel-collector`; `make test` через LB; `/health/ready`; свой `GET /users/me` и его
|
||||
запись в stdout; загрузка `sample.png` до `ready` (relay её опубликовал при недоступном
|
||||
Collector); окно не меньше 20 с, чтобы экспорт трёх сигналов успел отказать; записи логгеров
|
||||
`opentelemetry.*` в stdout каждого контейнера `api` и `worker` с начала фазы. Затем возврат
|
||||
Collector (`start` / `unpause`) — новые запись API в Loki, трасса API в Tempo, запись worker в
|
||||
Loki, span producer relay в Tempo, ряды API, worker и relay в Prometheus с отметкой времени позже
|
||||
возврата — и `StartedAt` / `RestartCount` тех же четырёх контейнеров прежние: сигналы приходят от
|
||||
процессов, переживших недоступность, без перезапуска. Затем второе окно того же вида: Collector
|
||||
снова недоступен, `make test` (свежие данные в буферах SDK), `docker stop --timeout 10` одной
|
||||
реплики, `docker compose stop worker`, `docker stop` relay (время, код выхода, «Stopped …»),
|
||||
`make replicas`, `docker start` worker и relay, возврат Collector. Последняя фаза — все пять
|
||||
сервисов телеметрии остановлены, `docker compose restart api worker relay`, `make test`, затем
|
||||
возврат телеметрии и сигналы снова. В `finally` — `unpause` и `start` телеметрии, `make
|
||||
replicas`, worker и relay.
|
||||
- Время остановки при недоступном Collector: реплика API — 1,71 с (stop) и 1,50 с (pause), worker —
|
||||
0,74 и 0,73 с, relay — 0,64 и 0,63 с; срок закрытия SDK — 1 с у API и окно 0,5 с у worker и relay
|
||||
- Время остановки при недоступном Collector: реплика API — 1,64 с (stop) и 1,66 с (pause), worker —
|
||||
0,72 и 0,71 с, relay — 0,63 и 0,64 с; срок закрытия SDK — 1 с у API и окно 0,5 с у worker и relay
|
||||
(`TELEMETRY_SHUTDOWN_TIMEOUT_SECONDS`, design change `otel-telemetry`, решение 13; design change
|
||||
`job-trace-propagation`, решение 15).
|
||||
- Записи SDK об отказе — по одной на модуль экспортёра (трассы, метрики, логи) на контейнер `api` и
|
||||
`worker` за окно: фильтр пропускает не больше одной в минуту на логгер. В фазе `pause` экспортёр
|
||||
логов к моменту подсчёта ещё не дождался срока отправки — записей по две. Записи SDK relay в
|
||||
этом подсчёте не участвуют.
|
||||
- Ограничение проверки: relay — единственный, и `docker stop` relay (сценарии `graceful-shutdown`)
|
||||
идёт до возврата Collector, поэтому сигналы relay после возврата приходят от того же контейнера
|
||||
после `docker start`, а не от непрерывно живущего процесса (design, «Уточнения apply», п. 9). Что
|
||||
relay переживает недоступный Collector без перезапуска, показывает строка «relay start time and
|
||||
restarts unchanged».
|
||||
`worker` за окно: фильтр пропускает не больше одной в минуту на логгер. В фазе `pause` к моменту
|
||||
подсчёта отказать успели не все экспортёры (запрос к замороженному Collector ждёт срока), и
|
||||
записей меньше — у второй реплики ни одной. Записи SDK relay в этом подсчёте не участвуют.
|
||||
|
||||
```
|
||||
[collector stop] SDK records in 20 s: {'a4a1f203ef70 opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, 'a4a1f203ef70 opentelemetry.exporter.otlp.proto.http._log_exporter': 1, 'a4a1f203ef70 opentelemetry.exporter.otlp.proto.http.metric_exporter': 1, '48b425abbd15 opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, '48b425abbd15 opentelemetry.exporter.otlp.proto.http._log_exporter': 1, '48b425abbd15 opentelemetry.exporter.otlp.proto.http.metric_exporter': 1, 'b221c6088b4b opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, 'b221c6088b4b opentelemetry.exporter.otlp.proto.http._log_exporter': 1, 'b221c6088b4b opentelemetry.exporter.otlp.proto.http.metric_exporter': 1}
|
||||
[collector stop] docker stop api a4a1f203ef70: 1.71 s
|
||||
[collector stop] docker compose stop worker: 0.74 s
|
||||
[collector stop] docker stop relay: 0.64 s
|
||||
[collector pause] SDK records in 20 s: {'a4a1f203ef70 opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, 'a4a1f203ef70 opentelemetry.exporter.otlp.proto.http.metric_exporter': 1, '48b425abbd15 opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, '48b425abbd15 opentelemetry.exporter.otlp.proto.http.metric_exporter': 1, 'b221c6088b4b opentelemetry.exporter.otlp.proto.http.metric_exporter': 1, 'b221c6088b4b opentelemetry.exporter.otlp.proto.http.trace_exporter': 1}
|
||||
[collector pause] docker stop api a4a1f203ef70: 1.50 s
|
||||
[collector pause] docker compose stop worker: 0.73 s
|
||||
[collector pause] docker stop relay: 0.63 s
|
||||
[collector stop] docker compose stop otel-collector
|
||||
[collector stop] make test: 0 PASS
|
||||
[collector stop] /health/ready: 200
|
||||
[collector stop] upload becomes ready: yes
|
||||
[collector stop] own request record in stdout: yes
|
||||
[collector stop] SDK records in 20 s: {'820012004c61 opentelemetry.exporter.otlp.proto.http._log_exporter': 1, '820012004c61 opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, '820012004c61 opentelemetry.exporter.otlp.proto.http.metric_exporter': 1, '6c86638cdbce opentelemetry.exporter.otlp.proto.http._log_exporter': 1, '6c86638cdbce opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, '6c86638cdbce opentelemetry.exporter.otlp.proto.http.metric_exporter': 1, 'd04ac10e81df opentelemetry.exporter.otlp.proto.http._log_exporter': 1, 'd04ac10e81df opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, 'd04ac10e81df opentelemetry.exporter.otlp.proto.http.metric_exporter': 1}
|
||||
[collector stop] SDK records per logger at most 1: yes
|
||||
[collector stop] otel-collector back
|
||||
[collector stop] API record reaches Loki again: yes
|
||||
[collector stop] API trace reaches Tempo again: yes
|
||||
[collector stop] worker record reaches Loki again: yes
|
||||
[collector stop] relay producer span reaches Tempo again: yes
|
||||
[collector stop] relay metrics scraped again: yes
|
||||
[collector stop] API metrics scraped again: yes
|
||||
[collector stop] worker metrics scraped again: yes
|
||||
[collector stop] containers (StartedAt, RestartCount): 4 before, 4 after
|
||||
[collector stop] api, worker and relay start time and restarts unchanged: yes
|
||||
[collector stop] docker compose stop otel-collector (stops)
|
||||
[collector stop] make test before stops: 0 PASS
|
||||
[collector stop] docker stop api 820012004c61: 1.64 s
|
||||
[collector stop] docker stop api faster than 10 s: yes
|
||||
[collector stop] api exit code: 0
|
||||
[collector stop] Stopped API in stdout: yes
|
||||
[collector stop] docker compose stop worker: 0.72 s
|
||||
[collector stop] stop worker faster than 10 s: yes
|
||||
[collector stop] worker exit code: 0
|
||||
[collector stop] Stopped worker in stdout: yes
|
||||
[collector stop] docker stop relay: 0.63 s
|
||||
[collector stop] stop relay faster than 10 s: yes
|
||||
[collector stop] relay exit code: 0
|
||||
[collector stop] Stopped relay in stdout: yes
|
||||
[collector stop] worker and relay started again: yes
|
||||
[collector stop] LB ready: yes
|
||||
[collector stop] otel-collector back (after stops)
|
||||
[collector pause] docker compose pause otel-collector
|
||||
[collector pause] make test: 0 PASS
|
||||
[collector pause] /health/ready: 200
|
||||
[collector pause] upload becomes ready: yes
|
||||
[collector pause] own request record in stdout: yes
|
||||
[collector pause] SDK records in 20 s: {'820012004c61 opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, '820012004c61 opentelemetry.exporter.otlp.proto.http.metric_exporter': 1, 'd04ac10e81df opentelemetry.exporter.otlp.proto.http.trace_exporter': 1, 'd04ac10e81df opentelemetry.exporter.otlp.proto.http.metric_exporter': 1}
|
||||
[collector pause] SDK records per logger at most 1: yes
|
||||
[collector pause] otel-collector back
|
||||
[collector pause] API record reaches Loki again: yes
|
||||
[collector pause] API trace reaches Tempo again: yes
|
||||
[collector pause] worker record reaches Loki again: yes
|
||||
[collector pause] relay producer span reaches Tempo again: yes
|
||||
[collector pause] relay metrics scraped again: yes
|
||||
[collector pause] API metrics scraped again: yes
|
||||
[collector pause] worker metrics scraped again: yes
|
||||
[collector pause] containers (StartedAt, RestartCount): 4 before, 4 after
|
||||
[collector pause] api, worker and relay start time and restarts unchanged: yes
|
||||
[collector pause] docker compose pause otel-collector (stops)
|
||||
[collector pause] make test before stops: 0 PASS
|
||||
[collector pause] docker stop api 820012004c61: 1.66 s
|
||||
[collector pause] docker stop api faster than 10 s: yes
|
||||
[collector pause] api exit code: 0
|
||||
[collector pause] Stopped API in stdout: yes
|
||||
[collector pause] docker compose stop worker: 0.71 s
|
||||
[collector pause] stop worker faster than 10 s: yes
|
||||
[collector pause] worker exit code: 0
|
||||
[collector pause] Stopped worker in stdout: yes
|
||||
[collector pause] docker stop relay: 0.64 s
|
||||
[collector pause] stop relay faster than 10 s: yes
|
||||
[collector pause] relay exit code: 0
|
||||
[collector pause] Stopped relay in stdout: yes
|
||||
[collector pause] worker and relay started again: yes
|
||||
[collector pause] LB ready: yes
|
||||
[collector pause] otel-collector back (after stops)
|
||||
[telemetry down] app services depend on telemetry: []
|
||||
[telemetry down] docker compose stop otel-collector prometheus loki tempo grafana
|
||||
[telemetry down] docker compose restart api worker relay: 0
|
||||
[telemetry down] LB ready: yes
|
||||
[telemetry down] replicas started: 2
|
||||
[telemetry down] worker started: yes
|
||||
[telemetry down] relay started: yes
|
||||
[telemetry down] make test: 0 PASS
|
||||
[telemetry down] Grafana back: yes
|
||||
[telemetry down] API record reaches Loki again: yes
|
||||
[telemetry down] API trace reaches Tempo again: yes
|
||||
[telemetry down] worker record reaches Loki again: yes
|
||||
[telemetry down] relay producer span reaches Tempo again: yes
|
||||
[telemetry down] relay metrics scraped again: yes
|
||||
[telemetry down] API metrics scraped again: yes
|
||||
[telemetry down] worker metrics scraped again: yes
|
||||
|
||||
[collector stop] make test expected 0 PASS got 0 PASS
|
||||
[collector stop] /health/ready expected 200 got 200
|
||||
[collector stop] upload becomes ready expected yes got yes
|
||||
[collector stop] own request record in stdout expected yes got yes
|
||||
[collector stop] SDK records per logger at most 1 expected yes got yes
|
||||
[collector stop] docker stop api faster than 10 s expected yes got yes
|
||||
[collector stop] api exit code expected 0 got 0
|
||||
[collector stop] Stopped API in stdout expected yes got yes
|
||||
[collector stop] stop worker faster than 10 s expected yes got yes
|
||||
[collector stop] worker exit code expected 0 got 0
|
||||
[collector stop] Stopped worker in stdout expected yes got yes
|
||||
[collector stop] relay start time and restarts unchanged expected yes got yes
|
||||
[collector stop] stop relay faster than 10 s expected yes got yes
|
||||
[collector stop] relay exit code expected 0 got 0
|
||||
[collector stop] Stopped relay in stdout expected yes got yes
|
||||
[collector stop] worker and relay started again expected yes got yes
|
||||
[collector stop] LB ready expected yes got yes
|
||||
[collector stop] API record reaches Loki again expected yes got yes
|
||||
[collector stop] API trace reaches Tempo again expected yes got yes
|
||||
[collector stop] worker record reaches Loki again expected yes got yes
|
||||
[collector stop] relay producer span reaches Tempo again expected yes got yes
|
||||
[collector stop] relay metrics scraped again expected yes got yes
|
||||
[collector stop] API metrics scraped again expected yes got yes
|
||||
[collector stop] worker metrics scraped again expected yes got yes
|
||||
[collector pause] make test expected 0 PASS got 0 PASS
|
||||
[collector pause] /health/ready expected 200 got 200
|
||||
[collector pause] upload becomes ready expected yes got yes
|
||||
[collector pause] own request record in stdout expected yes got yes
|
||||
[collector pause] SDK records per logger at most 1 expected yes got yes
|
||||
[collector pause] docker stop api faster than 10 s expected yes got yes
|
||||
[collector pause] api exit code expected 0 got 0
|
||||
[collector pause] Stopped API in stdout expected yes got yes
|
||||
[collector pause] stop worker faster than 10 s expected yes got yes
|
||||
[collector pause] worker exit code expected 0 got 0
|
||||
[collector pause] Stopped worker in stdout expected yes got yes
|
||||
[collector pause] relay start time and restarts unchanged expected yes got yes
|
||||
[collector pause] stop relay faster than 10 s expected yes got yes
|
||||
[collector pause] relay exit code expected 0 got 0
|
||||
[collector pause] Stopped relay in stdout expected yes got yes
|
||||
[collector pause] worker and relay started again expected yes got yes
|
||||
[collector pause] LB ready expected yes got yes
|
||||
[collector pause] API record reaches Loki again expected yes got yes
|
||||
[collector pause] API trace reaches Tempo again expected yes got yes
|
||||
[collector pause] worker record reaches Loki again expected yes got yes
|
||||
[collector pause] relay producer span reaches Tempo again expected yes got yes
|
||||
[collector pause] relay metrics scraped again expected yes got yes
|
||||
[collector pause] API metrics scraped again expected yes got yes
|
||||
[collector pause] worker metrics scraped again expected yes got yes
|
||||
[telemetry down] app services depend on telemetry expected [] got []
|
||||
[telemetry down] docker compose restart api worker relay expected 0 got 0
|
||||
[telemetry down] LB ready expected yes got yes
|
||||
[telemetry down] replicas started expected 2 got 2
|
||||
[telemetry down] worker started expected yes got yes
|
||||
[telemetry down] relay started expected yes got yes
|
||||
[telemetry down] make test expected 0 PASS got 0 PASS
|
||||
[telemetry down] Grafana back expected yes got yes
|
||||
[telemetry down] API record reaches Loki again expected yes got yes
|
||||
[telemetry down] API trace reaches Tempo again expected yes got yes
|
||||
[telemetry down] worker record reaches Loki again expected yes got yes
|
||||
[telemetry down] relay producer span reaches Tempo again expected yes got yes
|
||||
[telemetry down] relay metrics scraped again expected yes got yes
|
||||
[telemetry down] API metrics scraped again expected yes got yes
|
||||
[telemetry down] worker metrics scraped again expected yes got yes
|
||||
PASS: API, worker and relay work and stop in time without the Collector, signals come back without restarts
|
||||
[collector stop] make test expected 0 PASS got 0 PASS
|
||||
[collector stop] /health/ready expected 200 got 200
|
||||
[collector stop] upload becomes ready expected yes got yes
|
||||
[collector stop] own request record in stdout expected yes got yes
|
||||
[collector stop] SDK records per logger at most 1 expected yes got yes
|
||||
[collector stop] API record reaches Loki again expected yes got yes
|
||||
[collector stop] API trace reaches Tempo again expected yes got yes
|
||||
[collector stop] worker record reaches Loki again expected yes got yes
|
||||
[collector stop] relay producer span reaches Tempo again expected yes got yes
|
||||
[collector stop] relay metrics scraped again expected yes got yes
|
||||
[collector stop] API metrics scraped again expected yes got yes
|
||||
[collector stop] worker metrics scraped again expected yes got yes
|
||||
[collector stop] api, worker and relay start time and restarts unchanged expected yes got yes
|
||||
[collector stop] make test before stops expected 0 PASS got 0 PASS
|
||||
[collector stop] docker stop api faster than 10 s expected yes got yes
|
||||
[collector stop] api exit code expected 0 got 0
|
||||
[collector stop] Stopped API in stdout expected yes got yes
|
||||
[collector stop] stop worker faster than 10 s expected yes got yes
|
||||
[collector stop] worker exit code expected 0 got 0
|
||||
[collector stop] Stopped worker in stdout expected yes got yes
|
||||
[collector stop] stop relay faster than 10 s expected yes got yes
|
||||
[collector stop] relay exit code expected 0 got 0
|
||||
[collector stop] Stopped relay in stdout expected yes got yes
|
||||
[collector stop] worker and relay started again expected yes got yes
|
||||
[collector stop] LB ready expected yes got yes
|
||||
[collector pause] make test expected 0 PASS got 0 PASS
|
||||
[collector pause] /health/ready expected 200 got 200
|
||||
[collector pause] upload becomes ready expected yes got yes
|
||||
[collector pause] own request record in stdout expected yes got yes
|
||||
[collector pause] SDK records per logger at most 1 expected yes got yes
|
||||
[collector pause] API record reaches Loki again expected yes got yes
|
||||
[collector pause] API trace reaches Tempo again expected yes got yes
|
||||
[collector pause] worker record reaches Loki again expected yes got yes
|
||||
[collector pause] relay producer span reaches Tempo again expected yes got yes
|
||||
[collector pause] relay metrics scraped again expected yes got yes
|
||||
[collector pause] API metrics scraped again expected yes got yes
|
||||
[collector pause] worker metrics scraped again expected yes got yes
|
||||
[collector pause] api, worker and relay start time and restarts unchanged expected yes got yes
|
||||
[collector pause] make test before stops expected 0 PASS got 0 PASS
|
||||
[collector pause] docker stop api faster than 10 s expected yes got yes
|
||||
[collector pause] api exit code expected 0 got 0
|
||||
[collector pause] Stopped API in stdout expected yes got yes
|
||||
[collector pause] stop worker faster than 10 s expected yes got yes
|
||||
[collector pause] worker exit code expected 0 got 0
|
||||
[collector pause] Stopped worker in stdout expected yes got yes
|
||||
[collector pause] stop relay faster than 10 s expected yes got yes
|
||||
[collector pause] relay exit code expected 0 got 0
|
||||
[collector pause] Stopped relay in stdout expected yes got yes
|
||||
[collector pause] worker and relay started again expected yes got yes
|
||||
[collector pause] LB ready expected yes got yes
|
||||
[telemetry down] app services depend on telemetry expected [] got []
|
||||
[telemetry down] docker compose restart api worker relay expected 0 got 0
|
||||
[telemetry down] LB ready expected yes got yes
|
||||
[telemetry down] replicas started expected 2 got 2
|
||||
[telemetry down] worker started expected yes got yes
|
||||
[telemetry down] relay started expected yes got yes
|
||||
[telemetry down] make test expected 0 PASS got 0 PASS
|
||||
[telemetry down] Grafana back expected yes got yes
|
||||
[telemetry down] API record reaches Loki again expected yes got yes
|
||||
[telemetry down] API trace reaches Tempo again expected yes got yes
|
||||
[telemetry down] worker record reaches Loki again expected yes got yes
|
||||
[telemetry down] relay producer span reaches Tempo again expected yes got yes
|
||||
[telemetry down] relay metrics scraped again expected yes got yes
|
||||
[telemetry down] API metrics scraped again expected yes got yes
|
||||
[telemetry down] worker metrics scraped again expected yes got yes
|
||||
PASS: API, worker and relay work and stop in time without the Collector; after it returns their signals come back without restarts; restarted without telemetry services, they start and work
|
||||
restored: telemetry services started, replicas and worker running: yes
|
||||
```
|
||||
|
||||
|
||||
@@ -1,12 +1,14 @@
|
||||
# `driver.py --port 8080 trace-chain` (change `job-trace-propagation`, задача 12.1)
|
||||
# `driver.py --port 8080 trace-chain` (change `job-trace-propagation`, задачи 12.1 и 13.11)
|
||||
|
||||
- Основа: commit `72cfa9a` (feat: propagate trace context through outbox and add pipeline saturation
|
||||
metrics), рабочее дерево чистое (`git status --short` пуст до прогона; после — только новый файл
|
||||
`evidence/`). Файлы `src/` и `migrations/` в контейнерах `api` (2), `relay`, `worker` и `cleaner`
|
||||
побайтно совпадают с деревом commit (SHA-256 каждого файла); ревизия схемы `e4b8c2d17f90`;
|
||||
Grafana запущена после последней правки dashboard JSON.
|
||||
- Дата: 2026-11-11, 16:55:56Z – 16:59:00Z (~3 мин); рабочий стенд `mtusi-lab6` (две реплики API,
|
||||
relay, worker, уборщик, Collector, Prometheus, Loki, Tempo, Grafana), команда из корня репозитория:
|
||||
- Основа: commit `72e2598` (fix: address job-trace-propagation verify findings), рабочее дерево
|
||||
чистое (`git status --short` пуст до прогона; после — только файлы `evidence/`). Файлы `src/` и
|
||||
`migrations/` (151) в контейнерах `api` (2), `relay`, `worker` и `cleaner` побайтно совпадают с
|
||||
деревом commit (SHA-256 каждого файла); ревизия схемы `e4b8c2d17f90`; Grafana запущена
|
||||
(17:55:46Z, рестарт в `telemetry-outage`) после последней правки dashboard JSON. Прежний прогон
|
||||
этого файла — на `72cfa9a`, до правок verify.
|
||||
- Дата: 2026-11-11, 17:56:20Z – 17:59:11Z (~3 мин), сразу после `telemetry-outage`; рабочий стенд
|
||||
`mtusi-lab6` (две реплики API, relay, worker, уборщик, Collector, Prometheus, Loki, Tempo,
|
||||
Grafana), команда из корня репозитория:
|
||||
`python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 trace-chain`, код выхода 0.
|
||||
- Всё — через HTTP API Grafana на `127.0.0.1:3000` с паролем из `.env` (источники-прокси uid
|
||||
`prometheus`, `loki`, `tempo`); PostgreSQL — только чтение, кроме одного `UPDATE` контекста своего
|
||||
@@ -18,16 +20,22 @@
|
||||
сокета Docker», «Панели уровня 4 после трафика», «Время до готовности в метрике» (шаг 5), «Проверка
|
||||
цепочки» (команда целиком); `service-logging` — «Записи задания ведут к трассе» (записи worker по F
|
||||
несут `trace_id` T записи «Uploaded attachment» запроса R).
|
||||
- С правками verify (design, «Уточнения apply», п. 19, 22, 23): предусловие — глубина очереди 0
|
||||
с ожиданием до 120 с (здесь дождалась сразу; не дождалась бы — код 2 «stand is not quiet»); CPU
|
||||
каждого контейнера `api`, `worker`, `relay` вырос за прогон (`increase(container_cpu_time_seconds_total[161s])`
|
||||
> 0 — сценарий «Пределы как в compose», «CPU растёт под нагрузкой»); панель пула с двумя
|
||||
единицами (ожидающие и отказы в секунду) — среди 19 запросов панелей уровня 4 без ошибок и с
|
||||
данными.
|
||||
- Шаги 2 и 3: Tempo 3.1 собрал span producer и consumer разных попыток (повтор после `pause s3`,
|
||||
replay) в одной трассе загрузки (design, «Уточнения apply», п. 7). Время до `ready`: worker перезапускался дважды, проверка складывает «тихие» значения двух
|
||||
процессов (п. 8).
|
||||
replay) в одной трассе загрузки (п. 7). Время до `ready`: worker перезапускался дважды, проверка
|
||||
складывает «тихие» значения двух процессов (п. 8).
|
||||
- Расхождений с design нет.
|
||||
|
||||
Ход прогона:
|
||||
|
||||
```text
|
||||
2026-11-11T16:55:56Z
|
||||
queue depth before (no unfinished jobs): 0
|
||||
queue depth 0 waited: 0 s
|
||||
queue depth before reaches 0 within 120 s (no unfinished jobs): yes
|
||||
docker compose stop worker
|
||||
upload queue0: 202
|
||||
upload queue1: 202
|
||||
@@ -53,7 +61,7 @@ root: server span of the upload: POST /api/v1/chats/{chat_id}/attachments messen
|
||||
producer spans: 2
|
||||
consumer spans: 2
|
||||
producers: messenger-relay, children of the root: yes
|
||||
producer service.instance.id = relay instance_id: f1a1d8fef0bc
|
||||
producer service.instance.id = relay instance_id: 0ac09c58168f
|
||||
producer attempts: [1, 2]
|
||||
retry reason of the second producer: storage_unavailable
|
||||
consumers: messenger-worker, parent and link = own producer: yes
|
||||
@@ -82,86 +90,96 @@ pool max: API replicas, worker, relay: ['10', '10'] ['2'] ['1']
|
||||
pools with used above max: 0
|
||||
worker capacity: 2
|
||||
queue series only from relay: ['messenger-relay']
|
||||
api a4a1f203ef70 limits = docker inspect: 1 536870912
|
||||
api a4a1f203ef70 memory below the limit: yes
|
||||
api 48b425abbd15 limits = docker inspect: 1 536870912
|
||||
api 48b425abbd15 memory below the limit: yes
|
||||
worker b221c6088b4b limits = docker inspect: 1 1073741824
|
||||
worker b221c6088b4b memory below the limit: yes
|
||||
relay f1a1d8fef0bc limits = docker inspect: 0.25 268435456
|
||||
relay f1a1d8fef0bc memory below the limit: yes
|
||||
api 820012004c61 limits = docker inspect: 1 536870912
|
||||
api 820012004c61 memory below the limit: yes
|
||||
api 820012004c61 CPU grows during the run (increase over 161 s > 0): yes
|
||||
api 6c86638cdbce limits = docker inspect: 1 536870912
|
||||
api 6c86638cdbce memory below the limit: yes
|
||||
api 6c86638cdbce CPU grows during the run (increase over 161 s > 0): yes
|
||||
worker d04ac10e81df limits = docker inspect: 1 1073741824
|
||||
worker d04ac10e81df memory below the limit: yes
|
||||
worker d04ac10e81df CPU grows during the run (increase over 161 s > 0): yes
|
||||
relay 0ac09c58168f limits = docker inspect: 0.25 268435456
|
||||
relay 0ac09c58168f memory below the limit: yes
|
||||
relay 0ac09c58168f CPU grows during the run (increase over 161 s > 0): yes
|
||||
Docker socket mounted by any service: no
|
||||
dashboard loaded: 200
|
||||
level 4 panel queries (19) without error: []
|
||||
level 4 panel queries with data: []
|
||||
waiting for the time-to-ready histogram to settle
|
||||
time to ready count grew by the uploads without replay: 5
|
||||
time to ready sum grew by at least 3 x 41 s (worker stopped): yes
|
||||
time to ready sum grew by at least 3 x 32 s (worker stopped): yes
|
||||
```
|
||||
|
||||
Итог («with data: []» — список панелей без данных пуст):
|
||||
|
||||
```text
|
||||
queue depth before (no unfinished jobs) expected 0 got 0
|
||||
upload queue0 expected 202 got 202
|
||||
upload queue1 expected 202 got 202
|
||||
upload queue2 expected 202 got 202
|
||||
outbox + published with worker stopped expected 3 got 3
|
||||
oldest age not less than time since first upload - 35 s expected yes got yes
|
||||
oldest age grows expected yes got yes
|
||||
all queued attachments ready after worker start expected yes got yes
|
||||
queue depth and age 0 after ready expected yes got yes
|
||||
upload retry expected 202 got 202
|
||||
job of F published with worker stopped expected yes got yes
|
||||
worker stdout: Retrying job of F (storage_unavailable) expected storage_unavailable got storage_unavailable
|
||||
F becomes ready after S3 is back expected ready got ready
|
||||
Loki: Uploaded attachment of R has trace_id T expected yes got yes
|
||||
worker records of F carry trace_id T expected yes got yes
|
||||
root: server span of the upload expected POST /api/v1/chats/{chat_id}/attachments messenger-api got POST /api/v1/chats/{chat_id}/attachments messenger-api
|
||||
producer spans expected 2 got 2
|
||||
consumer spans expected 2 got 2
|
||||
producers: messenger-relay, children of the root expected yes got yes
|
||||
producer service.instance.id = relay instance_id expected f1a1d8fef0bc got f1a1d8fef0bc
|
||||
producer attempts expected [1, 2] got [1, 2]
|
||||
retry reason of the second producer expected storage_unavailable got storage_unavailable
|
||||
consumers: messenger-worker, parent and link = own producer expected yes got yes
|
||||
consumer outcomes and attempts expected [('retry', 1), ('ready', 2)] got [('retry', 1), ('ready', 2)]
|
||||
retry consumer reason expected storage_unavailable got storage_unavailable
|
||||
retry consumer: S3 span with error expected yes got yes
|
||||
ready consumer: S3.PutObject >= 4, render, COMMIT expected yes got yes
|
||||
undecodable JPEG generated expected yes got yes
|
||||
upload replay expected 202 got 202
|
||||
undecodable JPEG becomes failed expected failed got failed
|
||||
replay --apply exit code expected 0 got 0
|
||||
replayed job failed again expected yes got yes
|
||||
producer span with messenger.job.replayed expected 1 got 1
|
||||
consumer span of the replayed attempt (failed) expected failed got failed
|
||||
two consumer spans failed in the upload trace expected 2 got 2
|
||||
upload plain expected 202 got 202
|
||||
context of one job removed (UPDATE rows) expected 1 got 1
|
||||
job without context becomes ready expected ready got ready
|
||||
worker trace_id differs from the upload trace expected yes got yes
|
||||
its root is the consumer span expected SPAN_KIND_CONSUMER messenger-worker got SPAN_KIND_CONSUMER messenger-worker
|
||||
producer spans in the worker trace expected 0 got 0
|
||||
producer and consumer spans in the upload trace expected 0 got 0
|
||||
pool max: API replicas, worker, relay expected ['10', '10'] ['2'] ['1'] got ['10', '10'] ['2'] ['1']
|
||||
pools with used above max expected 0 got 0
|
||||
worker capacity expected 2 got 2
|
||||
queue series only from relay expected ['messenger-relay'] got ['messenger-relay']
|
||||
api a4a1f203ef70 limits = docker inspect expected 1 536870912 got 1 536870912
|
||||
api a4a1f203ef70 memory below the limit expected yes got yes
|
||||
api 48b425abbd15 limits = docker inspect expected 1 536870912 got 1 536870912
|
||||
api 48b425abbd15 memory below the limit expected yes got yes
|
||||
worker b221c6088b4b limits = docker inspect expected 1 1073741824 got 1 1073741824
|
||||
worker b221c6088b4b memory below the limit expected yes got yes
|
||||
relay f1a1d8fef0bc limits = docker inspect expected 0.25 268435456 got 0.25 268435456
|
||||
relay f1a1d8fef0bc memory below the limit expected yes got yes
|
||||
Docker socket mounted by any service expected no got no
|
||||
dashboard loaded expected 200 got 200
|
||||
level 4 panel queries (19) without error expected [] got []
|
||||
level 4 panel queries with data expected [] got []
|
||||
time to ready count grew by the uploads without replay expected 5 got 5
|
||||
time to ready sum grew by at least 3 x 41 s (worker stopped) expected yes got yes
|
||||
queue depth before reaches 0 within 120 s (no unfinished jobs) expected yes got yes
|
||||
upload queue0 expected 202 got 202
|
||||
upload queue1 expected 202 got 202
|
||||
upload queue2 expected 202 got 202
|
||||
outbox + published with worker stopped expected 3 got 3
|
||||
oldest age not less than time since first upload - 35 s expected yes got yes
|
||||
oldest age grows expected yes got yes
|
||||
all queued attachments ready after worker start expected yes got yes
|
||||
queue depth and age 0 after ready expected yes got yes
|
||||
upload retry expected 202 got 202
|
||||
job of F published with worker stopped expected yes got yes
|
||||
worker stdout: Retrying job of F (storage_unavailable) expected storage_unavailable got storage_unavailable
|
||||
F becomes ready after S3 is back expected ready got ready
|
||||
Loki: Uploaded attachment of R has trace_id T expected yes got yes
|
||||
worker records of F carry trace_id T expected yes got yes
|
||||
root: server span of the upload expected POST /api/v1/chats/{chat_id}/attachments messenger-api got POST /api/v1/chats/{chat_id}/attachments messenger-api
|
||||
producer spans expected 2 got 2
|
||||
consumer spans expected 2 got 2
|
||||
producers: messenger-relay, children of the root expected yes got yes
|
||||
producer service.instance.id = relay instance_id expected 0ac09c58168f got 0ac09c58168f
|
||||
producer attempts expected [1, 2] got [1, 2]
|
||||
retry reason of the second producer expected storage_unavailable got storage_unavailable
|
||||
consumers: messenger-worker, parent and link = own producer expected yes got yes
|
||||
consumer outcomes and attempts expected [('retry', 1), ('ready', 2)] got [('retry', 1), ('ready', 2)]
|
||||
retry consumer reason expected storage_unavailable got storage_unavailable
|
||||
retry consumer: S3 span with error expected yes got yes
|
||||
ready consumer: S3.PutObject >= 4, render, COMMIT expected yes got yes
|
||||
undecodable JPEG generated expected yes got yes
|
||||
upload replay expected 202 got 202
|
||||
undecodable JPEG becomes failed expected failed got failed
|
||||
replay --apply exit code expected 0 got 0
|
||||
replayed job failed again expected yes got yes
|
||||
producer span with messenger.job.replayed expected 1 got 1
|
||||
consumer span of the replayed attempt (failed) expected failed got failed
|
||||
two consumer spans failed in the upload trace expected 2 got 2
|
||||
upload plain expected 202 got 202
|
||||
context of one job removed (UPDATE rows) expected 1 got 1
|
||||
job without context becomes ready expected ready got ready
|
||||
worker trace_id differs from the upload trace expected yes got yes
|
||||
its root is the consumer span expected SPAN_KIND_CONSUMER messenger-worker got SPAN_KIND_CONSUMER messenger-worker
|
||||
producer spans in the worker trace expected 0 got 0
|
||||
producer and consumer spans in the upload trace expected 0 got 0
|
||||
pool max: API replicas, worker, relay expected ['10', '10'] ['2'] ['1'] got ['10', '10'] ['2'] ['1']
|
||||
pools with used above max expected 0 got 0
|
||||
worker capacity expected 2 got 2
|
||||
queue series only from relay expected ['messenger-relay'] got ['messenger-relay']
|
||||
api 820012004c61 limits = docker inspect expected 1 536870912 got 1 536870912
|
||||
api 820012004c61 memory below the limit expected yes got yes
|
||||
api 820012004c61 CPU grows during the run (increase over 161 s > 0) expected yes got yes
|
||||
api 6c86638cdbce limits = docker inspect expected 1 536870912 got 1 536870912
|
||||
api 6c86638cdbce memory below the limit expected yes got yes
|
||||
api 6c86638cdbce CPU grows during the run (increase over 161 s > 0) expected yes got yes
|
||||
worker d04ac10e81df limits = docker inspect expected 1 1073741824 got 1 1073741824
|
||||
worker d04ac10e81df memory below the limit expected yes got yes
|
||||
worker d04ac10e81df CPU grows during the run (increase over 161 s > 0) expected yes got yes
|
||||
relay 0ac09c58168f limits = docker inspect expected 0.25 268435456 got 0.25 268435456
|
||||
relay 0ac09c58168f memory below the limit expected yes got yes
|
||||
relay 0ac09c58168f CPU grows during the run (increase over 161 s > 0) expected yes got yes
|
||||
Docker socket mounted by any service expected no got no
|
||||
dashboard loaded expected 200 got 200
|
||||
level 4 panel queries (19) without error expected [] got []
|
||||
level 4 panel queries with data expected [] got []
|
||||
time to ready count grew by the uploads without replay expected 5 got 5
|
||||
time to ready sum grew by at least 3 x 32 s (worker stopped) expected yes got yes
|
||||
PASS: one upload is one trace: producer and consumer per attempt, retry and replay in it, level 4 signals found
|
||||
restored: s3 unpaused, worker and relay running: yes
|
||||
```
|
||||
|
||||
Код выхода — 0.
|
||||
|
||||
@@ -237,7 +237,7 @@ evidence — после коммита этих правок основной с
|
||||
`make rolling_update`), `make test`, `make student_test` (красный только `test_exclusion_evidence_exists`), `docker
|
||||
compose config --quiet`, пробные `trace-chain` и `telemetry-outage` (вывод в scratchpad), `openspec validate
|
||||
job-trace-propagation --strict`.
|
||||
- [ ] 13.11 Пересъёмка evidence (после коммита правок основной сессией): `driver.py --port 8080 telemetry-outage` →
|
||||
- [x] 13.11 Пересъёмка evidence (после коммита правок основной сессией): `driver.py --port 8080 telemetry-outage` →
|
||||
`evidence/telemetry/collector-outage.md` (итоговая строка без ложного «without restarts»), `driver.py --port 8080
|
||||
trace-chain` → `evidence/telemetry/trace-chain.md` (затронут: предусловие, CPU, панель пула); ссылки commit в `REPORT.md`
|
||||
и `AGENTS.md`.
|
||||
|
||||
Reference in New Issue
Block a user