215 lines
22 KiB
Markdown
215 lines
22 KiB
Markdown
# `driver.py --port 8080 telemetry-check` (change `otel-telemetry`, задачи 11.1 и 12; change `job-trace-propagation`, задача 12.2; change `slo-alerts`, задачи 11.4, 12.11 и 12.12)
|
||
|
||
- Основа: commit `4a2f9c9` (fix: wait for all relay pass records in telemetry-check Loki comparison),
|
||
рабочее дерево чистое (`git status --short` до прогона пуст). Код приложения, правила Prometheus и
|
||
dashboard — те же, что на `ef60b47` (правки verify change `slo-alerts`): `4a2f9c9` и
|
||
промежуточный `a92dff3` меняют только драйвер скилла, документы и evidence. Файлы `src/` и
|
||
`migrations/` (151) в контейнерах `api` (2), `relay`, `worker` и `cleaner` побайтно совпадают с
|
||
деревом commit (SHA-256 каждого файла); ревизия схемы `e4b8c2d17f90`; `rules.yml` и
|
||
`prometheus.yml` в контейнере Prometheus и `messenger-overview.json` в контейнере Grafana равны
|
||
дереву `4a2f9c9` (`cmp`). Прежние прогоны этого файла — на `68f1e97` (до change
|
||
`job-trace-propagation`), на `72cfa9a` (до change `slo-alerts`) и на `15ea156` (до правок verify:
|
||
без проверки панели потерь на тихом стенде).
|
||
- Две попытки на `ef60b47` (2026-11-14, 01:43:53Z и 01:50:17Z) дали код 1 одной строкой — «same relay
|
||
pass records in Loki»: проверка ждала первой записи прохода relay в Loki, а сравнивала со всеми
|
||
записями stdout, вторая запись (через ~1,3 с) приходила в Loki следующим пакетом SDK, и через
|
||
секунды обе записи там были. Исправлено в драйвере (`4a2f9c9`: ожидание всех записей stdout со
|
||
сроком 60 с, условие проверки прежнее; design change `slo-alerts`, «Уточнения apply», п. 23).
|
||
- Дата: 2026-11-14, 01:58:59Z – 02:01:09Z (~2,2 мин), после `alerts-check` и `backend-outage` на
|
||
`ef60b47`; рабочий стенд `mtusi-lab6` (две реплики API, relay, worker, уборщик, Collector,
|
||
Prometheus, Loki, Tempo, Grafana), команда из корня репозитория:
|
||
`python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 telemetry-check`, код выхода 0.
|
||
- Всё — через HTTP API Grafana на `127.0.0.1:3000` с паролем из `.env` (источники-прокси uid
|
||
`prometheus`, `loki`, `tempo`); пароль, токены и тела в вывод не попадают. Пароль, `username`,
|
||
токен, текст, ключ и имя файла пробы — случайные синтетические, в выводе только их метки
|
||
U, P, T, M, K, F.
|
||
- Сценарии: `observability` — «Стенд с телеметрией», «Хранилища недоступны приложению»,
|
||
«История после рестарта хранилищ», «Provisioning после старта», «Источник правды — git», «Вход
|
||
в Grafana», «От лога к трассе и обратно» (строки «Loki field trace_id…», «Tempo span links…»,
|
||
«span-to-logs…»), «Ресурс сигналов», «Чужой traceparent», «Запросы в метрике», «Health не
|
||
учитывается» (нет трасс и рядов `http_route=~"/health.*"` за срок хранения), «Необычный метод и
|
||
неизвестный путь» (`PROPFIND` и `GET` к пути с UUID; `BREW` — ограничение, 400 парсера,
|
||
`KNOWN_ISSUES.md`), «Готовое вложение в метриках», «Неудачное вложение в метриках», «Трасса
|
||
запроса», «Трасса задания», «Отказ S3 в трассе», «Health и фоновые циклы без трасс», «Проба
|
||
приватности», «Запрос найден», «Задание найдено»; `service-logging` — «Записи запроса ведут к
|
||
трассе», «Записи задания ведут к трассе», «Запись в Loki совпадает со stdout»;
|
||
`service-configuration` — «Переменные OTEL_* не действуют». С change `job-trace-propagation`:
|
||
`observability` «Ресурс relay», `service-logging` «Записи relay в Loki» и «Проход relay без полей
|
||
трассы»; шаги задания — по consumer span `ready` трассы загрузки (его родитель — span producer
|
||
relay).
|
||
- С change `slo-alerts` (design, «Уточнения apply», пп. 5, 6, 22): `observability` «Метрики
|
||
балансировщика без доступа к сети хранилищ» (строки `LB networks` — только `default`, `LB stats port
|
||
published on host` — нет, `prometheus has an LB target` — нет, `LB series with job haproxy…`, цель
|
||
`haproxy` в списке `up` = 1) и `load-balancing` «Метрики балансировщика» (`LB frontend responses
|
||
grew by 20 or more` — 20 своих запросов через LB); «Без потерь» — `loss panel …`: все три запроса
|
||
панели потерь (id 40) на тихом стенде — 0, `TelemetryDataLoss` неактивно (тишина наступила сразу,
|
||
ожидание 0 с из 330); «Новые панели после трафика» — `dashboard panel queries (57) without error: 0`
|
||
и `panel queries with data after traffic: []`: все 57 запросов панелей к Prometheus (из них новых с
|
||
id от 25 — 31, макрос `$__range` подменён на `1h`) выполнены без ошибки и с данными на стенде без
|
||
оповещений, в том числе панели потерь и истории оповещений (`or on() vector(0)`).
|
||
- Свой запрос: `X-Request-Id` `6ad3d17993a34f829b83854d8bb24658`, `X-Instance-Id` `4b9782f7240b`,
|
||
trace-id `f21d0427d8ac0fbbb47ace9b69f462e3`; трасса загрузки `89bb3ac8152d9fc70ef3c4540a331e7f`;
|
||
проба приватности просмотрела 49 записей Loki, 30 трасс Tempo и 968 рядов Prometheus окна.
|
||
|
||
```
|
||
grafana without credentials expected 401 got 401
|
||
grafana wrong password expected 401 got 401
|
||
grafana password from .env expected 200 got 200
|
||
grafana published only on 127.0.0.1 expected yes got yes
|
||
datasource prometheus expected 200 prometheus got 200 prometheus
|
||
datasource prometheus health expected OK got OK
|
||
datasource loki expected 200 loki got 200 loki
|
||
datasource loki health expected OK got OK
|
||
datasource tempo expected 200 tempo got 200 tempo
|
||
datasource tempo health expected OK got OK
|
||
dashboard from git expected 200 got 200
|
||
dashboard provisioned expected True got True
|
||
dashboard folder expected Messenger got Messenger
|
||
save of the provisioned dashboard rejected expected yes got yes
|
||
dashboard equals git after the attempt expected yes got yes
|
||
prometheus target otel-collector up expected 1 got 1
|
||
prometheus target otel-collector-exporter up expected 1 got 1
|
||
prometheus target loki up expected 1 got 1
|
||
prometheus target tempo up expected 1 got 1
|
||
prometheus target prometheus up expected 1 got 1
|
||
prometheus target haproxy up expected 1 got 1
|
||
LB networks expected ['default'] got ['default']
|
||
LB stats port published on host expected no got no
|
||
prometheus has an LB target expected no got no
|
||
LB series with job haproxy (frontend, backend, ready) expected yes got yes
|
||
LB frontend responses grew by 20 or more expected yes got yes
|
||
loss panel queries expected 3 got 3
|
||
loss panel shows zero on a quiet stand expected [0.0, 0.0, 0.0] got [0.0, 0.0, 0.0]
|
||
TelemetryDataLoss on a quiet stand expected inactive got inactive
|
||
loss panel quiet within 330 s expected yes got yes
|
||
register probe user expected 201 got 201
|
||
login by Basic expected 201 got 201
|
||
create chat expected 201 got 201
|
||
send message expected 201 got 201
|
||
401 without token expected 401 got 401
|
||
404 of an unknown chat expected 404 got 404
|
||
PROPFIND of an unknown path with a UUID expected 404 got 404
|
||
GET of an unknown path with a UUID expected 404 got 404
|
||
BREW rejected by the HTTP parser (limitation) expected 400 got 400
|
||
request with traceparent expected 200 got 200
|
||
upload sample.png expected 202 got 202
|
||
attachment becomes ready expected ready got ready
|
||
undecodable JPEG generated expected yes got yes
|
||
upload undecodable JPEG expected 202 got 202
|
||
undecodable JPEG becomes failed expected failed got failed
|
||
stdout records of the request expected yes got yes
|
||
one trace_id and span_id in them expected 1 got 1
|
||
trace_id format expected yes got yes
|
||
Loki record of the request expected 1 got 1
|
||
Loki service_name expected messenger-api got messenger-api
|
||
Loki service_instance_id = X-Instance-Id expected 4b9782f7240b got 4b9782f7240b
|
||
Loki instance_id = X-Instance-Id expected 4b9782f7240b got 4b9782f7240b
|
||
Loki trace_id = stdout expected f21d0427d8ac0fbbb47ace9b69f462e3 got f21d0427d8ac0fbbb47ace9b69f462e3
|
||
Loki event, method, path, status, duration = stdout expected yes got yes
|
||
Tempo server span expected POST /api/v1/chats/{chat_id}/messages SPAN_KIND_SERVER got POST /api/v1/chats/{chat_id}/messages SPAN_KIND_SERVER
|
||
server span status code expected 201 got 201
|
||
server span request id = X-Request-Id expected 6ad3d17993a34f829b83854d8bb24658 got 6ad3d17993a34f829b83854d8bb24658
|
||
resource service.name expected messenger-api got messenger-api
|
||
resource service.instance.id expected 4b9782f7240b got 4b9782f7240b
|
||
resource service.version = contract expected 1.6.0 got 1.6.0
|
||
resource deployment.environment.name expected lab got lab
|
||
PostgreSQL query spans under it expected yes got yes
|
||
COMMIT span under it expected yes got yes
|
||
Loki field trace_id links to datasource expected tempo label got tempo label
|
||
Tempo span links to logs in datasource expected loki got loki
|
||
span-to-logs query fully substituted expected yes got yes
|
||
span-to-logs query returns records of trace T expected yes got yes
|
||
traceparent request has its own trace expected yes got yes
|
||
spans of the foreign trace-id in Tempo expected 0 got 0
|
||
its server span has no parent expected yes got yes
|
||
Loki worker record of the attachment expected 1 got 1
|
||
job trace = upload trace (trace_id of Uploaded attachment) expected 89bb3ac8152d9fc70ef3c4540a331e7f got 89bb3ac8152d9fc70ef3c4540a331e7f
|
||
worker stdout records carry the job trace_id expected yes got yes
|
||
Tempo consumer span in the upload trace expected SPAN_KIND_CONSUMER ready got SPAN_KIND_CONSUMER ready
|
||
worker resource service.name expected messenger-worker got messenger-worker
|
||
worker service.instance.id = instance_id of its stdout expected dd039bfb3324 got dd039bfb3324
|
||
consumer span parent: relay producer span expected SPAN_KIND_PRODUCER got SPAN_KIND_PRODUCER
|
||
producer resource service.name expected messenger-relay got messenger-relay
|
||
producer service.instance.id = instance_id of relay stdout expected b90816dcb1cd got b90816dcb1cd
|
||
relay pass record with publication in stdout expected yes got yes
|
||
same relay pass records in Loki (event, published) expected yes got yes
|
||
relay pass records without trace_id in stdout and Loki expected yes got yes
|
||
job spans: PostgreSQL expected yes got yes
|
||
job spans: S3.GetObject (staging) expected yes got yes
|
||
job spans: S3.PutObject (original and 3 variants) expected yes got yes
|
||
job spans: render process expected yes got yes
|
||
upload with S3 paused expected 503 got 503
|
||
its server span status expected STATUS_CODE_ERROR got STATUS_CODE_ERROR
|
||
S3 span with error expected yes got yes
|
||
S3 span duration within 4.0-7.5 s expected yes got yes
|
||
error.type of the S3 span set expected yes got yes
|
||
health traces in Tempo expected 0 got 0
|
||
trace roots other than requests and deliveries expected 0 got 0
|
||
traces whose root is a relay span expected 0 got 0
|
||
api -> prometheus expected not reachable got not reachable
|
||
api -> loki expected not reachable got not reachable
|
||
api -> tempo expected not reachable got not reachable
|
||
api -> grafana expected not reachable got not reachable
|
||
api -> otel-collector:4318 expected connected got connected
|
||
worker -> prometheus expected not reachable got not reachable
|
||
worker -> loki expected not reachable got not reachable
|
||
worker -> tempo expected not reachable got not reachable
|
||
worker -> grafana expected not reachable got not reachable
|
||
worker -> otel-collector:4318 expected connected got connected
|
||
relay -> prometheus expected not reachable got not reachable
|
||
relay -> loki expected not reachable got not reachable
|
||
relay -> tempo expected not reachable got not reachable
|
||
relay -> grafana expected not reachable got not reachable
|
||
relay -> otel-collector:4318 expected connected got connected
|
||
probe container with OTEL_* started expected 0 got 0
|
||
probe container healthy expected yes got yes
|
||
probe request inside expected 401 got 401
|
||
warning level expected warning got warning
|
||
warning names the variables expected ['OTEL_EXPORTER_OTLP_ENDPOINT', 'OTEL_SDK_DISABLED'] got ['OTEL_EXPORTER_OTLP_ENDPOINT', 'OTEL_SDK_DISABLED']
|
||
no variable values in stdout expected yes got yes
|
||
probe records reach Loki from [telemetry] expected yes got yes
|
||
probe service_name expected messenger-api got messenger-api
|
||
chat GET 200 count grew by 10 expected 10 got 10
|
||
count grew on every replica expected 2 got 2
|
||
jobs ready grew by 1 expected 1 got 1
|
||
jobs failed grew by 1 expected 1 got 1
|
||
_OTHER 404 count without http_route grew by 1 expected 1 got 1
|
||
GET 404 count without http_route grew by 1 expected 1 got 1
|
||
Prometheus series with health routes expected 0 got 0
|
||
_OTHER series without http_route expected yes got yes
|
||
401 series of /api/v1/users/me expected yes got yes
|
||
404 series of /api/v1/chats/{chat_id} expected yes got yes
|
||
forbidden label names expected [] got []
|
||
label values that are UUIDs expected 0 got 0
|
||
label values with BREW, PROPFIND or the odd UUID expected 0 got 0
|
||
dashboard panel queries (57) without error expected 0 got 0
|
||
panel queries with data after traffic expected [] got []
|
||
U in telemetry expected [] got []
|
||
P in telemetry expected [] got []
|
||
T in telemetry expected [] got []
|
||
M in telemetry expected [] got []
|
||
K in telemetry expected [] got []
|
||
F in telemetry expected [] got []
|
||
Bearer in telemetry expected [] got []
|
||
Basic in telemetry expected [] got []
|
||
foreign trace-id in telemetry expected [] got []
|
||
pipeline back after restart expected yes got yes
|
||
same Loki record after restart expected yes got yes
|
||
same trace after restart expected yes got yes
|
||
same metric value at the saved time expected 50 got 50
|
||
PASS: own request and job are found in Loki, Tempo and Prometheus through Grafana, no secrets in telemetry
|
||
```
|
||
|
||
Код выхода — 0. «Same metric value at the saved time» — значение суммы ряда GET карточки чата на
|
||
момент снимка до рестарта, запрошенное снова после `docker compose restart otel-collector
|
||
prometheus loki tempo` (параметр `time` запроса). «Span-to-logs query» — запрос `tracesToLogsV2`
|
||
источника Tempo из `infra/otel/grafana/datasources.yaml`, в который подставлены `service.name`
|
||
ресурса и trace-id своей трассы: он возвращает записи только этой трассы — тот же переход, что
|
||
открывает ссылка из span в Grafana. «Dashboard panel queries (57) without error: 0» — число
|
||
запросов панелей к Prometheus с ошибкой; «panel queries with data after traffic: []» — список
|
||
запросов без данных пуст (57 запросов dashboard: уровни 3 и 4 и новые строки change `slo-alerts`).
|
||
«Loss panel …» — запросы A, B и C панели 40 из dashboard в Grafana на момент проверки (до своих
|
||
рестартов Collector и хранилищ); «same relay pass records in Loki» — все записи прохода relay с
|
||
публикацией из stdout с начала прогона (своя загрузка и недекодируемый JPEG) есть в Loki с тем же
|
||
`published` и `duration_ms`.
|