Files
mtusi2026lab6/evidence/telemetry/check.md
T
2026-11-14 05:03:22 +03:00

215 lines
22 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# `driver.py --port 8080 telemetry-check` (change `otel-telemetry`, задачи 11.1 и 12; change `job-trace-propagation`, задача 12.2; change `slo-alerts`, задачи 11.4, 12.11 и 12.12)
- Основа: commit `4a2f9c9` (fix: wait for all relay pass records in telemetry-check Loki comparison),
рабочее дерево чистое (`git status --short` до прогона пуст). Код приложения, правила Prometheus и
dashboard — те же, что на `ef60b47` (правки verify change `slo-alerts`): `4a2f9c9` и
промежуточный `a92dff3` меняют только драйвер скилла, документы и evidence. Файлы `src/` и
`migrations/` (151) в контейнерах `api` (2), `relay`, `worker` и `cleaner` побайтно совпадают с
деревом commit (SHA-256 каждого файла); ревизия схемы `e4b8c2d17f90`; `rules.yml` и
`prometheus.yml` в контейнере Prometheus и `messenger-overview.json` в контейнере Grafana равны
дереву `4a2f9c9` (`cmp`). Прежние прогоны этого файла — на `68f1e97` (до change
`job-trace-propagation`), на `72cfa9a` (до change `slo-alerts`) и на `15ea156` (до правок verify:
без проверки панели потерь на тихом стенде).
- Две попытки на `ef60b47` (2026-11-14, 01:43:53Z и 01:50:17Z) дали код 1 одной строкой — «same relay
pass records in Loki»: проверка ждала первой записи прохода relay в Loki, а сравнивала со всеми
записями stdout, вторая запись (через ~1,3 с) приходила в Loki следующим пакетом SDK, и через
секунды обе записи там были. Исправлено в драйвере (`4a2f9c9`: ожидание всех записей stdout со
сроком 60 с, условие проверки прежнее; design change `slo-alerts`, «Уточнения apply», п. 23).
- Дата: 2026-11-14, 01:58:59Z – 02:01:09Z (~2,2 мин), после `alerts-check` и `backend-outage` на
`ef60b47`; рабочий стенд `mtusi-lab6` (две реплики API, relay, worker, уборщик, Collector,
Prometheus, Loki, Tempo, Grafana), команда из корня репозитория:
`python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 telemetry-check`, код выхода 0.
- Всё — через HTTP API Grafana на `127.0.0.1:3000` с паролем из `.env` (источники-прокси uid
`prometheus`, `loki`, `tempo`); пароль, токены и тела в вывод не попадают. Пароль, `username`,
токен, текст, ключ и имя файла пробы — случайные синтетические, в выводе только их метки
U, P, T, M, K, F.
- Сценарии: `observability` — «Стенд с телеметрией», «Хранилища недоступны приложению»,
«История после рестарта хранилищ», «Provisioning после старта», «Источник правды — git», «Вход
в Grafana», «От лога к трассе и обратно» (строки «Loki field trace_id…», «Tempo span links…»,
«span-to-logs…»), «Ресурс сигналов», «Чужой traceparent», «Запросы в метрике», «Health не
учитывается» (нет трасс и рядов `http_route=~"/health.*"` за срок хранения), «Необычный метод и
неизвестный путь» (`PROPFIND` и `GET` к пути с UUID; `BREW` — ограничение, 400 парсера,
`KNOWN_ISSUES.md`), «Готовое вложение в метриках», «Неудачное вложение в метриках», «Трасса
запроса», «Трасса задания», «Отказ S3 в трассе», «Health и фоновые циклы без трасс», «Проба
приватности», «Запрос найден», «Задание найдено»; `service-logging` — «Записи запроса ведут к
трассе», «Записи задания ведут к трассе», «Запись в Loki совпадает со stdout»;
`service-configuration` — «Переменные OTEL_* не действуют». С change `job-trace-propagation`:
`observability` «Ресурс relay», `service-logging` «Записи relay в Loki» и «Проход relay без полей
трассы»; шаги задания — по consumer span `ready` трассы загрузки (его родитель — span producer
relay).
- С change `slo-alerts` (design, «Уточнения apply», пп. 5, 6, 22): `observability` «Метрики
балансировщика без доступа к сети хранилищ» (строки `LB networks` — только `default`, `LB stats port
published on host` — нет, `prometheus has an LB target` — нет, `LB series with job haproxy…`, цель
`haproxy` в списке `up` = 1) и `load-balancing` «Метрики балансировщика» (`LB frontend responses
grew by 20 or more` — 20 своих запросов через LB); «Без потерь» — `loss panel …`: все три запроса
панели потерь (id 40) на тихом стенде — 0, `TelemetryDataLoss` неактивно (тишина наступила сразу,
ожидание 0 с из 330); «Новые панели после трафика» — `dashboard panel queries (57) without error: 0`
и `panel queries with data after traffic: []`: все 57 запросов панелей к Prometheus (из них новых с
id от 25 — 31, макрос `$__range` подменён на `1h`) выполнены без ошибки и с данными на стенде без
оповещений, в том числе панели потерь и истории оповещений (`or on() vector(0)`).
- Свой запрос: `X-Request-Id` `6ad3d17993a34f829b83854d8bb24658`, `X-Instance-Id` `4b9782f7240b`,
trace-id `f21d0427d8ac0fbbb47ace9b69f462e3`; трасса загрузки `89bb3ac8152d9fc70ef3c4540a331e7f`;
проба приватности просмотрела 49 записей Loki, 30 трасс Tempo и 968 рядов Prometheus окна.
```
grafana without credentials expected 401 got 401
grafana wrong password expected 401 got 401
grafana password from .env expected 200 got 200
grafana published only on 127.0.0.1 expected yes got yes
datasource prometheus expected 200 prometheus got 200 prometheus
datasource prometheus health expected OK got OK
datasource loki expected 200 loki got 200 loki
datasource loki health expected OK got OK
datasource tempo expected 200 tempo got 200 tempo
datasource tempo health expected OK got OK
dashboard from git expected 200 got 200
dashboard provisioned expected True got True
dashboard folder expected Messenger got Messenger
save of the provisioned dashboard rejected expected yes got yes
dashboard equals git after the attempt expected yes got yes
prometheus target otel-collector up expected 1 got 1
prometheus target otel-collector-exporter up expected 1 got 1
prometheus target loki up expected 1 got 1
prometheus target tempo up expected 1 got 1
prometheus target prometheus up expected 1 got 1
prometheus target haproxy up expected 1 got 1
LB networks expected ['default'] got ['default']
LB stats port published on host expected no got no
prometheus has an LB target expected no got no
LB series with job haproxy (frontend, backend, ready) expected yes got yes
LB frontend responses grew by 20 or more expected yes got yes
loss panel queries expected 3 got 3
loss panel shows zero on a quiet stand expected [0.0, 0.0, 0.0] got [0.0, 0.0, 0.0]
TelemetryDataLoss on a quiet stand expected inactive got inactive
loss panel quiet within 330 s expected yes got yes
register probe user expected 201 got 201
login by Basic expected 201 got 201
create chat expected 201 got 201
send message expected 201 got 201
401 without token expected 401 got 401
404 of an unknown chat expected 404 got 404
PROPFIND of an unknown path with a UUID expected 404 got 404
GET of an unknown path with a UUID expected 404 got 404
BREW rejected by the HTTP parser (limitation) expected 400 got 400
request with traceparent expected 200 got 200
upload sample.png expected 202 got 202
attachment becomes ready expected ready got ready
undecodable JPEG generated expected yes got yes
upload undecodable JPEG expected 202 got 202
undecodable JPEG becomes failed expected failed got failed
stdout records of the request expected yes got yes
one trace_id and span_id in them expected 1 got 1
trace_id format expected yes got yes
Loki record of the request expected 1 got 1
Loki service_name expected messenger-api got messenger-api
Loki service_instance_id = X-Instance-Id expected 4b9782f7240b got 4b9782f7240b
Loki instance_id = X-Instance-Id expected 4b9782f7240b got 4b9782f7240b
Loki trace_id = stdout expected f21d0427d8ac0fbbb47ace9b69f462e3 got f21d0427d8ac0fbbb47ace9b69f462e3
Loki event, method, path, status, duration = stdout expected yes got yes
Tempo server span expected POST /api/v1/chats/{chat_id}/messages SPAN_KIND_SERVER got POST /api/v1/chats/{chat_id}/messages SPAN_KIND_SERVER
server span status code expected 201 got 201
server span request id = X-Request-Id expected 6ad3d17993a34f829b83854d8bb24658 got 6ad3d17993a34f829b83854d8bb24658
resource service.name expected messenger-api got messenger-api
resource service.instance.id expected 4b9782f7240b got 4b9782f7240b
resource service.version = contract expected 1.6.0 got 1.6.0
resource deployment.environment.name expected lab got lab
PostgreSQL query spans under it expected yes got yes
COMMIT span under it expected yes got yes
Loki field trace_id links to datasource expected tempo label got tempo label
Tempo span links to logs in datasource expected loki got loki
span-to-logs query fully substituted expected yes got yes
span-to-logs query returns records of trace T expected yes got yes
traceparent request has its own trace expected yes got yes
spans of the foreign trace-id in Tempo expected 0 got 0
its server span has no parent expected yes got yes
Loki worker record of the attachment expected 1 got 1
job trace = upload trace (trace_id of Uploaded attachment) expected 89bb3ac8152d9fc70ef3c4540a331e7f got 89bb3ac8152d9fc70ef3c4540a331e7f
worker stdout records carry the job trace_id expected yes got yes
Tempo consumer span in the upload trace expected SPAN_KIND_CONSUMER ready got SPAN_KIND_CONSUMER ready
worker resource service.name expected messenger-worker got messenger-worker
worker service.instance.id = instance_id of its stdout expected dd039bfb3324 got dd039bfb3324
consumer span parent: relay producer span expected SPAN_KIND_PRODUCER got SPAN_KIND_PRODUCER
producer resource service.name expected messenger-relay got messenger-relay
producer service.instance.id = instance_id of relay stdout expected b90816dcb1cd got b90816dcb1cd
relay pass record with publication in stdout expected yes got yes
same relay pass records in Loki (event, published) expected yes got yes
relay pass records without trace_id in stdout and Loki expected yes got yes
job spans: PostgreSQL expected yes got yes
job spans: S3.GetObject (staging) expected yes got yes
job spans: S3.PutObject (original and 3 variants) expected yes got yes
job spans: render process expected yes got yes
upload with S3 paused expected 503 got 503
its server span status expected STATUS_CODE_ERROR got STATUS_CODE_ERROR
S3 span with error expected yes got yes
S3 span duration within 4.0-7.5 s expected yes got yes
error.type of the S3 span set expected yes got yes
health traces in Tempo expected 0 got 0
trace roots other than requests and deliveries expected 0 got 0
traces whose root is a relay span expected 0 got 0
api -> prometheus expected not reachable got not reachable
api -> loki expected not reachable got not reachable
api -> tempo expected not reachable got not reachable
api -> grafana expected not reachable got not reachable
api -> otel-collector:4318 expected connected got connected
worker -> prometheus expected not reachable got not reachable
worker -> loki expected not reachable got not reachable
worker -> tempo expected not reachable got not reachable
worker -> grafana expected not reachable got not reachable
worker -> otel-collector:4318 expected connected got connected
relay -> prometheus expected not reachable got not reachable
relay -> loki expected not reachable got not reachable
relay -> tempo expected not reachable got not reachable
relay -> grafana expected not reachable got not reachable
relay -> otel-collector:4318 expected connected got connected
probe container with OTEL_* started expected 0 got 0
probe container healthy expected yes got yes
probe request inside expected 401 got 401
warning level expected warning got warning
warning names the variables expected ['OTEL_EXPORTER_OTLP_ENDPOINT', 'OTEL_SDK_DISABLED'] got ['OTEL_EXPORTER_OTLP_ENDPOINT', 'OTEL_SDK_DISABLED']
no variable values in stdout expected yes got yes
probe records reach Loki from [telemetry] expected yes got yes
probe service_name expected messenger-api got messenger-api
chat GET 200 count grew by 10 expected 10 got 10
count grew on every replica expected 2 got 2
jobs ready grew by 1 expected 1 got 1
jobs failed grew by 1 expected 1 got 1
_OTHER 404 count without http_route grew by 1 expected 1 got 1
GET 404 count without http_route grew by 1 expected 1 got 1
Prometheus series with health routes expected 0 got 0
_OTHER series without http_route expected yes got yes
401 series of /api/v1/users/me expected yes got yes
404 series of /api/v1/chats/{chat_id} expected yes got yes
forbidden label names expected [] got []
label values that are UUIDs expected 0 got 0
label values with BREW, PROPFIND or the odd UUID expected 0 got 0
dashboard panel queries (57) without error expected 0 got 0
panel queries with data after traffic expected [] got []
U in telemetry expected [] got []
P in telemetry expected [] got []
T in telemetry expected [] got []
M in telemetry expected [] got []
K in telemetry expected [] got []
F in telemetry expected [] got []
Bearer in telemetry expected [] got []
Basic in telemetry expected [] got []
foreign trace-id in telemetry expected [] got []
pipeline back after restart expected yes got yes
same Loki record after restart expected yes got yes
same trace after restart expected yes got yes
same metric value at the saved time expected 50 got 50
PASS: own request and job are found in Loki, Tempo and Prometheus through Grafana, no secrets in telemetry
```
Код выхода — 0. «Same metric value at the saved time» — значение суммы ряда GET карточки чата на
момент снимка до рестарта, запрошенное снова после `docker compose restart otel-collector
prometheus loki tempo` (параметр `time` запроса). «Span-to-logs query» — запрос `tracesToLogsV2`
источника Tempo из `infra/otel/grafana/datasources.yaml`, в который подставлены `service.name`
ресурса и trace-id своей трассы: он возвращает записи только этой трассы — тот же переход, что
открывает ссылка из span в Grafana. «Dashboard panel queries (57) without error: 0» — число
запросов панелей к Prometheus с ошибкой; «panel queries with data after traffic: []» — список
запросов без данных пуст (57 запросов dashboard: уровни 3 и 4 и новые строки change `slo-alerts`).
«Loss panel …» — запросы A, B и C панели 40 из dashboard в Grafana на момент проверки (до своих
рестартов Collector и хранилищ); «same relay pass records in Loki» — все записи прохода relay с
публикацией из stdout с начала прогона (своя загрузка и недекодируемый JPEG) есть в Loki с тем же
`published` и `duration_ms`.