Files
mtusi2026lab6/evidence/telemetry/check.md
T
2026-11-14 05:03:22 +03:00

22 KiB
Raw Blame History

driver.py --port 8080 telemetry-check (change otel-telemetry, задачи 11.1 и 12; change job-trace-propagation, задача 12.2; change slo-alerts, задачи 11.4, 12.11 и 12.12)

  • Основа: commit 4a2f9c9 (fix: wait for all relay pass records in telemetry-check Loki comparison), рабочее дерево чистое (git status --short до прогона пуст). Код приложения, правила Prometheus и dashboard — те же, что на ef60b47 (правки verify change slo-alerts): 4a2f9c9 и промежуточный a92dff3 меняют только драйвер скилла, документы и evidence. Файлы src/ и migrations/ (151) в контейнерах api (2), relay, worker и cleaner побайтно совпадают с деревом commit (SHA-256 каждого файла); ревизия схемы e4b8c2d17f90; rules.yml и prometheus.yml в контейнере Prometheus и messenger-overview.json в контейнере Grafana равны дереву 4a2f9c9 (cmp). Прежние прогоны этого файла — на 68f1e97 (до change job-trace-propagation), на 72cfa9a (до change slo-alerts) и на 15ea156 (до правок verify: без проверки панели потерь на тихом стенде).
  • Две попытки на ef60b47 (2026-11-14, 01:43:53Z и 01:50:17Z) дали код 1 одной строкой — «same relay pass records in Loki»: проверка ждала первой записи прохода relay в Loki, а сравнивала со всеми записями stdout, вторая запись (через ~1,3 с) приходила в Loki следующим пакетом SDK, и через секунды обе записи там были. Исправлено в драйвере (4a2f9c9: ожидание всех записей stdout со сроком 60 с, условие проверки прежнее; design change slo-alerts, «Уточнения apply», п. 23).
  • Дата: 2026-11-14, 01:58:59Z – 02:01:09Z (~2,2 мин), после alerts-check и backend-outage на ef60b47; рабочий стенд mtusi-lab6 (две реплики API, relay, worker, уборщик, Collector, Prometheus, Loki, Tempo, Grafana), команда из корня репозитория: python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 telemetry-check, код выхода 0.
  • Всё — через HTTP API Grafana на 127.0.0.1:3000 с паролем из .env (источники-прокси uid prometheus, loki, tempo); пароль, токены и тела в вывод не попадают. Пароль, username, токен, текст, ключ и имя файла пробы — случайные синтетические, в выводе только их метки U, P, T, M, K, F.
  • Сценарии: observability — «Стенд с телеметрией», «Хранилища недоступны приложению», «История после рестарта хранилищ», «Provisioning после старта», «Источник правды — git», «Вход в Grafana», «От лога к трассе и обратно» (строки «Loki field trace_id…», «Tempo span links…», «span-to-logs…»), «Ресурс сигналов», «Чужой traceparent», «Запросы в метрике», «Health не учитывается» (нет трасс и рядов http_route=~"/health.*" за срок хранения), «Необычный метод и неизвестный путь» (PROPFIND и GET к пути с UUID; BREW — ограничение, 400 парсера, KNOWN_ISSUES.md), «Готовое вложение в метриках», «Неудачное вложение в метриках», «Трасса запроса», «Трасса задания», «Отказ S3 в трассе», «Health и фоновые циклы без трасс», «Проба приватности», «Запрос найден», «Задание найдено»; service-logging — «Записи запроса ведут к трассе», «Записи задания ведут к трассе», «Запись в Loki совпадает со stdout»; service-configuration — «Переменные OTEL_* не действуют». С change job-trace-propagation: observability «Ресурс relay», service-logging «Записи relay в Loki» и «Проход relay без полей трассы»; шаги задания — по consumer span ready трассы загрузки (его родитель — span producer relay).
  • С change slo-alerts (design, «Уточнения apply», пп. 5, 6, 22): observability «Метрики балансировщика без доступа к сети хранилищ» (строки LB networks — только default, LB stats port published on host — нет, prometheus has an LB target — нет, LB series with job haproxy…, цель haproxy в списке up = 1) и load-balancing «Метрики балансировщика» (LB frontend responses grew by 20 or more — 20 своих запросов через LB); «Без потерь» — loss panel …: все три запроса панели потерь (id 40) на тихом стенде — 0, TelemetryDataLoss неактивно (тишина наступила сразу, ожидание 0 с из 330); «Новые панели после трафика» — dashboard panel queries (57) without error: 0 и panel queries with data after traffic: []: все 57 запросов панелей к Prometheus (из них новых с id от 25 — 31, макрос $__range подменён на 1h) выполнены без ошибки и с данными на стенде без оповещений, в том числе панели потерь и истории оповещений (or on() vector(0)).
  • Свой запрос: X-Request-Id 6ad3d17993a34f829b83854d8bb24658, X-Instance-Id 4b9782f7240b, trace-id f21d0427d8ac0fbbb47ace9b69f462e3; трасса загрузки 89bb3ac8152d9fc70ef3c4540a331e7f; проба приватности просмотрела 49 записей Loki, 30 трасс Tempo и 968 рядов Prometheus окна.
grafana without credentials                                expected 401             got 401
grafana wrong password                                     expected 401             got 401
grafana password from .env                                 expected 200             got 200
grafana published only on 127.0.0.1                        expected yes             got yes
datasource prometheus                                      expected 200 prometheus  got 200 prometheus
datasource prometheus health                               expected OK              got OK
datasource loki                                            expected 200 loki        got 200 loki
datasource loki health                                     expected OK              got OK
datasource tempo                                           expected 200 tempo       got 200 tempo
datasource tempo health                                    expected OK              got OK
dashboard from git                                         expected 200             got 200
dashboard provisioned                                      expected True            got True
dashboard folder                                           expected Messenger       got Messenger
save of the provisioned dashboard rejected                 expected yes             got yes
dashboard equals git after the attempt                     expected yes             got yes
prometheus target otel-collector up                        expected 1               got 1
prometheus target otel-collector-exporter up               expected 1               got 1
prometheus target loki up                                  expected 1               got 1
prometheus target tempo up                                 expected 1               got 1
prometheus target prometheus up                            expected 1               got 1
prometheus target haproxy up                               expected 1               got 1
LB networks                                                expected ['default']     got ['default']
LB stats port published on host                            expected no              got no
prometheus has an LB target                                expected no              got no
LB series with job haproxy (frontend, backend, ready)      expected yes             got yes
LB frontend responses grew by 20 or more                   expected yes             got yes
loss panel queries                                         expected 3               got 3
loss panel shows zero on a quiet stand                     expected [0.0, 0.0, 0.0] got [0.0, 0.0, 0.0]
TelemetryDataLoss on a quiet stand                         expected inactive        got inactive
loss panel quiet within 330 s                              expected yes             got yes
register probe user                                        expected 201             got 201
login by Basic                                             expected 201             got 201
create chat                                                expected 201             got 201
send message                                               expected 201             got 201
401 without token                                          expected 401             got 401
404 of an unknown chat                                     expected 404             got 404
PROPFIND of an unknown path with a UUID                    expected 404             got 404
GET of an unknown path with a UUID                         expected 404             got 404
BREW rejected by the HTTP parser (limitation)              expected 400             got 400
request with traceparent                                   expected 200             got 200
upload sample.png                                          expected 202             got 202
attachment becomes ready                                   expected ready           got ready
undecodable JPEG generated                                 expected yes             got yes
upload undecodable JPEG                                    expected 202             got 202
undecodable JPEG becomes failed                            expected failed          got failed
stdout records of the request                              expected yes             got yes
one trace_id and span_id in them                           expected 1               got 1
trace_id format                                            expected yes             got yes
Loki record of the request                                 expected 1               got 1
Loki service_name                                          expected messenger-api   got messenger-api
Loki service_instance_id = X-Instance-Id                   expected 4b9782f7240b    got 4b9782f7240b
Loki instance_id = X-Instance-Id                           expected 4b9782f7240b    got 4b9782f7240b
Loki trace_id = stdout                                     expected f21d0427d8ac0fbbb47ace9b69f462e3 got f21d0427d8ac0fbbb47ace9b69f462e3
Loki event, method, path, status, duration = stdout        expected yes             got yes
Tempo server span                                          expected POST /api/v1/chats/{chat_id}/messages SPAN_KIND_SERVER got POST /api/v1/chats/{chat_id}/messages SPAN_KIND_SERVER
server span status code                                    expected 201             got 201
server span request id = X-Request-Id                      expected 6ad3d17993a34f829b83854d8bb24658 got 6ad3d17993a34f829b83854d8bb24658
resource service.name                                      expected messenger-api   got messenger-api
resource service.instance.id                               expected 4b9782f7240b    got 4b9782f7240b
resource service.version = contract                        expected 1.6.0           got 1.6.0
resource deployment.environment.name                       expected lab             got lab
PostgreSQL query spans under it                            expected yes             got yes
COMMIT span under it                                       expected yes             got yes
Loki field trace_id links to datasource                    expected tempo label     got tempo label
Tempo span links to logs in datasource                     expected loki            got loki
span-to-logs query fully substituted                       expected yes             got yes
span-to-logs query returns records of trace T              expected yes             got yes
traceparent request has its own trace                      expected yes             got yes
spans of the foreign trace-id in Tempo                     expected 0               got 0
its server span has no parent                              expected yes             got yes
Loki worker record of the attachment                       expected 1               got 1
job trace = upload trace (trace_id of Uploaded attachment) expected 89bb3ac8152d9fc70ef3c4540a331e7f got 89bb3ac8152d9fc70ef3c4540a331e7f
worker stdout records carry the job trace_id               expected yes             got yes
Tempo consumer span in the upload trace                    expected SPAN_KIND_CONSUMER ready got SPAN_KIND_CONSUMER ready
worker resource service.name                               expected messenger-worker got messenger-worker
worker service.instance.id = instance_id of its stdout     expected dd039bfb3324    got dd039bfb3324
consumer span parent: relay producer span                  expected SPAN_KIND_PRODUCER got SPAN_KIND_PRODUCER
producer resource service.name                             expected messenger-relay got messenger-relay
producer service.instance.id = instance_id of relay stdout expected b90816dcb1cd    got b90816dcb1cd
relay pass record with publication in stdout               expected yes             got yes
same relay pass records in Loki (event, published)         expected yes             got yes
relay pass records without trace_id in stdout and Loki     expected yes             got yes
job spans: PostgreSQL                                      expected yes             got yes
job spans: S3.GetObject (staging)                          expected yes             got yes
job spans: S3.PutObject (original and 3 variants)          expected yes             got yes
job spans: render process                                  expected yes             got yes
upload with S3 paused                                      expected 503             got 503
its server span status                                     expected STATUS_CODE_ERROR got STATUS_CODE_ERROR
S3 span with error                                         expected yes             got yes
S3 span duration within 4.0-7.5 s                          expected yes             got yes
error.type of the S3 span set                              expected yes             got yes
health traces in Tempo                                     expected 0               got 0
trace roots other than requests and deliveries             expected 0               got 0
traces whose root is a relay span                          expected 0               got 0
api -> prometheus                                          expected not reachable   got not reachable
api -> loki                                                expected not reachable   got not reachable
api -> tempo                                               expected not reachable   got not reachable
api -> grafana                                             expected not reachable   got not reachable
api -> otel-collector:4318                                 expected connected       got connected
worker -> prometheus                                       expected not reachable   got not reachable
worker -> loki                                             expected not reachable   got not reachable
worker -> tempo                                            expected not reachable   got not reachable
worker -> grafana                                          expected not reachable   got not reachable
worker -> otel-collector:4318                              expected connected       got connected
relay -> prometheus                                        expected not reachable   got not reachable
relay -> loki                                              expected not reachable   got not reachable
relay -> tempo                                             expected not reachable   got not reachable
relay -> grafana                                           expected not reachable   got not reachable
relay -> otel-collector:4318                               expected connected       got connected
probe container with OTEL_* started                        expected 0               got 0
probe container healthy                                    expected yes             got yes
probe request inside                                       expected 401             got 401
warning level                                              expected warning         got warning
warning names the variables                                expected ['OTEL_EXPORTER_OTLP_ENDPOINT', 'OTEL_SDK_DISABLED'] got ['OTEL_EXPORTER_OTLP_ENDPOINT', 'OTEL_SDK_DISABLED']
no variable values in stdout                               expected yes             got yes
probe records reach Loki from [telemetry]                  expected yes             got yes
probe service_name                                         expected messenger-api   got messenger-api
chat GET 200 count grew by 10                              expected 10              got 10
count grew on every replica                                expected 2               got 2
jobs ready grew by 1                                       expected 1               got 1
jobs failed grew by 1                                      expected 1               got 1
_OTHER 404 count without http_route grew by 1              expected 1               got 1
GET 404 count without http_route grew by 1                 expected 1               got 1
Prometheus series with health routes                       expected 0               got 0
_OTHER series without http_route                           expected yes             got yes
401 series of /api/v1/users/me                             expected yes             got yes
404 series of /api/v1/chats/{chat_id}                      expected yes             got yes
forbidden label names                                      expected []              got []
label values that are UUIDs                                expected 0               got 0
label values with BREW, PROPFIND or the odd UUID           expected 0               got 0
dashboard panel queries (57) without error                 expected 0               got 0
panel queries with data after traffic                      expected []              got []
U in telemetry                                             expected []              got []
P in telemetry                                             expected []              got []
T in telemetry                                             expected []              got []
M in telemetry                                             expected []              got []
K in telemetry                                             expected []              got []
F in telemetry                                             expected []              got []
Bearer in telemetry                                        expected []              got []
Basic in telemetry                                         expected []              got []
foreign trace-id in telemetry                              expected []              got []
pipeline back after restart                                expected yes             got yes
same Loki record after restart                             expected yes             got yes
same trace after restart                                   expected yes             got yes
same metric value at the saved time                        expected 50              got 50
PASS: own request and job are found in Loki, Tempo and Prometheus through Grafana, no secrets in telemetry

Код выхода — 0. «Same metric value at the saved time» — значение суммы ряда GET карточки чата на момент снимка до рестарта, запрошенное снова после docker compose restart otel-collector prometheus loki tempo (параметр time запроса). «Span-to-logs query» — запрос tracesToLogsV2 источника Tempo из infra/otel/grafana/datasources.yaml, в который подставлены service.name ресурса и trace-id своей трассы: он возвращает записи только этой трассы — тот же переход, что открывает ссылка из span в Grafana. «Dashboard panel queries (57) without error: 0» — число запросов панелей к Prometheus с ошибкой; «panel queries with data after traffic: []» — список запросов без данных пуст (57 запросов dashboard: уровни 3 и 4 и новые строки change slo-alerts). «Loss panel …» — запросы A, B и C панели 40 из dashboard в Grafana на момент проверки (до своих рестартов Collector и хранилищ); «same relay pass records in Loki» — все записи прохода relay с публикацией из stdout с начала прогона (своя загрузка и недекодируемый JPEG) есть в Loki с тем же published и duration_ms.