forked from mtusi2026/mtusi2026lab6
Grade 4
This commit is contained in:
+3
-6
@@ -23,9 +23,6 @@ MINIO_ROOT_PASSWORD=replace-with-admin-password
|
||||
S3_ACCESS_KEY_ID=messenger-api
|
||||
S3_SECRET_ACCESS_KEY=replace-with-api-password
|
||||
|
||||
# Reserved for lab6 observability setup.
|
||||
OTEL_SERVICE_NAME=messenger-api
|
||||
# Worker must set a distinct service.name.
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
|
||||
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
|
||||
OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=lab
|
||||
# Compose sets service.name separately for API, dispatcher and worker.
|
||||
OTEL_SDK_DISABLED=false
|
||||
GRAFANA_ADMIN_PASSWORD=replace-with-grafana-password
|
||||
|
||||
@@ -1,18 +1,32 @@
|
||||
## Лабораторная 6
|
||||
|
||||
Автор / группа: TODO
|
||||
Целевая оценка (3/4/5): TODO
|
||||
Трек 5, если заявлен: TODO
|
||||
Автор / группа: Ромашко Владимир, БВТ2301.
|
||||
|
||||
Что реализовано и что перенесено: TODO
|
||||
Команды запуска из чистого clone: TODO
|
||||
Результаты проверок и ссылка на REPORT.md: TODO
|
||||
Ограничения / известные ошибки: TODO
|
||||
Источники и использование ИИ: TODO
|
||||
Целевая оценка (3/4/5): **4**.
|
||||
|
||||
- [ ] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
|
||||
- [ ] Контракт и публичные тесты не ослаблены.
|
||||
- [ ] make check и make test выполнены; приложен реальный вывод.
|
||||
- [ ] Сценарии отказов/безопасности своего уровня воспроизведены.
|
||||
- [ ] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
|
||||
- [ ] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
|
||||
Что реализовано и что перенесено: приложение lab5 с API-репликами, БД, S3, брокером и worker сохранено. Добавлены OTLP metrics/logs/traces, Collector, Prometheus/Loki/Tempo, Grafana provisioning, сквозной trace context с retry, SLO и два алерта.
|
||||
|
||||
Команды запуска из чистого clone:
|
||||
|
||||
```sh
|
||||
cp .env.example .env
|
||||
# Задать локальные пароли в .env.
|
||||
docker compose up -d --build
|
||||
curl http://127.0.0.1:8080/health/ready # дождаться 200
|
||||
python -X utf8 scripts/check_contract.py
|
||||
python -X utf8 tests/smoke.py --base-url http://127.0.0.1:8080
|
||||
python -X utf8 student-tests/observability.py --scenario all
|
||||
```
|
||||
|
||||
Результаты проверок: check, публичный smoke и проверка сигналов — PASS; сценарии worker/БД/S3/Tempo пройдены, алерты firing → resolved. Нагрузка: 2389 запросов за 30s, без ошибок. [REPORT.md](../REPORT.md), [вывод](../evidence/verification.txt), [сценарии](../evidence/observability.json), [нагрузка](../evidence/load.json).
|
||||
|
||||
Ограничения / известные ошибки: один узел, без HA/TLS и внешней доставки алертов; при длительном отказе backend допустима потеря телеметрии. Блокирующих ошибок в проверенных сценариях не выявлено.
|
||||
|
||||
Источники и использование ИИ: контракт лабораторной, документация OpenTelemetry и Loki; Codex — реализация, тестирование и оформление. Ссылки в REPORT.md.
|
||||
|
||||
- [x] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
|
||||
- [x] Контракт и публичные тесты не ослаблены.
|
||||
- [x] make check и make test выполнены эквивалентными Python-командами; приложен реальный вывод.
|
||||
- [x] Сценарии отказов/безопасности своего уровня воспроизведены.
|
||||
- [x] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
|
||||
- [x] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
|
||||
|
||||
@@ -1,37 +1,72 @@
|
||||
# Отчёт — лабораторная 6
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
- Автор, группа: Ромашко Владимир, БВТ2301.
|
||||
- Целевая оценка: **4**.
|
||||
- Источники: [контракт](contracts/observability.md), [OpenTelemetry](https://opentelemetry.io/docs/collector/configuration/), [Loki OTLP](https://grafana.com/docs/loki/latest/send-data/otel/). Библиотеки: FastAPI, SQLAlchemy, Redis, boto3, Pillow, OpenTelemetry SDK, psutil. Codex использовался для реализации, проверок и оформления.
|
||||
|
||||
## Запуск из чистого clone
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
Нужны Docker с Compose и Python 3.9+ на хосте; свободные порты 8080, 3000, 9090, 3100, 3200. В контейнерах Python 3.12.8; PostgreSQL 17.4, Redis 7.4.2, Nginx 1.28.0; MinIO закреплён digest. Телеметрия: SDK 1.38.0 / semantic conventions 0.59b0, Collector 0.136.0, Prometheus 3.6.0, Loki 3.5.7, Tempo 2.9.0, Grafana 12.2.0.
|
||||
|
||||
```sh
|
||||
cp .env.example .env
|
||||
# Задать свои POSTGRES_PASSWORD, MINIO_ROOT_PASSWORD,
|
||||
# S3_SECRET_ACCESS_KEY и GRAFANA_ADMIN_PASSWORD.
|
||||
docker compose config --quiet
|
||||
docker compose up -d --build
|
||||
curl http://127.0.0.1:8080/health/ready # дождаться 200 {"status":"ok"}
|
||||
python -X utf8 scripts/check_contract.py
|
||||
python -X utf8 tests/smoke.py --base-url http://127.0.0.1:8080
|
||||
python -X utf8 student-tests/observability.py --scenario all
|
||||
python -X utf8 tests/load.py --base-url http://127.0.0.1:8080 --duration 30 --concurrency 4 --output evidence/load.json
|
||||
docker compose down # безопасная остановка, volumes остаются
|
||||
```
|
||||
|
||||
Миграции (включая 008 с trace context) и создание S3 bucket выполняются автоматически. Grafana: http://localhost:3000, `admin`, пароль из `.env`; dashboard **Lab 6 / Messenger / Lab 6** импортируется автоматически. Python-команды check/smoke эквивалентны `make check` / `make test`; на проверявшемся Windows-хосте make отсутствует. Для **намеренного удаления всех учебных данных**: `docker compose down -v` (при проверках не выполнялось).
|
||||
|
||||
## Архитектура и решения
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab5, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
Из lab5 сохранены Nginx → две API-реплики, PostgreSQL, Redis для сессий/кеша, S3, durable Redis Streams, outbox/dispatcher и worker. PostgreSQL — источник истины для заданий; attachment и outbox фиксируются одной транзакцией после записи staging, ack выполняется после устойчивого результата.
|
||||
|
||||
Добавлены API/worker/dispatcher → OTLP → Collector → Prometheus (метрики), Loki (логи), Tempo (трассы) → Grafana. История хранится в volumes, retention 72 часа. W3C context сохраняется в outbox и переносится через broker в worker и image process; retry остаётся в той же трассе. Экспортируются шаблоны маршрутов и технические поля, без SQL-параметров, токенов и содержимого сообщений.
|
||||
|
||||
Метрики: `http_requests_total` и `jobs_processed_total` — счётчики; `http_duration_seconds`, `job_duration_seconds`, `attachment_time_to_ready_seconds` — histogram в секундах; `queue_depth`, `db_pool_used/waiting`, `worker_busy/capacity` — количества; `queue_oldest_age_seconds` — секунды; `container_memory_bytes` / `container_cpu_seconds_total` — память/CPU. Точные имена, labels и запросы: [описание сигналов](infra/otel/README.md#сигналы), [dashboard JSON](infra/grafana/dashboards/messenger.json).
|
||||
|
||||
Переменные `.env.example` подставляются в Compose: пароли — соответствующим сервисам, `MAX_*`, `JOB_*`, `WORKER_CONCURRENCY` — приложению, `APP_PORT` — Nginx. `OTEL_SDK_DISABLED=false` включает телеметрию; service.name и внутренние адреса заданы отдельно по ролям.
|
||||
|
||||
SLI = 1 − 5xx / все запросы без health за **30 минут**, SLO ≥ **99%**. `ApiHighErrorRate`: >5% ошибок за 2m, минимум 20 запросов, `for: 20s`. `QueueLagHigh`: возраст queued >30s при свежем sample, `for: 15s`. Правила и [инструкции диагностики](infra/otel/README.md#queue-lag) сохранены в репозитории.
|
||||
|
||||
## Доказательства критериев
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
| Структура и публичный smoke | Команды check/smoke выше | PASS при включённой телеметрии | [verification.txt](evidence/verification.txt) |
|
||||
| Минимум lab5 и общий стенд | `docker compose up -d --build`; smoke | API, upload 202, ready, оригинал и thumbnail работают; хранилища имеют volumes | [compose.yaml](compose.yaml), verification.txt |
|
||||
| OTLP: метрики, логи, трассы API/worker | `observability.py --scenario smoke` | Все обязательные метрики найдены; связанные логи трёх сервисов | [observability.json: smoke](evidence/observability.json) |
|
||||
| Импортируемый dashboard и безопасные атрибуты | Provisioning; сценарий smoke | 18 панелей, 3 datasource; проверены labels и отсутствие приватных входных данных | [provisioning](infra/grafana/provisioning), observability.json |
|
||||
| Корреляция HTTP/job → log/trace | Сценарий smoke | Найдены request_id, trace_id и parent-связи до S3/DB | observability.json: smoke |
|
||||
| W3C propagation, включая retry | `observability.py --scenario retry` | Попытки 1 и 2 в одной трассе, tracestate сохранён | observability.json: retry |
|
||||
| p50/p95/p99, очередь, DB pool, worker, CPU/RAM, time-to-ready | Сценарий smoke; dashboard | Метрики существуют; percentile вычисляется по сумме histogram buckets | dashboard JSON, observability.json |
|
||||
| SLI/SLO и два проверенных алерта | `--scenario queue`, `--scenario database` | Оба алерта firing → resolved после восстановления | [alerts.yaml](infra/otel/alerts.yaml), observability.json |
|
||||
| Ограниченная буферизация и отказ telemetry backend | `--scenario backend` | API продолжает работу; очередь ≤32 batches, RSS ≤199.03 MiB | [collector.yaml](infra/otel/collector.yaml), observability.json |
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
Здесь `observability.py` — `python -X utf8 student-tests/observability.py`. Сценарии останавливают реальные контейнеры текущего Compose-проекта и восстанавливают их в `finally`; volumes не удаляют.
|
||||
|
||||
## Эксперименты и отказы
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
Проверки 16.11.2026. Гипотеза: телеметрия показывает причину отказа, а её backend не блокирует бизнес-операции. Docker Desktop: 8 CPU, 7.76 GiB RAM; лимиты API — по 1 CPU/512 MiB, worker — 2 CPU/2 GiB, Collector — 0.5 CPU/256 MiB.
|
||||
|
||||
Нагрузка: один прогон closed-loop POST сообщений, 4 клиента, 30 секунд, без отдельного прогрева и автоматических retry. **2389 успешных запросов, 0 ошибок, 79.219 RPS; p50/p95/p99 = 40.784/99.520/129.314 ms**. Исходная сводка: [load.json](evidence/load.json). Это локальная проверка, не исследование предельной производительности; CPU во время прогона отдельно не фиксировался, RSS Collector измерен в сценарии отказа ниже.
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
| Остановить worker | Рост возраста очереди и QueueLagHigh | Depth=1, возраст 55s; producer есть, consumer появляется после запуска; затем ready, depth=0 | firing → resolved: 20.3s; принятое вложение сохранено |
|
||||
| Остановить PostgreSQL | 503 и ApiHighErrorRate | Error ratio 42.6%; в trace `db.acquire`/`OperationalError` и связанный лог — отказ БД, а не медленный handler | После запуска запросы успешны; firing → resolved: 109.1s с учётом окна 2m |
|
||||
| Временно остановить S3 | Retry без потери задания | Первая попытка failed, вторая ready, общий trace | Задание успешно завершено |
|
||||
| Остановить Tempo на 60s | Приложение работает, буферы ограничены | 165 записей и обработка вложения успешны; очередь ≤32, RSS 198.52–199.02 MiB; после запуска новая трасса найдена | Потеря 808 принятых Collector spans, бизнес-операции успешны |
|
||||
|
||||
Порядок диагностики: dashboard → лог по request_id/trace_id → ошибочный или отсутствующий span зависимости → восстановление сервиса → проверка ready/успешных запросов и resolved. Для Tempo потеря рассчитана как Δaccepted − Δsent после опустошения очереди; повторы отправки не считаются отдельными потерянными spans. Подробные timestamps и ID — в [observability.json](evidence/observability.json).
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
Один локальный узел, без HA и TLS; алерты только локальные. SDK ограничивает очереди 512 записями, Collector использует memory limiter/batch и retry до 15s: длительный отказ может терять телеметрию. Потери до приёма Collector указанной дельтой не измеряются. История удаляется по retention; метрики не заменяют журнал транзакций. При недоступной БД API возвращает 503, после восстановления работает снова. Треки оценки 5 не реализовывались.
|
||||
|
||||
+86
-1
@@ -1,9 +1,10 @@
|
||||
x-api: &api
|
||||
image: mtusi2026lab6-api:local
|
||||
pull_policy: never
|
||||
dns_opt:
|
||||
- timeout:1
|
||||
- attempts:1
|
||||
environment:
|
||||
environment: &app-env
|
||||
APP_HOST: 0.0.0.0
|
||||
APP_PORT: 8080
|
||||
POSTGRES_HOST: postgres
|
||||
@@ -29,6 +30,9 @@ x-api: &api
|
||||
JOB_TIMEOUT_SECONDS: ${JOB_TIMEOUT_SECONDS:-60}
|
||||
JOB_MEMORY_MB: ${JOB_MEMORY_MB:-768}
|
||||
STAGING_RETENTION_SECONDS: ${STAGING_RETENTION_SECONDS:-86400}
|
||||
OTEL_SERVICE_NAME: messenger-api
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4318
|
||||
OTEL_SDK_DISABLED: ${OTEL_SDK_DISABLED:-false}
|
||||
networks: [backend]
|
||||
cpus: 1.0
|
||||
mem_limit: 512m
|
||||
@@ -51,6 +55,9 @@ services:
|
||||
|
||||
dispatcher:
|
||||
<<: *api
|
||||
environment:
|
||||
<<: *app-env
|
||||
OTEL_SERVICE_NAME: messenger-dispatcher
|
||||
command: [python, -m, src.jobs, dispatch]
|
||||
healthcheck:
|
||||
disable: true
|
||||
@@ -64,6 +71,9 @@ services:
|
||||
|
||||
worker:
|
||||
<<: *api
|
||||
environment:
|
||||
<<: *app-env
|
||||
OTEL_SERVICE_NAME: messenger-worker
|
||||
command: [python, -m, src.worker]
|
||||
init: true
|
||||
healthcheck:
|
||||
@@ -101,6 +111,9 @@ services:
|
||||
|
||||
migrate:
|
||||
<<: *api
|
||||
environment:
|
||||
<<: *app-env
|
||||
OTEL_SDK_DISABLED: 'true'
|
||||
command: [python, -m, src.database]
|
||||
healthcheck:
|
||||
disable: true
|
||||
@@ -150,6 +163,9 @@ services:
|
||||
|
||||
cleanup:
|
||||
<<: *api
|
||||
environment:
|
||||
<<: *app-env
|
||||
OTEL_SDK_DISABLED: 'true'
|
||||
command: [python, -m, src.cleanup]
|
||||
healthcheck:
|
||||
disable: true
|
||||
@@ -217,6 +233,71 @@ services:
|
||||
cpus: 0.5
|
||||
mem_limit: 256m
|
||||
|
||||
otel-collector:
|
||||
image: otel/opentelemetry-collector-contrib:0.136.0
|
||||
command: [--config=/etc/otel/collector.yaml]
|
||||
volumes:
|
||||
- ./infra/otel/collector.yaml:/etc/otel/collector.yaml:ro
|
||||
networks: [backend]
|
||||
mem_limit: 256m
|
||||
cpus: 0.5
|
||||
restart: unless-stopped
|
||||
|
||||
prometheus:
|
||||
image: prom/prometheus:v3.6.0
|
||||
command: [--config.file=/etc/prometheus/prometheus.yaml, --storage.tsdb.retention.time=3d]
|
||||
volumes:
|
||||
- ./infra/otel/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro
|
||||
- ./infra/otel/alerts.yaml:/etc/prometheus/alerts.yaml:ro
|
||||
- prometheus_data:/prometheus
|
||||
ports: ['127.0.0.1:9090:9090']
|
||||
networks: [edge, backend]
|
||||
mem_limit: 512m
|
||||
cpus: 0.5
|
||||
restart: unless-stopped
|
||||
|
||||
loki:
|
||||
image: grafana/loki:3.5.7
|
||||
command: [-config.file=/etc/loki/local.yaml]
|
||||
volumes:
|
||||
- ./infra/otel/loki.yaml:/etc/loki/local.yaml:ro
|
||||
- loki_data:/loki
|
||||
ports: ['127.0.0.1:3100:3100']
|
||||
networks: [edge, backend]
|
||||
mem_limit: 384m
|
||||
cpus: 0.5
|
||||
restart: unless-stopped
|
||||
|
||||
tempo:
|
||||
image: grafana/tempo:2.9.0
|
||||
command: [-config.file=/etc/tempo.yaml]
|
||||
volumes:
|
||||
- ./infra/otel/tempo.yaml:/etc/tempo.yaml:ro
|
||||
- tempo_data:/var/tempo
|
||||
ports: ['127.0.0.1:3200:3200']
|
||||
networks: [edge, backend]
|
||||
mem_limit: 384m
|
||||
cpus: 0.5
|
||||
restart: unless-stopped
|
||||
|
||||
grafana:
|
||||
image: grafana/grafana:12.2.0
|
||||
environment:
|
||||
GF_SECURITY_ADMIN_USER: admin
|
||||
GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:?Set GRAFANA_ADMIN_PASSWORD in .env}
|
||||
GF_USERS_ALLOW_SIGN_UP: 'false'
|
||||
GF_ANALYTICS_REPORTING_ENABLED: 'false'
|
||||
GF_ANALYTICS_CHECK_FOR_UPDATES: 'false'
|
||||
volumes:
|
||||
- ./infra/grafana/provisioning:/etc/grafana/provisioning:ro
|
||||
- ./infra/grafana/dashboards:/var/lib/grafana/dashboards:ro
|
||||
- grafana_data:/var/lib/grafana
|
||||
ports: ['127.0.0.1:3000:3000']
|
||||
networks: [edge, backend]
|
||||
mem_limit: 384m
|
||||
cpus: 0.5
|
||||
restart: unless-stopped
|
||||
|
||||
networks:
|
||||
edge:
|
||||
backend:
|
||||
@@ -226,3 +307,7 @@ volumes:
|
||||
postgres_data:
|
||||
redis_data:
|
||||
s3_data:
|
||||
prometheus_data:
|
||||
loki_data:
|
||||
tempo_data:
|
||||
grafana_data:
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"lab": 6,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 30.0,
|
||||
"elapsed_seconds": 30.157,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 2389,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 79.219,
|
||||
"successful_latency_ms": {
|
||||
"p50": 40.784,
|
||||
"p95": 99.52,
|
||||
"p99": 129.314
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,5 @@
|
||||
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||
PASS: lab6 public API scenario; infrastructure and grading evidence remain separate.
|
||||
Running smoke
|
||||
PASS: smoke
|
||||
PASS: Grafana provisioning (18 panels, 3 datasources), all dashboard queries, no active alerts; local secrets absent from Git candidates; assignment/contracts/public tests unchanged.
|
||||
@@ -0,0 +1,716 @@
|
||||
{
|
||||
"uid": "messenger-lab6",
|
||||
"title": "Messenger / Lab 6",
|
||||
"schemaVersion": 41,
|
||||
"version": 1,
|
||||
"refresh": "5s",
|
||||
"time": {
|
||||
"from": "now-30m",
|
||||
"to": "now"
|
||||
},
|
||||
"tags": [
|
||||
"lab6"
|
||||
],
|
||||
"panels": [
|
||||
{
|
||||
"id": 1,
|
||||
"title": "API requests / second",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 0,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"description": "Business requests only; health probes are excluded.",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum by (method,route) (rate(http_requests_total{service_name=\"messenger-api\"}[2m]))",
|
||||
"legendFormat": "{{method}} {{route}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "reqps"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"title": "API server error ratio",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 0,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"description": "5xx / all non-health requests, rolling 2m. 4xx are not server errors.",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "messenger:http_error_ratio:2m",
|
||||
"legendFormat": "5xx / all"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"title": "API duration p50 / p95 / p99",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 8,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"description": "Quantiles from merged histogram buckets across API replicas. Never averages of per-instance percentiles.",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "histogram_quantile(0.5, sum by (le) (rate(http_duration_seconds_bucket{service_name=\"messenger-api\"}[2m])))",
|
||||
"legendFormat": "p50"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "histogram_quantile(0.95, sum by (le) (rate(http_duration_seconds_bucket{service_name=\"messenger-api\"}[2m])))",
|
||||
"legendFormat": "p95"
|
||||
},
|
||||
{
|
||||
"refId": "C",
|
||||
"expr": "histogram_quantile(0.99, sum by (le) (rate(http_duration_seconds_bucket{service_name=\"messenger-api\"}[2m])))",
|
||||
"legendFormat": "p99"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"title": "Job attempts / second",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 8,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"description": "One count per claimed attempt; failed includes retryable failures, skipped is a fenced attempt.",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum by (outcome) (rate(jobs_processed_total[2m]))",
|
||||
"legendFormat": "{{outcome}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"title": "Worker processing duration p50 / p95 / p99",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 16,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"description": "Includes isolated image process and durable result check; excludes queue wait.",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "histogram_quantile(0.5, sum by (le) (rate(job_duration_seconds_bucket[2m])))",
|
||||
"legendFormat": "p50"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "histogram_quantile(0.95, sum by (le) (rate(job_duration_seconds_bucket[2m])))",
|
||||
"legendFormat": "p95"
|
||||
},
|
||||
{
|
||||
"refId": "C",
|
||||
"expr": "histogram_quantile(0.99, sum by (le) (rate(job_duration_seconds_bucket[2m])))",
|
||||
"legendFormat": "p99"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"title": "Time from accepted upload to ready",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 16,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"description": "Database timestamps: durable attachment creation to ready update. Includes queue wait and retries; observed on successful attempts.",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "histogram_quantile(0.5, sum by (le) (rate(attachment_time_to_ready_seconds_bucket[5m])))",
|
||||
"legendFormat": "p50"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "histogram_quantile(0.95, sum by (le) (rate(attachment_time_to_ready_seconds_bucket[5m])))",
|
||||
"legendFormat": "p95"
|
||||
},
|
||||
{
|
||||
"refId": "C",
|
||||
"expr": "histogram_quantile(0.99, sum by (le) (rate(attachment_time_to_ready_seconds_bucket[5m])))",
|
||||
"legendFormat": "p99"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"title": "Durable queue depth",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 24,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"description": "PostgreSQL queued attachments, including unpublished outbox/retries; excludes processing jobs.",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "max(queue_depth)",
|
||||
"legendFormat": "queued"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"title": "Oldest queued attachment age",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 24,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"description": "Age since acceptance. A stale sample is not zero; check queue sample age below.",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "max(queue_oldest_age_seconds)",
|
||||
"legendFormat": "age"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 9,
|
||||
"title": "DB pool used / acquiring",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 32,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"description": "Per long-lived process. Acquiring includes connection creation and pool wait; max pool size is 10.",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "db_pool_used",
|
||||
"legendFormat": "{{service_name}} {{service_instance_id}} used"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "db_pool_waiting",
|
||||
"legendFormat": "{{service_name}} {{service_instance_id}} acquiring"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 10,
|
||||
"title": "Worker saturation",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 32,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"description": "Busy consumers / configured concurrency. Missing capacity after 30s indicates stopped worker.",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum(worker_busy)",
|
||||
"legendFormat": "busy"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "sum(worker_capacity)",
|
||||
"legendFormat": "capacity"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 11,
|
||||
"title": "Container memory (includes image children)",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 40,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"description": "cgroup v2 memory.current. Includes child processes and charged cache.",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "container_memory_bytes",
|
||||
"legendFormat": "{{service_name}} {{service_instance_id}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "bytes"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 12,
|
||||
"title": "Container CPU",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 40,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"description": "cgroup v2 CPU seconds / second; 1.0 is one fully used CPU core.",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "rate(container_cpu_seconds_total[2m])",
|
||||
"legendFormat": "{{service_name}} {{service_instance_id}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 13,
|
||||
"title": "API availability SLI / SLO",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 48,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"description": "Target: >=99% non-5xx business requests in a rolling 30m window. No traffic is not evidence that SLO is met.",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "1 - ((sum(increase(http_requests_total{service_name=\"messenger-api\",status_class=\"5xx\"}[30m])) or vector(0)) / clamp_min(sum(increase(http_requests_total{service_name=\"messenger-api\"}[30m])), 1))",
|
||||
"legendFormat": "SLI over 30m"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "vector(0.99)",
|
||||
"legendFormat": "SLO"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 14,
|
||||
"title": "Local alerts",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 48,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"description": "1 means pending/firing. Series disappears after resolution. Rules and runbooks are stored in infra/otel.",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "ALERTS{alertname=~\"ApiHighErrorRate|QueueLagHigh\"}",
|
||||
"legendFormat": "{{alertname}} {{alertstate}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 15,
|
||||
"title": "Collector memory",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 56,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"description": "",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "otelcol_process_memory_rss_bytes",
|
||||
"legendFormat": "RSS"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "bytes"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 16,
|
||||
"title": "Collector exporter queue / capacity",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 56,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"description": "Bounded queues: 32 batches per backend, up to 256 records per batch.",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "otelcol_exporter_queue_size",
|
||||
"legendFormat": "{{exporter}} queued"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "otelcol_exporter_queue_capacity",
|
||||
"legendFormat": "{{exporter}} capacity"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 17,
|
||||
"title": "Collector failed sends / rejected enqueue",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 64,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"description": "Send failures may include retries, not unique lost spans. Exact experiment loss is measured by accepted/sent deltas after draining.",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "rate(otelcol_exporter_send_failed_spans_total[2m])",
|
||||
"legendFormat": "{{exporter}} span send failures"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "rate(otelcol_exporter_send_failed_log_records_total[2m])",
|
||||
"legendFormat": "{{exporter}} log send failures"
|
||||
},
|
||||
{
|
||||
"refId": "C",
|
||||
"expr": "rate(otelcol_exporter_enqueue_failed_spans_total[2m])",
|
||||
"legendFormat": "{{exporter}} span queue rejections"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 18,
|
||||
"title": "Queue sample age",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 64,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"description": "Above 15s: queue measurements are stale (e.g. DB outage).",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "time() - max(queue_sample_timestamp_seconds)",
|
||||
"legendFormat": "seconds since DB sample"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"links": [
|
||||
{
|
||||
"title": "Explore logs",
|
||||
"url": "/explore?left=%7B%22datasource%22%3A%22loki%22%7D"
|
||||
},
|
||||
{
|
||||
"title": "Explore traces",
|
||||
"url": "/explore?left=%7B%22datasource%22%3A%22tempo%22%7D"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
apiVersion: 1
|
||||
providers:
|
||||
- name: messenger
|
||||
folder: Lab 6
|
||||
type: file
|
||||
options:
|
||||
path: /var/lib/grafana/dashboards
|
||||
@@ -0,0 +1,41 @@
|
||||
apiVersion: 1
|
||||
datasources:
|
||||
- name: Prometheus
|
||||
uid: prometheus
|
||||
type: prometheus
|
||||
access: proxy
|
||||
url: http://prometheus:9090
|
||||
isDefault: true
|
||||
jsonData:
|
||||
timeInterval: 5s
|
||||
exemplarTraceIdDestinations:
|
||||
- name: trace_id
|
||||
datasourceUid: tempo
|
||||
- name: Loki
|
||||
uid: loki
|
||||
type: loki
|
||||
access: proxy
|
||||
url: http://loki:3100
|
||||
jsonData:
|
||||
derivedFields:
|
||||
- name: trace_id
|
||||
matcherType: label
|
||||
matcherRegex: trace_id
|
||||
datasourceUid: tempo
|
||||
url: '$${__value.raw}'
|
||||
urlDisplayLabel: View trace
|
||||
- name: Tempo
|
||||
uid: tempo
|
||||
type: tempo
|
||||
access: proxy
|
||||
url: http://tempo:3200
|
||||
jsonData:
|
||||
tracesToLogsV2:
|
||||
datasourceUid: loki
|
||||
spanStartTimeShift: '-5m'
|
||||
spanEndTimeShift: '5m'
|
||||
tags: [{key: service.name, value: service_name}]
|
||||
filterByTraceID: true
|
||||
filterBySpanID: false
|
||||
nodeGraph:
|
||||
enabled: true
|
||||
+137
-4
@@ -1,7 +1,140 @@
|
||||
# Конфигурация наблюдаемости — реализовать
|
||||
# Лаба 6: запуск и диагностика
|
||||
|
||||
Добавьте `collector.yaml` с OTLP receivers, processors и exporters выбранной версии Collector. Укажите явные bind endpoints, внутренние сети, memory limits/batch и поведение при backpressure по уровню. Сопоставьте все три pipeline с реально развёрнутыми backend.
|
||||
```sh
|
||||
cp .env.example .env
|
||||
# Задать локальные пароли в .env (в Git не добавлять).
|
||||
docker compose up -d --build
|
||||
python -X utf8 tests/smoke.py --base-url http://127.0.0.1:8080
|
||||
python -X utf8 tests/load.py --base-url http://127.0.0.1:8080 --duration 30 --concurrency 4 --output evidence/load.json
|
||||
python -X utf8 student-tests/observability.py --scenario all
|
||||
```
|
||||
|
||||
Добавьте конфигурации storage, Grafana provisioning/datasources, provisioning/dashboards и импортируемые dashboard JSON. На 4 — alert rules и runbooks. Пустой dashboard или debug exporter, который только печатает данные в stdout, не заменяют хранилища/доказательства из `contracts/observability.md`.
|
||||
Последняя команда создаёт синтетические данные и временно останавливает **только
|
||||
сервисы текущего Compose-проекта**. В `finally` запускает остановленные сервисы.
|
||||
Не запускать параллельно с другими проверками этого стенда. Volumes не удаляются.
|
||||
Санитизированные результаты сохраняются в `evidence/observability.json`.
|
||||
|
||||
Не публикуйте OTLP/metrics/административные ports в общую сеть без необходимости. В lab7 реализуйте TLS/mTLS по trust matrix; экспорт Authorization и message bodies запрещён.
|
||||
Grafana: http://localhost:3000, пользователь `admin`, пароль `GRAFANA_ADMIN_PASSWORD`
|
||||
из `.env`. Dashboard **Lab 6 / Messenger / Lab 6** и три datasource загружаются
|
||||
автоматически. Prometheus: http://localhost:9090; Loki API: порт 3100; Tempo API:
|
||||
3200. Все опубликованные порты слушают только loopback, OTLP наружу не опубликован.
|
||||
|
||||
Collector принимает OTLP/HTTP от API, dispatcher и worker. Prometheus хранит
|
||||
метрики, Loki — логи, Tempo — трассы; Grafana обращается к этим хранилищам.
|
||||
История сохраняется в именованных volumes, retention — 72 часа.
|
||||
Версии закреплены в Compose и requirements.txt: OTel Python 1.38.0, semantic
|
||||
conventions 0.59b0 (зависимость SDK); наши имена метрик перечислены ниже.
|
||||
Это явная инструментация: auto-instrumentation, сбор SQL и HTTP headers отключены.
|
||||
|
||||
## Сигналы
|
||||
|
||||
| Prometheus name | Единицы и смысл |
|
||||
| --- | --- |
|
||||
| `http_requests_total` | Число бизнес-запросов; method, route template, status_class |
|
||||
| `http_duration_seconds` | Histogram полного HTTP-запроса, включая отдачу stream |
|
||||
| `jobs_processed_total` | Попытки обработки; job_type, outcome=ready/failed/skipped |
|
||||
| `job_duration_seconds` | Histogram времени попытки, без ожидания в очереди |
|
||||
| `attachment_time_to_ready_seconds` | Histogram от записи attachment до durable ready, включая retry |
|
||||
| `queue_depth` | Количество queued в PostgreSQL, включая outbox и отложенные retry |
|
||||
| `queue_oldest_age_seconds` | Возраст старейшего queued от момента приёма, не время обработки |
|
||||
| `queue_sample_timestamp_seconds` | Unix-время последнего успешного опроса очереди |
|
||||
| `db_pool_used`, `db_pool_waiting` | Занятые соединения / потоки, получающие соединение (ожидание + reconnect) |
|
||||
| `worker_busy`, `worker_capacity` | Число занятых consumer / лимит concurrency |
|
||||
| `process_memory_bytes`, `process_cpu_seconds_total` | RSS и CPU процесса |
|
||||
| `container_memory_bytes`, `container_cpu_seconds_total` | cgroup v2, включая дочерние процессы обработки |
|
||||
|
||||
Histogram buckets, секунды: .005, .01, .025, .05, .1, .25, .5, 1, 2, 3, 5, 10,
|
||||
30, 60, 120, 300, +Inf. Dashboard содержит PromQL, единицы и описания каждого
|
||||
графика. p50/p95/p99 считаются `histogram_quantile` по сумме buckets всех реплик.
|
||||
Метрики очереди отдаёт только dispatcher; это не длина Redis Stream (в нём
|
||||
ещё могут быть уже взятые сообщения). При недоступной БД sample становится
|
||||
устаревшим; возраст sample выведен отдельно, QueueLagHigh не использует stale data.
|
||||
|
||||
Resource: service.name, service.version=1.6.0, service.instance.id=hostname,
|
||||
deployment.environment.name=lab. API: messenger-api; worker и его дочерний
|
||||
процесс: messenger-worker; publisher: messenger-dispatcher. Дочерний процесс
|
||||
экспортирует только traces/logs, чтобы не создавать отдельные краткоживущие
|
||||
ряды метрик. Parent измеряет длительность попытки и результат после COMMIT.
|
||||
Метрики наблюдательные: авария после COMMIT до записи счётчика может потерять
|
||||
измерение, но не результат обработки. Предметных ID в labels метрик нет.
|
||||
|
||||
## Связь сигналов
|
||||
|
||||
SERVER span API → traceparent/tracestate в transactional outbox → PRODUCER span
|
||||
dispatcher → W3C context в JSON сообщения Redis → CONSUMER span worker → context
|
||||
через окружение изолированного image process → S3/DB CLIENT spans. Retry публикует
|
||||
новый producer/consumer в той же трассе, с `job.attempt`. Baggage не переносится.
|
||||
Context сохранён в миграции 008, поэтому рестарт API не разрывает причинную связь.
|
||||
|
||||
Логи содержат timestamp/severity/service.name/event и trace_id/span_id активного
|
||||
span. `http_request` содержит ограниченный `request_id`, совпадающий с
|
||||
X-Request-Id ответа. Loki сохраняет технические поля как structured metadata;
|
||||
индексирует только service.name и окружение. Для корреляции достаточно request_id,
|
||||
trace_id и номера попытки: имена файлов и предметные ID не экспортируются.
|
||||
|
||||
В Grafana Explore / Loki:
|
||||
|
||||
```logql
|
||||
{service_name="messenger-api"} | request_id="ЗНАЧЕНИЕ_X_REQUEST_ID"
|
||||
{service_name=~"messenger-.*"} | trace_id="TRACE_ID"
|
||||
```
|
||||
|
||||
Поле trace_id открывает Tempo; из Tempo доступен переход к логам. В трассе upload
|
||||
видны `POST /api/v1/chats/{chat_id}/attachments`, `attachment-jobs publish`,
|
||||
`attachment-jobs process`, `image.process`, `s3.GetObject`, `s3.PutObject`,
|
||||
`db.SELECT`/`db.UPDATE`. Исключения содержат только тип, без message/stack/SQL.
|
||||
|
||||
## SLI/SLO и alerts
|
||||
|
||||
SLI доступности = 1 − число 5xx / число всех бизнес-запросов за скользящие **30 минут**.
|
||||
SLO: **не менее 99%**. Health исключён из числителя и знаменателя; 4xx не считается
|
||||
отказом сервера. Без запросов SLI не оцениваем. Это учебное окно, а не месячный SLO.
|
||||
|
||||
`ApiHighErrorRate`: 5xx > 5% за 2 минуты, минимум 20 запросов, `for: 20s`.
|
||||
`QueueLagHigh`: queued > 0 и возраст > 30s, свежий sample, `for: 15s`.
|
||||
Пороги короткие для воспроизводимой локальной демонстрации; алерты не являются
|
||||
burn-rate исследованием. Rules вычисляет Prometheus; firing/pending видны в
|
||||
http://localhost:9090/alerts и на dashboard. После восстановления alert пропадает
|
||||
из instant `ALERTS`; история переходов остаётся в range query. Доставка наружу не настроена.
|
||||
|
||||
## Queue lag
|
||||
|
||||
1. Открыть queue depth/age и worker capacity/busy. Растущая очередь при отсутствии
|
||||
capacity означает остановку worker; свежий sample подтверждает доступность БД.
|
||||
2. В Loki найти `job_published`, затем открыть trace: producer есть, consumer ещё нет.
|
||||
3. `docker compose start worker`; дождаться `job_processed` outcome=ready, depth=0.
|
||||
4. Проверить, что QueueLagHigh больше не firing и в той же трассе появился consumer.
|
||||
|
||||
## API errors
|
||||
|
||||
1. При ApiHighErrorRate найти http_request со status_class=5xx и взять trace_id.
|
||||
2. В Tempo проверить `db.acquire`/`db.SELECT`: error.type и длительность покажут
|
||||
отказ подключения/БД. В Loki есть `db_error`/`operation_failed`. Медленный handler
|
||||
без отказа зависимости не имел бы ошибочного DB span. Посмотреть db_pool_waiting.
|
||||
3. Проверить PostgreSQL: `docker compose ps postgres`; восстановить
|
||||
`docker compose start postgres`, дождаться readiness и успешного бизнес-запроса.
|
||||
4. Через окно 2m error ratio снизится, alert перейдёт в resolved. Sample очереди
|
||||
тоже должен стать свежим. Сценарий проверки сохраняет трассу ошибочного запроса.
|
||||
|
||||
## Недоступность telemetry backend
|
||||
|
||||
SDK: очереди по 512 spans/logs, batch 64, HTTP timeout 1s; metric export каждые 5s.
|
||||
Отправка выполняется в фоновых потоках. Collector: memory limiter 192 MiB,
|
||||
spike 48 MiB, container limit 256 MiB; batch ≤256 записей; отдельная очередь
|
||||
32 batches / 2 consumers на Loki и Tempo; retry до 15s, timeout 2s, overflow
|
||||
не блокирует приём. Конечные ограничения защищают память, но допускают потерю
|
||||
телеметрии при длительном отказе. API и pipeline не зависят от готовности backend.
|
||||
|
||||
Сценарий `backend` останавливает Tempo, продолжает создавать сообщения и
|
||||
обрабатывать изображения, снимает RSS/очередь Collector, затем запускает Tempo
|
||||
и проверяет новую трассу. `send_failed_spans` может считать повторные попытки:
|
||||
его нельзя выдавать за число уникально потерянных spans. Для отдельного окна
|
||||
тест вычисляет accepted − sent после опустошения очереди (без рестартов Collector)
|
||||
и сохраняет эти дельты вместе с наблюдаемой памятью. Это потери уже принятых
|
||||
Collector spans; отброшенные до приёма SDK/receiver в этот счёт не входят.
|
||||
Prometheus всё это время
|
||||
работает, поэтому сохраняется история отказа и восстановления.
|
||||
|
||||
Первичные источники: [Collector](https://opentelemetry.io/docs/collector/configuration/),
|
||||
[native Loki OTLP](https://grafana.com/docs/loki/latest/send-data/otel/),
|
||||
[bounded exporter queues](https://github.com/open-telemetry/opentelemetry-collector/blob/v0.136.0/exporter/exporterhelper/README.md).
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
groups:
|
||||
- name: messenger
|
||||
rules:
|
||||
- record: messenger:http_error_ratio:2m
|
||||
expr: |
|
||||
(sum(rate(http_requests_total{service_name="messenger-api",status_class="5xx"}[2m])) or vector(0))
|
||||
/ clamp_min(sum(rate(http_requests_total{service_name="messenger-api"}[2m])), 0.001)
|
||||
- alert: ApiHighErrorRate
|
||||
expr: messenger:http_error_ratio:2m > 0.05 and sum(increase(http_requests_total{service_name="messenger-api"}[2m])) >= 20
|
||||
for: 20s
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: 'API: more than 5% server errors over 2 minutes'
|
||||
description: 'SLI is 5xx / all non-health requests. At least 20 requests in the window.'
|
||||
runbook: 'infra/otel/README.md#api-errors'
|
||||
- alert: QueueLagHigh
|
||||
expr: |
|
||||
max(queue_oldest_age_seconds) > 30
|
||||
and max(queue_depth) > 0
|
||||
and time() - max(queue_sample_timestamp_seconds) < 15
|
||||
for: 15s
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: 'Oldest queued attachment waits more than 30 seconds'
|
||||
description: 'Check worker capacity/busy and job errors; backlog is sampled by dispatcher.'
|
||||
runbook: 'infra/otel/README.md#queue-lag'
|
||||
@@ -0,0 +1,69 @@
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
http:
|
||||
endpoint: 0.0.0.0:4318
|
||||
|
||||
processors:
|
||||
memory_limiter:
|
||||
check_interval: 1s
|
||||
limit_mib: 192
|
||||
spike_limit_mib: 48
|
||||
batch:
|
||||
timeout: 1s
|
||||
send_batch_size: 128
|
||||
send_batch_max_size: 256
|
||||
|
||||
exporters:
|
||||
prometheus:
|
||||
endpoint: 0.0.0.0:8889
|
||||
resource_to_telemetry_conversion:
|
||||
enabled: true
|
||||
enable_open_metrics: true
|
||||
metric_expiration: 30s
|
||||
translation_strategy: UnderscoreEscapingWithSuffixes
|
||||
otlphttp/loki:
|
||||
endpoint: http://loki:3100/otlp
|
||||
timeout: 2s
|
||||
sending_queue:
|
||||
queue_size: 32
|
||||
num_consumers: 2
|
||||
block_on_overflow: false
|
||||
retry_on_failure:
|
||||
initial_interval: 1s
|
||||
max_interval: 5s
|
||||
max_elapsed_time: 15s
|
||||
otlphttp/tempo:
|
||||
endpoint: http://tempo:4318
|
||||
timeout: 2s
|
||||
sending_queue:
|
||||
queue_size: 32
|
||||
num_consumers: 2
|
||||
block_on_overflow: false
|
||||
retry_on_failure:
|
||||
initial_interval: 1s
|
||||
max_interval: 5s
|
||||
max_elapsed_time: 15s
|
||||
|
||||
service:
|
||||
telemetry:
|
||||
metrics:
|
||||
readers:
|
||||
- pull:
|
||||
exporter:
|
||||
prometheus:
|
||||
host: 0.0.0.0
|
||||
port: 8888
|
||||
pipelines:
|
||||
metrics:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, batch]
|
||||
exporters: [prometheus]
|
||||
logs:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, batch]
|
||||
exporters: [otlphttp/loki]
|
||||
traces:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, batch]
|
||||
exporters: [otlphttp/tempo]
|
||||
@@ -0,0 +1,37 @@
|
||||
auth_enabled: false
|
||||
server:
|
||||
http_listen_port: 3100
|
||||
common:
|
||||
path_prefix: /loki
|
||||
replication_factor: 1
|
||||
ring:
|
||||
kvstore:
|
||||
store: inmemory
|
||||
storage:
|
||||
filesystem:
|
||||
chunks_directory: /loki/chunks
|
||||
rules_directory: /loki/rules
|
||||
schema_config:
|
||||
configs:
|
||||
- from: 2024-01-01
|
||||
store: tsdb
|
||||
object_store: filesystem
|
||||
schema: v13
|
||||
index:
|
||||
prefix: index_
|
||||
period: 24h
|
||||
limits_config:
|
||||
allow_structured_metadata: true
|
||||
retention_period: 72h
|
||||
otlp_config:
|
||||
resource_attributes:
|
||||
ignore_defaults: true
|
||||
attributes_config:
|
||||
- action: index_label
|
||||
attributes: [service.name, deployment.environment.name]
|
||||
compactor:
|
||||
working_directory: /loki/compactor
|
||||
retention_enabled: true
|
||||
delete_request_store: filesystem
|
||||
analytics:
|
||||
reporting_enabled: false
|
||||
@@ -0,0 +1,12 @@
|
||||
global:
|
||||
scrape_interval: 5s
|
||||
evaluation_interval: 5s
|
||||
rule_files:
|
||||
- /etc/prometheus/alerts.yaml
|
||||
scrape_configs:
|
||||
- job_name: messenger
|
||||
static_configs:
|
||||
- targets: ['otel-collector:8889']
|
||||
- job_name: collector
|
||||
static_configs:
|
||||
- targets: ['otel-collector:8888']
|
||||
@@ -0,0 +1,23 @@
|
||||
server:
|
||||
http_listen_port: 3200
|
||||
distributor:
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
http:
|
||||
endpoint: 0.0.0.0:4318
|
||||
ingester:
|
||||
trace_idle_period: 10s
|
||||
max_block_duration: 5m
|
||||
compactor:
|
||||
compaction:
|
||||
block_retention: 72h
|
||||
storage:
|
||||
trace:
|
||||
backend: local
|
||||
wal:
|
||||
path: /var/tempo/wal
|
||||
local:
|
||||
path: /var/tempo/blocks
|
||||
usage_report:
|
||||
reporting_enabled: false
|
||||
@@ -0,0 +1 @@
|
||||
ALTER TABLE attachment_jobs ADD COLUMN trace_context jsonb NOT NULL DEFAULT '{}';
|
||||
@@ -17,3 +17,7 @@ redis==5.2.1
|
||||
argon2-cffi==23.1.0
|
||||
boto3==1.40.70
|
||||
Pillow==12.0.0
|
||||
opentelemetry-api==1.38.0
|
||||
opentelemetry-sdk==1.38.0
|
||||
opentelemetry-exporter-otlp-proto-http==1.38.0
|
||||
psutil==7.1.2
|
||||
|
||||
@@ -9,5 +9,6 @@ if __name__ == '__main__':
|
||||
host=os.getenv('APP_HOST', '0.0.0.0'),
|
||||
port=int(os.getenv('APP_PORT', '8080')),
|
||||
workers=1,
|
||||
access_log=False, # Structured logs use route templates, never query strings.
|
||||
timeout_graceful_shutdown=5,
|
||||
)
|
||||
|
||||
+2
-1
@@ -10,6 +10,7 @@ from .admission import reserve_capacity
|
||||
from .models import Attachment, AttachmentJob, ChatMember
|
||||
from .object_storage import BUCKET, PREFIX, s3_client
|
||||
from .store import ensure_uuid, serialize, store
|
||||
from .telemetry import carrier
|
||||
|
||||
MAX_BYTES = min(int(os.getenv('MAX_UPLOAD_BYTES', '10485760')), 10485760)
|
||||
MAX_PIXELS = min(int(os.getenv('MAX_IMAGE_PIXELS', '40000000')), 40000000)
|
||||
@@ -71,7 +72,7 @@ def save_attachment(source, chat_id, actor_id, filename, content_type, size, dig
|
||||
size_bytes=size, sha256=digest, staging_key=staging_key)
|
||||
session.add(attachment)
|
||||
session.flush()
|
||||
session.add(AttachmentJob(attachment_id=attachment_id))
|
||||
session.add(AttachmentJob(attachment_id=attachment_id, trace_context=carrier()))
|
||||
result = metadata(attachment)
|
||||
# An uncertain COMMIT is not followed by DELETE: cleanup checks the DB later.
|
||||
return result
|
||||
|
||||
+57
-2
@@ -1,8 +1,12 @@
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy import create_engine, func, insert, select
|
||||
from sqlalchemy import create_engine, event, func, insert, select
|
||||
from sqlalchemy.engine import URL, make_url
|
||||
from sqlalchemy.pool import QueuePool
|
||||
from threading import Lock
|
||||
|
||||
from . import telemetry as otel
|
||||
|
||||
from .models import SchemaMigration
|
||||
|
||||
@@ -23,12 +27,63 @@ def database_url():
|
||||
)
|
||||
|
||||
|
||||
class ObservedPool(QueuePool):
|
||||
"""Count threads acquiring a connection, including pool wait and reconnect time."""
|
||||
def __init__(self, *args, **kwargs):
|
||||
super().__init__(*args, **kwargs)
|
||||
self.waiting = 0
|
||||
self.wait_lock = Lock()
|
||||
|
||||
def connect(self):
|
||||
with self.wait_lock:
|
||||
self.waiting += 1
|
||||
try:
|
||||
with otel.dependency_span('db.acquire', **{'db.system': 'postgresql'}):
|
||||
return super().connect()
|
||||
finally:
|
||||
with self.wait_lock:
|
||||
self.waiting -= 1
|
||||
|
||||
|
||||
def create_db_engine():
|
||||
return create_engine(
|
||||
engine = create_engine(
|
||||
database_url(), pool_size=10, max_overflow=0, pool_timeout=2, pool_pre_ping=True,
|
||||
poolclass=ObservedPool, hide_parameters=True,
|
||||
connect_args={'connect_timeout': 2,
|
||||
'options': '-c statement_timeout=3000 -c lock_timeout=2000'},
|
||||
)
|
||||
# Only operation names; SQL text, bound values and connection URLs are never exported.
|
||||
@event.listens_for(engine, 'before_cursor_execute')
|
||||
def before(conn, cursor, statement, parameters, context, executemany):
|
||||
if not otel.trace.get_current_span().get_span_context().is_valid:
|
||||
return
|
||||
operation = statement.lstrip().split(None, 1)[0].upper() if statement.strip() else 'OTHER'
|
||||
if operation not in {'SELECT', 'INSERT', 'UPDATE', 'DELETE'}:
|
||||
operation = 'OTHER'
|
||||
context.otel_span = otel.tracer.start_span('db.' + operation,
|
||||
kind=otel.SpanKind.CLIENT, attributes={'db.system': 'postgresql', 'db.operation.name': operation})
|
||||
|
||||
@event.listens_for(engine, 'after_cursor_execute')
|
||||
def after(conn, cursor, statement, parameters, context, executemany):
|
||||
current = getattr(context, 'otel_span', None)
|
||||
if current:
|
||||
current.end()
|
||||
|
||||
@event.listens_for(engine, 'handle_error')
|
||||
def error(context):
|
||||
current = getattr(context.execution_context, 'otel_span', None)
|
||||
if current:
|
||||
current.set_status(otel.Status(otel.StatusCode.ERROR))
|
||||
current.set_attribute('error.type', type(context.original_exception).__name__)
|
||||
current.end()
|
||||
otel.event('db_error', severity=otel.logging.ERROR, error_type=type(context.original_exception).__name__)
|
||||
|
||||
if os.getenv('OTEL_CHILD_PROCESS') != 'true':
|
||||
otel.meter.create_observable_gauge('db_pool_used', callbacks=[
|
||||
lambda _: [otel.metrics.Observation(engine.pool.checkedout())]])
|
||||
otel.meter.create_observable_gauge('db_pool_waiting', callbacks=[
|
||||
lambda _: [otel.metrics.Observation(engine.pool.waiting)]])
|
||||
return engine
|
||||
|
||||
|
||||
def migrate(engine):
|
||||
|
||||
@@ -12,6 +12,7 @@ from .database import create_db_engine
|
||||
from .jobs import now
|
||||
from .models import Attachment, AttachmentJob
|
||||
from .object_storage import BUCKET, s3_client
|
||||
from . import telemetry as otel
|
||||
|
||||
|
||||
def fit_size(size, bound):
|
||||
@@ -98,9 +99,13 @@ def main():
|
||||
import resource
|
||||
memory = int(os.getenv('JOB_MEMORY_MB', '768')) * 1024 * 1024
|
||||
resource.setrlimit(resource.RLIMIT_AS, (memory, memory))
|
||||
otel.setup(child=True)
|
||||
engine = create_db_engine()
|
||||
try:
|
||||
process(sessionmaker(engine, expire_on_commit=False), sys.argv[1], sys.argv[2])
|
||||
with otel.span('image.process', context=otel.propagator.extract({
|
||||
'traceparent': os.getenv('TRACEPARENT', ''), 'tracestate': os.getenv('TRACESTATE', '')})):
|
||||
process(sessionmaker(engine, expire_on_commit=False), sys.argv[1], sys.argv[2])
|
||||
otel.event('image_completed')
|
||||
except (UnidentifiedImageError, Image.DecompressionBombError, ValueError, SyntaxError):
|
||||
return 20
|
||||
except MemoryError:
|
||||
@@ -109,6 +114,7 @@ def main():
|
||||
return 22 # No credentials or decoder/database internals in logs.
|
||||
finally:
|
||||
engine.dispose()
|
||||
otel.shutdown()
|
||||
return 0
|
||||
|
||||
|
||||
|
||||
+23
-4
@@ -10,13 +10,14 @@ from uuid import UUID, uuid4
|
||||
|
||||
from redis import Redis
|
||||
from redis.exceptions import ResponseError
|
||||
from sqlalchemy import and_, or_, select
|
||||
from sqlalchemy import and_, or_, select, func
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from .database import create_db_engine
|
||||
from .admission import reserve_capacity
|
||||
from .models import Attachment, AttachmentJob
|
||||
from .object_storage import BUCKET, s3_client
|
||||
from . import telemetry as otel
|
||||
|
||||
STREAM = 'attachment-jobs'
|
||||
GROUP = 'image-workers'
|
||||
@@ -62,7 +63,8 @@ def fail(session, job, attachment, code, permanent=False):
|
||||
attachment.error_code = None
|
||||
delay = min(60, 2 ** min(job.failures, 6)) * random.uniform(0.5, 1.5)
|
||||
job.available_at = now() + timedelta(seconds=delay)
|
||||
logging.info('job=%s attempt=%s result=%s code=%s', job.id, job.attempt, attachment.status, code)
|
||||
otel.event('job_retry' if attachment.status == 'queued' else 'job_failed',
|
||||
attempt=job.attempt, outcome=attachment.status, error_code=code)
|
||||
|
||||
|
||||
def dispatch_once(sessions, client):
|
||||
@@ -80,10 +82,24 @@ def dispatch_once(sessions, client):
|
||||
fail(session, job, attachment, 'lease_expired')
|
||||
continue
|
||||
if job.published_at is None:
|
||||
client.xadd(STREAM, {'job': json.dumps(envelope(job))})
|
||||
with otel.span('attachment-jobs publish', kind=otel.SpanKind.PRODUCER,
|
||||
context=otel.propagator.extract(job.trace_context), attributes={
|
||||
'messaging.system': 'redis', 'messaging.destination.name': STREAM,
|
||||
'messaging.operation.type': 'publish', 'job.attempt': job.attempt + 1}):
|
||||
client.xadd(STREAM, {'job': json.dumps({**envelope(job), **otel.carrier()})})
|
||||
otel.event('job_published', attempt=job.attempt + 1)
|
||||
job.published_at = now()
|
||||
|
||||
|
||||
def sample_queue(sessions):
|
||||
# Database is the durable backlog: includes outbox rows not yet in Redis and retries.
|
||||
with sessions() as session:
|
||||
depth, oldest = session.execute(select(func.count(), func.min(Attachment.created_at))
|
||||
.select_from(Attachment).where(Attachment.status == 'queued')).one()
|
||||
otel.queue_state.update(depth=depth, age=max(0, (now() - oldest).total_seconds()) if oldest else 0,
|
||||
updated=time.time())
|
||||
|
||||
|
||||
def claim(sessions, message):
|
||||
if (message.get('schema_version') != 1 or message.get('type') != 'attachment.process.v1'
|
||||
or message.get('pipeline_version') != 1):
|
||||
@@ -148,6 +164,7 @@ def main():
|
||||
retry.add_argument('--reason', required=True)
|
||||
args = parser.parse_args()
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
otel.setup()
|
||||
engine = create_db_engine()
|
||||
sessions = sessionmaker(engine, expire_on_commit=False)
|
||||
try:
|
||||
@@ -169,11 +186,13 @@ def main():
|
||||
try:
|
||||
ensure_group(client)
|
||||
dispatch_once(sessions, client)
|
||||
sample_queue(sessions)
|
||||
except Exception as exc:
|
||||
logging.warning('Dispatch deferred: %s', type(exc).__name__)
|
||||
otel.event('dispatch_deferred', error_type=type(exc).__name__)
|
||||
time.sleep(1)
|
||||
finally:
|
||||
engine.dispose()
|
||||
otel.shutdown()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
||||
@@ -18,10 +18,13 @@ from .routers.users import router as users_router
|
||||
from .routers.chats import router as chats_router
|
||||
from .routers.auth import router as auth_router
|
||||
from .routers.attachments import router as attachments_router
|
||||
from . import telemetry
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app):
|
||||
telemetry.setup()
|
||||
telemetry.seed_http(app.routes)
|
||||
engine = create_db_engine()
|
||||
try:
|
||||
store.sessions = sessionmaker(engine, expire_on_commit=False)
|
||||
@@ -29,6 +32,7 @@ async def lifespan(app):
|
||||
finally:
|
||||
engine.dispose()
|
||||
sessions.client.close()
|
||||
telemetry.shutdown()
|
||||
|
||||
|
||||
app = FastAPI(title="MTUSI Messenger", lifespan=lifespan)
|
||||
@@ -46,3 +50,6 @@ app.include_router(users_router)
|
||||
app.include_router(chats_router)
|
||||
app.include_router(auth_router)
|
||||
app.include_router(attachments_router)
|
||||
|
||||
# Outside FastAPI's error middleware: includes generated 500 responses and full streams.
|
||||
app = telemetry.HttpTelemetry(app)
|
||||
|
||||
@@ -97,6 +97,7 @@ class AttachmentJob(Base):
|
||||
lease_token: Mapped[str | None] = mapped_column(Uuid(as_uuid=False))
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.clock_timestamp())
|
||||
replay_audit: Mapped[list] = mapped_column(JSONB, server_default='[]')
|
||||
trace_context: Mapped[dict] = mapped_column(JSONB, server_default='{}')
|
||||
|
||||
|
||||
class IdempotencyRecord(Base):
|
||||
|
||||
+11
-1
@@ -3,6 +3,7 @@ from functools import lru_cache
|
||||
|
||||
import boto3
|
||||
from botocore.config import Config
|
||||
from .telemetry import dependency_span
|
||||
|
||||
BUCKET = os.getenv('S3_BUCKET', 'messenger-attachments')
|
||||
PREFIX = 'originals/'
|
||||
@@ -10,7 +11,7 @@ PREFIX = 'originals/'
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def s3_client():
|
||||
return boto3.client(
|
||||
client = boto3.client(
|
||||
's3', endpoint_url=os.getenv('S3_ENDPOINT', 'http://s3:9000'),
|
||||
region_name=os.getenv('S3_REGION', 'us-east-1'),
|
||||
aws_access_key_id=os.environ['S3_ACCESS_KEY_ID'],
|
||||
@@ -21,3 +22,12 @@ def s3_client():
|
||||
request_checksum_calculation='when_required',
|
||||
response_checksum_validation='when_required'),
|
||||
)
|
||||
# Instrument SDK operations without object keys, signed URLs, headers or bytes.
|
||||
original = client._make_api_call
|
||||
|
||||
def call(operation, parameters):
|
||||
with dependency_span('s3.' + operation, **{'rpc.system': 'aws-api', 'rpc.service': 'S3',
|
||||
'rpc.method': operation}):
|
||||
return original(operation, parameters)
|
||||
client._make_api_call = call
|
||||
return client
|
||||
|
||||
@@ -0,0 +1,206 @@
|
||||
"""Small, explicit OTel instrumentation. Only allowlisted technical fields leave the app."""
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import socket
|
||||
import time
|
||||
from contextlib import contextmanager
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
import psutil
|
||||
from opentelemetry import metrics, trace
|
||||
from opentelemetry.trace import SpanKind, Status, StatusCode
|
||||
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
|
||||
from opentelemetry.sdk.resources import Resource
|
||||
from opentelemetry.sdk.trace import TracerProvider
|
||||
from opentelemetry.sdk.trace.export import BatchSpanProcessor
|
||||
from opentelemetry.sdk.metrics import MeterProvider, Histogram
|
||||
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
|
||||
from opentelemetry.sdk.metrics.view import View, ExplicitBucketHistogramAggregation
|
||||
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
|
||||
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
|
||||
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
|
||||
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
|
||||
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
|
||||
|
||||
tracer = trace.get_tracer('messenger', '1.6.0')
|
||||
meter = metrics.get_meter('messenger', '1.6.0')
|
||||
propagator = TraceContextTextMapPropagator() # No baggage: it may contain user data.
|
||||
logger = logging.getLogger('messenger.events')
|
||||
logger.propagate = False
|
||||
logger.setLevel(logging.INFO)
|
||||
providers = []
|
||||
LOG_FIELDS = {'method', 'route', 'status_class', 'status', 'duration_seconds', 'request_id',
|
||||
'operation', 'error_type', 'attempt', 'outcome', 'error_code', 'job_type'}
|
||||
http_requests = meter.create_counter('http_requests')
|
||||
http_duration = meter.create_histogram('http_duration', unit='s')
|
||||
jobs_processed = meter.create_counter('jobs_processed')
|
||||
job_duration = meter.create_histogram('job_duration', unit='s')
|
||||
time_to_ready = meter.create_histogram('attachment_time_to_ready', unit='s')
|
||||
worker_busy = meter.create_up_down_counter('worker_busy')
|
||||
queue_state = {'depth': 0, 'age': 0, 'updated': 0}
|
||||
|
||||
|
||||
class JsonFormatter(logging.Formatter):
|
||||
def format(self, record):
|
||||
context = trace.get_current_span().get_span_context()
|
||||
return json.dumps({
|
||||
'timestamp': datetime.fromtimestamp(record.created, timezone.utc).isoformat(),
|
||||
'severity': record.levelname, 'service.name': os.getenv('OTEL_SERVICE_NAME', 'messenger-api'),
|
||||
'event': record.getMessage(),
|
||||
'trace_id': format(context.trace_id, '032x') if context.is_valid else '',
|
||||
'span_id': format(context.span_id, '016x') if context.is_valid else '',
|
||||
**{key: record.__dict__[key] for key in LOG_FIELDS if key in record.__dict__},
|
||||
})
|
||||
|
||||
|
||||
def event(name, *, severity=logging.INFO, **fields):
|
||||
# Call sites pass fixed event names, safe enums/numbers and technical correlation IDs.
|
||||
logger.log(severity, name, extra={'event': name, **{k: v for k, v in fields.items() if k in LOG_FIELDS}})
|
||||
|
||||
|
||||
def setup(*, child=False):
|
||||
if providers or os.getenv('OTEL_SDK_DISABLED', 'false').lower() == 'true':
|
||||
return
|
||||
resource = Resource({
|
||||
'service.name': os.getenv('OTEL_SERVICE_NAME', 'messenger-api'),
|
||||
'service.version': '1.6.0', 'service.instance.id': socket.gethostname(),
|
||||
'deployment.environment.name': 'lab',
|
||||
})
|
||||
endpoint = os.getenv('OTEL_EXPORTER_OTLP_ENDPOINT', 'http://otel-collector:4318').rstrip('/')
|
||||
traces = TracerProvider(resource=resource)
|
||||
traces.add_span_processor(BatchSpanProcessor(
|
||||
OTLPSpanExporter(endpoint=endpoint + '/v1/traces', timeout=1),
|
||||
max_queue_size=512, max_export_batch_size=64, schedule_delay_millis=1000))
|
||||
trace.set_tracer_provider(traces)
|
||||
logs = LoggerProvider(resource=resource)
|
||||
logs.add_log_record_processor(BatchLogRecordProcessor(
|
||||
OTLPLogExporter(endpoint=endpoint + '/v1/logs', timeout=1),
|
||||
max_queue_size=512, max_export_batch_size=64, schedule_delay_millis=1000))
|
||||
console = logging.StreamHandler()
|
||||
console.setFormatter(JsonFormatter())
|
||||
logger.addHandler(console)
|
||||
logger.addHandler(LoggingHandler(logger_provider=logs))
|
||||
providers.extend([traces, logs])
|
||||
if child:
|
||||
return # Short-lived image processes emit spans/logs; parent owns worker metrics.
|
||||
reader = PeriodicExportingMetricReader(
|
||||
OTLPMetricExporter(endpoint=endpoint + '/v1/metrics', timeout=1),
|
||||
export_interval_millis=5000, export_timeout_millis=2000)
|
||||
buckets = [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2, 3, 5, 10, 30, 60, 120, 300]
|
||||
metric_provider = MeterProvider(resource=resource, metric_readers=[reader], views=[
|
||||
View(instrument_type=Histogram,
|
||||
aggregation=ExplicitBucketHistogramAggregation(buckets))])
|
||||
metrics.set_meter_provider(metric_provider)
|
||||
providers.append(metric_provider)
|
||||
process = psutil.Process()
|
||||
meter.create_observable_gauge('process_memory', unit='By', callbacks=[
|
||||
lambda _: [metrics.Observation(process.memory_info().rss)]])
|
||||
meter.create_observable_counter('process_cpu', unit='s', callbacks=[
|
||||
lambda _: [metrics.Observation(sum(process.cpu_times()[:2]))]])
|
||||
cgroup = Path('/sys/fs/cgroup')
|
||||
if (cgroup / 'memory.current').exists():
|
||||
meter.create_observable_gauge('container_memory', unit='By', callbacks=[
|
||||
lambda _: [metrics.Observation(int((cgroup / 'memory.current').read_text()))]])
|
||||
def cpu_usage(_):
|
||||
values = dict(line.split() for line in (cgroup / 'cpu.stat').read_text().splitlines())
|
||||
return [metrics.Observation(int(values['usage_usec']) / 1_000_000)]
|
||||
meter.create_observable_counter('container_cpu', unit='s', callbacks=[cpu_usage])
|
||||
if os.getenv('OTEL_SERVICE_NAME') == 'messenger-worker':
|
||||
capacity = min(4, max(1, int(os.getenv('WORKER_CONCURRENCY', '2'))))
|
||||
meter.create_observable_gauge('worker_capacity', callbacks=[
|
||||
lambda _: [metrics.Observation(capacity)]])
|
||||
worker_busy.add(0)
|
||||
if os.getenv('OTEL_SERVICE_NAME') == 'messenger-dispatcher':
|
||||
for name, key, unit in [('queue_depth', 'depth', ''), ('queue_oldest_age', 'age', 's'),
|
||||
('queue_sample_timestamp', 'updated', 's')]:
|
||||
meter.create_observable_gauge(name, unit=unit, callbacks=[
|
||||
lambda _, key=key: [metrics.Observation(queue_state[key], {'queue': 'attachment-jobs'})]])
|
||||
|
||||
|
||||
def shutdown():
|
||||
for provider in reversed(providers):
|
||||
provider.shutdown()
|
||||
providers.clear()
|
||||
|
||||
|
||||
def seed_http(routes):
|
||||
# Export zero before the first failure: Prometheus rate needs a prior sample.
|
||||
for route in routes:
|
||||
if route.path.startswith('/health/'):
|
||||
continue
|
||||
for method in route.methods or ():
|
||||
for status in ('2xx', '3xx', '4xx', '5xx'):
|
||||
http_requests.add(0, {'method': method, 'route': route.path, 'status_class': status})
|
||||
|
||||
|
||||
def carrier():
|
||||
result = {}
|
||||
propagator.inject(result)
|
||||
return result
|
||||
|
||||
|
||||
@contextmanager
|
||||
def span(name, *, kind=SpanKind.INTERNAL, context=None, attributes=None):
|
||||
# Never export exception messages/stack traces (SQL errors can embed parameters).
|
||||
with tracer.start_as_current_span(name, kind=kind, context=context, attributes=attributes,
|
||||
record_exception=False, set_status_on_exception=False) as current:
|
||||
try:
|
||||
yield current
|
||||
except Exception as exc:
|
||||
current.set_status(Status(StatusCode.ERROR))
|
||||
current.set_attribute('error.type', type(exc).__name__)
|
||||
event('operation_failed', severity=logging.ERROR, operation=name, error_type=type(exc).__name__)
|
||||
raise
|
||||
|
||||
|
||||
@contextmanager
|
||||
def dependency_span(name, **attributes):
|
||||
# Background health probes and queue polling must not create disconnected trace noise.
|
||||
if not trace.get_current_span().get_span_context().is_valid:
|
||||
yield
|
||||
else:
|
||||
with span(name, kind=SpanKind.CLIENT, attributes=attributes):
|
||||
yield
|
||||
|
||||
|
||||
class HttpTelemetry:
|
||||
def __init__(self, app):
|
||||
self.app = app
|
||||
|
||||
async def __call__(self, scope, receive, send):
|
||||
if scope['type'] != 'http' or scope['path'].startswith('/health/'):
|
||||
return await self.app(scope, receive, send)
|
||||
headers = {k.decode('latin1'): v.decode('latin1') for k, v in scope['headers']
|
||||
if k in (b'traceparent', b'tracestate')}
|
||||
method = scope['method'] if scope['method'] in {'GET', 'POST', 'PUT', 'PATCH', 'DELETE', 'HEAD', 'OPTIONS'} else 'OTHER'
|
||||
status = 500
|
||||
started = time.monotonic()
|
||||
with span('HTTP ' + method, kind=SpanKind.SERVER, context=propagator.extract(headers)) as current:
|
||||
async def response(message):
|
||||
nonlocal status
|
||||
if message['type'] == 'http.response.start':
|
||||
status = message['status']
|
||||
await send(message)
|
||||
try:
|
||||
await self.app(scope, receive, response)
|
||||
finally:
|
||||
route = getattr(scope.get('route'), 'path', 'unmatched')
|
||||
labels = {'method': method, 'route': route, 'status_class': f'{status // 100}xx'}
|
||||
elapsed = time.monotonic() - started
|
||||
http_requests.add(1, labels)
|
||||
http_duration.record(elapsed, labels)
|
||||
current.update_name(method + ' ' + route)
|
||||
current.set_attributes({'http.request.method': method, 'http.route': route,
|
||||
'http.response.status_code': status})
|
||||
if status >= 500:
|
||||
current.set_status(Status(StatusCode.ERROR))
|
||||
request_id = scope.get('state', {}).get('request_id', '')
|
||||
if not re.fullmatch(r'[A-Za-z0-9_-]{1,64}', request_id):
|
||||
request_id = ''
|
||||
current.set_attribute('request_id', request_id)
|
||||
event('http_request', severity=logging.ERROR if status >= 500 else logging.INFO,
|
||||
**labels, status=status, duration_seconds=elapsed,
|
||||
request_id=request_id)
|
||||
+61
-15
@@ -12,30 +12,75 @@ from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from .database import create_db_engine
|
||||
from .jobs import GROUP, STREAM, LEASE_SECONDS, TIMEOUT, broker, claim, ensure_group, record_failure
|
||||
from .models import Attachment, AttachmentJob
|
||||
from . import telemetry as otel
|
||||
|
||||
|
||||
def handle(sessions, client, message_id, fields):
|
||||
try:
|
||||
claimed = claim(sessions, json.loads(fields['job']))
|
||||
message = json.loads(fields['job'])
|
||||
if not isinstance(message, dict):
|
||||
raise ValueError('invalid envelope')
|
||||
except (ValueError, KeyError, TypeError):
|
||||
logging.error('Invalid broker envelope: id=%s', message_id)
|
||||
otel.event('invalid_job')
|
||||
acknowledge(client, message_id)
|
||||
return
|
||||
with otel.span('attachment-jobs process', kind=otel.SpanKind.CONSUMER,
|
||||
context=otel.propagator.extract(message), attributes={
|
||||
'messaging.system': 'redis', 'messaging.destination.name': STREAM,
|
||||
'messaging.operation.type': 'process', 'job.attempt': message.get('attempt', 0)}):
|
||||
process_message(sessions, client, message_id, message)
|
||||
|
||||
|
||||
def process_message(sessions, client, message_id, message):
|
||||
try:
|
||||
claimed = claim(sessions, message)
|
||||
except (ValueError, KeyError, TypeError):
|
||||
otel.event('invalid_job')
|
||||
claimed = None
|
||||
if claimed:
|
||||
job_id, token = claimed
|
||||
started = time.monotonic()
|
||||
outcome = 'failed'
|
||||
otel.worker_busy.add(1)
|
||||
try:
|
||||
result = subprocess.run([sys.executable, '-m', 'src.image_processor', job_id, token],
|
||||
timeout=TIMEOUT, check=False)
|
||||
code = result.returncode
|
||||
except subprocess.TimeoutExpired:
|
||||
code = 23
|
||||
if code in (91, 92):
|
||||
# Simulate a lost worker before ack: leave the Redis pending entry and lease.
|
||||
return
|
||||
if code:
|
||||
record_failure(sessions, job_id, token,
|
||||
{20: 'invalid_image', 21: 'resource_limit', 23: 'processing_timeout'}
|
||||
.get(code, 'processing_failed'), permanent=code in (20, 21))
|
||||
environment = {**os.environ, 'OTEL_CHILD_PROCESS': 'true'}
|
||||
environment.update({key.upper(): value for key, value in otel.carrier().items()})
|
||||
try:
|
||||
result = subprocess.run([sys.executable, '-m', 'src.image_processor', job_id, token],
|
||||
timeout=TIMEOUT, check=False, env=environment)
|
||||
code = result.returncode
|
||||
except subprocess.TimeoutExpired:
|
||||
code = 23
|
||||
if code in (91, 92):
|
||||
# Simulate a lost worker before ack: leave the pending entry and lease.
|
||||
return
|
||||
if code:
|
||||
record_failure(sessions, job_id, token,
|
||||
{20: 'invalid_image', 21: 'resource_limit', 23: 'processing_timeout'}
|
||||
.get(code, 'processing_failed'), permanent=code in (20, 21))
|
||||
otel.trace.get_current_span().set_status(otel.Status(otel.StatusCode.ERROR))
|
||||
else:
|
||||
with sessions() as session:
|
||||
job = session.get(AttachmentJob, job_id)
|
||||
attachment = session.get(Attachment, job.attachment_id)
|
||||
outcome = 'ready' if attachment.status == 'ready' else 'skipped'
|
||||
if outcome == 'ready':
|
||||
otel.time_to_ready.record((attachment.updated_at - attachment.created_at).total_seconds(),
|
||||
{'job_type': 'attachment.process.v1'})
|
||||
finally:
|
||||
otel.worker_busy.add(-1)
|
||||
labels = {'job_type': 'attachment.process.v1', 'outcome': outcome}
|
||||
elapsed = time.monotonic() - started
|
||||
otel.jobs_processed.add(1, labels)
|
||||
otel.job_duration.record(elapsed, labels)
|
||||
otel.event('job_processed', severity=logging.ERROR if outcome == 'failed' else logging.INFO,
|
||||
**labels, duration_seconds=elapsed)
|
||||
# Only after a durable ready/failed/retry (or an existing durable lease).
|
||||
acknowledge(client, message_id)
|
||||
|
||||
|
||||
def acknowledge(client, message_id):
|
||||
with client.pipeline(transaction=True) as pipe:
|
||||
pipe.xack(STREAM, GROUP, message_id)
|
||||
pipe.xdel(STREAM, message_id)
|
||||
@@ -58,12 +103,13 @@ def consume(sessions):
|
||||
for message_id, fields in messages:
|
||||
handle(sessions, client, message_id, fields)
|
||||
except Exception as exc:
|
||||
logging.warning('Worker deferred: %s', type(exc).__name__)
|
||||
otel.event('worker_deferred', error_type=type(exc).__name__)
|
||||
time.sleep(1)
|
||||
|
||||
|
||||
def main():
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
otel.setup()
|
||||
engine = create_db_engine()
|
||||
sessions = sessionmaker(engine, expire_on_commit=False)
|
||||
concurrency = min(4, max(1, int(os.getenv('WORKER_CONCURRENCY', '2'))))
|
||||
|
||||
@@ -0,0 +1,290 @@
|
||||
"""Lab 6 acceptance: real signals, two incidents, retry propagation and bounded outage.
|
||||
|
||||
Uses only the standard library. Stops services only in this Compose project and
|
||||
always starts them again; never deletes volumes. Run separately from other tests.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / 'tests'))
|
||||
from client import Client, check
|
||||
|
||||
PROM = 'http://127.0.0.1:9090'
|
||||
LOKI = 'http://127.0.0.1:3100'
|
||||
TEMPO = 'http://127.0.0.1:3200'
|
||||
|
||||
|
||||
def stamp():
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def get(url, **params):
|
||||
if params:
|
||||
url += '?' + urllib.parse.urlencode(params)
|
||||
with urllib.request.urlopen(urllib.request.Request(url, headers={'Accept': 'application/json'}), timeout=10) as r:
|
||||
return json.load(r)
|
||||
|
||||
|
||||
def query(expr):
|
||||
data = get(PROM + '/api/v1/query', query=expr)
|
||||
check(data['status'] == 'success', 'PromQL failed')
|
||||
return data['data']['result']
|
||||
|
||||
|
||||
def number(expr):
|
||||
values = query(expr)
|
||||
return float(values[0]['value'][1]) if values else 0.0
|
||||
|
||||
|
||||
def wait(label, fn, timeout=90):
|
||||
end = time.monotonic() + timeout
|
||||
last = None
|
||||
while time.monotonic() < end:
|
||||
try:
|
||||
value = fn()
|
||||
if value:
|
||||
return value
|
||||
except (urllib.error.URLError, TimeoutError, AssertionError) as exc:
|
||||
last = type(exc).__name__
|
||||
time.sleep(2)
|
||||
raise AssertionError(f'Timeout: {label} ({last})')
|
||||
|
||||
|
||||
def compose(*args):
|
||||
subprocess.run(['docker', 'compose', *args], cwd=ROOT, check=True,
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=90)
|
||||
|
||||
|
||||
def trace_data(trace_id):
|
||||
data = get(TEMPO + '/api/traces/' + trace_id)
|
||||
spans = []
|
||||
for batch in data.get('batches', data.get('resourceSpans', [])):
|
||||
resource = {a['key']: a['value'].get('stringValue') for a in batch.get('resource', {}).get('attributes', [])}
|
||||
for scope in batch.get('scopeSpans', batch.get('instrumentationLibrarySpans', [])):
|
||||
for span in scope.get('spans', []):
|
||||
spans.append({**span, 'service': resource.get('service.name')})
|
||||
return data, spans
|
||||
|
||||
|
||||
def logs(trace_id):
|
||||
return get(LOKI + '/loki/api/v1/query_range',
|
||||
query='{service_name=~"messenger-.*"} | trace_id="' + trace_id + '"',
|
||||
start=str(time.time_ns() - 3_600_000_000_000), limit=1000)['data']['result']
|
||||
|
||||
|
||||
def attrs(span):
|
||||
return {a['key']: next(iter(a['value'].values())) for a in span.get('attributes', [])}
|
||||
|
||||
|
||||
def pipeline(trace_id, *, retry=False):
|
||||
data, spans = trace_data(trace_id)
|
||||
names = {s['name'] for s in spans}
|
||||
required = {'POST /api/v1/chats/{chat_id}/attachments', 'attachment-jobs publish',
|
||||
'attachment-jobs process', 'image.process', 's3.GetObject', 's3.PutObject', 'db.UPDATE'}
|
||||
if not required <= names:
|
||||
return None
|
||||
by_id = {s['spanId']: s for s in spans}
|
||||
for name, parent_name in [('attachment-jobs publish', 'POST /api/v1/chats/{chat_id}/attachments'),
|
||||
('attachment-jobs process', 'attachment-jobs publish'),
|
||||
('image.process', 'attachment-jobs process')]:
|
||||
for s in (s for s in spans if s['name'] == name):
|
||||
check(by_id.get(s.get('parentSpanId'), {}).get('name') == parent_name,
|
||||
'Broken trace parent for ' + name)
|
||||
check(s.get('traceState') == 'lab=check', 'W3C tracestate lost at ' + name)
|
||||
attempts = sorted({int(attrs(s)['job.attempt']) for s in spans
|
||||
if s['name'] == 'attachment-jobs process'})
|
||||
if retry and len(attempts) < 2:
|
||||
return None
|
||||
records = logs(trace_id)
|
||||
services = {r['stream'].get('service_name') for r in records}
|
||||
if not {'messenger-api', 'messenger-worker', 'messenger-dispatcher'} <= services:
|
||||
return None
|
||||
return {'trace_id': trace_id, 'span_count': len(spans), 'span_names': sorted(names),
|
||||
'log_services': sorted(services), 'attempts': attempts, 'parent_chain_verified': True,
|
||||
'tracestate_preserved': True,
|
||||
'_raw': json.dumps([data, records])}
|
||||
|
||||
|
||||
class Scenario:
|
||||
def __init__(self, base_url):
|
||||
self.client = Client(base_url)
|
||||
self.alice, _, _, self.chat = self.client.bootstrap()
|
||||
self.private_markers = [self.alice['token'], self.alice['registration']['password']]
|
||||
|
||||
def upload(self):
|
||||
trace_id, request_id = uuid.uuid4().hex, str(uuid.uuid4())
|
||||
filename = 'private_filename_' + uuid.uuid4().hex + '.png'
|
||||
self.private_markers.append(filename)
|
||||
result = self.client.call('POST', '/api/v1/chats/' + self.chat['id'] + '/attachments?filename=' + filename,
|
||||
202, actor=self.alice, raw=(ROOT / 'tests/fixtures/sample.png').read_bytes(),
|
||||
headers={'Content-Type': 'image/png', 'traceparent': f'00-{trace_id}-{uuid.uuid4().hex[:16]}-01',
|
||||
'tracestate': 'lab=check', 'X-Request-Id': request_id}, schema='Attachment')
|
||||
check(result['headers']['X-Request-Id'] == request_id, 'Request ID changed')
|
||||
return result['json'], trace_id, request_id
|
||||
|
||||
def check_pipeline(self, trace_id, request_id, retry=False):
|
||||
result = wait('pipeline spans/logs', lambda: pipeline(trace_id, retry=retry))
|
||||
raw = result.pop('_raw')
|
||||
check(all(value not in raw for value in self.private_markers), 'Private input leaked into telemetry')
|
||||
check(request_id in raw, 'Request ID is missing from trace/log correlation')
|
||||
result['request_id'] = request_id
|
||||
result['private_inputs_absent'] = True
|
||||
return result
|
||||
|
||||
def business(self, expected=200, trace_id=None):
|
||||
headers = {'traceparent': f'00-{trace_id}-{uuid.uuid4().hex[:16]}-01'} if trace_id else {}
|
||||
return self.client.call('GET', '/api/v1/chats/' + self.chat['id'], expected,
|
||||
actor=self.alice, headers=headers, schema='Chat' if expected == 200 else None)
|
||||
|
||||
def smoke(self):
|
||||
att, trace_id, request_id = self.upload()
|
||||
self.client.wait_ready(self.alice, att)
|
||||
result = self.check_pipeline(trace_id, request_id)
|
||||
names = ['http_requests_total', 'http_duration_seconds_bucket', 'jobs_processed_total',
|
||||
'job_duration_seconds_bucket', 'attachment_time_to_ready_seconds_bucket',
|
||||
'queue_depth', 'queue_oldest_age_seconds', 'db_pool_used', 'db_pool_waiting',
|
||||
'worker_busy', 'worker_capacity', 'container_cpu_seconds_total', 'container_memory_bytes']
|
||||
for name in names:
|
||||
wait(name, lambda name=name: query(name))
|
||||
series = get(PROM + '/api/v1/series', **{'match[]': '{job="messenger"}', 'start': str(time.time() - 300)})['data']
|
||||
forbidden = {'user_id', 'chat_id', 'message_id', 'attachment_id', 'request_id', 'filename', 'url'}
|
||||
check(all(not forbidden.intersection(row) for row in series), 'High-cardinality metric label')
|
||||
check(all(self.chat['id'] not in json.dumps(row) for row in series), 'Raw route ID in metrics')
|
||||
result['metrics_present'] = names
|
||||
return result
|
||||
|
||||
def queue(self):
|
||||
result = {'started_at': stamp()}
|
||||
compose('stop', 'worker')
|
||||
try:
|
||||
att, trace_id, request_id = self.upload()
|
||||
self.client.message(self.alice, self.chat, 'queued attachment', attachments=[att['id']])
|
||||
wait('QueueLagHigh firing', lambda: number('ALERTS{alertname="QueueLagHigh",alertstate="firing"}') == 1)
|
||||
result.update(firing_at=stamp(), depth=number('max(queue_depth)'),
|
||||
oldest_age_seconds=number('max(queue_oldest_age_seconds)'))
|
||||
finally:
|
||||
compose('start', 'worker')
|
||||
ready = self.client.wait_ready(self.alice, att)
|
||||
wait('queue alert resolved', lambda: not query('ALERTS{alertname="QueueLagHigh"}'))
|
||||
result.update(resolved_at=stamp(), ready=ready['status'], depth_after=number('max(queue_depth)'),
|
||||
pipeline=self.check_pipeline(trace_id, request_id))
|
||||
return result
|
||||
|
||||
def database(self):
|
||||
result = {'started_at': stamp()}
|
||||
trace_id = uuid.uuid4().hex
|
||||
compose('stop', 'postgres')
|
||||
try:
|
||||
errors = 0
|
||||
def load_until_firing():
|
||||
nonlocal errors
|
||||
self.business(expected=503, trace_id=trace_id if errors == 0 else None)
|
||||
errors += 1
|
||||
return number('ALERTS{alertname="ApiHighErrorRate",alertstate="firing"}') == 1
|
||||
wait('ApiHighErrorRate firing under sustained traffic', load_until_firing, timeout=120)
|
||||
result['failed_requests'] = errors
|
||||
result.update(firing_at=stamp(), error_ratio=number('messenger:http_error_ratio:2m'))
|
||||
def failed_trace():
|
||||
_, spans = trace_data(trace_id)
|
||||
failed = [s for s in spans if s['name'].startswith('db.')
|
||||
and s.get('status', {}).get('code') in (2, 'STATUS_CODE_ERROR')]
|
||||
return [{'name': s['name'], 'error_type': attrs(s).get('error.type'),
|
||||
'duration_seconds': (int(s['endTimeUnixNano']) - int(s['startTimeUnixNano'])) / 1e9}
|
||||
for s in failed]
|
||||
result['db_error_spans'] = wait('failed DB span', failed_trace)
|
||||
result['trace_id'] = trace_id
|
||||
result['correlated_log_streams'] = len(wait('DB error logs', lambda: logs(trace_id)))
|
||||
finally:
|
||||
compose('start', 'postgres')
|
||||
wait('API recovered', self.business)
|
||||
for _ in range(10):
|
||||
self.business()
|
||||
wait('API alert resolved', lambda: not query('ALERTS{alertname="ApiHighErrorRate"}'), timeout=160)
|
||||
result['resolved_at'] = stamp()
|
||||
return result
|
||||
|
||||
def retry(self):
|
||||
failures = number('sum(jobs_processed_total{outcome="failed"})')
|
||||
compose('stop', 'worker')
|
||||
try:
|
||||
att, trace_id, request_id = self.upload()
|
||||
compose('stop', 's3')
|
||||
# `compose start worker` can also start its dependency graph, including S3.
|
||||
compose('up', '-d', '--no-deps', '--no-recreate', 'worker')
|
||||
wait('failed processing attempt', lambda: number('sum(jobs_processed_total{outcome="failed"})') > failures, timeout=40)
|
||||
finally:
|
||||
compose('start', 's3', 'worker')
|
||||
wait('S3/API ready', lambda: self.client.call('GET', '/health/ready', schema='Health'))
|
||||
self.client.wait_ready(self.alice, att)
|
||||
return self.check_pipeline(trace_id, request_id, retry=True)
|
||||
|
||||
def backend(self):
|
||||
accepted = 'sum(otelcol_receiver_accepted_spans_total)'
|
||||
sent = 'sum(otelcol_exporter_sent_spans_total{exporter="otlphttp/tempo"})'
|
||||
queue = 'sum(otelcol_exporter_queue_size{exporter="otlphttp/tempo"})'
|
||||
# A quiet interval on both ends makes accepted-minus-sent a loss count,
|
||||
# not a count of failed export attempts or currently buffered spans.
|
||||
time.sleep(8)
|
||||
wait('trace queue drained', lambda: number(queue) == 0)
|
||||
before = {'accepted': number(accepted), 'sent': number(sent)}
|
||||
result = {'started_at': stamp(), 'before': before}
|
||||
samples, writes = [], 0
|
||||
compose('stop', 'tempo')
|
||||
try:
|
||||
end = time.monotonic() + 60
|
||||
att, _, _ = self.upload()
|
||||
self.client.wait_ready(self.alice, att)
|
||||
while time.monotonic() < end:
|
||||
body = 'private_message_' + uuid.uuid4().hex
|
||||
self.private_markers.append(body)
|
||||
self.client.message(self.alice, self.chat, body)
|
||||
writes += 1
|
||||
samples.append({'at': stamp(), 'collector_rss_bytes': number('otelcol_process_memory_rss_bytes'),
|
||||
'queue_batches': number(queue)})
|
||||
time.sleep(0.3)
|
||||
check(writes > 10, 'Not enough writes during telemetry outage')
|
||||
check(max(s['queue_batches'] for s in samples) <= 32, 'Collector queue is unbounded')
|
||||
check(0 < max(s['collector_rss_bytes'] for s in samples) < 256 * 1024 * 1024,
|
||||
'Collector exceeded its memory budget')
|
||||
finally:
|
||||
compose('start', 'tempo')
|
||||
wait('trace queue drained', lambda: number(queue) == 0)
|
||||
time.sleep(10) # Include final Prometheus scrape after drain.
|
||||
after = {'accepted': number(accepted), 'sent': number(sent)}
|
||||
lost = (after['accepted'] - before['accepted']) - (after['sent'] - before['sent'])
|
||||
check(lost >= 0, 'Loss window was not quiescent; rerun backend separately')
|
||||
result.update(after=after, lost_accepted_spans=lost, successful_writes=writes,
|
||||
attachment_ready_during_outage=True, samples=samples, recovered_at=stamp())
|
||||
att, trace_id, request_id = self.upload()
|
||||
self.client.wait_ready(self.alice, att)
|
||||
result['recovered_pipeline'] = self.check_pipeline(trace_id, request_id)
|
||||
return result
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--scenario', choices=['smoke', 'queue', 'database', 'retry', 'backend', 'all'], default='smoke')
|
||||
parser.add_argument('--base-url', default='http://127.0.0.1:8080')
|
||||
args = parser.parse_args()
|
||||
scenario = Scenario(args.base_url)
|
||||
path = ROOT / 'evidence/observability.json'
|
||||
evidence = json.loads(path.read_text(encoding='utf-8')) if path.exists() else {}
|
||||
for name in (['smoke', 'queue', 'database', 'retry', 'backend'] if args.scenario == 'all' else [args.scenario]):
|
||||
print('Running ' + name, flush=True)
|
||||
evidence[name] = {'checked_at': stamp(), **getattr(scenario, name)()}
|
||||
path.write_text(json.dumps(evidence, indent=2, ensure_ascii=False) + '\n', encoding='utf-8')
|
||||
print('PASS: ' + name, flush=True)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
Reference in New Issue
Block a user