Реализовать уровень 4: сквозные трассы, SLO, отказы и отчёт
This commit is contained in:
+1
-1
@@ -23,4 +23,4 @@ infra/secrets
|
||||
Защита.md
|
||||
ЗАЩИТА.md
|
||||
|
||||
??????.md
|
||||
защита.md
|
||||
|
||||
+7
-4
@@ -7,7 +7,7 @@ SESSION_TTL_SECONDS=3600
|
||||
# Choose credentials locally. File paths contain no secret values.
|
||||
DATABASE_URL_FILE=/run/secrets/database_url
|
||||
# Optional on 3, required use on 4 in lab2; Redis OR Valkey:
|
||||
SESSION_STORE=postgres
|
||||
SESSION_STORE=valkey
|
||||
CACHE_URL_FILE=/run/secrets/cache_url
|
||||
API_REPLICAS=2
|
||||
IDEMPOTENCY_TTL_SECONDS=86400
|
||||
@@ -19,10 +19,13 @@ S3_SECRET_ACCESS_KEY_FILE=/run/secrets/s3_secret_access_key
|
||||
MAX_UPLOAD_BYTES=10485760
|
||||
MAX_IMAGE_PIXELS=40000000
|
||||
BROKER_URL_FILE=/run/secrets/broker_url
|
||||
WORKER_CONCURRENCY=2
|
||||
# The current consumer is sequential; this documents capacity, not a tuning knob.
|
||||
WORKER_CONCURRENCY=1
|
||||
JOB_MAX_ATTEMPTS=5
|
||||
JOB_TIMEOUT_SECONDS=60
|
||||
STAGING_RETENTION_SECONDS=86400
|
||||
JOB_TIMEOUT_SECONDS=30
|
||||
JOB_TOTAL_TIMEOUT_SECONDS=90
|
||||
JOB_LEASE_SECONDS=120
|
||||
STAGING_RETENTION_SECONDS=604800
|
||||
OTEL_SERVICE_NAME=messenger-api
|
||||
# Worker must set a distinct service.name.
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
name: Application smoke
|
||||
on: [push, pull_request]
|
||||
permissions:
|
||||
contents: read
|
||||
jobs:
|
||||
application:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
- run: sh scripts/init-secrets.sh
|
||||
- run: docker compose up -d --build --wait --wait-timeout 240
|
||||
- run: docker compose --profile tools run --rm -T --no-deps tester make check test
|
||||
- run: docker compose exec -T api python - < student-tests/observability_unit.py
|
||||
- run: python scripts/observe_lab6.py --baseline
|
||||
- if: always()
|
||||
run: docker compose down
|
||||
@@ -16,14 +16,52 @@ thumbnail JPEG 256×256, crop JPEG 128×128 и optimized WebP 1280×1280.
|
||||
Применяются at least once, publisher confirms, ack после commit,
|
||||
lease с fencing token, ограниченные retry с jitter и журнал ошибок.
|
||||
|
||||
## Запуск базового стенда
|
||||
## Запуск из чистого clone
|
||||
|
||||
На Windows: `powershell -ExecutionPolicy Bypass -File scripts/init-secrets.ps1`.
|
||||
На Linux/macOS: `sh scripts/init-secrets.sh`.
|
||||
Затем `docker compose up -d --build --wait --wait-timeout 240`.
|
||||
Smoke: `python -X utf8 tests/smoke.py`; дополнительные проверки:
|
||||
`python -X utf8 scripts/test_lab5.py --quick`.
|
||||
Остановка без потери данных: `docker compose down` (без `-v`).
|
||||
Нужны Docker Engine/Compose v2 (Linux containers), доступ к registry/PyPI,
|
||||
около 8 GiB RAM у Docker и Python 3.9+ для stdlib HTTP-проверок.
|
||||
Образ приложения: Python 3.12.7; PostgreSQL 16.4, Valkey 8.0.1,
|
||||
SeaweedFS 3.97, RabbitMQ 3.13.7, HAProxy 3.0.11. Python-зависимости
|
||||
закреплены в `requirements.txt`. `Dockerfile` общий для API/publisher/worker.
|
||||
|
||||
Windows PowerShell:
|
||||
|
||||
```powershell
|
||||
powershell -ExecutionPolicy Bypass -File scripts/init-secrets.ps1
|
||||
Copy-Item .env.example .env
|
||||
# Если заняты 8080/8404, изменить LB_PORT/LB_STATS_PORT в .env.
|
||||
docker compose config --quiet
|
||||
docker compose up -d --build --wait --wait-timeout 240
|
||||
docker compose --profile tools build tester
|
||||
docker compose --profile tools run --rm -T --no-deps tester make check test
|
||||
$env:BASE_URL = 'http://localhost:8080' # либо выбранный LB_PORT
|
||||
python -X utf8 scripts/observe_lab6.py --baseline
|
||||
python -X utf8 scripts/observe_lab6.py --chain-with-retry
|
||||
python -X utf8 scripts/observe_lab6.py --incidents
|
||||
```
|
||||
|
||||
Linux/macOS: `sh scripts/init-secrets.sh`, `cp .env.example .env`, те же
|
||||
команды Compose; `BASE_URL=http://localhost:8080 python3 scripts/observe_lab6.py ...`.
|
||||
Unit: `docker compose exec -T api python - < student-tests/observability_unit.py`
|
||||
(на PowerShell: `Get-Content -Raw student-tests/observability_unit.py | docker compose exec -T api python -`).
|
||||
Для полной схемы: `python -m venv .venv`, установить `requirements-dev.txt`
|
||||
в venv, затем `python -X utf8 scripts/validate_openapi.py` из этого venv.
|
||||
Собственные проверки изображений/восстановления: `python -X utf8 scripts/test_lab5.py`
|
||||
(без `--quick` выполняются намеренные отказы). Не запускать сценарии отказов параллельно.
|
||||
|
||||
Init-скрипты создают случайные локальные credentials в `infra/secrets/`,
|
||||
существующие сохраняют; файлы исключены из Git и Docker build context.
|
||||
`migrate` до запуска API последовательно применяет миграции 001–006 под
|
||||
блокировкой и журналирует их; повторный запуск пропускает уже применённые.
|
||||
`s3-init` создаёт приватный bucket и проверяет права. Ручной seed не нужен.
|
||||
`--wait` проверяет health основных компонентов; baseline дополнительно
|
||||
ожидает поступление реальных данных в backend.
|
||||
|
||||
Безопасная остановка: `docker compose down`. Данные сохраняются в volumes.
|
||||
**Только для намеренного сброса учебных данных:** `docker compose down -v`.
|
||||
Эта команда удалит и бизнес-данные, и сохранённую телеметрию.
|
||||
В текущем прогоне используются локальные порты API **8086**, LB stats **8406**,
|
||||
Grafana **3000**, Prometheus **9090**, Loki **3100**, Tempo **3200**.
|
||||
|
||||
## Наблюдаемость: уровень 3
|
||||
|
||||
@@ -65,3 +103,201 @@ traces/logs по 512 records, batches по 64, HTTP timeout 2 секунды, п
|
||||
[Loki OTLP](https://grafana.com/docs/loki/latest/send-data/otel/),
|
||||
[Grafana provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/).
|
||||
Реализация и документация подготовлены с помощью Codex.
|
||||
|
||||
## Уровень 4: причинная связь и насыщение
|
||||
|
||||
Миграция 006 добавляет nullable W3C carrier в attachments/outbox без изменения
|
||||
публичных схем. API сохраняет carrier в той же транзакции, что и metadata/outbox;
|
||||
publisher извлекает его, создаёт PRODUCER span, записывает новый carrier в
|
||||
envelope. Worker создаёт CONSUMER span с этим parent, сохраняет consumer
|
||||
context при claim для lease recovery. Следующая попытка получает новый span ID
|
||||
в той же трассе. Operator replay/redeliver сохраняет причинный контекст.
|
||||
Транзакционная граница outbox и семантика at least once остаются прежними.
|
||||
|
||||
Внутренние spans: `db.acquire`, `db.query`, `db.commit`, `db.health`,
|
||||
`s3.PutObject/GetObject/DeleteObject/HeadObject`, `image.generate`.
|
||||
SQL span содержит только тип операции, exception — только имя класса.
|
||||
GetObject измеряет получение headers; чтение streaming body продолжается
|
||||
внутри вызывающей операции, поэтому этот span не является полной download latency.
|
||||
|
||||
| Prometheus metric | Смысл и единица |
|
||||
| --- | --- |
|
||||
| `messenger_queue_depth` | queued + processing в PostgreSQL, jobs; включает outbox и delayed retry |
|
||||
| `messenger_queue_oldest_age_seconds` | возраст старейшей незавершённой job, секунды |
|
||||
| `messenger_queue_sample_age_seconds` | возраст последнего успешного наблюдения, секунды |
|
||||
| `messenger_db_pool_used` / `waiting` / `capacity` | занятые соединения / acquisition requests in progress / лимит соединений |
|
||||
| `messenger_worker_busy` / `capacity` | занятая обработка / доступная ёмкость (один consumer, capacity=1) |
|
||||
| `messenger_process_cpu_seconds_total` | CPU time родительского процесса, секунды |
|
||||
| `messenger_process_memory_bytes` | текущая RSS родительского процесса, байты |
|
||||
| `messenger_attachment_time_to_ready_seconds_*` | histogram от created_at принятого attachment до успешного COMMIT ready |
|
||||
|
||||
Waiting включает создание/проверку соединения, а не только очередь при
|
||||
исчерпанном pool. Метрики CPU/RSS не включают подпроцесс декодера; контейнер
|
||||
дополнительно ограничен Docker. Backlog измеряет publisher независимо от worker,
|
||||
sample age позволяет обнаружить устаревшее наблюдение при отказе БД.
|
||||
`WORKER_CONCURRENCY` из логических рекомендаций skeleton не используется:
|
||||
реальный worker последователен, prefetch=1; scale worker потребует отдельной проверки.
|
||||
Dashboard фильтрует worker resources по heartbeat не старше 15 секунд,
|
||||
затем суммирует busy/capacity. Так старые process resources после restart
|
||||
не завышают ёмкость и не показывают мёртвый busy worker. Collector удаляет
|
||||
не обновляемые серии через 5 минут; heartbeat делает эту задержку явной.
|
||||
|
||||
Пример p95 API в секундах (для p50/p99 меняется только первый аргумент):
|
||||
|
||||
```promql
|
||||
histogram_quantile(0.95,
|
||||
sum by (le) (rate(messenger_http_duration_seconds_bucket{
|
||||
service_name="messenger-api",route!~"/health/.*"}[5m])))
|
||||
```
|
||||
|
||||
Сначала объединяются buckets реплик с одинаковыми границами, потом вычисляется
|
||||
percentile. Buckets SDK: 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5,
|
||||
5, 10, 30, 60, 120, +Inf секунд. Это приближённая оценка, не среднее p95.
|
||||
Все запросы, легенды и единицы находятся в импортируемом dashboard JSON.
|
||||
|
||||
SLI доступности = 1 − (5xx / все бизнес-ответы), health исключены, 4xx не
|
||||
считаются отказом сервера. Учебный SLO: ≥99% за скользящие 5 минут.
|
||||
Дополнительный SLO: ≥95% успешных ready-завершений за ≤60 секунд в окне 5 минут
|
||||
по histogram; он условен по завершениям, pending/failed отдельно контролируются
|
||||
backlog/errors. При отсутствии трафика или завершений выполнение SLO не доказано.
|
||||
Alert high error rate: >5% за 1 минуту непрерывно 10 секунд. Queue lag:
|
||||
age >15 секунд при depth >0 и свежем sample <10 секунд непрерывно 10 секунд.
|
||||
Оценка каждые 5 секунд; короткие пороги выбраны для защиты, а не production SLA.
|
||||
Правила — `infra/prometheus/alerts.yaml`, диагностика —
|
||||
[RUNBOOK](infra/otel/RUNBOOK.md). Доставка уведомлений наружу не используется.
|
||||
|
||||
## Конфигурация и границы устойчивости
|
||||
|
||||
`.env.example` содержит логические рекомендации skeleton. Реально Compose
|
||||
передаёт DATABASE_URL_FILE, CACHE_URL_FILE, BROKER_URL_FILE и S3 *_FILE через
|
||||
secrets; сессии кэшируются в Valkey, источник истины PostgreSQL. DB pool
|
||||
10+5 соединений на процесс, pool/connect timeout 3 секунды, statement timeout
|
||||
5 секунд. Upload: ≤10 MiB и ≤40M pixels; spool RAM 1 MiB на upload,
|
||||
не более 8 concurrent uploads на API. Worker: image timeout 30 секунд,
|
||||
total job timeout 90, lease 120, максимум 5 попыток, exponential backoff+jitter.
|
||||
Thumbnail JPEG quality=85, crop quality=85, optimized WebP quality=80,
|
||||
metadata EXIF удаляется после применения orientation; original сохраняется побайтно.
|
||||
|
||||
Порты, API_REPLICAS, SESSION_TTL_SECONDS, IDEMPOTENCY_TTL_SECONDS,
|
||||
JOB_* и RETRY_BASE_SECONDS можно переопределить через `.env` по `compose.yaml`.
|
||||
Для локальных OTLP-ресурсов выбраны имена messenger-api/worker/publisher;
|
||||
endpoint и resource заданы явно в Compose. `OTEL_SDK_DISABLED=true` отключает SDK.
|
||||
API readiness зависит от PostgreSQL и drain; S3 влияет на вложения, а broker,
|
||||
worker и telemetry не выключают весь синхронный путь сообщений.
|
||||
|
||||
Две API-реплики выдерживают остановку одной. Postgres/S3/broker volumes
|
||||
переживают перезапуск контейнеров; outbox, publisher confirms, ack после commit,
|
||||
идемпотентная обработка и fencing token защищают от окон потери/дублей.
|
||||
Один Docker-хост, один LB, DB, S3, broker и Collector — точки отказа.
|
||||
Это не кластер с независимыми узлами и не backup; отказ диска/хоста здесь
|
||||
не покрыт. HTTP разрешён только для локального синтетического стенда.
|
||||
|
||||
Telemetry экспортируется фоново с ограниченной памятью. При недоступности
|
||||
backend Collector повторяет до 30 секунд, затем отбрасывает batch; при заполнении
|
||||
очереди возможны дополнительные потери. SDK counters export_failures и
|
||||
dropped_records отражают неуспешные exports; overflow SDK queue отдельно
|
||||
не подсчитывается. Collector send_failed — число неуспешных попыток с records,
|
||||
не число уникально потерянных spans. Отказ наблюдаемости не должен останавливать
|
||||
бизнес-записи. Семидневный retention задан для всех трёх сигналов;
|
||||
Prometheus дополнительно ограничен 512 MB, остальные backend ограничены
|
||||
контейнерами и свободным местом учебного Docker-диска.
|
||||
|
||||
## Методика и результаты проверки
|
||||
|
||||
Проверки выполнены 21.11.2026 на Windows/Docker Desktop 29.3.0, Linux
|
||||
containers, Intel Core Ultra 5 125U. Docker видит 14 CPU и 7 965 700 096 bytes
|
||||
RAM. Лимиты ресурсов каждого сервиса закреплены в Compose; это один хост.
|
||||
Телеметрия включена, sampling полный; данные синтетические.
|
||||
Результаты сняты из рабочего дерева уровня 4 после коммита `d0a062e` и
|
||||
включены в третий коммит. История: `git log -4 --oneline`.
|
||||
|
||||
Профиль нагрузки: публичный `tests/load.py`, closed loop POST messages,
|
||||
concurrency=4, длительность 20 секунд, один прогон, без автоматических retry.
|
||||
Отдельный длительный прогрев не выполнялся; до нагрузки стенд работал
|
||||
несколько минут и прошёл smoke. Создано **1645 сообщений**, ошибок **0**,
|
||||
успешные RPS **81.154**, p50 **38.796 ms**, p95 **114.354 ms**, p99 **194.742 ms**.
|
||||
Это client-side latency успешных записей; ошибки учитываются отдельно,
|
||||
closed loop имеет coordinated omission. Этот прогон не доказывает максимальную
|
||||
пропускную способность или выполнение SLO за длительный период.
|
||||
Данные: [load-summary.json](evidence/load-summary.json),
|
||||
[команда/stdout](evidence/load.txt).
|
||||
|
||||
За тот же интервал Prometheus зафиксировал максимальную RSS одного API
|
||||
128 040 960 bytes, worker 89 513 984, publisher 89 305 088. Максимальный
|
||||
15-секундный rate CPU одного API 0.389 CPU cores, worker 0.001,
|
||||
publisher 0.021. Это дискретные samples родительских процессов, не суммарные
|
||||
CPU/RAM всех контейнеров; image worker подпроцесс не включён.
|
||||
[Сырые запросы и range samples](evidence/load-resources.json).
|
||||
|
||||
| Критерий | Команда / сценарий | Доказательство |
|
||||
| --- | --- | --- |
|
||||
| Контракт и публичный smoke с telemetry | `python -X utf8 scripts/test_lab6.py` | [make-check-test.txt](evidence/make-check-test.txt) |
|
||||
| Полная OpenAPI / JSON Schema | venv Python, `scripts/validate_openapi.py` | [validate.txt](evidence/validate.txt) |
|
||||
| SDK, корреляция, W3C, отсутствие утечек | `student-tests/observability_unit.py` в API image | [5 tests](evidence/observability-unit.txt) |
|
||||
| Метрики HTTP / jobs, реальные API и worker logs/traces | `scripts/observe_lab6.py --baseline` | [baseline](evidence/observability-baseline.json), [trace](evidence/baseline-api-trace.json), [API logs](evidence/baseline-api-logs.json), [worker logs](evidence/baseline-worker-logs.json) |
|
||||
| Устойчивый upload, изображения и лимиты lab5 | `scripts/test_lab5.py --quick` | [images](evidence/images.txt), [decoder limits](evidence/limits.txt) |
|
||||
| Сохранность после restart | public persistence с `scripts/restart.sh` | [persistence.txt](evidence/persistence.txt) |
|
||||
| Смена API-реплики без потери сессии/данных | public failover с `scripts/stop-one.sh` | [failover.txt](evidence/failover.txt) |
|
||||
| Outbox gap, crash до commit / до ack, retry exhaustion, replay и duplicate no-op | `scripts/test_lab5.py` (полный прогон с telemetry) | [recovery.txt](evidence/recovery.txt) |
|
||||
|
||||
Метрики/логи/трассы из evidence запрошены у настоящих backend; это не
|
||||
статические фиктивные примеры. JSON dashboard и provisioning находятся в Git,
|
||||
а не только в скриншоте. Содержимое вывода и JSON проверяется на текущие
|
||||
синтетические secret markers и запрещённые telemetry keys/metric labels.
|
||||
Собственная проверка oversized upload отправляет заведомо чрезмерный
|
||||
Content-Length и проверяет ранний 413 до передачи body: urllib может потерять
|
||||
ранний ответ, продолжая писать уже отклонённые 10 MiB. Публичные тесты не менялись.
|
||||
|
||||
## Контролируемые отказы и диагностика
|
||||
|
||||
Общие данные: [observability-incidents.json](evidence/observability-incidents.json).
|
||||
Сценарии восстанавливают остановленные сервисы в `finally`; worker запускается
|
||||
через точные Docker container IDs, чтобы Compose не поднял одновременно
|
||||
искусственно остановленную зависимость. Перед проверкой alerts ожидаются
|
||||
пустой backlog и inactive alert, чтобы не принять старое срабатывание за новое.
|
||||
|
||||
| Воздействие | Наблюдение и причина | Восстановление |
|
||||
| --- | --- | --- |
|
||||
| Worker остановлен, приняты четыре upload (09:03:12–09:04:10 UTC) | Растут depth/oldest age при свежем publisher sample. `MessengerQueueLag` activeAt 09:03:35, затем firing. API продолжает выдавать 202; причина — отсутствие consumer, не длительность HTTP | После старта все четыре ready, оригиналы побайтно и SHA-256 совпали, thumbnail 256×160. Alert resolved в 09:04:10; полный сценарий 58 секунд |
|
||||
| PostgreSQL остановлен (08:58:31–09:00:26 UTC) | 18 прямых бизнес-запросов в две реальные API-реплики дали 503 unavailable. `MessengerHighErrorRate` activeAt 08:58:55, firing. DB error span и связанный log указывают на зависимость | БД запущена с тем же volume, readiness восстановился, сообщение записано. Alert resolved 09:00:25 после выхода ошибок из минутного окна |
|
||||
| Tempo и Loki остановлены на 65 секунд | 339 успешных записей сообщений; exporter send_failed растёт, очереди не превышают capacity=256, максимальная Collector RSS 204 MiB < контейнерного лимита 256 MiB | После старта и опустошения очередей новый upload снова имеет связанный API/worker trace и logs. Проверены все 339 контрольных trace ID и SERVER spans: отсутствующих 0; отсутствующих контрольных API logs 5 |
|
||||
|
||||
Для первого инцидента отдельный чистый прогон:
|
||||
[worker-lag-isolated.txt](evidence/worker-lag-isolated.txt).
|
||||
Для БД: [trace](evidence/incident-db-trace.json) и
|
||||
[logs](evidence/incident-db-logs.json). Запросы идут прямо в API-контейнеры,
|
||||
потому что после отрицательного readiness HAProxy сам возвращает 503;
|
||||
такой ответ LB не следует выдавать за ошибку, измеренную SDK API.
|
||||
Значение alert `activeAt` — начало pending, не точное время перехода firing;
|
||||
само состояние firing сохранено в JSON.
|
||||
|
||||
Потери измерены по известным request/trace IDs контрольных бизнес-записей
|
||||
после восстановления и опустошения очередей, с дополнительной проверкой
|
||||
наличия SERVER span. Это количество отсутствующих контрольных событий,
|
||||
а не всех внутренних spans системы. Отсутствующий log при сохранённой
|
||||
бизнес-записи показывает допустимую потерю наблюдений при ограниченном retry.
|
||||
[Сырые контрольные logs](evidence/telemetry-loss-control-logs.json) и список IDs
|
||||
с результатами поиска находятся в evidence. Нулевая потеря этих traces
|
||||
не означает гарантию сохранности telemetry при любом отказе.
|
||||
|
||||
Отдельно воспроизведены S3 outage, retry и lease recovery:
|
||||
[summary](evidence/observability-retry.json),
|
||||
[полная трасса](evidence/retry-api-trace.json),
|
||||
[worker logs](evidence/retry-worker-logs.json).
|
||||
Upload `8da6897d-7ad0-4a47-92a1-63e8820c6fb3` имеет общий trace
|
||||
`94af84b9ec1663416ac05526995fbd91`; видны error попытки 1/2,
|
||||
duplicate redelivery и success попытки 3. Проверен путь parent каждого
|
||||
consumer через producer до исходного SERVER span, а также DB/S3 children.
|
||||
Оригинал 111817 bytes сохранился, thumbnail имеет размер 256×160.
|
||||
Lease 120 секунд объясняет длительное восстановление после прерывания
|
||||
worker: бюджет smoke 60 секунд относится к свободному исправному worker,
|
||||
не к восстановлению аварийной попытки.
|
||||
|
||||
После массового чтения накопленных трасс настройки Tempo уточнены:
|
||||
GOMEMLIMIT=256 MiB, контейнер 512 MiB / 1 CPU. Итоговая конфигурация сохраняет
|
||||
ограниченную память и выдержала повторный поиск всех контрольных SERVER spans.
|
||||
Нагрузочный прогон и первоначальный backend outage сняты до этого изменения;
|
||||
бизнес-код и настройки Collector при этом не менялись.
|
||||
|
||||
Трек оценки 5 не заявлен. Нет сравнительного исследования sampling,
|
||||
доказанной production ёмкости или отказоустойчивого кластера хранилищ.
|
||||
|
||||
+114
@@ -0,0 +1,114 @@
|
||||
"""Low-cardinality saturation/backlog signals and safe dependency spans (lab6/4)."""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from datetime import datetime, timezone
|
||||
import os
|
||||
import resource
|
||||
import time
|
||||
|
||||
from opentelemetry import metrics
|
||||
from opentelemetry.metrics import Observation
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.telemetry import span
|
||||
|
||||
meter = metrics.get_meter("messenger")
|
||||
waiting = 0
|
||||
busy = 0
|
||||
snapshot: dict[str, float] = {}
|
||||
ready_duration = meter.create_histogram("messenger.attachment.time_to_ready", unit="s")
|
||||
retries = meter.create_counter("messenger.jobs.retries", unit="{retry}")
|
||||
|
||||
|
||||
class ObservedSession(AsyncSession):
|
||||
async def execute(self, statement, params=None, **kwargs):
|
||||
global waiting
|
||||
# Number of acquisition requests in progress (includes new connects).
|
||||
# Acquisition is separate from query execution to diagnose pool waits.
|
||||
waiting += 1
|
||||
try:
|
||||
with span("db.acquire", {"db.system": "postgresql"}):
|
||||
await self.connection()
|
||||
finally:
|
||||
waiting -= 1
|
||||
operation = str(statement).lstrip().split(None, 1)[0].upper()
|
||||
if operation not in {"SELECT", "INSERT", "UPDATE", "DELETE", "SET"}:
|
||||
operation = "OTHER"
|
||||
with span("db.query", {"db.system": "postgresql", "db.operation": operation}):
|
||||
return await super().execute(statement, params, **kwargs)
|
||||
|
||||
async def commit(self):
|
||||
with span("db.commit", {"db.system": "postgresql"}):
|
||||
return await super().commit()
|
||||
|
||||
|
||||
def install_gauges(engine) -> None:
|
||||
pool = engine.sync_engine.pool
|
||||
meter.create_observable_gauge("messenger.db.pool.used",
|
||||
callbacks=[lambda _: [Observation(pool.checkedout())]])
|
||||
meter.create_observable_gauge("messenger.db.pool.waiting",
|
||||
callbacks=[lambda _: [Observation(waiting)]])
|
||||
meter.create_observable_gauge("messenger.db.pool.capacity",
|
||||
callbacks=[lambda _: [Observation(pool.size() + pool._max_overflow)]])
|
||||
meter.create_observable_gauge("messenger.worker.busy",
|
||||
callbacks=[lambda _: [Observation(busy)]])
|
||||
capacity = 1 if os.getenv("OTEL_SERVICE_NAME") == "messenger-worker" else 0
|
||||
meter.create_observable_gauge("messenger.worker.capacity",
|
||||
callbacks=[lambda _: [Observation(capacity)]])
|
||||
if capacity:
|
||||
meter.create_observable_gauge("messenger.worker.heartbeat", unit="s",
|
||||
callbacks=[lambda _: [Observation(time.time())]])
|
||||
meter.create_observable_counter("messenger.process.cpu", unit="s",
|
||||
callbacks=[lambda _: [Observation(time.process_time())]])
|
||||
|
||||
def rss(_):
|
||||
# Linux /proc resident pages, current RSS (not ru_maxrss high watermark).
|
||||
try:
|
||||
with open("/proc/self/statm", encoding="ascii") as statm:
|
||||
value = int(statm.read().split()[1]) * os.sysconf("SC_PAGE_SIZE")
|
||||
except OSError:
|
||||
value = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss * 1024
|
||||
return [Observation(value)]
|
||||
|
||||
meter.create_observable_gauge("messenger.process.memory", unit="By", callbacks=[rss])
|
||||
if os.getenv("OTEL_SERVICE_NAME") == "messenger-publisher":
|
||||
for name, unit, key in (("depth", "", "depth"),
|
||||
("oldest_age", "s", "oldest_age"),
|
||||
("sample_age", "s", "sample_age")):
|
||||
def observe(_, key=key):
|
||||
if "sampled_at" not in snapshot:
|
||||
return []
|
||||
value = (time.monotonic() - snapshot["sampled_at"]
|
||||
if key == "sample_age" else snapshot[key])
|
||||
return [Observation(value, {"queue": "attachments"})]
|
||||
meter.create_observable_gauge("messenger.queue." + name, unit=unit,
|
||||
callbacks=[observe])
|
||||
|
||||
|
||||
async def backlog_loop(session_factory) -> None:
|
||||
# Authoritative logical queue = queued + processing, including durable
|
||||
# outbox and delayed retries. Remains observable when worker is stopped.
|
||||
while True:
|
||||
try:
|
||||
async with session_factory() as db:
|
||||
result = await db.execute(text("""
|
||||
SELECT count(*) AS depth,
|
||||
COALESCE(EXTRACT(EPOCH FROM now()-min(created_at)),0) AS oldest_age
|
||||
FROM attachments WHERE status IN ('queued','processing')
|
||||
"""))
|
||||
row = result.mappings().one()
|
||||
snapshot.update(depth=float(row["depth"]), oldest_age=float(row["oldest_age"]),
|
||||
sampled_at=time.monotonic())
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception:
|
||||
# Retain last sample; sample_age makes stale observations explicit.
|
||||
pass
|
||||
await asyncio.sleep(2)
|
||||
|
||||
|
||||
def observe_ready(row) -> None:
|
||||
elapsed = (datetime.now(timezone.utc) - row["created_at"]).total_seconds()
|
||||
ready_duration.record(max(0, elapsed), {"job_type": "attachment.process.v1"})
|
||||
@@ -7,6 +7,8 @@ from sqlalchemy.ext.asyncio import (
|
||||
)
|
||||
|
||||
from app.config import settings
|
||||
from app.advanced import ObservedSession, install_gauges
|
||||
from app.telemetry import span
|
||||
|
||||
|
||||
database_url = settings.database_url.replace(
|
||||
@@ -38,10 +40,13 @@ engine = create_async_engine(
|
||||
|
||||
SessionLocal = async_sessionmaker(
|
||||
bind=engine,
|
||||
class_=ObservedSession,
|
||||
expire_on_commit=False,
|
||||
autoflush=False,
|
||||
)
|
||||
|
||||
install_gauges(engine)
|
||||
|
||||
|
||||
async def get_db() -> AsyncGenerator[AsyncSession, None]:
|
||||
async with SessionLocal() as session:
|
||||
@@ -50,8 +55,9 @@ async def get_db() -> AsyncGenerator[AsyncSession, None]:
|
||||
|
||||
async def check_database_ready() -> bool:
|
||||
try:
|
||||
async with engine.connect() as connection:
|
||||
await connection.exec_driver_sql("SELECT 1")
|
||||
with span("db.health", {"db.system": "postgresql"}):
|
||||
async with engine.connect() as connection:
|
||||
await connection.exec_driver_sql("SELECT 1")
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
+4
-1
@@ -15,6 +15,7 @@ from app.pipeline import (RETENTION_SECONDS, arm_from_env, broker_channel,
|
||||
crash_if_armed, envelope, publish, retry_or_fail)
|
||||
from app.storage import storage
|
||||
from app.telemetry import configure, emit, install_termination_handler, shutdown
|
||||
from app import advanced
|
||||
|
||||
logger = logging.getLogger("messenger.jobs")
|
||||
|
||||
@@ -87,7 +88,6 @@ async def relay_one(channel) -> bool:
|
||||
{"id": row["id"]})
|
||||
await db.commit()
|
||||
logger.info("published job=%s attempt=%s", row["job_id"], row["attempt"])
|
||||
emit("job.published", job_id=str(row["job_id"]), attempt=row["attempt"])
|
||||
return True
|
||||
|
||||
|
||||
@@ -98,6 +98,7 @@ async def main() -> None:
|
||||
for name in ("aio_pika", "aiormq"):
|
||||
logging.getLogger(name).setLevel(logging.CRITICAL)
|
||||
await storage.start()
|
||||
backlog = asyncio.create_task(advanced.backlog_loop(SessionLocal))
|
||||
connection = None
|
||||
try:
|
||||
await arm_from_env("PUBLISHER_FAULT")
|
||||
@@ -128,6 +129,8 @@ async def main() -> None:
|
||||
connection = None
|
||||
await asyncio.sleep(1)
|
||||
finally:
|
||||
backlog.cancel()
|
||||
await asyncio.gather(backlog, return_exceptions=True)
|
||||
if connection:
|
||||
await connection.close()
|
||||
await storage.close()
|
||||
|
||||
+2
-2
@@ -60,8 +60,8 @@ async def run(args) -> None:
|
||||
# Ready effects are checked in PG by the worker and acked.
|
||||
await event(db, row, "operator_redeliver")
|
||||
await db.execute(text("""
|
||||
INSERT INTO outbox(job_id,attachment_id,attempt,available_at)
|
||||
SELECT job_id,id,attempt,GREATEST(now(),next_attempt_at)
|
||||
INSERT INTO outbox(job_id,attachment_id,attempt,available_at,traceparent,tracestate)
|
||||
SELECT job_id,id,attempt,GREATEST(now(),next_attempt_at),traceparent,tracestate
|
||||
FROM attachments WHERE id=:id
|
||||
"""), {"id": row["id"]})
|
||||
await db.commit()
|
||||
|
||||
+32
-10
@@ -15,6 +15,10 @@ from sqlalchemy import text
|
||||
|
||||
from app.config import read_value
|
||||
from app.db import SessionLocal
|
||||
from app import advanced
|
||||
from app.telemetry import emit, span
|
||||
from app.trace_context import carrier, parent_context
|
||||
from opentelemetry.trace import SpanKind
|
||||
|
||||
logger = logging.getLogger("messenger.pipeline")
|
||||
QUEUE = "attachments"
|
||||
@@ -56,13 +60,15 @@ async def broker_channel():
|
||||
|
||||
|
||||
def envelope(row) -> dict:
|
||||
return {
|
||||
job = {
|
||||
"schema_version": 1, "job_id": str(row["job_id"]),
|
||||
"type": "attachment.process.v1", "attachment_id": str(row["attachment_id"]),
|
||||
"pipeline_version": int(row.get("pipeline_version", 1)),
|
||||
"attempt": int(row["attempt"]),
|
||||
"created_at": row["created_at"].isoformat().replace("+00:00", "Z"),
|
||||
}
|
||||
job.update({name: row[name] for name in ("traceparent", "tracestate") if row.get(name)})
|
||||
return job
|
||||
|
||||
|
||||
def parse_envelope(raw: bytes) -> dict:
|
||||
@@ -94,12 +100,20 @@ def parse_envelope(raw: bytes) -> dict:
|
||||
|
||||
|
||||
async def publish(channel, job: dict) -> None:
|
||||
await channel.default_exchange.publish(
|
||||
aio_pika.Message(json.dumps(job).encode("utf-8"), content_type="application/json",
|
||||
delivery_mode=aio_pika.DeliveryMode.PERSISTENT,
|
||||
message_id=job["job_id"]),
|
||||
routing_key=QUEUE, mandatory=True, timeout=5,
|
||||
)
|
||||
with span("attachments publish", kind=SpanKind.PRODUCER, parent=parent_context(job), attributes={
|
||||
"messaging.system": "rabbitmq", "messaging.destination.name": QUEUE,
|
||||
"messaging.operation": "publish", "job_id": job["job_id"], "attempt": job["attempt"],
|
||||
}):
|
||||
# The confirmed envelope names this producer span as the consumer's
|
||||
# parent. Durable outbox remains unchanged if confirmation is lost.
|
||||
outgoing = {**job, **carrier()}
|
||||
await channel.default_exchange.publish(
|
||||
aio_pika.Message(json.dumps(outgoing).encode("utf-8"), content_type="application/json",
|
||||
delivery_mode=aio_pika.DeliveryMode.PERSISTENT,
|
||||
message_id=job["job_id"]),
|
||||
routing_key=QUEUE, mandatory=True, timeout=5,
|
||||
)
|
||||
emit("job.published", job_id=job["job_id"], attempt=job["attempt"], queue=QUEUE)
|
||||
|
||||
|
||||
async def event(db, row, name: str, detail: str = "") -> None:
|
||||
@@ -146,15 +160,23 @@ async def retry_or_fail(db, row, code: str, *, permanent: bool = False) -> None:
|
||||
logger.warning("failed attachment=%s code=%s", row["id"], final_code)
|
||||
return
|
||||
delay = retry_delay(row["attempts_run"])
|
||||
# A consumer retry continues from its active span. Lease recovery may have
|
||||
# no active span, so retain the context durably recorded by the old claim.
|
||||
context = carrier() or {name: row.get(name) for name in ("traceparent", "tracestate")}
|
||||
await db.execute(text("""
|
||||
UPDATE attachments SET status='queued',error_code=NULL,attempt=attempt+1,
|
||||
lease_token=NULL,lease_until=NULL,
|
||||
traceparent=:traceparent,tracestate=:tracestate,
|
||||
next_attempt_at=now()+make_interval(secs => CAST(:delay AS double precision)),
|
||||
updated_at=now() WHERE id=:id
|
||||
"""), {"id": row["id"], "delay": delay})
|
||||
"""), {"id": row["id"], "delay": delay,
|
||||
"traceparent": context.get("traceparent"), "tracestate": context.get("tracestate")})
|
||||
await db.execute(text("""
|
||||
INSERT INTO outbox(job_id,attachment_id,attempt,available_at)
|
||||
SELECT job_id,id,attempt,next_attempt_at FROM attachments WHERE id=:id
|
||||
INSERT INTO outbox(job_id,attachment_id,attempt,available_at,traceparent,tracestate)
|
||||
SELECT job_id,id,attempt,next_attempt_at,traceparent,tracestate FROM attachments WHERE id=:id
|
||||
"""), {"id": row["id"]})
|
||||
await event(db, row, "retry", f"{code}; delay={delay:.3f}s")
|
||||
logger.info("retry attachment=%s after=%.3fs", row["id"], delay)
|
||||
advanced.retries.add(1, {"job_type": "attachment.process.v1"})
|
||||
emit("job.retry", job_id=str(row["job_id"]), attachment_id=str(row["id"]),
|
||||
attempt=row["attempt"] + 1, outcome="retry", queue=QUEUE)
|
||||
|
||||
+12
-4
@@ -9,6 +9,7 @@ from uuid import UUID, uuid4
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from app.trace_context import carrier
|
||||
|
||||
|
||||
async def create_user(
|
||||
@@ -580,14 +581,17 @@ async def finalize_attachment(
|
||||
|
||||
job_id = uuid4()
|
||||
final_key = f"originals/{attachment_id}/v1/original"
|
||||
context = carrier()
|
||||
result = await db.execute(
|
||||
text(
|
||||
f"""
|
||||
INSERT INTO attachments AS a
|
||||
(id, chat_id, owner_id, filename, content_type, size_bytes,
|
||||
sha256, width, height, object_key, staging_key, job_id, status)
|
||||
sha256, width, height, object_key, staging_key, job_id, status,
|
||||
traceparent, tracestate)
|
||||
VALUES (:id, :chat_id, :owner_id, :filename, :content_type,
|
||||
:size_bytes, :sha256, :width, :height, :final_key, :object_key, :job_id, 'queued')
|
||||
:size_bytes, :sha256, :width, :height, :final_key, :object_key, :job_id, 'queued',
|
||||
:traceparent, :tracestate)
|
||||
RETURNING {ATTACHMENT_COLUMNS}
|
||||
"""
|
||||
),
|
||||
@@ -604,12 +608,16 @@ async def finalize_attachment(
|
||||
"object_key": object_key,
|
||||
"final_key": final_key,
|
||||
"job_id": job_id,
|
||||
"traceparent": context.get("traceparent"),
|
||||
"tracestate": context.get("tracestate"),
|
||||
},
|
||||
)
|
||||
row = dict(result.mappings().one())
|
||||
await db.execute(
|
||||
text("INSERT INTO outbox (job_id, attachment_id, attempt) VALUES (:job_id, :id, 1)"),
|
||||
{"job_id": job_id, "id": attachment_id},
|
||||
text("""INSERT INTO outbox (job_id, attachment_id, attempt, traceparent, tracestate)
|
||||
VALUES (:job_id, :id, 1, :traceparent, :tracestate)"""),
|
||||
{"job_id": job_id, "id": attachment_id,
|
||||
"traceparent": context.get("traceparent"), "tracestate": context.get("tracestate")},
|
||||
)
|
||||
await db.commit()
|
||||
return row
|
||||
|
||||
@@ -16,12 +16,25 @@ from collections.abc import AsyncIterator
|
||||
from contextlib import AsyncExitStack
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, BinaryIO
|
||||
from functools import wraps
|
||||
|
||||
from aiobotocore.config import AioConfig
|
||||
from aiobotocore.session import get_session
|
||||
from botocore.exceptions import BotoCoreError, ClientError
|
||||
|
||||
from app.config import settings
|
||||
from app.telemetry import span
|
||||
|
||||
|
||||
def observed(operation):
|
||||
"""Export operation/outcome only, never S3 URL, key or credentials."""
|
||||
def decorate(method):
|
||||
@wraps(method)
|
||||
async def run(*args, **kwargs):
|
||||
with span("s3." + operation, {"rpc.system": "aws-api", "rpc.method": operation}):
|
||||
return await method(*args, **kwargs)
|
||||
return run
|
||||
return decorate
|
||||
|
||||
|
||||
logger = logging.getLogger("messenger.storage")
|
||||
@@ -89,6 +102,7 @@ class ObjectStorage:
|
||||
raise StorageUnavailable("S3 client is not started")
|
||||
return self._client
|
||||
|
||||
@observed("PutObject")
|
||||
async def put(
|
||||
self,
|
||||
key: str,
|
||||
@@ -112,6 +126,7 @@ class ObjectStorage:
|
||||
except (BotoCoreError, ClientError, OSError, asyncio.TimeoutError) as exc:
|
||||
raise StorageUnavailable(type(exc).__name__) from exc
|
||||
|
||||
@observed("GetObject")
|
||||
async def open_download(self, key: str) -> tuple[AsyncIterator[bytes], int]:
|
||||
"""GET the object; S3 errors surface here, before any byte is sent."""
|
||||
client = self._require_client()
|
||||
@@ -136,6 +151,7 @@ class ObjectStorage:
|
||||
|
||||
return chunks(), int(response["ContentLength"])
|
||||
|
||||
@observed("DeleteObject")
|
||||
async def delete(self, key: str) -> None:
|
||||
"""Idempotent: deleting a missing key succeeds in S3."""
|
||||
client = self._require_client()
|
||||
@@ -145,6 +161,7 @@ class ObjectStorage:
|
||||
except (BotoCoreError, ClientError, OSError, asyncio.TimeoutError) as exc:
|
||||
raise StorageUnavailable(type(exc).__name__) from exc
|
||||
|
||||
@observed("HeadObject")
|
||||
async def exists(self, key: str) -> bool:
|
||||
client = self._require_client()
|
||||
try:
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
"""W3C context carried durably alongside a job, never its HTTP credentials.
|
||||
|
||||
The upload transaction stores ``carrier()``; publisher and retries restore it
|
||||
with ``parent_context``. Only a new producer/consumer span changes the parent
|
||||
span ID. Context-less legacy jobs deliberately start a new trace.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
|
||||
from opentelemetry.context import Context
|
||||
from opentelemetry.trace import get_current_span
|
||||
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
|
||||
|
||||
_PROPAGATOR = TraceContextTextMapPropagator()
|
||||
_FIELDS = ("traceparent", "tracestate")
|
||||
|
||||
|
||||
def carrier() -> dict[str, str]:
|
||||
"""Return the active span's W3C fields, or {} outside a valid span."""
|
||||
headers: dict[str, str] = {}
|
||||
_PROPAGATOR.inject(headers)
|
||||
return {name: headers[name] for name in _FIELDS if headers.get(name)}
|
||||
|
||||
|
||||
def parent_context(row_or_job: Mapping | None) -> Context:
|
||||
"""Extract nullable DB/envelope fields without inheriting ambient context.
|
||||
|
||||
W3C parsing validates version, nonzero trace/span IDs, flags and tracestate.
|
||||
Malformed or missing parent IDs become a new root, even if a publisher's
|
||||
unrelated maintenance span happens to be active.
|
||||
"""
|
||||
headers: dict[str, str] = {}
|
||||
if isinstance(row_or_job, Mapping):
|
||||
for name in _FIELDS:
|
||||
value = row_or_job.get(name)
|
||||
if isinstance(value, str) and value:
|
||||
headers[name] = value
|
||||
context = _PROPAGATOR.extract(headers, context=Context())
|
||||
return context if get_current_span(context).get_span_context().is_valid else Context()
|
||||
+12
-3
@@ -22,6 +22,8 @@ from app.storage import ObjectMissing, storage
|
||||
from app.telemetry import (configure, emit, install_termination_handler, jobs_duration,
|
||||
jobs_processed, shutdown, span)
|
||||
from opentelemetry.trace import SpanKind, Status, StatusCode
|
||||
from app import advanced
|
||||
from app.trace_context import carrier, parent_context
|
||||
|
||||
logger = logging.getLogger("messenger.worker")
|
||||
|
||||
@@ -46,9 +48,11 @@ async def claim(job: dict) -> dict | None:
|
||||
result = await db.execute(text("""
|
||||
UPDATE attachments SET status='processing',attempts_run=attempts_run+1,
|
||||
lease_token=:token,lease_until=now()+make_interval(secs => :lease),
|
||||
traceparent=:traceparent,tracestate=:tracestate,
|
||||
updated_at=now()
|
||||
WHERE id=:id AND next_attempt_at<=now() RETURNING *
|
||||
"""), {"id": row["id"], "token": uuid4(), "lease": LEASE_SECONDS})
|
||||
"""), {"id": row["id"], "token": uuid4(), "lease": LEASE_SECONDS,
|
||||
"traceparent": carrier().get("traceparent"), "tracestate": carrier().get("tracestate")})
|
||||
claimed = result.mappings().first()
|
||||
if claimed is None:
|
||||
await db.rollback()
|
||||
@@ -107,7 +111,8 @@ async def build_objects(row: dict) -> list[dict]:
|
||||
await chunks.aclose()
|
||||
if size != declared_length or size != row["size_bytes"] or digest.hexdigest() != row["sha256"]:
|
||||
raise PermanentFailure("source_checksum_mismatch")
|
||||
variants = await generate(source, directory)
|
||||
with span("image.generate"):
|
||||
variants = await generate(source, directory)
|
||||
# API never publishes directly to this final prefix. Repeating one
|
||||
# v1 job writes the same bytes to the same four keys, never appends.
|
||||
with source.open("rb") as body:
|
||||
@@ -146,6 +151,8 @@ async def finish(row: dict, *, variants=None, code: str | None = None,
|
||||
"""), {"id": row["id"], "variants": json.dumps(variants)})
|
||||
await event(db, current, "ready")
|
||||
await db.commit()
|
||||
if code is None:
|
||||
advanced.observe_ready(current)
|
||||
return True
|
||||
|
||||
|
||||
@@ -195,13 +202,15 @@ async def handle(message) -> None:
|
||||
pass
|
||||
started = time.perf_counter()
|
||||
outcome = "error"
|
||||
with span("attachments process", kind=SpanKind.CONSUMER, attributes={
|
||||
with span("attachments process", kind=SpanKind.CONSUMER, parent=parent_context(job), attributes={
|
||||
"messaging.system": "rabbitmq", "messaging.destination.name": "attachments",
|
||||
"messaging.operation": "process", "job_type": "attachment.process.v1",
|
||||
}) as active:
|
||||
try:
|
||||
advanced.busy = 1
|
||||
outcome = await _handle(message)
|
||||
finally:
|
||||
advanced.busy = 0
|
||||
duration = time.perf_counter() - started
|
||||
attrs = {"job_type": "attachment.process.v1", "outcome": outcome}
|
||||
jobs_processed.add(1, attrs)
|
||||
|
||||
+5
-2
@@ -299,7 +299,7 @@ services:
|
||||
volumes:
|
||||
- ./infra/haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
|
||||
ports:
|
||||
- "${LB_PORT:-8080}:8080"
|
||||
- "127.0.0.1:${LB_PORT:-8080}:8080"
|
||||
# Backend state page for the operator, only on the host loopback.
|
||||
- "127.0.0.1:${LB_STATS_PORT:-8404}:8404"
|
||||
networks: [edge]
|
||||
@@ -360,6 +360,9 @@ services:
|
||||
|
||||
tempo:
|
||||
image: grafana/tempo:2.6.1
|
||||
environment:
|
||||
# Bound the Go heap while searching many retained blocks on this small stand.
|
||||
GOMEMLIMIT: 256MiB
|
||||
# A new named volume needs write access for the local evaluation backend.
|
||||
user: "0:0"
|
||||
command: ["-config.file=/etc/tempo/tempo.yaml"]
|
||||
@@ -371,7 +374,7 @@ services:
|
||||
restart: unless-stopped
|
||||
deploy:
|
||||
resources:
|
||||
limits: { cpus: "0.5", memory: 384M }
|
||||
limits: { cpus: "1.0", memory: 512M }
|
||||
|
||||
grafana:
|
||||
image: grafana/grafana:11.2.2
|
||||
|
||||
@@ -0,0 +1,191 @@
|
||||
{
|
||||
"status": "success",
|
||||
"data": {
|
||||
"resultType": "streams",
|
||||
"result": [
|
||||
{
|
||||
"stream": {
|
||||
"code_filepath": "/app/app/telemetry.py",
|
||||
"code_function": "emit",
|
||||
"code_lineno": "180",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"duration_seconds": "0.14954320399965582",
|
||||
"event": "http.request.complete",
|
||||
"flags": "1",
|
||||
"method": "POST",
|
||||
"observed_timestamp": "1795252636588490258",
|
||||
"outcome": "success",
|
||||
"request_id": "observe-60a6c2da00814bf08cb3a801b8388ef4",
|
||||
"route": "/api/v1/chats/{chat_id}/attachments",
|
||||
"scope_name": "messenger.telemetry.events",
|
||||
"service_instance_id": "b33ff621673d",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "7f43d8d88f960fff",
|
||||
"status_code": "202",
|
||||
"trace_id": "115550cd6d2c36ddf2bea96f48e89c6b"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1795252636588292864",
|
||||
"http.request.complete"
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"stats": {
|
||||
"summary": {
|
||||
"bytesProcessedPerSecond": 3571455,
|
||||
"linesProcessedPerSecond": 18129,
|
||||
"totalBytesProcessed": 18124,
|
||||
"totalLinesProcessed": 92,
|
||||
"execTime": 0.005075,
|
||||
"queueTime": 0.000262,
|
||||
"subqueries": 0,
|
||||
"totalEntriesReturned": 1,
|
||||
"splits": 0,
|
||||
"shards": 1,
|
||||
"totalPostFilterLines": 1,
|
||||
"totalStructuredMetadataBytesProcessed": 16192
|
||||
},
|
||||
"querier": {
|
||||
"store": {
|
||||
"totalChunksRef": 0,
|
||||
"totalChunksDownloaded": 0,
|
||||
"chunksDownloadTime": 0,
|
||||
"queryReferencedStructuredMetadata": false,
|
||||
"chunk": {
|
||||
"headChunkBytes": 0,
|
||||
"headChunkLines": 0,
|
||||
"decompressedBytes": 0,
|
||||
"decompressedLines": 0,
|
||||
"compressedBytes": 0,
|
||||
"totalDuplicates": 0,
|
||||
"postFilterLines": 0,
|
||||
"headChunkStructuredMetadataBytes": 0,
|
||||
"decompressedStructuredMetadataBytes": 0
|
||||
},
|
||||
"chunkRefsFetchTime": 0,
|
||||
"congestionControlLatency": 0,
|
||||
"pipelineWrapperFilteredLines": 0
|
||||
}
|
||||
},
|
||||
"ingester": {
|
||||
"totalReached": 1,
|
||||
"totalChunksMatched": 1,
|
||||
"totalBatches": 2,
|
||||
"totalLinesSent": 1,
|
||||
"store": {
|
||||
"totalChunksRef": 0,
|
||||
"totalChunksDownloaded": 0,
|
||||
"chunksDownloadTime": 0,
|
||||
"queryReferencedStructuredMetadata": true,
|
||||
"chunk": {
|
||||
"headChunkBytes": 18124,
|
||||
"headChunkLines": 92,
|
||||
"decompressedBytes": 0,
|
||||
"decompressedLines": 0,
|
||||
"compressedBytes": 0,
|
||||
"totalDuplicates": 0,
|
||||
"postFilterLines": 1,
|
||||
"headChunkStructuredMetadataBytes": 16192,
|
||||
"decompressedStructuredMetadataBytes": 0
|
||||
},
|
||||
"chunkRefsFetchTime": 81728,
|
||||
"congestionControlLatency": 0,
|
||||
"pipelineWrapperFilteredLines": 0
|
||||
}
|
||||
},
|
||||
"cache": {
|
||||
"chunk": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"index": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"result": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"statsResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"volumeResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"seriesResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"labelResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"instantMetricResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
}
|
||||
},
|
||||
"index": {
|
||||
"totalChunks": 0,
|
||||
"postFilterChunks": 0,
|
||||
"shardsDuration": 0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,191 @@
|
||||
{
|
||||
"status": "success",
|
||||
"data": {
|
||||
"resultType": "streams",
|
||||
"result": [
|
||||
{
|
||||
"stream": {
|
||||
"attachment_id": "eb62cd3e-75a1-4dbe-8965-9e012b312282",
|
||||
"attempt": "1",
|
||||
"code_filepath": "/app/app/telemetry.py",
|
||||
"code_function": "emit",
|
||||
"code_lineno": "180",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"duration_seconds": "0.42506639099974564",
|
||||
"event": "job.complete",
|
||||
"flags": "1",
|
||||
"job_id": "e1b6a557-f9fa-462c-ac79-1ab71f0b35bf",
|
||||
"job_type": "attachment.process.v1",
|
||||
"observed_timestamp": "1795252637174010973",
|
||||
"outcome": "success",
|
||||
"scope_name": "messenger.telemetry.events",
|
||||
"service_instance_id": "4d9015b23ab54bde9d37384ddc4bf89b",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "c71f9d5977e9bd59",
|
||||
"trace_id": "115550cd6d2c36ddf2bea96f48e89c6b"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1795252637173831168",
|
||||
"job.complete"
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"stats": {
|
||||
"summary": {
|
||||
"bytesProcessedPerSecond": 79386,
|
||||
"linesProcessedPerSecond": 422,
|
||||
"totalBytesProcessed": 376,
|
||||
"totalLinesProcessed": 2,
|
||||
"execTime": 0.004736,
|
||||
"queueTime": 0.000244,
|
||||
"subqueries": 0,
|
||||
"totalEntriesReturned": 1,
|
||||
"splits": 0,
|
||||
"shards": 1,
|
||||
"totalPostFilterLines": 1,
|
||||
"totalStructuredMetadataBytesProcessed": 352
|
||||
},
|
||||
"querier": {
|
||||
"store": {
|
||||
"totalChunksRef": 0,
|
||||
"totalChunksDownloaded": 0,
|
||||
"chunksDownloadTime": 0,
|
||||
"queryReferencedStructuredMetadata": false,
|
||||
"chunk": {
|
||||
"headChunkBytes": 0,
|
||||
"headChunkLines": 0,
|
||||
"decompressedBytes": 0,
|
||||
"decompressedLines": 0,
|
||||
"compressedBytes": 0,
|
||||
"totalDuplicates": 0,
|
||||
"postFilterLines": 0,
|
||||
"headChunkStructuredMetadataBytes": 0,
|
||||
"decompressedStructuredMetadataBytes": 0
|
||||
},
|
||||
"chunkRefsFetchTime": 0,
|
||||
"congestionControlLatency": 0,
|
||||
"pipelineWrapperFilteredLines": 0
|
||||
}
|
||||
},
|
||||
"ingester": {
|
||||
"totalReached": 1,
|
||||
"totalChunksMatched": 1,
|
||||
"totalBatches": 2,
|
||||
"totalLinesSent": 1,
|
||||
"store": {
|
||||
"totalChunksRef": 0,
|
||||
"totalChunksDownloaded": 0,
|
||||
"chunksDownloadTime": 0,
|
||||
"queryReferencedStructuredMetadata": true,
|
||||
"chunk": {
|
||||
"headChunkBytes": 376,
|
||||
"headChunkLines": 2,
|
||||
"decompressedBytes": 0,
|
||||
"decompressedLines": 0,
|
||||
"compressedBytes": 0,
|
||||
"totalDuplicates": 0,
|
||||
"postFilterLines": 1,
|
||||
"headChunkStructuredMetadataBytes": 352,
|
||||
"decompressedStructuredMetadataBytes": 0
|
||||
},
|
||||
"chunkRefsFetchTime": 138740,
|
||||
"congestionControlLatency": 0,
|
||||
"pipelineWrapperFilteredLines": 0
|
||||
}
|
||||
},
|
||||
"cache": {
|
||||
"chunk": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"index": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"result": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"statsResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"volumeResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"seriesResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"labelResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"instantMetricResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
}
|
||||
},
|
||||
"index": {
|
||||
"totalChunks": 0,
|
||||
"postFilterChunks": 0,
|
||||
"shardsDuration": 0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,6 @@
|
||||
|
||||
UTC: 2026-11-21T09:16:46Z
|
||||
Command: C:\Users\masha\AppData\Local\Programs\Python\Python313\python.exe -X utf8 scripts/check_contract.py
|
||||
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:16:46Z
|
||||
@@ -0,0 +1,5 @@
|
||||
|
||||
UTC: 2026-11-21T09:16:45Z
|
||||
Command: docker compose config --quiet
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:16:46Z
|
||||
@@ -0,0 +1,220 @@
|
||||
|
||||
Before public persistence — 2026-11-21T09:14:39Z
|
||||
|
||||
UTC: 2026-11-21T09:14:39Z
|
||||
Command: docker compose --profile tools ps -a -q
|
||||
81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85
|
||||
0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8
|
||||
d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d
|
||||
0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084
|
||||
b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026
|
||||
5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4
|
||||
278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300
|
||||
68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf
|
||||
93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c
|
||||
13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b
|
||||
cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb
|
||||
a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612
|
||||
00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007
|
||||
fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05
|
||||
837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053
|
||||
27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:14:40Z
|
||||
|
||||
UTC: 2026-11-21T09:14:40Z
|
||||
Command: docker inspect --format "{{.Name}} status={{.State.Status}} started={{.State.StartedAt}} finished={{.State.FinishedAt}}" 81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85 0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8 d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d 0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084 b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026 5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4 278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300 68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf 93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c 13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612 00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007 fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05 837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053 27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
|
||||
/messenger-lab6-api-1 status=running started=2026-11-21T09:10:22.045070464Z finished=2026-11-21T09:10:21.647750535Z
|
||||
/messenger-lab6-api-2 status=running started=2026-11-21T09:10:18.318585259Z finished=2026-11-21T09:10:17.940277751Z
|
||||
/messenger-lab6-broker-1 status=running started=2026-11-21T09:10:24.158928332Z finished=2026-11-21T09:10:23.907977242Z
|
||||
/messenger-lab6-db-1 status=running started=2026-11-21T08:59:12.528963564Z finished=2026-11-21T08:58:31.826616128Z
|
||||
/messenger-lab6-grafana-1 status=running started=2026-11-21T08:51:11.412805651Z finished=2026-11-21T08:51:10.520984964Z
|
||||
/messenger-lab6-lb-1 status=running started=2026-11-21T08:51:52.621809468Z finished=0001-01-01T00:00:00Z
|
||||
/messenger-lab6-loki-1 status=running started=2026-11-21T09:01:34.707920007Z finished=2026-11-21T09:00:28.644856083Z
|
||||
/messenger-lab6-migrate-1 status=exited started=2026-11-21T08:52:49.834748909Z finished=2026-11-21T08:52:50.79798573Z
|
||||
/messenger-lab6-otel-collector-1 status=running started=2026-11-21T08:49:03.407564143Z finished=0001-01-01T00:00:00Z
|
||||
/messenger-lab6-prometheus-1 status=running started=2026-11-21T08:51:11.311294483Z finished=2026-11-21T08:51:10.512998838Z
|
||||
/messenger-lab6-publisher-1 status=running started=2026-11-21T09:10:55.825987305Z finished=2026-11-21T09:10:55.547390235Z
|
||||
/messenger-lab6-s3-1 status=running started=2026-11-21T09:14:09.569875679Z finished=2026-11-21T09:13:44.880718518Z
|
||||
/messenger-lab6-s3-init-1 status=exited started=2026-11-21T08:53:15.109417522Z finished=2026-11-21T08:53:17.005092363Z
|
||||
/messenger-lab6-tempo-1 status=running started=2026-11-21T09:06:37.030642992Z finished=0001-01-01T00:00:00Z
|
||||
/messenger-lab6-valkey-1 status=running started=2026-11-21T08:45:28.390651046Z finished=2026-11-21T08:45:27.109421727Z
|
||||
/messenger-lab6-worker-1 status=running started=2026-11-21T09:13:45.19691868Z finished=2026-11-21T09:13:41.499885247Z
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:14:40Z
|
||||
|
||||
After public persistence — 2026-11-21T09:15:30Z
|
||||
|
||||
UTC: 2026-11-21T09:15:30Z
|
||||
Command: docker compose --profile tools ps -a -q
|
||||
81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85
|
||||
0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8
|
||||
d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d
|
||||
0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084
|
||||
b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026
|
||||
5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4
|
||||
278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300
|
||||
68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf
|
||||
93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c
|
||||
13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b
|
||||
cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb
|
||||
a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612
|
||||
00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007
|
||||
fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05
|
||||
837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053
|
||||
27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:15:31Z
|
||||
|
||||
UTC: 2026-11-21T09:15:31Z
|
||||
Command: docker inspect --format "{{.Name}} status={{.State.Status}} started={{.State.StartedAt}} finished={{.State.FinishedAt}}" 81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85 0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8 d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d 0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084 b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026 5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4 278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300 68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf 93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c 13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612 00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007 fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05 837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053 27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
|
||||
/messenger-lab6-api-1 status=running started=2026-11-21T09:15:18.227427331Z finished=2026-11-21T09:14:49.40226195Z
|
||||
/messenger-lab6-api-2 status=running started=2026-11-21T09:15:17.875504619Z finished=2026-11-21T09:14:49.181726236Z
|
||||
/messenger-lab6-broker-1 status=running started=2026-11-21T09:14:51.311241575Z finished=2026-11-21T09:14:49.824073591Z
|
||||
/messenger-lab6-db-1 status=running started=2026-11-21T09:14:50.494933266Z finished=2026-11-21T09:14:49.814681938Z
|
||||
/messenger-lab6-grafana-1 status=running started=2026-11-21T08:51:11.412805651Z finished=2026-11-21T08:51:10.520984964Z
|
||||
/messenger-lab6-lb-1 status=running started=2026-11-21T09:15:24.826206264Z finished=2026-11-21T09:14:45.274514498Z
|
||||
/messenger-lab6-loki-1 status=running started=2026-11-21T09:01:34.707920007Z finished=2026-11-21T09:00:28.644856083Z
|
||||
/messenger-lab6-migrate-1 status=exited started=2026-11-21T08:52:49.834748909Z finished=2026-11-21T08:52:50.79798573Z
|
||||
/messenger-lab6-otel-collector-1 status=running started=2026-11-21T08:49:03.407564143Z finished=0001-01-01T00:00:00Z
|
||||
/messenger-lab6-prometheus-1 status=running started=2026-11-21T08:51:11.311294483Z finished=2026-11-21T08:51:10.512998838Z
|
||||
/messenger-lab6-publisher-1 status=running started=2026-11-21T09:15:17.332690697Z finished=2026-11-21T09:14:45.830823803Z
|
||||
/messenger-lab6-s3-1 status=running started=2026-11-21T09:14:51.024302658Z finished=2026-11-21T09:14:49.817292594Z
|
||||
/messenger-lab6-s3-init-1 status=exited started=2026-11-21T08:53:15.109417522Z finished=2026-11-21T08:53:17.005092363Z
|
||||
/messenger-lab6-tempo-1 status=running started=2026-11-21T09:06:37.030642992Z finished=0001-01-01T00:00:00Z
|
||||
/messenger-lab6-valkey-1 status=running started=2026-11-21T09:14:50.777977548Z finished=2026-11-21T09:14:49.803932669Z
|
||||
/messenger-lab6-worker-1 status=running started=2026-11-21T09:15:17.592030927Z finished=2026-11-21T09:14:45.554953067Z
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:15:31Z
|
||||
|
||||
Before public failover — 2026-11-21T09:15:31Z
|
||||
|
||||
UTC: 2026-11-21T09:15:31Z
|
||||
Command: docker compose --profile tools ps -a -q
|
||||
81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85
|
||||
0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8
|
||||
d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d
|
||||
0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084
|
||||
b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026
|
||||
5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4
|
||||
278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300
|
||||
68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf
|
||||
93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c
|
||||
13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b
|
||||
cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb
|
||||
a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612
|
||||
00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007
|
||||
fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05
|
||||
837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053
|
||||
27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:15:31Z
|
||||
|
||||
UTC: 2026-11-21T09:15:31Z
|
||||
Command: docker inspect --format "{{.Name}} status={{.State.Status}} started={{.State.StartedAt}} finished={{.State.FinishedAt}}" 81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85 0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8 d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d 0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084 b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026 5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4 278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300 68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf 93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c 13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612 00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007 fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05 837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053 27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
|
||||
/messenger-lab6-api-1 status=running started=2026-11-21T09:15:18.227427331Z finished=2026-11-21T09:14:49.40226195Z
|
||||
/messenger-lab6-api-2 status=running started=2026-11-21T09:15:17.875504619Z finished=2026-11-21T09:14:49.181726236Z
|
||||
/messenger-lab6-broker-1 status=running started=2026-11-21T09:14:51.311241575Z finished=2026-11-21T09:14:49.824073591Z
|
||||
/messenger-lab6-db-1 status=running started=2026-11-21T09:14:50.494933266Z finished=2026-11-21T09:14:49.814681938Z
|
||||
/messenger-lab6-grafana-1 status=running started=2026-11-21T08:51:11.412805651Z finished=2026-11-21T08:51:10.520984964Z
|
||||
/messenger-lab6-lb-1 status=running started=2026-11-21T09:15:24.826206264Z finished=2026-11-21T09:14:45.274514498Z
|
||||
/messenger-lab6-loki-1 status=running started=2026-11-21T09:01:34.707920007Z finished=2026-11-21T09:00:28.644856083Z
|
||||
/messenger-lab6-migrate-1 status=exited started=2026-11-21T08:52:49.834748909Z finished=2026-11-21T08:52:50.79798573Z
|
||||
/messenger-lab6-otel-collector-1 status=running started=2026-11-21T08:49:03.407564143Z finished=0001-01-01T00:00:00Z
|
||||
/messenger-lab6-prometheus-1 status=running started=2026-11-21T08:51:11.311294483Z finished=2026-11-21T08:51:10.512998838Z
|
||||
/messenger-lab6-publisher-1 status=running started=2026-11-21T09:15:17.332690697Z finished=2026-11-21T09:14:45.830823803Z
|
||||
/messenger-lab6-s3-1 status=running started=2026-11-21T09:14:51.024302658Z finished=2026-11-21T09:14:49.817292594Z
|
||||
/messenger-lab6-s3-init-1 status=exited started=2026-11-21T08:53:15.109417522Z finished=2026-11-21T08:53:17.005092363Z
|
||||
/messenger-lab6-tempo-1 status=running started=2026-11-21T09:06:37.030642992Z finished=0001-01-01T00:00:00Z
|
||||
/messenger-lab6-valkey-1 status=running started=2026-11-21T09:14:50.777977548Z finished=2026-11-21T09:14:49.803932669Z
|
||||
/messenger-lab6-worker-1 status=running started=2026-11-21T09:15:17.592030927Z finished=2026-11-21T09:14:45.554953067Z
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:15:31Z
|
||||
|
||||
After public failover, before API restoration — 2026-11-21T09:15:52Z
|
||||
|
||||
UTC: 2026-11-21T09:15:52Z
|
||||
Command: docker compose --profile tools ps -a -q
|
||||
81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85
|
||||
0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8
|
||||
d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d
|
||||
0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084
|
||||
b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026
|
||||
5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4
|
||||
278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300
|
||||
68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf
|
||||
93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c
|
||||
13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b
|
||||
cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb
|
||||
a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612
|
||||
00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007
|
||||
fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05
|
||||
837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053
|
||||
27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:15:53Z
|
||||
|
||||
UTC: 2026-11-21T09:15:53Z
|
||||
Command: docker inspect --format "{{.Name}} status={{.State.Status}} started={{.State.StartedAt}} finished={{.State.FinishedAt}}" 81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85 0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8 d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d 0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084 b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026 5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4 278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300 68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf 93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c 13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612 00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007 fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05 837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053 27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
|
||||
/messenger-lab6-api-1 status=exited started=2026-11-21T09:15:18.227427331Z finished=2026-11-21T09:15:40.679293151Z
|
||||
/messenger-lab6-api-2 status=running started=2026-11-21T09:15:17.875504619Z finished=2026-11-21T09:14:49.181726236Z
|
||||
/messenger-lab6-broker-1 status=running started=2026-11-21T09:14:51.311241575Z finished=2026-11-21T09:14:49.824073591Z
|
||||
/messenger-lab6-db-1 status=running started=2026-11-21T09:14:50.494933266Z finished=2026-11-21T09:14:49.814681938Z
|
||||
/messenger-lab6-grafana-1 status=running started=2026-11-21T08:51:11.412805651Z finished=2026-11-21T08:51:10.520984964Z
|
||||
/messenger-lab6-lb-1 status=running started=2026-11-21T09:15:24.826206264Z finished=2026-11-21T09:14:45.274514498Z
|
||||
/messenger-lab6-loki-1 status=running started=2026-11-21T09:01:34.707920007Z finished=2026-11-21T09:00:28.644856083Z
|
||||
/messenger-lab6-migrate-1 status=exited started=2026-11-21T08:52:49.834748909Z finished=2026-11-21T08:52:50.79798573Z
|
||||
/messenger-lab6-otel-collector-1 status=running started=2026-11-21T08:49:03.407564143Z finished=0001-01-01T00:00:00Z
|
||||
/messenger-lab6-prometheus-1 status=running started=2026-11-21T08:51:11.311294483Z finished=2026-11-21T08:51:10.512998838Z
|
||||
/messenger-lab6-publisher-1 status=running started=2026-11-21T09:15:17.332690697Z finished=2026-11-21T09:14:45.830823803Z
|
||||
/messenger-lab6-s3-1 status=running started=2026-11-21T09:14:51.024302658Z finished=2026-11-21T09:14:49.817292594Z
|
||||
/messenger-lab6-s3-init-1 status=exited started=2026-11-21T08:53:15.109417522Z finished=2026-11-21T08:53:17.005092363Z
|
||||
/messenger-lab6-tempo-1 status=running started=2026-11-21T09:06:37.030642992Z finished=0001-01-01T00:00:00Z
|
||||
/messenger-lab6-valkey-1 status=running started=2026-11-21T09:14:50.777977548Z finished=2026-11-21T09:14:49.803932669Z
|
||||
/messenger-lab6-worker-1 status=running started=2026-11-21T09:15:17.592030927Z finished=2026-11-21T09:14:45.554953067Z
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:15:53Z
|
||||
|
||||
After API restoration — 2026-11-21T09:16:00Z
|
||||
|
||||
UTC: 2026-11-21T09:16:00Z
|
||||
Command: docker compose --profile tools ps -a -q
|
||||
81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85
|
||||
0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8
|
||||
d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d
|
||||
0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084
|
||||
b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026
|
||||
5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4
|
||||
278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300
|
||||
68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf
|
||||
93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c
|
||||
13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b
|
||||
cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb
|
||||
a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612
|
||||
00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007
|
||||
fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05
|
||||
837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053
|
||||
27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:16:01Z
|
||||
|
||||
UTC: 2026-11-21T09:16:01Z
|
||||
Command: docker inspect --format "{{.Name}} status={{.State.Status}} started={{.State.StartedAt}} finished={{.State.FinishedAt}}" 81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85 0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8 d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d 0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084 b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026 5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4 278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300 68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf 93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c 13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612 00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007 fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05 837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053 27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
|
||||
/messenger-lab6-api-1 status=running started=2026-11-21T09:15:54.07308443Z finished=2026-11-21T09:15:40.679293151Z
|
||||
/messenger-lab6-api-2 status=running started=2026-11-21T09:15:17.875504619Z finished=2026-11-21T09:14:49.181726236Z
|
||||
/messenger-lab6-broker-1 status=running started=2026-11-21T09:14:51.311241575Z finished=2026-11-21T09:14:49.824073591Z
|
||||
/messenger-lab6-db-1 status=running started=2026-11-21T09:14:50.494933266Z finished=2026-11-21T09:14:49.814681938Z
|
||||
/messenger-lab6-grafana-1 status=running started=2026-11-21T08:51:11.412805651Z finished=2026-11-21T08:51:10.520984964Z
|
||||
/messenger-lab6-lb-1 status=running started=2026-11-21T09:15:24.826206264Z finished=2026-11-21T09:14:45.274514498Z
|
||||
/messenger-lab6-loki-1 status=running started=2026-11-21T09:01:34.707920007Z finished=2026-11-21T09:00:28.644856083Z
|
||||
/messenger-lab6-migrate-1 status=exited started=2026-11-21T08:52:49.834748909Z finished=2026-11-21T08:52:50.79798573Z
|
||||
/messenger-lab6-otel-collector-1 status=running started=2026-11-21T08:49:03.407564143Z finished=0001-01-01T00:00:00Z
|
||||
/messenger-lab6-prometheus-1 status=running started=2026-11-21T08:51:11.311294483Z finished=2026-11-21T08:51:10.512998838Z
|
||||
/messenger-lab6-publisher-1 status=running started=2026-11-21T09:15:17.332690697Z finished=2026-11-21T09:14:45.830823803Z
|
||||
/messenger-lab6-s3-1 status=running started=2026-11-21T09:14:51.024302658Z finished=2026-11-21T09:14:49.817292594Z
|
||||
/messenger-lab6-s3-init-1 status=exited started=2026-11-21T08:53:15.109417522Z finished=2026-11-21T08:53:17.005092363Z
|
||||
/messenger-lab6-tempo-1 status=running started=2026-11-21T09:06:37.030642992Z finished=0001-01-01T00:00:00Z
|
||||
/messenger-lab6-valkey-1 status=running started=2026-11-21T09:14:50.777977548Z finished=2026-11-21T09:14:49.803932669Z
|
||||
/messenger-lab6-worker-1 status=running started=2026-11-21T09:15:17.592030927Z finished=2026-11-21T09:14:45.554953067Z
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:16:01Z
|
||||
@@ -0,0 +1,187 @@
|
||||
{
|
||||
"captured_at": "2026-11-21T09:17:47.509294Z",
|
||||
"live_worker_capacity": 1.0,
|
||||
"panels": 24,
|
||||
"promql_checks": [
|
||||
{
|
||||
"panel": "Business API requests / second",
|
||||
"query": "sum(rate(messenger_http_requests_total{service_name=\"messenger-api\",route!~\"/health/.*\"}[1m]))",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "Business API 5xx / all responses",
|
||||
"query": "(sum(rate(messenger_http_requests_total{service_name=\"messenger-api\",route!~\"/health/.*\",status_class=\"5xx\"}[1m])) or vector(0)) / clamp_min(sum(rate(messenger_http_requests_total{service_name=\"messenger-api\",route!~\"/health/.*\"}[1m])), 1e-9)",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "API mean duration (seconds)",
|
||||
"query": "sum(rate(messenger_http_duration_seconds_sum{service_name=\"messenger-api\",route!~\"/health/.*\"}[1m])) / clamp_min(sum(rate(messenger_http_duration_seconds_count{service_name=\"messenger-api\",route!~\"/health/.*\"}[1m])), 1e-9)",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "Worker attempt outcomes / second",
|
||||
"query": "sum by (outcome) (rate(messenger_jobs_processed_total{service_name=\"messenger-worker\"}[1m]))",
|
||||
"series": 3
|
||||
},
|
||||
{
|
||||
"panel": "Worker mean attempt duration (seconds)",
|
||||
"query": "sum(rate(messenger_jobs_duration_seconds_sum{service_name=\"messenger-worker\"}[1m])) / clamp_min(sum(rate(messenger_jobs_duration_seconds_count{service_name=\"messenger-worker\"}[1m])), 1e-9)",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "Worker completed attempts (counter)",
|
||||
"query": "sum by (outcome) (messenger_jobs_processed_total{service_name=\"messenger-worker\"})",
|
||||
"series": 3
|
||||
},
|
||||
{
|
||||
"panel": "Structured application events (expand for trace_id / span_id / request_id)",
|
||||
"query": "{service_name=~\"messenger-api|messenger-worker|messenger-publisher\"}",
|
||||
"series": 1343
|
||||
},
|
||||
{
|
||||
"panel": "API duration p50 / p95 / p99 (seconds)",
|
||||
"query": "histogram_quantile(0.5, sum by (le) (rate(messenger_http_duration_seconds_bucket{service_name=\"messenger-api\",route!~\"/health/.*\"}[1m])))",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "API duration p50 / p95 / p99 (seconds)",
|
||||
"query": "histogram_quantile(0.95, sum by (le) (rate(messenger_http_duration_seconds_bucket{service_name=\"messenger-api\",route!~\"/health/.*\"}[1m])))",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "API duration p50 / p95 / p99 (seconds)",
|
||||
"query": "histogram_quantile(0.99, sum by (le) (rate(messenger_http_duration_seconds_bucket{service_name=\"messenger-api\",route!~\"/health/.*\"}[1m])))",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "Worker duration p50 / p95 / p99 (seconds)",
|
||||
"query": "histogram_quantile(0.5, sum by (le) (rate(messenger_jobs_duration_seconds_bucket{service_name=\"messenger-worker\"}[1m])))",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "Worker duration p50 / p95 / p99 (seconds)",
|
||||
"query": "histogram_quantile(0.95, sum by (le) (rate(messenger_jobs_duration_seconds_bucket{service_name=\"messenger-worker\"}[1m])))",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "Worker duration p50 / p95 / p99 (seconds)",
|
||||
"query": "histogram_quantile(0.99, sum by (le) (rate(messenger_jobs_duration_seconds_bucket{service_name=\"messenger-worker\"}[1m])))",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "Durable unfinished jobs (count)",
|
||||
"query": "max(messenger_queue_depth{service_name=\"messenger-publisher\"})",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "Oldest unfinished job / observer sample age (seconds)",
|
||||
"query": "max(messenger_queue_oldest_age_seconds{service_name=\"messenger-publisher\"})",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "Oldest unfinished job / observer sample age (seconds)",
|
||||
"query": "max(messenger_queue_sample_age_seconds{service_name=\"messenger-publisher\"})",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "DB pool used / capacity (connections)",
|
||||
"query": "messenger_db_pool_used",
|
||||
"series": 13
|
||||
},
|
||||
{
|
||||
"panel": "DB pool used / capacity (connections)",
|
||||
"query": "messenger_db_pool_capacity",
|
||||
"series": 13
|
||||
},
|
||||
{
|
||||
"panel": "DB acquisitions in progress (requests)",
|
||||
"query": "messenger_db_pool_waiting",
|
||||
"series": 13
|
||||
},
|
||||
{
|
||||
"panel": "Worker saturation (busy / capacity)",
|
||||
"query": "(sum(messenger_worker_busy{service_name=\"messenger-worker\"} and on(service_instance_id) (time() - messenger_worker_heartbeat_seconds{service_name=\"messenger-worker\"} < 15)) or vector(0)) / (sum(messenger_worker_capacity{service_name=\"messenger-worker\"} and on(service_instance_id) (time() - messenger_worker_heartbeat_seconds{service_name=\"messenger-worker\"} < 15)) or vector(0))",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "Worker busy / capacity (jobs)",
|
||||
"query": "sum(messenger_worker_busy{service_name=\"messenger-worker\"} and on(service_instance_id) (time() - messenger_worker_heartbeat_seconds{service_name=\"messenger-worker\"} < 15)) or vector(0)",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "Worker busy / capacity (jobs)",
|
||||
"query": "sum(messenger_worker_capacity{service_name=\"messenger-worker\"} and on(service_instance_id) (time() - messenger_worker_heartbeat_seconds{service_name=\"messenger-worker\"} < 15)) or vector(0)",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "Application CPU (cores)",
|
||||
"query": "rate(messenger_process_cpu_seconds_total[1m])",
|
||||
"series": 13
|
||||
},
|
||||
{
|
||||
"panel": "Application resident RAM (bytes)",
|
||||
"query": "messenger_process_memory_bytes",
|
||||
"series": 13
|
||||
},
|
||||
{
|
||||
"panel": "Acceptance to ready p50 / p95 / p99 (seconds)",
|
||||
"query": "histogram_quantile(0.5, sum by (le) (rate(messenger_attachment_time_to_ready_seconds_bucket{service_name=\"messenger-worker\"}[1m])))",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "Acceptance to ready p50 / p95 / p99 (seconds)",
|
||||
"query": "histogram_quantile(0.95, sum by (le) (rate(messenger_attachment_time_to_ready_seconds_bucket{service_name=\"messenger-worker\"}[1m])))",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "Acceptance to ready p50 / p95 / p99 (seconds)",
|
||||
"query": "histogram_quantile(0.99, sum by (le) (rate(messenger_attachment_time_to_ready_seconds_bucket{service_name=\"messenger-worker\"}[1m])))",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "API availability SLI / SLO (5 minutes)",
|
||||
"query": "1 - (sum(rate(messenger_http_requests_total{service_name=\"messenger-api\",route!~\"/health/.*\",status_class=\"5xx\"}[5m])) or vector(0)) / sum(rate(messenger_http_requests_total{service_name=\"messenger-api\",route!~\"/health/.*\"}[5m]))",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "API availability SLI / SLO (5 minutes)",
|
||||
"query": "vector(0.99)",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "Ready within 60 seconds SLI / SLO (5 minutes)",
|
||||
"query": "sum(rate(messenger_attachment_time_to_ready_seconds_bucket{service_name=\"messenger-worker\",le=\"60\"}[5m])) / sum(rate(messenger_attachment_time_to_ready_seconds_count{service_name=\"messenger-worker\"}[5m]))",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "Ready within 60 seconds SLI / SLO (5 minutes)",
|
||||
"query": "vector(0.95)",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "Local alert state: pending / firing",
|
||||
"query": "ALERTS{alertname=~\"MessengerHighErrorRate|MessengerQueueLag\"}",
|
||||
"series": 0
|
||||
},
|
||||
{
|
||||
"panel": "Collector queue occupancy (batches)",
|
||||
"query": "otelcol_exporter_queue_size{exporter=~\"otlphttp/.*\"}",
|
||||
"series": 2
|
||||
},
|
||||
{
|
||||
"panel": "Collector queue occupancy (batches)",
|
||||
"query": "otelcol_exporter_queue_capacity{exporter=~\"otlphttp/.*\"}",
|
||||
"series": 2
|
||||
},
|
||||
{
|
||||
"panel": "Collector failed export attempts / second",
|
||||
"query": "rate(otelcol_exporter_send_failed_spans[1m])",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "Collector failed export attempts / second",
|
||||
"query": "rate(otelcol_exporter_send_failed_log_records[1m])",
|
||||
"series": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
|
||||
UTC: 2026-11-21T09:15:31Z
|
||||
Command: docker compose --profile tools run --rm -T --no-deps tester python3 -u tests/resilience.py --mode failover --action-script scripts/stop-one.sh
|
||||
Container messenger-lab6-tester-run-0af389d5dbc3 Creating
|
||||
Container messenger-lab6-tester-run-0af389d5dbc3 Created
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||
{"observed_instances_before": 2, "successful_writes": 127, "failed_attempts": 0, "first_recovery_seconds": 4.189}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:15:52Z
|
||||
|
||||
UTC: 2026-11-21T09:15:53Z
|
||||
Command: docker compose --profile tools run --rm -T --no-deps tester sh scripts/start-api.sh
|
||||
Container messenger-lab6-tester-run-b9e9496603b9 Creating
|
||||
Container messenger-lab6-tester-run-b9e9496603b9 Created
|
||||
started api container 81ebb23d65ec
|
||||
running api replicas: 2
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:16:00Z
|
||||
@@ -0,0 +1,19 @@
|
||||
|
||||
UTC: 2026-11-21T09:09:53Z
|
||||
Command: docker compose --profile tools run --rm -T --no-deps tester python3 -u student-tests/image_validation_unit.py
|
||||
Container messenger-lab6-tester-run-56217a03ad55 Creating
|
||||
Container messenger-lab6-tester-run-56217a03ad55 Created
|
||||
test_adam7_pass_filter_boundaries (__main__.ImageValidationTests.test_adam7_pass_filter_boundaries) ... ok
|
||||
test_crc_compressed_tail_and_pixel_bomb (__main__.ImageValidationTests.test_crc_compressed_tail_and_pixel_bomb) ... ok
|
||||
test_critical_headers_and_chunk_order (__main__.ImageValidationTests.test_critical_headers_and_chunk_order) ... ok
|
||||
test_filter_values_and_row_boundaries (__main__.ImageValidationTests.test_filter_values_and_row_boundaries) ... ok
|
||||
test_jpeg_and_truncated_progressive_above_old_bypass_limit (__main__.ImageValidationTests.test_jpeg_and_truncated_progressive_above_old_bypass_limit) ... ok
|
||||
test_palette_shape_and_transparency (__main__.ImageValidationTests.test_palette_shape_and_transparency) ... ok
|
||||
test_valid_png_and_trailing_padding (__main__.ImageValidationTests.test_valid_png_and_trailing_padding) ... ok
|
||||
|
||||
----------------------------------------------------------------------
|
||||
Ran 7 tests in 0.462s
|
||||
|
||||
OK
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:09:55Z
|
||||
@@ -0,0 +1,14 @@
|
||||
|
||||
UTC: 2026-11-21T09:09:55Z
|
||||
Command: docker compose --profile tools run --rm -T --no-deps tester python3 -u student-tests/lab5_images.py
|
||||
Container messenger-lab6-tester-run-d0860115a1da Creating
|
||||
Container messenger-lab6-tester-run-d0860115a1da Created
|
||||
PASS centered crop, three real variants, checksums, original bytes and ACL
|
||||
PASS no upscaling and floor rounding for thumbnail/optimized
|
||||
PASS transparent PNG composited onto white JPEG
|
||||
PASS EXIF orientation applied to variants and metadata removed
|
||||
PASS malformed PNG filter and truncated >16MP progressive JPEG rejected before 202
|
||||
PASS invalid/MIME mismatch 415, >10MiB and >40M pixels 413, unsafe filenames 400
|
||||
PASS: lab5 grade4 image checks
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:10:05Z
|
||||
@@ -0,0 +1,191 @@
|
||||
{
|
||||
"status": "success",
|
||||
"data": {
|
||||
"resultType": "streams",
|
||||
"result": [
|
||||
{
|
||||
"stream": {
|
||||
"code_filepath": "/app/app/telemetry.py",
|
||||
"code_function": "emit",
|
||||
"code_lineno": "180",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "error",
|
||||
"duration_seconds": "3.3880072909998944",
|
||||
"event": "http.request.complete",
|
||||
"flags": "1",
|
||||
"method": "POST",
|
||||
"observed_timestamp": "1795251516034833239",
|
||||
"outcome": "error",
|
||||
"request_id": "f44f1b74-f3f1-4cbe-8b44-510cfbe21ebd",
|
||||
"route": "/api/v1/users",
|
||||
"scope_name": "messenger.telemetry.events",
|
||||
"service_instance_id": "0d13c5814dab",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "17",
|
||||
"severity_text": "ERROR",
|
||||
"span_id": "6389f79f96d8c840",
|
||||
"status_code": "503",
|
||||
"trace_id": "819eab187881e315742feb7f2ebb70ff"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1795251516034664192",
|
||||
"http.request.complete"
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"stats": {
|
||||
"summary": {
|
||||
"bytesProcessedPerSecond": 4455963,
|
||||
"linesProcessedPerSecond": 22619,
|
||||
"totalBytesProcessed": 70329,
|
||||
"totalLinesProcessed": 357,
|
||||
"execTime": 0.015783,
|
||||
"queueTime": 0.002886,
|
||||
"subqueries": 0,
|
||||
"totalEntriesReturned": 1,
|
||||
"splits": 2,
|
||||
"shards": 2,
|
||||
"totalPostFilterLines": 1,
|
||||
"totalStructuredMetadataBytesProcessed": 62832
|
||||
},
|
||||
"querier": {
|
||||
"store": {
|
||||
"totalChunksRef": 0,
|
||||
"totalChunksDownloaded": 0,
|
||||
"chunksDownloadTime": 0,
|
||||
"queryReferencedStructuredMetadata": false,
|
||||
"chunk": {
|
||||
"headChunkBytes": 0,
|
||||
"headChunkLines": 0,
|
||||
"decompressedBytes": 0,
|
||||
"decompressedLines": 0,
|
||||
"compressedBytes": 0,
|
||||
"totalDuplicates": 0,
|
||||
"postFilterLines": 0,
|
||||
"headChunkStructuredMetadataBytes": 0,
|
||||
"decompressedStructuredMetadataBytes": 0
|
||||
},
|
||||
"chunkRefsFetchTime": 0,
|
||||
"congestionControlLatency": 0,
|
||||
"pipelineWrapperFilteredLines": 0
|
||||
}
|
||||
},
|
||||
"ingester": {
|
||||
"totalReached": 2,
|
||||
"totalChunksMatched": 2,
|
||||
"totalBatches": 3,
|
||||
"totalLinesSent": 1,
|
||||
"store": {
|
||||
"totalChunksRef": 0,
|
||||
"totalChunksDownloaded": 0,
|
||||
"chunksDownloadTime": 0,
|
||||
"queryReferencedStructuredMetadata": true,
|
||||
"chunk": {
|
||||
"headChunkBytes": 70329,
|
||||
"headChunkLines": 357,
|
||||
"decompressedBytes": 0,
|
||||
"decompressedLines": 0,
|
||||
"compressedBytes": 0,
|
||||
"totalDuplicates": 0,
|
||||
"postFilterLines": 1,
|
||||
"headChunkStructuredMetadataBytes": 62832,
|
||||
"decompressedStructuredMetadataBytes": 0
|
||||
},
|
||||
"chunkRefsFetchTime": 584920,
|
||||
"congestionControlLatency": 0,
|
||||
"pipelineWrapperFilteredLines": 0
|
||||
}
|
||||
},
|
||||
"cache": {
|
||||
"chunk": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"index": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"result": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"statsResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"volumeResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"seriesResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"labelResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"instantMetricResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
}
|
||||
},
|
||||
"index": {
|
||||
"totalChunks": 0,
|
||||
"postFilterChunks": 0,
|
||||
"shardsDuration": 0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
{
|
||||
"batches": [
|
||||
{
|
||||
"resource": {
|
||||
"attributes": [
|
||||
{
|
||||
"key": "service.version",
|
||||
"value": {
|
||||
"stringValue": "1.6.0"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.instance.id",
|
||||
"value": {
|
||||
"stringValue": "0d13c5814dab"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "deployment.environment.name",
|
||||
"value": {
|
||||
"stringValue": "lab"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.name",
|
||||
"value": {
|
||||
"stringValue": "messenger-api"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"scopeSpans": [
|
||||
{
|
||||
"scope": {
|
||||
"name": "messenger",
|
||||
"version": "1.6.0"
|
||||
},
|
||||
"spans": [
|
||||
{
|
||||
"traceId": "gZ6rGHiB4xV0L+t/Lrtw/w==",
|
||||
"spanId": "z5kgvNQvytc=",
|
||||
"parentSpanId": "Y4n3n5bYyEA=",
|
||||
"name": "db.acquire",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1795251513033234050",
|
||||
"endTimeUnixNano": "1795251516033386648",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "db.system",
|
||||
"value": {
|
||||
"stringValue": "postgresql"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "exception_type",
|
||||
"value": {
|
||||
"stringValue": "TimeoutError"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {
|
||||
"code": "STATUS_CODE_ERROR"
|
||||
}
|
||||
},
|
||||
{
|
||||
"traceId": "gZ6rGHiB4xV0L+t/Lrtw/w==",
|
||||
"spanId": "Y4n3n5bYyEA=",
|
||||
"name": "POST /api/v1/users",
|
||||
"kind": "SPAN_KIND_SERVER",
|
||||
"startTimeUnixNano": "1795251512647658833",
|
||||
"endTimeUnixNano": "1795251516034910448",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "method",
|
||||
"value": {
|
||||
"stringValue": "POST"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "route",
|
||||
"value": {
|
||||
"stringValue": "/api/v1/users"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "status_class",
|
||||
"value": {
|
||||
"stringValue": "5xx"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "http.response.status_code",
|
||||
"value": {
|
||||
"intValue": "503"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {
|
||||
"code": "STATUS_CODE_ERROR"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,443 @@
|
||||
|
||||
UTC: 2026-11-21T09:09:39Z
|
||||
Command: C:\Users\masha\AppData\Local\Programs\Python\Python313\python.exe -X utf8 scripts/test_lab5.py
|
||||
|
||||
UTC: 2026-11-21T09:09:40Z
|
||||
Command: docker compose --profile tools ps --status running --services
|
||||
api
|
||||
broker
|
||||
db
|
||||
grafana
|
||||
lb
|
||||
loki
|
||||
otel-collector
|
||||
prometheus
|
||||
publisher
|
||||
s3
|
||||
tempo
|
||||
valkey
|
||||
worker
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:09:40Z
|
||||
|
||||
UTC: 2026-11-21T09:09:40Z
|
||||
Command: docker compose --profile tools ps -a
|
||||
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
|
||||
messenger-lab6-api-1 messenger-lab6-api:dev "python -m app.server" api 18 minutes ago Up 17 minutes (healthy) 8080/tcp
|
||||
messenger-lab6-api-2 messenger-lab6-api:dev "python -m app.server" api 18 minutes ago Up 17 minutes (healthy) 8080/tcp
|
||||
messenger-lab6-broker-1 rabbitmq:3.13.7-alpine "/bin/sh -ec 'export…" broker 34 minutes ago Up 24 minutes (healthy) 4369/tcp, 5671-5672/tcp, 15691-15692/tcp, 25672/tcp
|
||||
messenger-lab6-db-1 postgres:16.4-alpine "docker-entrypoint.s…" db 34 minutes ago Up 10 minutes (healthy) 5432/tcp
|
||||
messenger-lab6-grafana-1 grafana/grafana:11.2.2 "/run.sh" grafana 19 minutes ago Up 18 minutes 127.0.0.1:3000->3000/tcp
|
||||
messenger-lab6-lb-1 haproxy:3.0.11-alpine "docker-entrypoint.s…" lb 17 minutes ago Up 17 minutes (healthy) 127.0.0.1:8086->8080/tcp, 127.0.0.1:8406->8404/tcp
|
||||
messenger-lab6-loki-1 grafana/loki:3.2.1 "/usr/bin/loki -conf…" loki 19 minutes ago Up 8 minutes 127.0.0.1:3100->3100/tcp
|
||||
messenger-lab6-migrate-1 messenger-lab6-api:dev "python scripts/migr…" migrate 18 minutes ago Exited (0) 16 minutes ago
|
||||
messenger-lab6-otel-collector-1 otel/opentelemetry-collector-contrib:0.111.0 "/otelcol-contrib --…" otel-collector 20 minutes ago Up 20 minutes 4317/tcp, 55678-55679/tcp
|
||||
messenger-lab6-prometheus-1 prom/prometheus:v2.55.0 "/bin/prometheus --c…" prometheus 19 minutes ago Up 18 minutes 127.0.0.1:9090->9090/tcp
|
||||
messenger-lab6-publisher-1 messenger-lab6-api:dev "python -m app.jobs" publisher 18 minutes ago Up 17 minutes 8080/tcp
|
||||
messenger-lab6-s3-1 chrislusf/seaweedfs:3.97 "/bin/sh /opt/lab/st…" s3 34 minutes ago Up 13 minutes (healthy) 7333/tcp, 8080/tcp, 8333/tcp, 8888/tcp, 9333/tcp, 18080/tcp, 18888/tcp, 19333/tcp
|
||||
messenger-lab6-s3-init-1 messenger-lab6-api:dev "python scripts/s3_b…" s3-init 18 minutes ago Exited (0) 16 minutes ago
|
||||
messenger-lab6-tempo-1 grafana/tempo:2.6.1 "/tempo -config.file…" tempo 3 minutes ago Up 3 minutes 127.0.0.1:3200->3200/tcp
|
||||
messenger-lab6-valkey-1 valkey/valkey:8.0.1-alpine "docker-entrypoint.s…" valkey 34 minutes ago Up 24 minutes (healthy) 6379/tcp
|
||||
messenger-lab6-worker-1 messenger-lab6-api:dev "python -m app.worker" worker 18 minutes ago Up 5 minutes 8080/tcp
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:09:40Z
|
||||
|
||||
UTC: 2026-11-21T09:09:40Z
|
||||
Command: docker compose --profile tools build tester
|
||||
Image messenger-lab6-tester Building
|
||||
#1 [internal] load local bake definitions
|
||||
#1 reading from stdin 587B 0.0s done
|
||||
#1 DONE 0.0s
|
||||
|
||||
#2 [internal] load build definition from Dockerfile.tester
|
||||
#2 transferring dockerfile: 405B done
|
||||
#2 DONE 0.0s
|
||||
|
||||
#3 [internal] load metadata for docker.io/library/python:3.12.11-alpine3.22
|
||||
#3 DONE 0.2s
|
||||
|
||||
#4 [internal] load metadata for docker.io/library/docker:28.5.1-cli
|
||||
#4 DONE 0.2s
|
||||
|
||||
#5 [internal] load .dockerignore
|
||||
#5 transferring context: 286B done
|
||||
#5 DONE 0.0s
|
||||
|
||||
#6 [stage-1 1/5] FROM docker.io/library/python:3.12.11-alpine3.22@sha256:efcdfa6a6b2fd2afb9c7dfa9a5b288a6f68338b5cfdebe6b637d986067d85757
|
||||
#6 resolve docker.io/library/python:3.12.11-alpine3.22@sha256:efcdfa6a6b2fd2afb9c7dfa9a5b288a6f68338b5cfdebe6b637d986067d85757 0.0s done
|
||||
#6 DONE 0.1s
|
||||
|
||||
#7 [docker-cli 1/1] FROM docker.io/library/docker:28.5.1-cli@sha256:9190b0613792e658a7783cf14b2d5ace5941bb68ede7276922ea36ee457d76ad
|
||||
#7 resolve docker.io/library/docker:28.5.1-cli@sha256:9190b0613792e658a7783cf14b2d5ace5941bb68ede7276922ea36ee457d76ad 0.0s done
|
||||
#7 DONE 0.1s
|
||||
|
||||
#8 [stage-1 2/5] RUN apk add --no-cache make
|
||||
#8 CACHED
|
||||
|
||||
#9 [stage-1 3/5] RUN pip install --no-cache-dir Pillow==11.0.0
|
||||
#9 CACHED
|
||||
|
||||
#10 [stage-1 4/5] COPY --from=docker-cli /usr/local/bin/docker /usr/local/bin/docker
|
||||
#10 CACHED
|
||||
|
||||
#11 [stage-1 5/5] WORKDIR /work
|
||||
#11 CACHED
|
||||
|
||||
#12 exporting to image
|
||||
#12 exporting layers done
|
||||
#12 exporting manifest sha256:1c93bba11e9ded8b819bbcadafd9217f8b6b4fc5301deeca1e720c49a45f1b2e done
|
||||
#12 exporting config sha256:9ad2daad56b9cfccc6c03e7be0468faf51994e1604e86b502077a03f28844325 done
|
||||
#12 exporting attestation manifest sha256:96fa2816d4a538ed8e6e3ea5c702033990470ee27e842b91c5f2979f0304e95a 0.1s done
|
||||
#12 exporting manifest list sha256:3e586fe0bf4aeb3814ad182a780c6173a9e252da1c33e7e61f20187228c8b23f 0.0s done
|
||||
#12 naming to docker.io/library/messenger-lab6-tester:latest
|
||||
#12 naming to docker.io/library/messenger-lab6-tester:latest done
|
||||
#12 unpacking to docker.io/library/messenger-lab6-tester:latest 0.0s done
|
||||
#12 DONE 0.2s
|
||||
|
||||
#13 resolving provenance for metadata file
|
||||
#13 DONE 0.0s
|
||||
Image messenger-lab6-tester Built
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:09:42Z
|
||||
|
||||
UTC: 2026-11-21T09:09:42Z
|
||||
Command: docker compose --profile tools run --rm -T --no-deps tester python3 -u scripts/check_contract.py
|
||||
Container messenger-lab6-tester-run-e5ea96fea8b0 Creating
|
||||
Container messenger-lab6-tester-run-e5ea96fea8b0 Created
|
||||
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:09:46Z
|
||||
|
||||
UTC: 2026-11-21T09:09:46Z
|
||||
Command: docker compose --profile tools run --rm -T --no-deps tester python3 -u tests/smoke.py
|
||||
Container messenger-lab6-tester-run-42a477b98467 Creating
|
||||
Container messenger-lab6-tester-run-42a477b98467 Created
|
||||
PASS: lab6 public API scenario; infrastructure and grading evidence remain separate.
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:09:53Z
|
||||
|
||||
UTC: 2026-11-21T09:09:53Z
|
||||
Command: docker compose --profile tools run --rm -T --no-deps tester python3 -u student-tests/image_validation_unit.py
|
||||
Container messenger-lab6-tester-run-56217a03ad55 Creating
|
||||
Container messenger-lab6-tester-run-56217a03ad55 Created
|
||||
test_adam7_pass_filter_boundaries (__main__.ImageValidationTests.test_adam7_pass_filter_boundaries) ... ok
|
||||
test_crc_compressed_tail_and_pixel_bomb (__main__.ImageValidationTests.test_crc_compressed_tail_and_pixel_bomb) ... ok
|
||||
test_critical_headers_and_chunk_order (__main__.ImageValidationTests.test_critical_headers_and_chunk_order) ... ok
|
||||
test_filter_values_and_row_boundaries (__main__.ImageValidationTests.test_filter_values_and_row_boundaries) ... ok
|
||||
test_jpeg_and_truncated_progressive_above_old_bypass_limit (__main__.ImageValidationTests.test_jpeg_and_truncated_progressive_above_old_bypass_limit) ... ok
|
||||
test_palette_shape_and_transparency (__main__.ImageValidationTests.test_palette_shape_and_transparency) ... ok
|
||||
test_valid_png_and_trailing_padding (__main__.ImageValidationTests.test_valid_png_and_trailing_padding) ... ok
|
||||
|
||||
----------------------------------------------------------------------
|
||||
Ran 7 tests in 0.462s
|
||||
|
||||
OK
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:09:55Z
|
||||
|
||||
UTC: 2026-11-21T09:09:55Z
|
||||
Command: docker compose --profile tools run --rm -T --no-deps tester python3 -u student-tests/lab5_images.py
|
||||
Container messenger-lab6-tester-run-d0860115a1da Creating
|
||||
Container messenger-lab6-tester-run-d0860115a1da Created
|
||||
PASS centered crop, three real variants, checksums, original bytes and ACL
|
||||
PASS no upscaling and floor rounding for thumbnail/optimized
|
||||
PASS transparent PNG composited onto white JPEG
|
||||
PASS EXIF orientation applied to variants and metadata removed
|
||||
PASS malformed PNG filter and truncated >16MP progressive JPEG rejected before 202
|
||||
PASS invalid/MIME mismatch 415, >10MiB and >40M pixels 413, unsafe filenames 400
|
||||
PASS: lab5 grade4 image checks
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:10:05Z
|
||||
|
||||
UTC: 2026-11-21T09:10:05Z
|
||||
Command: docker compose --profile tools exec -T worker python -u - < student-tests/lab5_limits.py
|
||||
PASS memory: real 16MP image decoder stopped at RLIMIT_AS=48MiB
|
||||
PASS timeout: decoder killed and reaped; parent continues
|
||||
PASS: lab5 decoder resource limits
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:10:08Z
|
||||
|
||||
UTC: 2026-11-21T09:10:08Z
|
||||
Command: docker compose --profile tools run --rm -T --no-deps tester chmod +x scripts/restart.sh scripts/stop-one.sh
|
||||
Container messenger-lab6-tester-run-100a83886eb9 Creating
|
||||
Container messenger-lab6-tester-run-100a83886eb9 Created
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:10:09Z
|
||||
|
||||
UTC: 2026-11-21T09:10:09Z
|
||||
Command: docker compose --profile tools run --rm -T --no-deps tester python3 -u student-tests/lab5_recovery.py
|
||||
Container messenger-lab6-tester-run-6606f20c462f Creating
|
||||
Container messenger-lab6-tester-run-6606f20c462f Created
|
||||
PASS persistence: attachment=9d7de7ac-016a-49d7-a0f1-9b880cee3c83 same session/ID after API+broker restart; volumes preserved; queued content409; messages201; ready
|
||||
PASS crash_after_commit_before_publish: attachment=af40fc00-4da4-4fe6-80b3-27a8879d7661 hits=1 recovered_seconds=4.0; same original; three variants
|
||||
PASS crash_after_objects_before_commit: attachment=30ee3a05-086c-45fd-bbf1-78df42514413 hits=1 recovered_seconds=118.6; same original; three variants
|
||||
PASS crash_after_commit_before_ack: attachment=4aa3f19d-638f-492a-b17e-ccb103f1cae9 hits=1 recovered_seconds=0.4; same original; three variants
|
||||
PASS failed/replay: attachment=00b66546-aa21-4115-aca1-13fbc7f47f2c; permanent error fails once; failed registry; content/link409; audited replay preserves job_id; ready duplicate no-op
|
||||
PASS transient S3 outage: attachment=242a36c2-40f8-440f-899b-228164e5ddc4; attempts=5; durable retries, retry_exhausted, explicit replay after recovery
|
||||
PASS: lab5 grade4 recovery checks (all)
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:14:39Z
|
||||
|
||||
UTC: 2026-11-21T09:14:39Z
|
||||
Command: docker compose --profile tools ps -a -q
|
||||
81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85
|
||||
0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8
|
||||
d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d
|
||||
0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084
|
||||
b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026
|
||||
5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4
|
||||
278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300
|
||||
68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf
|
||||
93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c
|
||||
13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b
|
||||
cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb
|
||||
a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612
|
||||
00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007
|
||||
fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05
|
||||
837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053
|
||||
27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:14:40Z
|
||||
|
||||
UTC: 2026-11-21T09:14:40Z
|
||||
Command: docker inspect --format "{{.Name}} status={{.State.Status}} started={{.State.StartedAt}} finished={{.State.FinishedAt}}" 81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85 0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8 d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d 0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084 b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026 5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4 278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300 68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf 93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c 13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612 00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007 fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05 837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053 27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
|
||||
/messenger-lab6-api-1 status=running started=2026-11-21T09:10:22.045070464Z finished=2026-11-21T09:10:21.647750535Z
|
||||
/messenger-lab6-api-2 status=running started=2026-11-21T09:10:18.318585259Z finished=2026-11-21T09:10:17.940277751Z
|
||||
/messenger-lab6-broker-1 status=running started=2026-11-21T09:10:24.158928332Z finished=2026-11-21T09:10:23.907977242Z
|
||||
/messenger-lab6-db-1 status=running started=2026-11-21T08:59:12.528963564Z finished=2026-11-21T08:58:31.826616128Z
|
||||
/messenger-lab6-grafana-1 status=running started=2026-11-21T08:51:11.412805651Z finished=2026-11-21T08:51:10.520984964Z
|
||||
/messenger-lab6-lb-1 status=running started=2026-11-21T08:51:52.621809468Z finished=0001-01-01T00:00:00Z
|
||||
/messenger-lab6-loki-1 status=running started=2026-11-21T09:01:34.707920007Z finished=2026-11-21T09:00:28.644856083Z
|
||||
/messenger-lab6-migrate-1 status=exited started=2026-11-21T08:52:49.834748909Z finished=2026-11-21T08:52:50.79798573Z
|
||||
/messenger-lab6-otel-collector-1 status=running started=2026-11-21T08:49:03.407564143Z finished=0001-01-01T00:00:00Z
|
||||
/messenger-lab6-prometheus-1 status=running started=2026-11-21T08:51:11.311294483Z finished=2026-11-21T08:51:10.512998838Z
|
||||
/messenger-lab6-publisher-1 status=running started=2026-11-21T09:10:55.825987305Z finished=2026-11-21T09:10:55.547390235Z
|
||||
/messenger-lab6-s3-1 status=running started=2026-11-21T09:14:09.569875679Z finished=2026-11-21T09:13:44.880718518Z
|
||||
/messenger-lab6-s3-init-1 status=exited started=2026-11-21T08:53:15.109417522Z finished=2026-11-21T08:53:17.005092363Z
|
||||
/messenger-lab6-tempo-1 status=running started=2026-11-21T09:06:37.030642992Z finished=0001-01-01T00:00:00Z
|
||||
/messenger-lab6-valkey-1 status=running started=2026-11-21T08:45:28.390651046Z finished=2026-11-21T08:45:27.109421727Z
|
||||
/messenger-lab6-worker-1 status=running started=2026-11-21T09:13:45.19691868Z finished=2026-11-21T09:13:41.499885247Z
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:14:40Z
|
||||
|
||||
UTC: 2026-11-21T09:14:40Z
|
||||
Command: docker compose --profile tools run --rm -T --no-deps tester python3 -u tests/resilience.py --mode persistence --action-script scripts/restart.sh
|
||||
Container messenger-lab6-tester-run-e3bdaf9250f1 Creating
|
||||
Container messenger-lab6-tester-run-e3bdaf9250f1 Created
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:15:30Z
|
||||
|
||||
UTC: 2026-11-21T09:15:30Z
|
||||
Command: docker compose --profile tools ps -a -q
|
||||
81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85
|
||||
0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8
|
||||
d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d
|
||||
0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084
|
||||
b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026
|
||||
5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4
|
||||
278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300
|
||||
68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf
|
||||
93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c
|
||||
13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b
|
||||
cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb
|
||||
a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612
|
||||
00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007
|
||||
fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05
|
||||
837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053
|
||||
27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:15:31Z
|
||||
|
||||
UTC: 2026-11-21T09:15:31Z
|
||||
Command: docker inspect --format "{{.Name}} status={{.State.Status}} started={{.State.StartedAt}} finished={{.State.FinishedAt}}" 81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85 0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8 d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d 0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084 b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026 5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4 278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300 68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf 93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c 13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612 00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007 fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05 837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053 27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
|
||||
/messenger-lab6-api-1 status=running started=2026-11-21T09:15:18.227427331Z finished=2026-11-21T09:14:49.40226195Z
|
||||
/messenger-lab6-api-2 status=running started=2026-11-21T09:15:17.875504619Z finished=2026-11-21T09:14:49.181726236Z
|
||||
/messenger-lab6-broker-1 status=running started=2026-11-21T09:14:51.311241575Z finished=2026-11-21T09:14:49.824073591Z
|
||||
/messenger-lab6-db-1 status=running started=2026-11-21T09:14:50.494933266Z finished=2026-11-21T09:14:49.814681938Z
|
||||
/messenger-lab6-grafana-1 status=running started=2026-11-21T08:51:11.412805651Z finished=2026-11-21T08:51:10.520984964Z
|
||||
/messenger-lab6-lb-1 status=running started=2026-11-21T09:15:24.826206264Z finished=2026-11-21T09:14:45.274514498Z
|
||||
/messenger-lab6-loki-1 status=running started=2026-11-21T09:01:34.707920007Z finished=2026-11-21T09:00:28.644856083Z
|
||||
/messenger-lab6-migrate-1 status=exited started=2026-11-21T08:52:49.834748909Z finished=2026-11-21T08:52:50.79798573Z
|
||||
/messenger-lab6-otel-collector-1 status=running started=2026-11-21T08:49:03.407564143Z finished=0001-01-01T00:00:00Z
|
||||
/messenger-lab6-prometheus-1 status=running started=2026-11-21T08:51:11.311294483Z finished=2026-11-21T08:51:10.512998838Z
|
||||
/messenger-lab6-publisher-1 status=running started=2026-11-21T09:15:17.332690697Z finished=2026-11-21T09:14:45.830823803Z
|
||||
/messenger-lab6-s3-1 status=running started=2026-11-21T09:14:51.024302658Z finished=2026-11-21T09:14:49.817292594Z
|
||||
/messenger-lab6-s3-init-1 status=exited started=2026-11-21T08:53:15.109417522Z finished=2026-11-21T08:53:17.005092363Z
|
||||
/messenger-lab6-tempo-1 status=running started=2026-11-21T09:06:37.030642992Z finished=0001-01-01T00:00:00Z
|
||||
/messenger-lab6-valkey-1 status=running started=2026-11-21T09:14:50.777977548Z finished=2026-11-21T09:14:49.803932669Z
|
||||
/messenger-lab6-worker-1 status=running started=2026-11-21T09:15:17.592030927Z finished=2026-11-21T09:14:45.554953067Z
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:15:31Z
|
||||
|
||||
UTC: 2026-11-21T09:15:31Z
|
||||
Command: docker compose --profile tools ps -a -q
|
||||
81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85
|
||||
0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8
|
||||
d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d
|
||||
0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084
|
||||
b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026
|
||||
5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4
|
||||
278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300
|
||||
68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf
|
||||
93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c
|
||||
13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b
|
||||
cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb
|
||||
a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612
|
||||
00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007
|
||||
fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05
|
||||
837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053
|
||||
27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:15:31Z
|
||||
|
||||
UTC: 2026-11-21T09:15:31Z
|
||||
Command: docker inspect --format "{{.Name}} status={{.State.Status}} started={{.State.StartedAt}} finished={{.State.FinishedAt}}" 81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85 0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8 d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d 0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084 b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026 5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4 278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300 68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf 93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c 13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612 00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007 fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05 837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053 27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
|
||||
/messenger-lab6-api-1 status=running started=2026-11-21T09:15:18.227427331Z finished=2026-11-21T09:14:49.40226195Z
|
||||
/messenger-lab6-api-2 status=running started=2026-11-21T09:15:17.875504619Z finished=2026-11-21T09:14:49.181726236Z
|
||||
/messenger-lab6-broker-1 status=running started=2026-11-21T09:14:51.311241575Z finished=2026-11-21T09:14:49.824073591Z
|
||||
/messenger-lab6-db-1 status=running started=2026-11-21T09:14:50.494933266Z finished=2026-11-21T09:14:49.814681938Z
|
||||
/messenger-lab6-grafana-1 status=running started=2026-11-21T08:51:11.412805651Z finished=2026-11-21T08:51:10.520984964Z
|
||||
/messenger-lab6-lb-1 status=running started=2026-11-21T09:15:24.826206264Z finished=2026-11-21T09:14:45.274514498Z
|
||||
/messenger-lab6-loki-1 status=running started=2026-11-21T09:01:34.707920007Z finished=2026-11-21T09:00:28.644856083Z
|
||||
/messenger-lab6-migrate-1 status=exited started=2026-11-21T08:52:49.834748909Z finished=2026-11-21T08:52:50.79798573Z
|
||||
/messenger-lab6-otel-collector-1 status=running started=2026-11-21T08:49:03.407564143Z finished=0001-01-01T00:00:00Z
|
||||
/messenger-lab6-prometheus-1 status=running started=2026-11-21T08:51:11.311294483Z finished=2026-11-21T08:51:10.512998838Z
|
||||
/messenger-lab6-publisher-1 status=running started=2026-11-21T09:15:17.332690697Z finished=2026-11-21T09:14:45.830823803Z
|
||||
/messenger-lab6-s3-1 status=running started=2026-11-21T09:14:51.024302658Z finished=2026-11-21T09:14:49.817292594Z
|
||||
/messenger-lab6-s3-init-1 status=exited started=2026-11-21T08:53:15.109417522Z finished=2026-11-21T08:53:17.005092363Z
|
||||
/messenger-lab6-tempo-1 status=running started=2026-11-21T09:06:37.030642992Z finished=0001-01-01T00:00:00Z
|
||||
/messenger-lab6-valkey-1 status=running started=2026-11-21T09:14:50.777977548Z finished=2026-11-21T09:14:49.803932669Z
|
||||
/messenger-lab6-worker-1 status=running started=2026-11-21T09:15:17.592030927Z finished=2026-11-21T09:14:45.554953067Z
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:15:31Z
|
||||
|
||||
UTC: 2026-11-21T09:15:31Z
|
||||
Command: docker compose --profile tools run --rm -T --no-deps tester python3 -u tests/resilience.py --mode failover --action-script scripts/stop-one.sh
|
||||
Container messenger-lab6-tester-run-0af389d5dbc3 Creating
|
||||
Container messenger-lab6-tester-run-0af389d5dbc3 Created
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||
{"observed_instances_before": 2, "successful_writes": 127, "failed_attempts": 0, "first_recovery_seconds": 4.189}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:15:52Z
|
||||
|
||||
UTC: 2026-11-21T09:15:52Z
|
||||
Command: docker compose --profile tools ps -a -q
|
||||
81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85
|
||||
0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8
|
||||
d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d
|
||||
0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084
|
||||
b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026
|
||||
5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4
|
||||
278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300
|
||||
68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf
|
||||
93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c
|
||||
13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b
|
||||
cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb
|
||||
a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612
|
||||
00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007
|
||||
fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05
|
||||
837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053
|
||||
27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:15:53Z
|
||||
|
||||
UTC: 2026-11-21T09:15:53Z
|
||||
Command: docker inspect --format "{{.Name}} status={{.State.Status}} started={{.State.StartedAt}} finished={{.State.FinishedAt}}" 81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85 0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8 d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d 0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084 b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026 5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4 278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300 68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf 93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c 13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612 00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007 fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05 837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053 27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
|
||||
/messenger-lab6-api-1 status=exited started=2026-11-21T09:15:18.227427331Z finished=2026-11-21T09:15:40.679293151Z
|
||||
/messenger-lab6-api-2 status=running started=2026-11-21T09:15:17.875504619Z finished=2026-11-21T09:14:49.181726236Z
|
||||
/messenger-lab6-broker-1 status=running started=2026-11-21T09:14:51.311241575Z finished=2026-11-21T09:14:49.824073591Z
|
||||
/messenger-lab6-db-1 status=running started=2026-11-21T09:14:50.494933266Z finished=2026-11-21T09:14:49.814681938Z
|
||||
/messenger-lab6-grafana-1 status=running started=2026-11-21T08:51:11.412805651Z finished=2026-11-21T08:51:10.520984964Z
|
||||
/messenger-lab6-lb-1 status=running started=2026-11-21T09:15:24.826206264Z finished=2026-11-21T09:14:45.274514498Z
|
||||
/messenger-lab6-loki-1 status=running started=2026-11-21T09:01:34.707920007Z finished=2026-11-21T09:00:28.644856083Z
|
||||
/messenger-lab6-migrate-1 status=exited started=2026-11-21T08:52:49.834748909Z finished=2026-11-21T08:52:50.79798573Z
|
||||
/messenger-lab6-otel-collector-1 status=running started=2026-11-21T08:49:03.407564143Z finished=0001-01-01T00:00:00Z
|
||||
/messenger-lab6-prometheus-1 status=running started=2026-11-21T08:51:11.311294483Z finished=2026-11-21T08:51:10.512998838Z
|
||||
/messenger-lab6-publisher-1 status=running started=2026-11-21T09:15:17.332690697Z finished=2026-11-21T09:14:45.830823803Z
|
||||
/messenger-lab6-s3-1 status=running started=2026-11-21T09:14:51.024302658Z finished=2026-11-21T09:14:49.817292594Z
|
||||
/messenger-lab6-s3-init-1 status=exited started=2026-11-21T08:53:15.109417522Z finished=2026-11-21T08:53:17.005092363Z
|
||||
/messenger-lab6-tempo-1 status=running started=2026-11-21T09:06:37.030642992Z finished=0001-01-01T00:00:00Z
|
||||
/messenger-lab6-valkey-1 status=running started=2026-11-21T09:14:50.777977548Z finished=2026-11-21T09:14:49.803932669Z
|
||||
/messenger-lab6-worker-1 status=running started=2026-11-21T09:15:17.592030927Z finished=2026-11-21T09:14:45.554953067Z
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:15:53Z
|
||||
|
||||
UTC: 2026-11-21T09:15:53Z
|
||||
Command: docker compose --profile tools run --rm -T --no-deps tester sh scripts/start-api.sh
|
||||
Container messenger-lab6-tester-run-b9e9496603b9 Creating
|
||||
Container messenger-lab6-tester-run-b9e9496603b9 Created
|
||||
started api container 81ebb23d65ec
|
||||
running api replicas: 2
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:16:00Z
|
||||
|
||||
UTC: 2026-11-21T09:16:00Z
|
||||
Command: docker compose --profile tools ps -a -q
|
||||
81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85
|
||||
0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8
|
||||
d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d
|
||||
0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084
|
||||
b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026
|
||||
5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4
|
||||
278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300
|
||||
68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf
|
||||
93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c
|
||||
13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b
|
||||
cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb
|
||||
a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612
|
||||
00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007
|
||||
fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05
|
||||
837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053
|
||||
27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:16:01Z
|
||||
|
||||
UTC: 2026-11-21T09:16:01Z
|
||||
Command: docker inspect --format "{{.Name}} status={{.State.Status}} started={{.State.StartedAt}} finished={{.State.FinishedAt}}" 81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85 0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8 d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d 0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084 b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026 5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4 278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300 68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf 93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c 13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612 00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007 fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05 837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053 27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
|
||||
/messenger-lab6-api-1 status=running started=2026-11-21T09:15:54.07308443Z finished=2026-11-21T09:15:40.679293151Z
|
||||
/messenger-lab6-api-2 status=running started=2026-11-21T09:15:17.875504619Z finished=2026-11-21T09:14:49.181726236Z
|
||||
/messenger-lab6-broker-1 status=running started=2026-11-21T09:14:51.311241575Z finished=2026-11-21T09:14:49.824073591Z
|
||||
/messenger-lab6-db-1 status=running started=2026-11-21T09:14:50.494933266Z finished=2026-11-21T09:14:49.814681938Z
|
||||
/messenger-lab6-grafana-1 status=running started=2026-11-21T08:51:11.412805651Z finished=2026-11-21T08:51:10.520984964Z
|
||||
/messenger-lab6-lb-1 status=running started=2026-11-21T09:15:24.826206264Z finished=2026-11-21T09:14:45.274514498Z
|
||||
/messenger-lab6-loki-1 status=running started=2026-11-21T09:01:34.707920007Z finished=2026-11-21T09:00:28.644856083Z
|
||||
/messenger-lab6-migrate-1 status=exited started=2026-11-21T08:52:49.834748909Z finished=2026-11-21T08:52:50.79798573Z
|
||||
/messenger-lab6-otel-collector-1 status=running started=2026-11-21T08:49:03.407564143Z finished=0001-01-01T00:00:00Z
|
||||
/messenger-lab6-prometheus-1 status=running started=2026-11-21T08:51:11.311294483Z finished=2026-11-21T08:51:10.512998838Z
|
||||
/messenger-lab6-publisher-1 status=running started=2026-11-21T09:15:17.332690697Z finished=2026-11-21T09:14:45.830823803Z
|
||||
/messenger-lab6-s3-1 status=running started=2026-11-21T09:14:51.024302658Z finished=2026-11-21T09:14:49.817292594Z
|
||||
/messenger-lab6-s3-init-1 status=exited started=2026-11-21T08:53:15.109417522Z finished=2026-11-21T08:53:17.005092363Z
|
||||
/messenger-lab6-tempo-1 status=running started=2026-11-21T09:06:37.030642992Z finished=0001-01-01T00:00:00Z
|
||||
/messenger-lab6-valkey-1 status=running started=2026-11-21T09:14:50.777977548Z finished=2026-11-21T09:14:49.803932669Z
|
||||
/messenger-lab6-worker-1 status=running started=2026-11-21T09:15:17.592030927Z finished=2026-11-21T09:14:45.554953067Z
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:16:01Z
|
||||
|
||||
UTC: 2026-11-21T09:16:01Z
|
||||
Command: docker compose --profile tools ps -a
|
||||
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
|
||||
messenger-lab6-api-1 sha256:efa9adc78d01ce2a8b6b72223b3964036998ac290fd83f91c929d417c137e0a4 "python -m app.server" api 24 minutes ago Up 7 seconds (healthy) 8080/tcp
|
||||
messenger-lab6-api-2 sha256:efa9adc78d01ce2a8b6b72223b3964036998ac290fd83f91c929d417c137e0a4 "python -m app.server" api 24 minutes ago Up 43 seconds (healthy) 8080/tcp
|
||||
messenger-lab6-broker-1 rabbitmq:3.13.7-alpine "/bin/sh -ec 'export…" broker 40 minutes ago Up About a minute (healthy) 4369/tcp, 5671-5672/tcp, 15691-15692/tcp, 25672/tcp
|
||||
messenger-lab6-db-1 postgres:16.4-alpine "docker-entrypoint.s…" db 40 minutes ago Up About a minute (healthy) 5432/tcp
|
||||
messenger-lab6-grafana-1 grafana/grafana:11.2.2 "/run.sh" grafana 25 minutes ago Up 24 minutes 127.0.0.1:3000->3000/tcp
|
||||
messenger-lab6-lb-1 haproxy:3.0.11-alpine "docker-entrypoint.s…" lb 24 minutes ago Up 36 seconds (healthy) 127.0.0.1:8086->8080/tcp, 127.0.0.1:8406->8404/tcp
|
||||
messenger-lab6-loki-1 grafana/loki:3.2.1 "/usr/bin/loki -conf…" loki 25 minutes ago Up 14 minutes 127.0.0.1:3100->3100/tcp
|
||||
messenger-lab6-migrate-1 sha256:efa9adc78d01ce2a8b6b72223b3964036998ac290fd83f91c929d417c137e0a4 "python scripts/migr…" migrate 24 minutes ago Exited (0) 23 minutes ago
|
||||
messenger-lab6-otel-collector-1 otel/opentelemetry-collector-contrib:0.111.0 "/otelcol-contrib --…" otel-collector 27 minutes ago Up 26 minutes 4317/tcp, 55678-55679/tcp
|
||||
messenger-lab6-prometheus-1 prom/prometheus:v2.55.0 "/bin/prometheus --c…" prometheus 25 minutes ago Up 24 minutes 127.0.0.1:9090->9090/tcp
|
||||
messenger-lab6-publisher-1 sha256:efa9adc78d01ce2a8b6b72223b3964036998ac290fd83f91c929d417c137e0a4 "python -m app.jobs" publisher 24 minutes ago Up 44 seconds 8080/tcp
|
||||
messenger-lab6-s3-1 chrislusf/seaweedfs:3.97 "/bin/sh /opt/lab/st…" s3 40 minutes ago Up About a minute (healthy) 7333/tcp, 8080/tcp, 8333/tcp, 8888/tcp, 9333/tcp, 18080/tcp, 18888/tcp, 19333/tcp
|
||||
messenger-lab6-s3-init-1 sha256:efa9adc78d01ce2a8b6b72223b3964036998ac290fd83f91c929d417c137e0a4 "python scripts/s3_b…" s3-init 24 minutes ago Exited (0) 22 minutes ago
|
||||
messenger-lab6-tempo-1 grafana/tempo:2.6.1 "/tempo -config.file…" tempo 9 minutes ago Up 9 minutes 127.0.0.1:3200->3200/tcp
|
||||
messenger-lab6-valkey-1 valkey/valkey:8.0.1-alpine "docker-entrypoint.s…" valkey 40 minutes ago Up About a minute (healthy) 6379/tcp
|
||||
messenger-lab6-worker-1 sha256:efa9adc78d01ce2a8b6b72223b3964036998ac290fd83f91c929d417c137e0a4 "python -m app.worker" worker 24 minutes ago Up 44 seconds 8080/tcp
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:16:01Z
|
||||
|
||||
PASS: lab5 full checks. UTF-8 logs: C:\Users\masha\WebstormProjects\mtusi2026lab6\evidence
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:16:01Z
|
||||
@@ -0,0 +1,8 @@
|
||||
|
||||
UTC: 2026-11-21T09:10:05Z
|
||||
Command: docker compose --profile tools exec -T worker python -u - < student-tests/lab5_limits.py
|
||||
PASS memory: real 16MP image decoder stopped at RLIMIT_AS=48MiB
|
||||
PASS timeout: decoder killed and reaped; parent continues
|
||||
PASS: lab5 decoder resource limits
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:10:08Z
|
||||
@@ -0,0 +1,562 @@
|
||||
{
|
||||
"start": "2026-11-21T08:51:58Z",
|
||||
"end": "2026-11-21T08:52:21Z",
|
||||
"queries": {
|
||||
"rss_bytes": "messenger_process_memory_bytes",
|
||||
"cpu_cores": "rate(messenger_process_cpu_seconds_total[15s])"
|
||||
},
|
||||
"results": {
|
||||
"rss_bytes": {
|
||||
"status": "success",
|
||||
"data": {
|
||||
"resultType": "matrix",
|
||||
"result": [
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "messenger_process_memory_bytes",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "0d13c5814dab",
|
||||
"exported_job": "messenger-api",
|
||||
"instance": "otel-collector:8889",
|
||||
"job": "messenger",
|
||||
"service_instance_id": "0d13c5814dab",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
1795251118,
|
||||
"120827904"
|
||||
],
|
||||
[
|
||||
1795251123,
|
||||
"121270272"
|
||||
],
|
||||
[
|
||||
1795251128,
|
||||
"122515456"
|
||||
],
|
||||
[
|
||||
1795251133,
|
||||
"125259776"
|
||||
],
|
||||
[
|
||||
1795251138,
|
||||
"127827968"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "messenger_process_memory_bytes",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "0fa6e7066e3145a993d3264494361be3",
|
||||
"exported_job": "messenger-worker",
|
||||
"instance": "otel-collector:8889",
|
||||
"job": "messenger",
|
||||
"service_instance_id": "0fa6e7066e3145a993d3264494361be3",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "1.6.0"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
1795251118,
|
||||
"89513984"
|
||||
],
|
||||
[
|
||||
1795251123,
|
||||
"89513984"
|
||||
],
|
||||
[
|
||||
1795251128,
|
||||
"89513984"
|
||||
],
|
||||
[
|
||||
1795251133,
|
||||
"89513984"
|
||||
],
|
||||
[
|
||||
1795251138,
|
||||
"89513984"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "messenger_process_memory_bytes",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "551e5017a02c4426af0773950c83c10e",
|
||||
"exported_job": "messenger-worker",
|
||||
"instance": "otel-collector:8889",
|
||||
"job": "messenger",
|
||||
"service_instance_id": "551e5017a02c4426af0773950c83c10e",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "1.6.0"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
1795251118,
|
||||
"88707072"
|
||||
],
|
||||
[
|
||||
1795251123,
|
||||
"88707072"
|
||||
],
|
||||
[
|
||||
1795251128,
|
||||
"88707072"
|
||||
],
|
||||
[
|
||||
1795251133,
|
||||
"88707072"
|
||||
],
|
||||
[
|
||||
1795251138,
|
||||
"88707072"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "messenger_process_memory_bytes",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "6cacaff48df9",
|
||||
"exported_job": "messenger-api",
|
||||
"instance": "otel-collector:8889",
|
||||
"job": "messenger",
|
||||
"service_instance_id": "6cacaff48df9",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
1795251118,
|
||||
"122683392"
|
||||
],
|
||||
[
|
||||
1795251123,
|
||||
"122683392"
|
||||
],
|
||||
[
|
||||
1795251128,
|
||||
"122683392"
|
||||
],
|
||||
[
|
||||
1795251133,
|
||||
"122683392"
|
||||
],
|
||||
[
|
||||
1795251138,
|
||||
"122683392"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "messenger_process_memory_bytes",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "81ebb23d65ec",
|
||||
"exported_job": "messenger-api",
|
||||
"instance": "otel-collector:8889",
|
||||
"job": "messenger",
|
||||
"service_instance_id": "81ebb23d65ec",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
1795251118,
|
||||
"120791040"
|
||||
],
|
||||
[
|
||||
1795251123,
|
||||
"121085952"
|
||||
],
|
||||
[
|
||||
1795251128,
|
||||
"122593280"
|
||||
],
|
||||
[
|
||||
1795251133,
|
||||
"125325312"
|
||||
],
|
||||
[
|
||||
1795251138,
|
||||
"128040960"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "messenger_process_memory_bytes",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "a068b747445b44439e7e1b7bc72675c2",
|
||||
"exported_job": "messenger-publisher",
|
||||
"instance": "otel-collector:8889",
|
||||
"job": "messenger",
|
||||
"service_instance_id": "a068b747445b44439e7e1b7bc72675c2",
|
||||
"service_name": "messenger-publisher",
|
||||
"service_version": "1.6.0"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
1795251118,
|
||||
"88813568"
|
||||
],
|
||||
[
|
||||
1795251123,
|
||||
"88813568"
|
||||
],
|
||||
[
|
||||
1795251128,
|
||||
"88813568"
|
||||
],
|
||||
[
|
||||
1795251133,
|
||||
"88813568"
|
||||
],
|
||||
[
|
||||
1795251138,
|
||||
"88813568"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "messenger_process_memory_bytes",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "da2c8073a2144ecc9965ea4553926bf3",
|
||||
"exported_job": "messenger-publisher",
|
||||
"instance": "otel-collector:8889",
|
||||
"job": "messenger",
|
||||
"service_instance_id": "da2c8073a2144ecc9965ea4553926bf3",
|
||||
"service_name": "messenger-publisher",
|
||||
"service_version": "1.6.0"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
1795251118,
|
||||
"89305088"
|
||||
],
|
||||
[
|
||||
1795251123,
|
||||
"89305088"
|
||||
],
|
||||
[
|
||||
1795251128,
|
||||
"89305088"
|
||||
],
|
||||
[
|
||||
1795251133,
|
||||
"89305088"
|
||||
],
|
||||
[
|
||||
1795251138,
|
||||
"89305088"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "messenger_process_memory_bytes",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "e16c50966c56",
|
||||
"exported_job": "messenger-api",
|
||||
"instance": "otel-collector:8889",
|
||||
"job": "messenger",
|
||||
"service_instance_id": "e16c50966c56",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
1795251118,
|
||||
"123514880"
|
||||
],
|
||||
[
|
||||
1795251123,
|
||||
"123514880"
|
||||
],
|
||||
[
|
||||
1795251128,
|
||||
"123514880"
|
||||
],
|
||||
[
|
||||
1795251133,
|
||||
"123514880"
|
||||
],
|
||||
[
|
||||
1795251138,
|
||||
"123514880"
|
||||
]
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"cpu_cores": {
|
||||
"status": "success",
|
||||
"data": {
|
||||
"resultType": "matrix",
|
||||
"result": [
|
||||
{
|
||||
"metric": {
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "0d13c5814dab",
|
||||
"exported_job": "messenger-api",
|
||||
"instance": "otel-collector:8889",
|
||||
"job": "messenger",
|
||||
"service_instance_id": "0d13c5814dab",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
1795251123,
|
||||
"0.022415557507320008"
|
||||
],
|
||||
[
|
||||
1795251128,
|
||||
"0.17908626842684267"
|
||||
],
|
||||
[
|
||||
1795251133,
|
||||
"0.3572547659765976"
|
||||
],
|
||||
[
|
||||
1795251138,
|
||||
"0.3890083826"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "0fa6e7066e3145a993d3264494361be3",
|
||||
"exported_job": "messenger-worker",
|
||||
"instance": "otel-collector:8889",
|
||||
"job": "messenger",
|
||||
"service_instance_id": "0fa6e7066e3145a993d3264494361be3",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "1.6.0"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
1795251118,
|
||||
"0"
|
||||
],
|
||||
[
|
||||
1795251123,
|
||||
"0"
|
||||
],
|
||||
[
|
||||
1795251128,
|
||||
"0"
|
||||
],
|
||||
[
|
||||
1795251133,
|
||||
"0"
|
||||
],
|
||||
[
|
||||
1795251138,
|
||||
"0"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "551e5017a02c4426af0773950c83c10e",
|
||||
"exported_job": "messenger-worker",
|
||||
"instance": "otel-collector:8889",
|
||||
"job": "messenger",
|
||||
"service_instance_id": "551e5017a02c4426af0773950c83c10e",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "1.6.0"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
1795251123,
|
||||
"0.0005782936565199904"
|
||||
],
|
||||
[
|
||||
1795251128,
|
||||
"0.0007851155115511435"
|
||||
],
|
||||
[
|
||||
1795251133,
|
||||
"0.0005319559955995547"
|
||||
],
|
||||
[
|
||||
1795251138,
|
||||
"0.0004686850999999992"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "6cacaff48df9",
|
||||
"exported_job": "messenger-api",
|
||||
"instance": "otel-collector:8889",
|
||||
"job": "messenger",
|
||||
"service_instance_id": "6cacaff48df9",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
1795251118,
|
||||
"0"
|
||||
],
|
||||
[
|
||||
1795251123,
|
||||
"0"
|
||||
],
|
||||
[
|
||||
1795251128,
|
||||
"0"
|
||||
],
|
||||
[
|
||||
1795251133,
|
||||
"0"
|
||||
],
|
||||
[
|
||||
1795251138,
|
||||
"0"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "81ebb23d65ec",
|
||||
"exported_job": "messenger-api",
|
||||
"instance": "otel-collector:8889",
|
||||
"job": "messenger",
|
||||
"service_instance_id": "81ebb23d65ec",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
1795251123,
|
||||
"0.038017601540733366"
|
||||
],
|
||||
[
|
||||
1795251128,
|
||||
"0.17548991699169916"
|
||||
],
|
||||
[
|
||||
1795251133,
|
||||
"0.34144971107110705"
|
||||
],
|
||||
[
|
||||
1795251138,
|
||||
"0.3803775972000001"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "a068b747445b44439e7e1b7bc72675c2",
|
||||
"exported_job": "messenger-publisher",
|
||||
"instance": "otel-collector:8889",
|
||||
"job": "messenger",
|
||||
"service_instance_id": "a068b747445b44439e7e1b7bc72675c2",
|
||||
"service_name": "messenger-publisher",
|
||||
"service_version": "1.6.0"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
1795251118,
|
||||
"0"
|
||||
],
|
||||
[
|
||||
1795251123,
|
||||
"0"
|
||||
],
|
||||
[
|
||||
1795251128,
|
||||
"0"
|
||||
],
|
||||
[
|
||||
1795251133,
|
||||
"0"
|
||||
],
|
||||
[
|
||||
1795251138,
|
||||
"0"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "da2c8073a2144ecc9965ea4553926bf3",
|
||||
"exported_job": "messenger-publisher",
|
||||
"instance": "otel-collector:8889",
|
||||
"job": "messenger",
|
||||
"service_instance_id": "da2c8073a2144ecc9965ea4553926bf3",
|
||||
"service_name": "messenger-publisher",
|
||||
"service_version": "1.6.0"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
1795251123,
|
||||
"0.01116402495005333"
|
||||
],
|
||||
[
|
||||
1795251128,
|
||||
"0.020074461846184624"
|
||||
],
|
||||
[
|
||||
1795251133,
|
||||
"0.021264563956395647"
|
||||
],
|
||||
[
|
||||
1795251138,
|
||||
"0.021109743999999986"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "e16c50966c56",
|
||||
"exported_job": "messenger-api",
|
||||
"instance": "otel-collector:8889",
|
||||
"job": "messenger",
|
||||
"service_instance_id": "e16c50966c56",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
1795251118,
|
||||
"0"
|
||||
],
|
||||
[
|
||||
1795251123,
|
||||
"0"
|
||||
],
|
||||
[
|
||||
1795251128,
|
||||
"0"
|
||||
],
|
||||
[
|
||||
1795251133,
|
||||
"0"
|
||||
],
|
||||
[
|
||||
1795251138,
|
||||
"0"
|
||||
]
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"lab": 6,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 20.0,
|
||||
"elapsed_seconds": 20.27,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 1645,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 81.154,
|
||||
"successful_latency_ms": {
|
||||
"p50": 38.796,
|
||||
"p95": 114.354,
|
||||
"p99": 194.742
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
|
||||
UTC: 2026-11-21T08:51:58Z
|
||||
Command: C:\Users\masha\AppData\Local\Programs\Python\Python313\python.exe -X utf8 tests/load.py --base-url http://localhost:8086 --duration 20 --concurrency 4 --output evidence/load-summary.json
|
||||
{
|
||||
"lab": 6,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 20.0,
|
||||
"elapsed_seconds": 20.27,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 1645,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 81.154,
|
||||
"successful_latency_ms": {
|
||||
"p50": 38.796,
|
||||
"p95": 114.354,
|
||||
"p99": 194.742
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
|
||||
}
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T08:52:21Z
|
||||
@@ -0,0 +1,11 @@
|
||||
|
||||
UTC: 2026-11-21T09:16:46Z
|
||||
Command: docker compose --profile tools run --rm -T --no-deps tester make check test
|
||||
Container messenger-lab6-tester-run-f02cd878c6ba Creating
|
||||
Container messenger-lab6-tester-run-f02cd878c6ba Created
|
||||
python3 scripts/check_contract.py
|
||||
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||
python3 tests/smoke.py --base-url "http://lb:8080"
|
||||
PASS: lab6 public API scenario; infrastructure and grading evidence remain separate.
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:16:55Z
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,6 @@
|
||||
|
||||
UTC: 2026-11-21T09:16:58Z
|
||||
Command: C:\Users\masha\AppData\Local\Programs\Python\Python313\python.exe -X utf8 scripts/observe_lab6.py --baseline
|
||||
PASS: real API/worker logs, traces and metrics; evidence/observability-baseline.json
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:17:18Z
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,8 @@
|
||||
|
||||
UTC: 2026-11-21T08:58:20Z
|
||||
Command: C:\Users\masha\AppData\Local\Programs\Python\Python313\python.exe -X utf8 scripts/observe_lab6.py --incidents
|
||||
PASS: worker stopped
|
||||
PASS: PostgreSQL unavailable
|
||||
PASS: Tempo and Loki unavailable
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:02:41Z
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"scenario": "S3 unavailable during first attempt, then restored",
|
||||
"attachment_id": "8da6897d-7ad0-4a47-92a1-63e8820c6fb3",
|
||||
"request_id": "observe-927e3831c3d241ac80d3d43391ad1ff4",
|
||||
"api_trace_id": "94af84b9ec1663416ac05526995fbd91",
|
||||
"accepted_at": "2026-11-21T08:56:07.085875Z",
|
||||
"worker_trace_id": "94af84b9ec1663416ac05526995fbd91",
|
||||
"job_id": "c1561046-eb77-42b6-a915-e8f6e4c0a1bc",
|
||||
"worker_attempts": [
|
||||
"1",
|
||||
"2",
|
||||
"2",
|
||||
"3"
|
||||
],
|
||||
"same_trace": true,
|
||||
"correlated_at": "2026-11-21T08:58:20.029926Z",
|
||||
"content": {
|
||||
"attachment_id": "8da6897d-7ad0-4a47-92a1-63e8820c6fb3",
|
||||
"original_sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
|
||||
"original_bytes": 111817,
|
||||
"thumbnail_size": [
|
||||
256,
|
||||
160
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
|
||||
UTC: 2026-11-21T08:56:05Z
|
||||
Command: C:\Users\masha\AppData\Local\Programs\Python\Python313\python.exe -X utf8 scripts/observe_lab6.py --chain-with-retry
|
||||
PASS: propagated upload/publish/consume/S3/DB trace including retry
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T08:58:20Z
|
||||
@@ -0,0 +1,15 @@
|
||||
|
||||
UTC: 2026-11-21T09:16:55Z
|
||||
Command: docker compose exec -T api python - < student-tests/observability_unit.py
|
||||
test_http_route_template_and_request_trace_log_correlation_without_secrets (__main__.ObservabilityTests.test_http_route_template_and_request_trace_log_correlation_without_secrets) ... ok
|
||||
test_missing_or_invalid_context_starts_new_root_even_inside_span (__main__.ObservabilityTests.test_missing_or_invalid_context_starts_new_root_even_inside_span) ... ok
|
||||
test_unknown_route_and_exception_message_do_not_escape_into_telemetry (__main__.ObservabilityTests.test_unknown_route_and_exception_message_do_not_escape_into_telemetry) ... ok
|
||||
test_unsampled_trace_flags_and_tracestate_survive (__main__.ObservabilityTests.test_unsampled_trace_flags_and_tracestate_survive) ... ok
|
||||
test_w3c_parent_child_and_durable_retry_preserve_trace (__main__.ObservabilityTests.test_w3c_parent_child_and_durable_retry_preserve_trace) ... ok
|
||||
|
||||
----------------------------------------------------------------------
|
||||
Ran 5 tests in 0.694s
|
||||
|
||||
OK
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:16:58Z
|
||||
@@ -0,0 +1,9 @@
|
||||
|
||||
UTC: 2026-11-21T09:14:40Z
|
||||
Command: docker compose --profile tools run --rm -T --no-deps tester python3 -u tests/resilience.py --mode persistence --action-script scripts/restart.sh
|
||||
Container messenger-lab6-tester-run-e3bdaf9250f1 Creating
|
||||
Container messenger-lab6-tester-run-e3bdaf9250f1 Created
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:15:30Z
|
||||
@@ -0,0 +1,14 @@
|
||||
|
||||
UTC: 2026-11-21T09:10:09Z
|
||||
Command: docker compose --profile tools run --rm -T --no-deps tester python3 -u student-tests/lab5_recovery.py
|
||||
Container messenger-lab6-tester-run-6606f20c462f Creating
|
||||
Container messenger-lab6-tester-run-6606f20c462f Created
|
||||
PASS persistence: attachment=9d7de7ac-016a-49d7-a0f1-9b880cee3c83 same session/ID after API+broker restart; volumes preserved; queued content409; messages201; ready
|
||||
PASS crash_after_commit_before_publish: attachment=af40fc00-4da4-4fe6-80b3-27a8879d7661 hits=1 recovered_seconds=4.0; same original; three variants
|
||||
PASS crash_after_objects_before_commit: attachment=30ee3a05-086c-45fd-bbf1-78df42514413 hits=1 recovered_seconds=118.6; same original; three variants
|
||||
PASS crash_after_commit_before_ack: attachment=4aa3f19d-638f-492a-b17e-ccb103f1cae9 hits=1 recovered_seconds=0.4; same original; three variants
|
||||
PASS failed/replay: attachment=00b66546-aa21-4115-aca1-13fbc7f47f2c; permanent error fails once; failed registry; content/link409; audited replay preserves job_id; ready duplicate no-op
|
||||
PASS transient S3 outage: attachment=242a36c2-40f8-440f-899b-228164e5ddc4; attempts=5; durable retries, retry_exhausted, explicit replay after recovery
|
||||
PASS: lab5 grade4 recovery checks (all)
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:14:39Z
|
||||
@@ -0,0 +1,191 @@
|
||||
{
|
||||
"status": "success",
|
||||
"data": {
|
||||
"resultType": "streams",
|
||||
"result": [
|
||||
{
|
||||
"stream": {
|
||||
"code_filepath": "/app/app/telemetry.py",
|
||||
"code_function": "emit",
|
||||
"code_lineno": "180",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"duration_seconds": "0.06001993599988964",
|
||||
"event": "http.request.complete",
|
||||
"flags": "1",
|
||||
"method": "POST",
|
||||
"observed_timestamp": "1795251367084401557",
|
||||
"outcome": "success",
|
||||
"request_id": "observe-927e3831c3d241ac80d3d43391ad1ff4",
|
||||
"route": "/api/v1/chats/{chat_id}/attachments",
|
||||
"scope_name": "messenger.telemetry.events",
|
||||
"service_instance_id": "81ebb23d65ec",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "10167513c3917aba",
|
||||
"status_code": "202",
|
||||
"trace_id": "94af84b9ec1663416ac05526995fbd91"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1795251367084238848",
|
||||
"http.request.complete"
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"stats": {
|
||||
"summary": {
|
||||
"bytesProcessedPerSecond": 6425057,
|
||||
"linesProcessedPerSecond": 32614,
|
||||
"totalBytesProcessed": 86680,
|
||||
"totalLinesProcessed": 440,
|
||||
"execTime": 0.013491,
|
||||
"queueTime": 0.001181,
|
||||
"subqueries": 0,
|
||||
"totalEntriesReturned": 1,
|
||||
"splits": 0,
|
||||
"shards": 1,
|
||||
"totalPostFilterLines": 1,
|
||||
"totalStructuredMetadataBytesProcessed": 77440
|
||||
},
|
||||
"querier": {
|
||||
"store": {
|
||||
"totalChunksRef": 0,
|
||||
"totalChunksDownloaded": 0,
|
||||
"chunksDownloadTime": 0,
|
||||
"queryReferencedStructuredMetadata": false,
|
||||
"chunk": {
|
||||
"headChunkBytes": 0,
|
||||
"headChunkLines": 0,
|
||||
"decompressedBytes": 0,
|
||||
"decompressedLines": 0,
|
||||
"compressedBytes": 0,
|
||||
"totalDuplicates": 0,
|
||||
"postFilterLines": 0,
|
||||
"headChunkStructuredMetadataBytes": 0,
|
||||
"decompressedStructuredMetadataBytes": 0
|
||||
},
|
||||
"chunkRefsFetchTime": 0,
|
||||
"congestionControlLatency": 0,
|
||||
"pipelineWrapperFilteredLines": 0
|
||||
}
|
||||
},
|
||||
"ingester": {
|
||||
"totalReached": 1,
|
||||
"totalChunksMatched": 1,
|
||||
"totalBatches": 2,
|
||||
"totalLinesSent": 1,
|
||||
"store": {
|
||||
"totalChunksRef": 0,
|
||||
"totalChunksDownloaded": 0,
|
||||
"chunksDownloadTime": 0,
|
||||
"queryReferencedStructuredMetadata": true,
|
||||
"chunk": {
|
||||
"headChunkBytes": 86680,
|
||||
"headChunkLines": 440,
|
||||
"decompressedBytes": 0,
|
||||
"decompressedLines": 0,
|
||||
"compressedBytes": 0,
|
||||
"totalDuplicates": 0,
|
||||
"postFilterLines": 1,
|
||||
"headChunkStructuredMetadataBytes": 77440,
|
||||
"decompressedStructuredMetadataBytes": 0
|
||||
},
|
||||
"chunkRefsFetchTime": 190898,
|
||||
"congestionControlLatency": 0,
|
||||
"pipelineWrapperFilteredLines": 0
|
||||
}
|
||||
},
|
||||
"cache": {
|
||||
"chunk": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"index": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"result": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"statsResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"volumeResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"seriesResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"labelResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"instantMetricResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
}
|
||||
},
|
||||
"index": {
|
||||
"totalChunks": 0,
|
||||
"postFilterChunks": 0,
|
||||
"shardsDuration": 0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,318 @@
|
||||
{
|
||||
"status": "success",
|
||||
"data": {
|
||||
"resultType": "streams",
|
||||
"result": [
|
||||
{
|
||||
"stream": {
|
||||
"attachment_id": "8da6897d-7ad0-4a47-92a1-63e8820c6fb3",
|
||||
"attempt": "1",
|
||||
"code_filepath": "/app/app/telemetry.py",
|
||||
"code_function": "emit",
|
||||
"code_lineno": "180",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "warn",
|
||||
"duration_seconds": "1.1696733750000021",
|
||||
"event": "job.complete",
|
||||
"flags": "1",
|
||||
"job_id": "c1561046-eb77-42b6-a915-e8f6e4c0a1bc",
|
||||
"job_type": "attachment.process.v1",
|
||||
"observed_timestamp": "1795251373761214222",
|
||||
"outcome": "error",
|
||||
"scope_name": "messenger.telemetry.events",
|
||||
"service_instance_id": "fbfa8249e8c241d98e76ac1b870fab3e",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "13",
|
||||
"severity_text": "WARN",
|
||||
"span_id": "604b1094f4b8ea5a",
|
||||
"trace_id": "94af84b9ec1663416ac05526995fbd91"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1795251373761101312",
|
||||
"job.complete"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"attachment_id": "8da6897d-7ad0-4a47-92a1-63e8820c6fb3",
|
||||
"attempt": "2",
|
||||
"code_filepath": "/app/app/telemetry.py",
|
||||
"code_function": "emit",
|
||||
"code_lineno": "180",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"duration_seconds": "0.04622686900006556",
|
||||
"event": "job.complete",
|
||||
"flags": "1",
|
||||
"job_id": "c1561046-eb77-42b6-a915-e8f6e4c0a1bc",
|
||||
"job_type": "attachment.process.v1",
|
||||
"observed_timestamp": "1795251405312162771",
|
||||
"outcome": "duplicate",
|
||||
"scope_name": "messenger.telemetry.events",
|
||||
"service_instance_id": "860a8bda1f144a65878ee73824659b78",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "ba1e0fd901467175",
|
||||
"trace_id": "94af84b9ec1663416ac05526995fbd91"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1795251405311928576",
|
||||
"job.complete"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"attachment_id": "8da6897d-7ad0-4a47-92a1-63e8820c6fb3",
|
||||
"attempt": "2",
|
||||
"code_filepath": "/app/app/telemetry.py",
|
||||
"code_function": "emit",
|
||||
"code_lineno": "180",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"event": "job.retry",
|
||||
"flags": "1",
|
||||
"job_id": "c1561046-eb77-42b6-a915-e8f6e4c0a1bc",
|
||||
"observed_timestamp": "1795251373753026344",
|
||||
"outcome": "retry",
|
||||
"queue": "attachments",
|
||||
"scope_name": "messenger.telemetry.events",
|
||||
"service_instance_id": "fbfa8249e8c241d98e76ac1b870fab3e",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "604b1094f4b8ea5a",
|
||||
"trace_id": "94af84b9ec1663416ac05526995fbd91"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1795251373752916480",
|
||||
"job.retry"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"attachment_id": "8da6897d-7ad0-4a47-92a1-63e8820c6fb3",
|
||||
"attempt": "2",
|
||||
"code_filepath": "/app/app/telemetry.py",
|
||||
"code_function": "emit",
|
||||
"code_lineno": "180",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "warn",
|
||||
"duration_seconds": "1.4581017409998367",
|
||||
"event": "job.complete",
|
||||
"flags": "1",
|
||||
"job_id": "c1561046-eb77-42b6-a915-e8f6e4c0a1bc",
|
||||
"job_type": "attachment.process.v1",
|
||||
"observed_timestamp": "1795251375868803562",
|
||||
"outcome": "error",
|
||||
"scope_name": "messenger.telemetry.events",
|
||||
"service_instance_id": "fbfa8249e8c241d98e76ac1b870fab3e",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "13",
|
||||
"severity_text": "WARN",
|
||||
"span_id": "83adf2393f95e484",
|
||||
"trace_id": "94af84b9ec1663416ac05526995fbd91"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1795251375868673792",
|
||||
"job.complete"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"attachment_id": "8da6897d-7ad0-4a47-92a1-63e8820c6fb3",
|
||||
"attempt": "3",
|
||||
"code_filepath": "/app/app/telemetry.py",
|
||||
"code_function": "emit",
|
||||
"code_lineno": "180",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"duration_seconds": "0.36952172699989205",
|
||||
"event": "job.complete",
|
||||
"flags": "1",
|
||||
"job_id": "c1561046-eb77-42b6-a915-e8f6e4c0a1bc",
|
||||
"job_type": "attachment.process.v1",
|
||||
"observed_timestamp": "1795251497430982971",
|
||||
"outcome": "success",
|
||||
"scope_name": "messenger.telemetry.events",
|
||||
"service_instance_id": "860a8bda1f144a65878ee73824659b78",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "ba826640e32b3505",
|
||||
"trace_id": "94af84b9ec1663416ac05526995fbd91"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1795251497430907392",
|
||||
"job.complete"
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"stats": {
|
||||
"summary": {
|
||||
"bytesProcessedPerSecond": 142743,
|
||||
"linesProcessedPerSecond": 766,
|
||||
"totalBytesProcessed": 1117,
|
||||
"totalLinesProcessed": 6,
|
||||
"execTime": 0.007825,
|
||||
"queueTime": 0.000908,
|
||||
"subqueries": 0,
|
||||
"totalEntriesReturned": 5,
|
||||
"splits": 0,
|
||||
"shards": 1,
|
||||
"totalPostFilterLines": 5,
|
||||
"totalStructuredMetadataBytesProcessed": 1048
|
||||
},
|
||||
"querier": {
|
||||
"store": {
|
||||
"totalChunksRef": 0,
|
||||
"totalChunksDownloaded": 0,
|
||||
"chunksDownloadTime": 0,
|
||||
"queryReferencedStructuredMetadata": false,
|
||||
"chunk": {
|
||||
"headChunkBytes": 0,
|
||||
"headChunkLines": 0,
|
||||
"decompressedBytes": 0,
|
||||
"decompressedLines": 0,
|
||||
"compressedBytes": 0,
|
||||
"totalDuplicates": 0,
|
||||
"postFilterLines": 0,
|
||||
"headChunkStructuredMetadataBytes": 0,
|
||||
"decompressedStructuredMetadataBytes": 0
|
||||
},
|
||||
"chunkRefsFetchTime": 0,
|
||||
"congestionControlLatency": 0,
|
||||
"pipelineWrapperFilteredLines": 0
|
||||
}
|
||||
},
|
||||
"ingester": {
|
||||
"totalReached": 1,
|
||||
"totalChunksMatched": 1,
|
||||
"totalBatches": 2,
|
||||
"totalLinesSent": 5,
|
||||
"store": {
|
||||
"totalChunksRef": 0,
|
||||
"totalChunksDownloaded": 0,
|
||||
"chunksDownloadTime": 0,
|
||||
"queryReferencedStructuredMetadata": true,
|
||||
"chunk": {
|
||||
"headChunkBytes": 1117,
|
||||
"headChunkLines": 6,
|
||||
"decompressedBytes": 0,
|
||||
"decompressedLines": 0,
|
||||
"compressedBytes": 0,
|
||||
"totalDuplicates": 0,
|
||||
"postFilterLines": 5,
|
||||
"headChunkStructuredMetadataBytes": 1048,
|
||||
"decompressedStructuredMetadataBytes": 0
|
||||
},
|
||||
"chunkRefsFetchTime": 103240,
|
||||
"congestionControlLatency": 0,
|
||||
"pipelineWrapperFilteredLines": 0
|
||||
}
|
||||
},
|
||||
"cache": {
|
||||
"chunk": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"index": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"result": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"statsResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"volumeResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"seriesResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"labelResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"instantMetricResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
}
|
||||
},
|
||||
"index": {
|
||||
"totalChunks": 0,
|
||||
"postFilterChunks": 0,
|
||||
"shardsDuration": 0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,73 @@
|
||||
{
|
||||
"captured_at": "2026-11-21T09:17:48.990874Z",
|
||||
"host": "Windows, Intel Core Ultra 5 125U",
|
||||
"docker": {
|
||||
"ServerVersion": "29.3.0",
|
||||
"NCPU": 14,
|
||||
"MemTotal": 7965700096,
|
||||
"OperatingSystem": "Docker Desktop"
|
||||
},
|
||||
"revision_before_final_commit": "d0a062e2d2fbeda54c2a24f518eb3687650f7c9e",
|
||||
"working_tree_grade4": true,
|
||||
"api_url": "http://localhost:8086",
|
||||
"code_sha256": {
|
||||
".env.example": "f4d81ed6a4559e9f26c0c5a3d8accc495adb330c107c4f47cc34d47321a81dfe",
|
||||
".github/workflows/application.yml": "b486fe8a8299d363cf4e403dc2987de70c35a3ac6cc7e39f7656b6b77e6763ea",
|
||||
".github/workflows/contracts.yml": "a45d7932ef14130aff8020bce79f043cccaf9692d697088a754d309a03f7d3e8",
|
||||
"app/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
|
||||
"app/advanced.py": "27762f863185cb6b51a64ae7698e60350b5aba0f924992e7eecaf58511d6e5cc",
|
||||
"app/attachments.py": "fae87bae0835fe6cafe8e85b685d1a1a0039d67516d52a633bac66e26c7fb8c1",
|
||||
"app/cache.py": "c7d5a210c4b85624ec089c2626cb69a84154790d325a6308cc8b0f578db3178b",
|
||||
"app/config.py": "0ee78d89095878150de4c195e28ffc17c44ab40f5e6b40fc6cb9ef23672d10aa",
|
||||
"app/db.py": "28eade9a5dda08887f9efeb786e52040832dd28975619a0c4b23988be2ba1ca3",
|
||||
"app/deps.py": "40b9353387289b7b3ff964a216446f13bbde66ee5a26bf5c847e39fe1948b062",
|
||||
"app/errors.py": "b06a0001aa2f1410cdcd4c385719ad3b90a0dacae62784649129c1e1c6dd933c",
|
||||
"app/image_worker.py": "66821f54709707fb8ad7d336eeceaa97cb69d6e6b1ae369147b59241a1e099f5",
|
||||
"app/images.py": "9e0cd83ef5d673356bde672d77ec106f8f3bb86812aaf46fbe3c932db2f7df54",
|
||||
"app/jobs.py": "b1b6dd46529042b72a088a61a3e153f6c8861d41550dbe713056be301dc5c3dc",
|
||||
"app/lifecycle.py": "097fcd74d178654fa91b3704d45c5215cfdec2f0ab3bb21b432e028160c9aa5e",
|
||||
"app/main.py": "fabcf21af59875defa0c130b03bf14df8db3ac9352df654dc3da4fcf38b84f50",
|
||||
"app/models.py": "69eb6bd83683c7ab1329e3d0d331577ce40612c156dc17c8ae1bdd231a921ecc",
|
||||
"app/operator.py": "d9046d7c757bc9c40721f11569f4ca1a7c9e52586ad05a04f0b6179b31b551b6",
|
||||
"app/pipeline.py": "3bcdd1484d4633c901a7bcfcd9f800db9f5e53db6d1c610d82d556d1337fbd4c",
|
||||
"app/repositories.py": "86b2774b89e2f7b875283cd684c7b346eeba1da2dd76d836661f6b45812ebcd8",
|
||||
"app/security.py": "b5e0fa73943a046501d1015d9b0288272b93e6a8458a47651c1bd9fa3b03a680",
|
||||
"app/server.py": "092af34b790f3103eb715f956c686655e18309cf6bab7073a015222f709d61b4",
|
||||
"app/services.py": "128a6668ad74697c05612fd9ad13107d14ba0ed0c9074338fb2d14437b54c785",
|
||||
"app/storage.py": "8573d43b3993c9f1e0e5ecbf82a9f65f58041e6bd50d4a315875d24b67042bd1",
|
||||
"app/telemetry.py": "232fde35c8445e3a80a067f41a1ac264fd41c662b3ae1ebbc3c7e661200fe9bc",
|
||||
"app/trace_context.py": "4ad1e98c346c6542df0bba808f0449922127ced410289b05ca2a3d94354b9740",
|
||||
"app/worker.py": "8eb05a2c37325a6d1da77102a683a19cc25fc7cb8adeeaa2b1c525a4029c7bbc",
|
||||
"compose.yaml": "a29a6af5228d9c2a1a529818071787690d3108d82cb83d7edc668ea0656fd969",
|
||||
"Dockerfile": "a0029707f843a15501a7d05142c06662b2705e16eb0292954148674cfe739b87",
|
||||
"infra/grafana/dashboards/messenger.json": "826c301adef62987d040a4f0d764d7f3dc89275de0c488979a03737ad73dbce9",
|
||||
"infra/grafana/provisioning/dashboards/dashboards.yaml": "e764b58e795886455758da0c93b1199e75d220bdf9e0c087f859a4ce67f744ef",
|
||||
"infra/grafana/provisioning/datasources/datasources.yaml": "8234d0c4b273127f3f5aad084f942dcc4dc498640f796bcc34feaae3a1eae1cb",
|
||||
"infra/loki/loki.yaml": "4614b904e6297165bebdafb4004e2c4ba93ea6bb278b4f53f196566621740dbd",
|
||||
"infra/otel/collector.yaml": "52fd1266cb78fd1e73bdc2a36057ec913abba6f50e7ce93bb3a31db69e25b8ba",
|
||||
"infra/prometheus/alerts.yaml": "252499c676f9fa9564c6466e7f268a9975c9044df0bdd82de78565f90548bbbe",
|
||||
"infra/prometheus/prometheus.yaml": "362d7f375ef6bb85fbd2e2e1e9c09c1a8507e94a3b115e13b0869ffcbc777d47",
|
||||
"infra/tempo/tempo.yaml": "a87b8a536ee4a42d8f83716c753b0380498ea17893082c1b7d1d21aafc21fc51",
|
||||
"migrations/001_initial_schema.sql": "e5c956278405d4f65be50b2e0b92c37f0870589e9c0d972c70d6733f1ae439e2",
|
||||
"migrations/002_indexes.sql": "6b0a375c09505f5220eb3add3ff3e4115b8426bbe1e1d97556effb8f64e4c7ff",
|
||||
"migrations/003_idempotency_keys.sql": "9554fd815d4b0e3085d2fe725c0e0e8b2a36e212fe3a486345e28afaf38e8b69",
|
||||
"migrations/004_attachments.sql": "2f421cbfb0cbf50d80d4bb8812378cdbc2619a5b8fa7aa5116238e287a8c6c6b",
|
||||
"migrations/005_async_pipeline.sql": "ce928ea8304c95a8bf2ad0ca92e6ffce81070362020de1a3373e381fd3dab247",
|
||||
"migrations/006_trace_context.sql": "a452982ba83fb0c270c54a3796d5c257bd86f74bdd8627474dcb3b022e33ad33",
|
||||
"requirements.txt": "4abdbc52e41c0ac0cfe5d91718558e5e9ee3951e27efe99d8c31a83c736f2924",
|
||||
"scripts/check_contract.py": "7c4402c2a6ab80d3b4676e1866d6ecc87b6daea14c080bcb9be9a7e8b959d671",
|
||||
"scripts/migrate.py": "9f095a2181ed2a84584b09fe8cdfbf896c5c26306c7c5c5c262b4bbbb3f28d88",
|
||||
"scripts/observe_lab6.py": "bbfa84f0c360c538db68396afd97d228e272bd9ca68456c72a7a89eb77b91025",
|
||||
"scripts/s3_bootstrap.py": "f7bddb150df0e94e97a21efa2cad22c426e00fc935e710c16e11070f15344ef2",
|
||||
"scripts/test_lab5.py": "957546f3939f1d120d119d61a6df672034c0d234a32d014c413152490bec59ef",
|
||||
"scripts/test_lab6.py": "fd2ee01770b601c3fb0c89ea8e79c50d18a859e20a06c19f873ae38b1b6d06d0",
|
||||
"scripts/validate_openapi.py": "4e981acbfc10dac58817bf922d5896e996798551fedc1b2385130e1cddb13d54",
|
||||
"student-tests/image_validation_unit.py": "531a2de31f90c92e094f7e5ef8bea663f2d70ecee41464a461e0e54d9800f795",
|
||||
"student-tests/lab5_helpers.py": "68a8a0cc9fc644d883342cb82642b2a679a32ed2db252f2e5a5b845d9ad2cec1",
|
||||
"student-tests/lab5_images.py": "048cc6dbb674869608c13ae290f20d57a8356da79a510f4517242bf57f5a505d",
|
||||
"student-tests/lab5_limits.py": "c2095797ce002e78a68e2d00afb2b10aa46a85787e8f1a43e6fb2b5a70045336",
|
||||
"student-tests/lab5_recovery.py": "8d7aa7de8dc09e18f206e32cac3001f4a916e9448e3887d0de7154aa832553d9",
|
||||
"student-tests/observability_unit.py": "9759b4ca7e312d656a2a0fb6f886a6dcb19ca50983a3f1733371142b24579d09"
|
||||
},
|
||||
"notes": "Collected before third commit; source hashes normalize CRLF to LF. Synthetic data only."
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
|
||||
UTC: 2026-11-21T09:09:40Z
|
||||
Command: docker compose --profile tools ps --status running --services
|
||||
api
|
||||
broker
|
||||
db
|
||||
grafana
|
||||
lb
|
||||
loki
|
||||
otel-collector
|
||||
prometheus
|
||||
publisher
|
||||
s3
|
||||
tempo
|
||||
valkey
|
||||
worker
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:09:40Z
|
||||
|
||||
UTC: 2026-11-21T09:09:40Z
|
||||
Command: docker compose --profile tools build tester
|
||||
Image messenger-lab6-tester Building
|
||||
#1 [internal] load local bake definitions
|
||||
#1 reading from stdin 587B 0.0s done
|
||||
#1 DONE 0.0s
|
||||
|
||||
#2 [internal] load build definition from Dockerfile.tester
|
||||
#2 transferring dockerfile: 405B done
|
||||
#2 DONE 0.0s
|
||||
|
||||
#3 [internal] load metadata for docker.io/library/python:3.12.11-alpine3.22
|
||||
#3 DONE 0.2s
|
||||
|
||||
#4 [internal] load metadata for docker.io/library/docker:28.5.1-cli
|
||||
#4 DONE 0.2s
|
||||
|
||||
#5 [internal] load .dockerignore
|
||||
#5 transferring context: 286B done
|
||||
#5 DONE 0.0s
|
||||
|
||||
#6 [stage-1 1/5] FROM docker.io/library/python:3.12.11-alpine3.22@sha256:efcdfa6a6b2fd2afb9c7dfa9a5b288a6f68338b5cfdebe6b637d986067d85757
|
||||
#6 resolve docker.io/library/python:3.12.11-alpine3.22@sha256:efcdfa6a6b2fd2afb9c7dfa9a5b288a6f68338b5cfdebe6b637d986067d85757 0.0s done
|
||||
#6 DONE 0.1s
|
||||
|
||||
#7 [docker-cli 1/1] FROM docker.io/library/docker:28.5.1-cli@sha256:9190b0613792e658a7783cf14b2d5ace5941bb68ede7276922ea36ee457d76ad
|
||||
#7 resolve docker.io/library/docker:28.5.1-cli@sha256:9190b0613792e658a7783cf14b2d5ace5941bb68ede7276922ea36ee457d76ad 0.0s done
|
||||
#7 DONE 0.1s
|
||||
|
||||
#8 [stage-1 2/5] RUN apk add --no-cache make
|
||||
#8 CACHED
|
||||
|
||||
#9 [stage-1 3/5] RUN pip install --no-cache-dir Pillow==11.0.0
|
||||
#9 CACHED
|
||||
|
||||
#10 [stage-1 4/5] COPY --from=docker-cli /usr/local/bin/docker /usr/local/bin/docker
|
||||
#10 CACHED
|
||||
|
||||
#11 [stage-1 5/5] WORKDIR /work
|
||||
#11 CACHED
|
||||
|
||||
#12 exporting to image
|
||||
#12 exporting layers done
|
||||
#12 exporting manifest sha256:1c93bba11e9ded8b819bbcadafd9217f8b6b4fc5301deeca1e720c49a45f1b2e done
|
||||
#12 exporting config sha256:9ad2daad56b9cfccc6c03e7be0468faf51994e1604e86b502077a03f28844325 done
|
||||
#12 exporting attestation manifest sha256:96fa2816d4a538ed8e6e3ea5c702033990470ee27e842b91c5f2979f0304e95a 0.1s done
|
||||
#12 exporting manifest list sha256:3e586fe0bf4aeb3814ad182a780c6173a9e252da1c33e7e61f20187228c8b23f 0.0s done
|
||||
#12 naming to docker.io/library/messenger-lab6-tester:latest
|
||||
#12 naming to docker.io/library/messenger-lab6-tester:latest done
|
||||
#12 unpacking to docker.io/library/messenger-lab6-tester:latest 0.0s done
|
||||
#12 DONE 0.2s
|
||||
|
||||
#13 resolving provenance for metadata file
|
||||
#13 DONE 0.0s
|
||||
Image messenger-lab6-tester Built
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:09:42Z
|
||||
|
||||
UTC: 2026-11-21T09:10:08Z
|
||||
Command: docker compose --profile tools run --rm -T --no-deps tester chmod +x scripts/restart.sh scripts/stop-one.sh
|
||||
Container messenger-lab6-tester-run-100a83886eb9 Creating
|
||||
Container messenger-lab6-tester-run-100a83886eb9 Created
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:10:09Z
|
||||
@@ -0,0 +1,8 @@
|
||||
|
||||
UTC: 2026-11-21T09:09:46Z
|
||||
Command: docker compose --profile tools run --rm -T --no-deps tester python3 -u tests/smoke.py
|
||||
Container messenger-lab6-tester-run-42a477b98467 Creating
|
||||
Container messenger-lab6-tester-run-42a477b98467 Created
|
||||
PASS: lab6 public API scenario; infrastructure and grading evidence remain separate.
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:09:53Z
|
||||
@@ -0,0 +1,17 @@
|
||||
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
|
||||
messenger-lab6-api-1 messenger-lab6-api:dev "python -m app.server" api About a minute ago Up About a minute (healthy) 8080/tcp
|
||||
messenger-lab6-api-2 messenger-lab6-api:dev "python -m app.server" api About a minute ago Up About a minute (healthy) 8080/tcp
|
||||
messenger-lab6-broker-1 rabbitmq:3.13.7-alpine "/bin/sh -ec 'export…" broker 42 minutes ago Up 2 minutes (healthy) 4369/tcp, 5671-5672/tcp, 15691-15692/tcp, 25672/tcp
|
||||
messenger-lab6-db-1 postgres:16.4-alpine "docker-entrypoint.s…" db 42 minutes ago Up 2 minutes (healthy) 5432/tcp
|
||||
messenger-lab6-grafana-1 grafana/grafana:11.2.2 "/run.sh" grafana 27 minutes ago Up 26 minutes 127.0.0.1:3000->3000/tcp
|
||||
messenger-lab6-lb-1 haproxy:3.0.11-alpine "docker-entrypoint.s…" lb 26 minutes ago Up 2 minutes (healthy) 127.0.0.1:8086->8080/tcp, 127.0.0.1:8406->8404/tcp
|
||||
messenger-lab6-loki-1 grafana/loki:3.2.1 "/usr/bin/loki -conf…" loki 27 minutes ago Up 16 minutes 127.0.0.1:3100->3100/tcp
|
||||
messenger-lab6-migrate-1 messenger-lab6-api:dev "python scripts/migr…" migrate About a minute ago Exited (0) About a minute ago
|
||||
messenger-lab6-otel-collector-1 otel/opentelemetry-collector-contrib:0.111.0 "/otelcol-contrib --…" otel-collector 29 minutes ago Up 28 minutes 4317/tcp, 55678-55679/tcp
|
||||
messenger-lab6-prometheus-1 prom/prometheus:v2.55.0 "/bin/prometheus --c…" prometheus 27 minutes ago Up 26 minutes 127.0.0.1:9090->9090/tcp
|
||||
messenger-lab6-publisher-1 messenger-lab6-api:dev "python -m app.jobs" publisher About a minute ago Up About a minute 8080/tcp
|
||||
messenger-lab6-s3-1 chrislusf/seaweedfs:3.97 "/bin/sh /opt/lab/st…" s3 42 minutes ago Up 2 minutes (healthy) 7333/tcp, 8080/tcp, 8333/tcp, 8888/tcp, 9333/tcp, 18080/tcp, 18888/tcp, 19333/tcp
|
||||
messenger-lab6-s3-init-1 messenger-lab6-api:dev "python scripts/s3_b…" s3-init About a minute ago Exited (0) About a minute ago
|
||||
messenger-lab6-tempo-1 grafana/tempo:2.6.1 "/tempo -config.file…" tempo About a minute ago Up About a minute 127.0.0.1:3200->3200/tcp
|
||||
messenger-lab6-valkey-1 valkey/valkey:8.0.1-alpine "docker-entrypoint.s…" valkey 42 minutes ago Up 2 minutes (healthy) 6379/tcp
|
||||
messenger-lab6-worker-1 messenger-lab6-api:dev "python -m app.worker" worker About a minute ago Up About a minute 8080/tcp
|
||||
@@ -0,0 +1,44 @@
|
||||
|
||||
UTC: 2026-11-21T09:09:40Z
|
||||
Command: docker compose --profile tools ps -a
|
||||
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
|
||||
messenger-lab6-api-1 messenger-lab6-api:dev "python -m app.server" api 18 minutes ago Up 17 minutes (healthy) 8080/tcp
|
||||
messenger-lab6-api-2 messenger-lab6-api:dev "python -m app.server" api 18 minutes ago Up 17 minutes (healthy) 8080/tcp
|
||||
messenger-lab6-broker-1 rabbitmq:3.13.7-alpine "/bin/sh -ec 'export…" broker 34 minutes ago Up 24 minutes (healthy) 4369/tcp, 5671-5672/tcp, 15691-15692/tcp, 25672/tcp
|
||||
messenger-lab6-db-1 postgres:16.4-alpine "docker-entrypoint.s…" db 34 minutes ago Up 10 minutes (healthy) 5432/tcp
|
||||
messenger-lab6-grafana-1 grafana/grafana:11.2.2 "/run.sh" grafana 19 minutes ago Up 18 minutes 127.0.0.1:3000->3000/tcp
|
||||
messenger-lab6-lb-1 haproxy:3.0.11-alpine "docker-entrypoint.s…" lb 17 minutes ago Up 17 minutes (healthy) 127.0.0.1:8086->8080/tcp, 127.0.0.1:8406->8404/tcp
|
||||
messenger-lab6-loki-1 grafana/loki:3.2.1 "/usr/bin/loki -conf…" loki 19 minutes ago Up 8 minutes 127.0.0.1:3100->3100/tcp
|
||||
messenger-lab6-migrate-1 messenger-lab6-api:dev "python scripts/migr…" migrate 18 minutes ago Exited (0) 16 minutes ago
|
||||
messenger-lab6-otel-collector-1 otel/opentelemetry-collector-contrib:0.111.0 "/otelcol-contrib --…" otel-collector 20 minutes ago Up 20 minutes 4317/tcp, 55678-55679/tcp
|
||||
messenger-lab6-prometheus-1 prom/prometheus:v2.55.0 "/bin/prometheus --c…" prometheus 19 minutes ago Up 18 minutes 127.0.0.1:9090->9090/tcp
|
||||
messenger-lab6-publisher-1 messenger-lab6-api:dev "python -m app.jobs" publisher 18 minutes ago Up 17 minutes 8080/tcp
|
||||
messenger-lab6-s3-1 chrislusf/seaweedfs:3.97 "/bin/sh /opt/lab/st…" s3 34 minutes ago Up 13 minutes (healthy) 7333/tcp, 8080/tcp, 8333/tcp, 8888/tcp, 9333/tcp, 18080/tcp, 18888/tcp, 19333/tcp
|
||||
messenger-lab6-s3-init-1 messenger-lab6-api:dev "python scripts/s3_b…" s3-init 18 minutes ago Exited (0) 16 minutes ago
|
||||
messenger-lab6-tempo-1 grafana/tempo:2.6.1 "/tempo -config.file…" tempo 3 minutes ago Up 3 minutes 127.0.0.1:3200->3200/tcp
|
||||
messenger-lab6-valkey-1 valkey/valkey:8.0.1-alpine "docker-entrypoint.s…" valkey 34 minutes ago Up 24 minutes (healthy) 6379/tcp
|
||||
messenger-lab6-worker-1 messenger-lab6-api:dev "python -m app.worker" worker 18 minutes ago Up 5 minutes 8080/tcp
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:09:40Z
|
||||
|
||||
UTC: 2026-11-21T09:16:01Z
|
||||
Command: docker compose --profile tools ps -a
|
||||
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
|
||||
messenger-lab6-api-1 sha256:efa9adc78d01ce2a8b6b72223b3964036998ac290fd83f91c929d417c137e0a4 "python -m app.server" api 24 minutes ago Up 7 seconds (healthy) 8080/tcp
|
||||
messenger-lab6-api-2 sha256:efa9adc78d01ce2a8b6b72223b3964036998ac290fd83f91c929d417c137e0a4 "python -m app.server" api 24 minutes ago Up 43 seconds (healthy) 8080/tcp
|
||||
messenger-lab6-broker-1 rabbitmq:3.13.7-alpine "/bin/sh -ec 'export…" broker 40 minutes ago Up About a minute (healthy) 4369/tcp, 5671-5672/tcp, 15691-15692/tcp, 25672/tcp
|
||||
messenger-lab6-db-1 postgres:16.4-alpine "docker-entrypoint.s…" db 40 minutes ago Up About a minute (healthy) 5432/tcp
|
||||
messenger-lab6-grafana-1 grafana/grafana:11.2.2 "/run.sh" grafana 25 minutes ago Up 24 minutes 127.0.0.1:3000->3000/tcp
|
||||
messenger-lab6-lb-1 haproxy:3.0.11-alpine "docker-entrypoint.s…" lb 24 minutes ago Up 36 seconds (healthy) 127.0.0.1:8086->8080/tcp, 127.0.0.1:8406->8404/tcp
|
||||
messenger-lab6-loki-1 grafana/loki:3.2.1 "/usr/bin/loki -conf…" loki 25 minutes ago Up 14 minutes 127.0.0.1:3100->3100/tcp
|
||||
messenger-lab6-migrate-1 sha256:efa9adc78d01ce2a8b6b72223b3964036998ac290fd83f91c929d417c137e0a4 "python scripts/migr…" migrate 24 minutes ago Exited (0) 23 minutes ago
|
||||
messenger-lab6-otel-collector-1 otel/opentelemetry-collector-contrib:0.111.0 "/otelcol-contrib --…" otel-collector 27 minutes ago Up 26 minutes 4317/tcp, 55678-55679/tcp
|
||||
messenger-lab6-prometheus-1 prom/prometheus:v2.55.0 "/bin/prometheus --c…" prometheus 25 minutes ago Up 24 minutes 127.0.0.1:9090->9090/tcp
|
||||
messenger-lab6-publisher-1 sha256:efa9adc78d01ce2a8b6b72223b3964036998ac290fd83f91c929d417c137e0a4 "python -m app.jobs" publisher 24 minutes ago Up 44 seconds 8080/tcp
|
||||
messenger-lab6-s3-1 chrislusf/seaweedfs:3.97 "/bin/sh /opt/lab/st…" s3 40 minutes ago Up About a minute (healthy) 7333/tcp, 8080/tcp, 8333/tcp, 8888/tcp, 9333/tcp, 18080/tcp, 18888/tcp, 19333/tcp
|
||||
messenger-lab6-s3-init-1 sha256:efa9adc78d01ce2a8b6b72223b3964036998ac290fd83f91c929d417c137e0a4 "python scripts/s3_b…" s3-init 24 minutes ago Exited (0) 22 minutes ago
|
||||
messenger-lab6-tempo-1 grafana/tempo:2.6.1 "/tempo -config.file…" tempo 9 minutes ago Up 9 minutes 127.0.0.1:3200->3200/tcp
|
||||
messenger-lab6-valkey-1 valkey/valkey:8.0.1-alpine "docker-entrypoint.s…" valkey 40 minutes ago Up About a minute (healthy) 6379/tcp
|
||||
messenger-lab6-worker-1 sha256:efa9adc78d01ce2a8b6b72223b3964036998ac290fd83f91c929d417c137e0a4 "python -m app.worker" worker 24 minutes ago Up 44 seconds 8080/tcp
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:16:01Z
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,191 @@
|
||||
{
|
||||
"status": "success",
|
||||
"data": {
|
||||
"resultType": "streams",
|
||||
"result": [
|
||||
{
|
||||
"stream": {
|
||||
"code_filepath": "/app/app/telemetry.py",
|
||||
"code_function": "emit",
|
||||
"code_lineno": "180",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"duration_seconds": "0.05064601900016896",
|
||||
"event": "http.request.complete",
|
||||
"flags": "1",
|
||||
"method": "POST",
|
||||
"observed_timestamp": "1795251728469179436",
|
||||
"outcome": "success",
|
||||
"request_id": "observe-81adf6fa6db94a2caf9dbdf96d62b259",
|
||||
"route": "/api/v1/chats/{chat_id}/attachments",
|
||||
"scope_name": "messenger.telemetry.events",
|
||||
"service_instance_id": "0d13c5814dab",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "0ff19193d60d0a11",
|
||||
"status_code": "202",
|
||||
"trace_id": "167d5623ef64fb0ed646ec1e2fe01ad8"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1795251728468912896",
|
||||
"http.request.complete"
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"stats": {
|
||||
"summary": {
|
||||
"bytesProcessedPerSecond": 87014288,
|
||||
"linesProcessedPerSecond": 188878,
|
||||
"totalBytesProcessed": 1367324,
|
||||
"totalLinesProcessed": 2968,
|
||||
"execTime": 0.015714,
|
||||
"queueTime": 0.000907,
|
||||
"subqueries": 0,
|
||||
"totalEntriesReturned": 1,
|
||||
"splits": 2,
|
||||
"shards": 2,
|
||||
"totalPostFilterLines": 1,
|
||||
"totalStructuredMetadataBytesProcessed": 1251756
|
||||
},
|
||||
"querier": {
|
||||
"store": {
|
||||
"totalChunksRef": 0,
|
||||
"totalChunksDownloaded": 0,
|
||||
"chunksDownloadTime": 0,
|
||||
"queryReferencedStructuredMetadata": false,
|
||||
"chunk": {
|
||||
"headChunkBytes": 0,
|
||||
"headChunkLines": 0,
|
||||
"decompressedBytes": 0,
|
||||
"decompressedLines": 0,
|
||||
"compressedBytes": 0,
|
||||
"totalDuplicates": 0,
|
||||
"postFilterLines": 0,
|
||||
"headChunkStructuredMetadataBytes": 0,
|
||||
"decompressedStructuredMetadataBytes": 0
|
||||
},
|
||||
"chunkRefsFetchTime": 0,
|
||||
"congestionControlLatency": 0,
|
||||
"pipelineWrapperFilteredLines": 0
|
||||
}
|
||||
},
|
||||
"ingester": {
|
||||
"totalReached": 2,
|
||||
"totalChunksMatched": 2,
|
||||
"totalBatches": 3,
|
||||
"totalLinesSent": 1,
|
||||
"store": {
|
||||
"totalChunksRef": 0,
|
||||
"totalChunksDownloaded": 0,
|
||||
"chunksDownloadTime": 0,
|
||||
"queryReferencedStructuredMetadata": true,
|
||||
"chunk": {
|
||||
"headChunkBytes": 60282,
|
||||
"headChunkLines": 306,
|
||||
"decompressedBytes": 1307042,
|
||||
"decompressedLines": 2662,
|
||||
"compressedBytes": 58630,
|
||||
"totalDuplicates": 0,
|
||||
"postFilterLines": 1,
|
||||
"headChunkStructuredMetadataBytes": 53856,
|
||||
"decompressedStructuredMetadataBytes": 1197900
|
||||
},
|
||||
"chunkRefsFetchTime": 386158,
|
||||
"congestionControlLatency": 0,
|
||||
"pipelineWrapperFilteredLines": 0
|
||||
}
|
||||
},
|
||||
"cache": {
|
||||
"chunk": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"index": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"result": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"statsResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"volumeResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"seriesResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"labelResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"instantMetricResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
}
|
||||
},
|
||||
"index": {
|
||||
"totalChunks": 0,
|
||||
"postFilterChunks": 0,
|
||||
"shardsDuration": 0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,191 @@
|
||||
{
|
||||
"status": "success",
|
||||
"data": {
|
||||
"resultType": "streams",
|
||||
"result": [
|
||||
{
|
||||
"stream": {
|
||||
"attachment_id": "567d9b19-91b2-468c-95bb-8317c6ef90f2",
|
||||
"attempt": "1",
|
||||
"code_filepath": "/app/app/telemetry.py",
|
||||
"code_function": "emit",
|
||||
"code_lineno": "180",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"duration_seconds": "0.4046185140000489",
|
||||
"event": "job.complete",
|
||||
"flags": "1",
|
||||
"job_id": "2c0f93ca-74a0-4191-9b63-85fd07008e86",
|
||||
"job_type": "attachment.process.v1",
|
||||
"observed_timestamp": "1795251729011236193",
|
||||
"outcome": "success",
|
||||
"scope_name": "messenger.telemetry.events",
|
||||
"service_instance_id": "b5902353dfd648c39ac966e0780e90a3",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "59e8500a32f934ab",
|
||||
"trace_id": "167d5623ef64fb0ed646ec1e2fe01ad8"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1795251729011139328",
|
||||
"job.complete"
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"stats": {
|
||||
"summary": {
|
||||
"bytesProcessedPerSecond": 197625,
|
||||
"linesProcessedPerSecond": 1051,
|
||||
"totalBytesProcessed": 1128,
|
||||
"totalLinesProcessed": 6,
|
||||
"execTime": 0.005708,
|
||||
"queueTime": 0.001056,
|
||||
"subqueries": 0,
|
||||
"totalEntriesReturned": 1,
|
||||
"splits": 2,
|
||||
"shards": 2,
|
||||
"totalPostFilterLines": 1,
|
||||
"totalStructuredMetadataBytesProcessed": 1056
|
||||
},
|
||||
"querier": {
|
||||
"store": {
|
||||
"totalChunksRef": 0,
|
||||
"totalChunksDownloaded": 0,
|
||||
"chunksDownloadTime": 0,
|
||||
"queryReferencedStructuredMetadata": false,
|
||||
"chunk": {
|
||||
"headChunkBytes": 0,
|
||||
"headChunkLines": 0,
|
||||
"decompressedBytes": 0,
|
||||
"decompressedLines": 0,
|
||||
"compressedBytes": 0,
|
||||
"totalDuplicates": 0,
|
||||
"postFilterLines": 0,
|
||||
"headChunkStructuredMetadataBytes": 0,
|
||||
"decompressedStructuredMetadataBytes": 0
|
||||
},
|
||||
"chunkRefsFetchTime": 0,
|
||||
"congestionControlLatency": 0,
|
||||
"pipelineWrapperFilteredLines": 0
|
||||
}
|
||||
},
|
||||
"ingester": {
|
||||
"totalReached": 2,
|
||||
"totalChunksMatched": 2,
|
||||
"totalBatches": 3,
|
||||
"totalLinesSent": 1,
|
||||
"store": {
|
||||
"totalChunksRef": 0,
|
||||
"totalChunksDownloaded": 0,
|
||||
"chunksDownloadTime": 0,
|
||||
"queryReferencedStructuredMetadata": true,
|
||||
"chunk": {
|
||||
"headChunkBytes": 1128,
|
||||
"headChunkLines": 6,
|
||||
"decompressedBytes": 0,
|
||||
"decompressedLines": 0,
|
||||
"compressedBytes": 0,
|
||||
"totalDuplicates": 0,
|
||||
"postFilterLines": 1,
|
||||
"headChunkStructuredMetadataBytes": 1056,
|
||||
"decompressedStructuredMetadataBytes": 0
|
||||
},
|
||||
"chunkRefsFetchTime": 227539,
|
||||
"congestionControlLatency": 0,
|
||||
"pipelineWrapperFilteredLines": 0
|
||||
}
|
||||
},
|
||||
"cache": {
|
||||
"chunk": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"index": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"result": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"statsResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"volumeResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"seriesResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"labelResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"instantMetricResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
}
|
||||
},
|
||||
"index": {
|
||||
"totalChunks": 0,
|
||||
"postFilterChunks": 0,
|
||||
"shardsDuration": 0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1 @@
|
||||
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
|
||||
@@ -0,0 +1,20 @@
|
||||
|
||||
UTC: 2026-11-21T09:03:11Z
|
||||
Command: C:\Users\masha\AppData\Local\Programs\Python\Python313\python.exe -X utf8 -c "import json
|
||||
import scripts.observe_lab6 as m
|
||||
client,stand=m.Business(),m.Stand()
|
||||
try:
|
||||
m.wait_backends();client.bootstrap()
|
||||
result=m.lag_incident(client,stand,'MessengerQueueLag')
|
||||
path=m.EVIDENCE/'observability-incidents.json'
|
||||
data=json.loads(path.read_text(encoding='utf-8'))
|
||||
data['incidents'][0]=result
|
||||
data['captured_at']=m.now()
|
||||
m.save('observability-incidents.json',data)
|
||||
print('PASS: clean queue and inactive alert -> newly firing -> resolved; original bytes verified')
|
||||
finally:
|
||||
stand.restore()
|
||||
"
|
||||
PASS: clean queue and inactive alert -> newly firing -> resolved; original bytes verified
|
||||
|
||||
RESULT: PASS (exit 0) at 2026-11-21T09:04:11Z
|
||||
@@ -1,13 +1,13 @@
|
||||
{
|
||||
"uid": "messenger-lab6",
|
||||
"title": "Messenger lab6 - Observability",
|
||||
"description": "Application OTLP metrics, structured logs and traces. HTTP error ratio is business 5xx / all business responses; health excluded. Duration units are seconds. All API replicas are aggregated.",
|
||||
"description": "Application OTLP metrics, structured logs and traces. HTTP error ratio is business 5xx / all business responses; health excluded. Duration units are seconds. All API replicas are aggregated. Grade 4: histogram percentiles, durable queue age/depth, pool/acquisition pressure, worker saturation, process CPU/RAM, ready latency, 5-minute SLIs and local alerts. See infra/otel/RUNBOOK.md.",
|
||||
"tags": [
|
||||
"lab6",
|
||||
"opentelemetry"
|
||||
],
|
||||
"schemaVersion": 39,
|
||||
"version": 1,
|
||||
"version": 2,
|
||||
"editable": false,
|
||||
"timezone": "browser",
|
||||
"refresh": "5s",
|
||||
@@ -74,7 +74,7 @@
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum(rate(messenger_http_requests_total{service_name=\"messenger-api\",route!~\"/health/.*\",status_class=\"5xx\"}[$__rate_interval])) / clamp_min(sum(rate(messenger_http_requests_total{service_name=\"messenger-api\",route!~\"/health/.*\"}[$__rate_interval])), 1e-9)",
|
||||
"expr": "(sum(rate(messenger_http_requests_total{service_name=\"messenger-api\",route!~\"/health/.*\",status_class=\"5xx\"}[$__rate_interval])) or vector(0)) / clamp_min(sum(rate(messenger_http_requests_total{service_name=\"messenger-api\",route!~\"/health/.*\"}[$__rate_interval])), 1e-9)",
|
||||
"legendFormat": ""
|
||||
}
|
||||
],
|
||||
@@ -317,6 +317,756 @@
|
||||
"options": {
|
||||
"showHeader": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 9,
|
||||
"type": "timeseries",
|
||||
"title": "API duration p50 / p95 / p99 (seconds)",
|
||||
"description": "Percentiles from aggregated histogram buckets across all API replicas. Never an average of replica percentiles.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 41,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "histogram_quantile(0.5, sum by (le) (rate(messenger_http_duration_seconds_bucket{service_name=\"messenger-api\",route!~\"/health/.*\"}[$__rate_interval])))",
|
||||
"legendFormat": "p50"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "histogram_quantile(0.95, sum by (le) (rate(messenger_http_duration_seconds_bucket{service_name=\"messenger-api\",route!~\"/health/.*\"}[$__rate_interval])))",
|
||||
"legendFormat": "p95"
|
||||
},
|
||||
{
|
||||
"refId": "C",
|
||||
"expr": "histogram_quantile(0.99, sum by (le) (rate(messenger_http_duration_seconds_bucket{service_name=\"messenger-api\",route!~\"/health/.*\"}[$__rate_interval])))",
|
||||
"legendFormat": "p99"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 10,
|
||||
"type": "timeseries",
|
||||
"title": "Worker duration p50 / p95 / p99 (seconds)",
|
||||
"description": "",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 41,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "histogram_quantile(0.5, sum by (le) (rate(messenger_jobs_duration_seconds_bucket{service_name=\"messenger-worker\"}[$__rate_interval])))",
|
||||
"legendFormat": "p50"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "histogram_quantile(0.95, sum by (le) (rate(messenger_jobs_duration_seconds_bucket{service_name=\"messenger-worker\"}[$__rate_interval])))",
|
||||
"legendFormat": "p95"
|
||||
},
|
||||
{
|
||||
"refId": "C",
|
||||
"expr": "histogram_quantile(0.99, sum by (le) (rate(messenger_jobs_duration_seconds_bucket{service_name=\"messenger-worker\"}[$__rate_interval])))",
|
||||
"legendFormat": "p99"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 11,
|
||||
"type": "timeseries",
|
||||
"title": "Durable unfinished jobs (count)",
|
||||
"description": "Authoritative PostgreSQL backlog, including pending outbox and delayed retries; not RabbitMQ ready count.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 49,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "max(messenger_queue_depth{service_name=\"messenger-publisher\"})",
|
||||
"legendFormat": "queued + processing"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 12,
|
||||
"type": "timeseries",
|
||||
"title": "Oldest unfinished job / observer sample age (seconds)",
|
||||
"description": "A sample older than 10 seconds is stale. Oldest age differs from processing latency.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 49,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "max(messenger_queue_oldest_age_seconds{service_name=\"messenger-publisher\"})",
|
||||
"legendFormat": "oldest unfinished"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "max(messenger_queue_sample_age_seconds{service_name=\"messenger-publisher\"})",
|
||||
"legendFormat": "sample age"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 15
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 13,
|
||||
"type": "timeseries",
|
||||
"title": "DB pool used / capacity (connections)",
|
||||
"description": "Bounded SQLAlchemy pool by process, max size includes overflow.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 57,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "messenger_db_pool_used",
|
||||
"legendFormat": "used {{service_name}} {{service_instance_id}}"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "messenger_db_pool_capacity",
|
||||
"legendFormat": "capacity {{service_name}} {{service_instance_id}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 14,
|
||||
"type": "timeseries",
|
||||
"title": "DB acquisitions in progress (requests)",
|
||||
"description": "Acquisitions in progress include waiting for a pool slot and opening a new DB connection. They are not active SQL queries.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 57,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "messenger_db_pool_waiting",
|
||||
"legendFormat": "{{service_name}} {{service_instance_id}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 15,
|
||||
"type": "timeseries",
|
||||
"title": "Worker saturation (busy / capacity)",
|
||||
"description": "Busy / capacity for workers with heartbeat younger than 15 seconds. Zero live capacity gives no ratio; do not interpret absence as successful operation.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 65,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "(sum(messenger_worker_busy{service_name=\"messenger-worker\"} and on(service_instance_id) (time() - messenger_worker_heartbeat_seconds{service_name=\"messenger-worker\"} < 15)) or vector(0)) / (sum(messenger_worker_capacity{service_name=\"messenger-worker\"} and on(service_instance_id) (time() - messenger_worker_heartbeat_seconds{service_name=\"messenger-worker\"} < 15)) or vector(0))",
|
||||
"legendFormat": "saturation"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 16,
|
||||
"type": "timeseries",
|
||||
"title": "Worker busy / capacity (jobs)",
|
||||
"description": "Sum only worker process instances with heartbeat younger than 15 seconds; old cached resources after crashes do not count.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 65,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum(messenger_worker_busy{service_name=\"messenger-worker\"} and on(service_instance_id) (time() - messenger_worker_heartbeat_seconds{service_name=\"messenger-worker\"} < 15)) or vector(0)",
|
||||
"legendFormat": "busy"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "sum(messenger_worker_capacity{service_name=\"messenger-worker\"} and on(service_instance_id) (time() - messenger_worker_heartbeat_seconds{service_name=\"messenger-worker\"} < 15)) or vector(0)",
|
||||
"legendFormat": "capacity"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 17,
|
||||
"type": "timeseries",
|
||||
"title": "Application CPU (cores)",
|
||||
"description": "CPU seconds / second = cores used by the Python application process. Worker image subprocess CPU is outside this process signal; container limits apply separately.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 73,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "rate(messenger_process_cpu_seconds_total[$__rate_interval])",
|
||||
"legendFormat": "{{service_name}} {{service_instance_id}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 18,
|
||||
"type": "timeseries",
|
||||
"title": "Application resident RAM (bytes)",
|
||||
"description": "Current resident memory of the Python application process; excludes the separate image subprocess.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 73,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "messenger_process_memory_bytes",
|
||||
"legendFormat": "{{service_name}} {{service_instance_id}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "bytes",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 19,
|
||||
"type": "timeseries",
|
||||
"title": "Acceptance to ready p50 / p95 / p99 (seconds)",
|
||||
"description": "From durable attachment acceptance timestamp to COMMIT ready, including queue wait and retries. Only completed ready jobs enter this histogram.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 81,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "histogram_quantile(0.5, sum by (le) (rate(messenger_attachment_time_to_ready_seconds_bucket{service_name=\"messenger-worker\"}[$__rate_interval])))",
|
||||
"legendFormat": "p50"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "histogram_quantile(0.95, sum by (le) (rate(messenger_attachment_time_to_ready_seconds_bucket{service_name=\"messenger-worker\"}[$__rate_interval])))",
|
||||
"legendFormat": "p95"
|
||||
},
|
||||
{
|
||||
"refId": "C",
|
||||
"expr": "histogram_quantile(0.99, sum by (le) (rate(messenger_attachment_time_to_ready_seconds_bucket{service_name=\"messenger-worker\"}[$__rate_interval])))",
|
||||
"legendFormat": "p99"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 20,
|
||||
"type": "timeseries",
|
||||
"title": "API availability SLI / SLO (5 minutes)",
|
||||
"description": "5-minute demonstration window. Health excluded, client 4xx not service failure. No traffic gives no SLI value; lower is worse.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 81,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "1 - (sum(rate(messenger_http_requests_total{service_name=\"messenger-api\",route!~\"/health/.*\",status_class=\"5xx\"}[5m])) or vector(0)) / sum(rate(messenger_http_requests_total{service_name=\"messenger-api\",route!~\"/health/.*\"}[5m]))",
|
||||
"legendFormat": "non-5xx / all business"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "vector(0.99)",
|
||||
"legendFormat": "SLO 99%"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 21,
|
||||
"type": "timeseries",
|
||||
"title": "Ready within 60 seconds SLI / SLO (5 minutes)",
|
||||
"description": "Conditional SLI over completed ready jobs. Missing completions produce no successful SLI. Inspect unfinished queue and failed attempts alongside this chart.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 89,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum(rate(messenger_attachment_time_to_ready_seconds_bucket{service_name=\"messenger-worker\",le=\"60\"}[5m])) / sum(rate(messenger_attachment_time_to_ready_seconds_count{service_name=\"messenger-worker\"}[5m]))",
|
||||
"legendFormat": "ready <=60s / ready"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "vector(0.95)",
|
||||
"legendFormat": "SLO 95%"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 22,
|
||||
"type": "timeseries",
|
||||
"title": "Local alert state: pending / firing",
|
||||
"description": "A series at 1 indicates pending or firing. No series means inactive/resolved. Evaluate rules at http://localhost:9090/alerts; for=10s, rule interval=5s.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 89,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "ALERTS{alertname=~\"MessengerHighErrorRate|MessengerQueueLag\"}",
|
||||
"legendFormat": "{{alertname}} {{alertstate}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 23,
|
||||
"type": "timeseries",
|
||||
"title": "Collector queue occupancy (batches)",
|
||||
"description": "Bounded in-memory sending queue: 256 batches per traces/logs exporter. Retries expire after 30 seconds.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 97,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "otelcol_exporter_queue_size{exporter=~\"otlphttp/.*\"}",
|
||||
"legendFormat": "{{exporter}} size"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "otelcol_exporter_queue_capacity{exporter=~\"otlphttp/.*\"}",
|
||||
"legendFormat": "{{exporter}} capacity"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 24,
|
||||
"type": "timeseries",
|
||||
"title": "Collector failed export attempts / second",
|
||||
"description": "Failed export attempts can include retries; they do not directly count distinct lost events. See evidence and runbook for outage loss accounting.",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 97,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "rate(otelcol_exporter_send_failed_spans[$__rate_interval])",
|
||||
"legendFormat": "spans {{exporter}}"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "rate(otelcol_exporter_send_failed_log_records[$__rate_interval])",
|
||||
"legendFormat": "logs {{exporter}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"annotations": {
|
||||
|
||||
@@ -18,7 +18,8 @@ datasources:
|
||||
jsonData:
|
||||
derivedFields:
|
||||
- name: TraceID
|
||||
matcherRegex: '"trace_id"\s*:\s*"([0-9a-f]{32})"'
|
||||
matcherType: label
|
||||
matcherRegex: '^trace_id$'
|
||||
datasourceUid: tempo
|
||||
url: '$${__value.raw}'
|
||||
- name: Tempo
|
||||
@@ -32,8 +33,10 @@ datasources:
|
||||
datasourceUid: loki
|
||||
spanStartTimeShift: -5m
|
||||
spanEndTimeShift: 5m
|
||||
filterByTraceID: true
|
||||
filterByTraceID: false
|
||||
filterBySpanID: false
|
||||
customQuery: true
|
||||
query: '{$${__tags}} | trace_id="$${__span.traceId}"'
|
||||
tags: [{ key: service.name, value: service_name }]
|
||||
nodeGraph:
|
||||
enabled: true
|
||||
|
||||
+16
-4
@@ -1,7 +1,19 @@
|
||||
# Конфигурация наблюдаемости — реализовать
|
||||
# Наблюдаемость lab6
|
||||
|
||||
Добавьте `collector.yaml` с OTLP receivers, processors и exporters выбранной версии Collector. Укажите явные bind endpoints, внутренние сети, memory limits/batch и поведение при backpressure по уровню. Сопоставьте все три pipeline с реально развёрнутыми backend.
|
||||
`API / publisher / worker → OTLP HTTP → Collector → Prometheus / Loki / Tempo → Grafana`.
|
||||
|
||||
Добавьте конфигурации storage, Grafana provisioning/datasources, provisioning/dashboards и импортируемые dashboard JSON. На 4 — alert rules и runbooks. Пустой dashboard или debug exporter, который только печатает данные в stdout, не заменяют хранилища/доказательства из `contracts/observability.md`.
|
||||
* [Collector](collector.yaml): OTLP receivers, memory limiter, bounded batch,
|
||||
sending queues и retry. Collector доступен только во внутренней сети Compose.
|
||||
* [Prometheus](../prometheus/prometheus.yaml) хранит метрики приложения и самого
|
||||
Collector; [alerts](../prometheus/alerts.yaml) оцениваются каждые 5 секунд.
|
||||
* [Loki](../loki/loki.yaml) принимает native OTLP logs; технические ID —
|
||||
structured metadata, не индексные labels.
|
||||
* [Tempo](../tempo/tempo.yaml) хранит реальные API/producer/consumer spans.
|
||||
* [Grafana](../grafana/provisioning/datasources/datasources.yaml) подключает
|
||||
три datasource; [dashboard](../grafana/dashboards/messenger.json) импортируется
|
||||
provisioning автоматически. Backend ports и Grafana опубликованы на loopback.
|
||||
|
||||
Не публикуйте OTLP/metrics/административные ports в общую сеть без необходимости. В lab7 реализуйте TLS/mTLS по trust matrix; экспорт Authorization и message bodies запрещён.
|
||||
Сценарии диагностики и точные SLI/SLO: [RUNBOOK](RUNBOOK.md).
|
||||
Запуск, семантика метрик, результаты и ограничения: [отчёт](../../REPORT.md).
|
||||
Воспроизводимая проверка с evidence: `python -X utf8 scripts/test_lab6.py --incidents`.
|
||||
Подробные сценарии находятся в `scripts/observe_lab6.py`.
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
# Диагностика стенда lab6
|
||||
|
||||
Правила предназначены для локального учебного стенда и синтетической нагрузки.
|
||||
Пороги 15 секунд и `for: 10s` сокращены для демонстрации, а не выведены из
|
||||
производственной статистики. Alert оценивается Prometheus каждые 5 секунд.
|
||||
Firing и последующее отсутствие активного alert сохраняет сценарий проверки;
|
||||
внешняя доставка уведомлений не требуется.
|
||||
|
||||
## SLI и SLO
|
||||
|
||||
- Доступность API: доля бизнес-ответов с кодом ниже 500 среди всех
|
||||
бизнес-ответов. `/health/*` исключается, 4xx считаются доступным сервисом.
|
||||
Учебный SLO: не менее 99% в скользящем окне 5 минут.
|
||||
- Готовность вложения: доля завершений `ready` с временем не более 60 секунд
|
||||
от устойчивого принятия до COMMIT `ready`. SLO: не менее 95% в окне 5 минут.
|
||||
Доля вычисляется по `messenger_attachment_time_to_ready_seconds_bucket{le="60"}`
|
||||
и `_count`; незавершённые и failed задачи отдельно видны в backlog/errors.
|
||||
Поэтому этот SLI условен по успешным завершениям и один не доказывает,
|
||||
что 95% всех принятых загрузок достигают ready. При остановленном worker
|
||||
отсутствие завершений не интерпретируется как выполненный SLO.
|
||||
- Percentile считается `histogram_quantile` после `sum by (le)` bucket rate
|
||||
всех нужных реплик. Средние p95 отдельных реплик не усредняются.
|
||||
|
||||
## API errors
|
||||
|
||||
Ссылка alert: `MessengerHighErrorRate`.
|
||||
|
||||
1. Открыть Grafana `http://localhost:3000`, dashboard **Messenger lab6**.
|
||||
Проверить интервал инцидента, RPS и 5xx ratio. Ошибки входа 401 и
|
||||
недоступность PostgreSQL 503 отвечают на разные вопросы.
|
||||
2. Выбрать ошибочный API trace из Tempo и связанный структурный лог по
|
||||
`trace_id`; `request_id` связывает лог с `X-Request-Id` ответа.
|
||||
3. Найти DB span: ошибка соединения или большая длительность DB при
|
||||
коротком handler указывает на зависимость. Сравнить pool used/waiting
|
||||
и DB operation duration. В spans нет текста SQL и параметров.
|
||||
Если DB spans быстрые, искать время между ними/в handler, S3 или
|
||||
ожидание свободного ресурса; одно значение p95 причины не устанавливает.
|
||||
4. Проверить `docker compose ps db api`. В контролируемом сценарии
|
||||
`docker compose stop db` ожидаются отказы синхронного пути и readiness.
|
||||
HAProxy может вернуть собственный 503, не попадающий в счётчик API:
|
||||
сценарий диагностики должен сохранять также прямой ответ API.
|
||||
5. Восстановить `docker compose start db`, дождаться API ready, повторить
|
||||
запросы. В Prometheus `http://localhost:9090/alerts` alert сначала
|
||||
становится inactive после выхода ошибок из окна 1 минуты.
|
||||
|
||||
Сохранять ответ без токена, trace JSON, лог, запросы метрик и состояние alert
|
||||
до воздействия, во время и после. Не считать текст ошибки DB в stdout
|
||||
единственным доказательством причины: нужны сохранённые сигналы приложения.
|
||||
|
||||
## Queue lag
|
||||
|
||||
Ссылка alert: `MessengerQueueLag`.
|
||||
|
||||
1. Сравнить queue depth, oldest age, свежесть наблюдения, busy/capacity worker,
|
||||
успешные/ошибочные jobs и длительность попыток. Backlog измерен по
|
||||
устойчивым незавершённым заданиям PostgreSQL, включая outbox и retry;
|
||||
это не только число сообщений ready в RabbitMQ.
|
||||
2. Если depth > 0 и возраст растёт при отсутствии завершений и busy=0,
|
||||
проверить `docker compose ps worker publisher broker`. Остановка worker
|
||||
не останавливает publisher-наблюдатель.
|
||||
3. Если busy=capacity, искать медленные DB/S3/image spans в worker trace.
|
||||
Если растут retry/errors, найти соответствующий consumer span и событие
|
||||
retry. По traceparent пройти producer → consumer → DB/S3, включая попытки.
|
||||
4. Контролируемое воздействие: `docker compose stop worker`, затем несколько
|
||||
upload стандартного fixture. Ответ 202 означает устойчивое принятие;
|
||||
рост oldest age и firing ожидаются через 15 секунд плюс `for` и интервалы
|
||||
сбора/оценки. Счётчик duration не заменяет возраст ожидающей задачи.
|
||||
5. Восстановить `docker compose start worker`. Дождаться ready вложений,
|
||||
depth=0 и inactive alert. Скачать original и thumbnail, проверить байты
|
||||
и размеры. Уменьшение backlog само по себе ещё не доказывает сохранность.
|
||||
|
||||
Sample age защищает от интерпретации устаревшей очереди как свежей. Если DB
|
||||
недоступна и sample age растёт, сначала расследовать DB; автоматически
|
||||
приписывать это остановленному worker нельзя.
|
||||
|
||||
## Недоступен backend telemetry
|
||||
|
||||
1. Перед воздействием сохранить Collector exporter queue/failed counters,
|
||||
память API/worker/Collector и контрольное сообщение/вложение.
|
||||
2. `docker compose stop tempo loki` отключает хранилища traces/logs.
|
||||
Prometheus остаётся включённым и сохраняет self metrics Collector:
|
||||
очередь, ошибки отправки и память. Отдельно проверить недоступность
|
||||
Collector, если нужно исследовать именно переполнение SDK.
|
||||
3. Продолжать бизнес-запросы и загрузки. Проверить 201 сообщений, 202 → ready
|
||||
вложений и чтение данных. Telemetry отсутствует в readiness; зависимость
|
||||
от неё не должна отбирать все API из балансировки.
|
||||
4. Collector: memory limiter 192 MiB, spike 48 MiB, batch максимум 512,
|
||||
sending queue максимум 256 batches на exporter, 2 consumers; timeout 3s,
|
||||
retry не более 30s на batch. После исчерпания retries данные отбрасываются.
|
||||
SDK также имеет конечные очереди и timeout; экспорт выполняется фоново.
|
||||
Ограничения RAM контейнеров — дополнительный предел, а не замена очередям.
|
||||
5. `docker compose start tempo loki`; дождаться backend ready и повторить
|
||||
контрольный запрос. Новые trace/log должны появиться. Часть старых
|
||||
сигналов может потеряться: сохранённый рост failed/dropped counters и
|
||||
границы измерения описываются в отчёте. Нельзя обещать полное восстановление
|
||||
telemetry после перезапуска Collector: его очереди находятся в памяти.
|
||||
|
||||
Команды `docker compose down` сохраняют volumes. `down -v` удаляет учебные
|
||||
данные и историю, для восстановления после инцидента эта команда не нужна.
|
||||
@@ -0,0 +1,32 @@
|
||||
groups:
|
||||
- name: messenger-lab6
|
||||
interval: 5s
|
||||
rules:
|
||||
- alert: MessengerHighErrorRate
|
||||
expr: |
|
||||
(
|
||||
sum(rate(messenger_http_requests_total{service_name="messenger-api",route!~"/health/.*",status_class="5xx"}[1m]))
|
||||
/
|
||||
clamp_min(sum(rate(messenger_http_requests_total{service_name="messenger-api",route!~"/health/.*"}[1m])), 1e-9)
|
||||
) > 0.05
|
||||
for: 10s
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Business API 5xx ratio exceeds 5%"
|
||||
description: "5xx / all business HTTP responses over 1 minute, continuously above 5% for 10 seconds. Health and client 4xx are not failures in this SLI."
|
||||
runbook: "infra/otel/RUNBOOK.md#api-errors"
|
||||
- alert: MessengerQueueLag
|
||||
expr: |
|
||||
max(messenger_queue_oldest_age_seconds{service_name="messenger-publisher"}) > 15
|
||||
and
|
||||
max(messenger_queue_depth{service_name="messenger-publisher"}) > 0
|
||||
and
|
||||
max(messenger_queue_sample_age_seconds{service_name="messenger-publisher"}) < 10
|
||||
for: 10s
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Oldest unfinished attachment job exceeds 15 seconds"
|
||||
description: "Durable PostgreSQL backlog is nonempty and oldest job age is above 15 seconds for 10 seconds; the observer sample is less than 10 seconds old. This is queue age, not worker processing duration."
|
||||
runbook: "infra/otel/RUNBOOK.md#queue-lag"
|
||||
@@ -2,6 +2,9 @@ global:
|
||||
scrape_interval: 5s
|
||||
evaluation_interval: 5s
|
||||
|
||||
rule_files:
|
||||
- /etc/prometheus/alerts.yaml
|
||||
|
||||
scrape_configs:
|
||||
- job_name: messenger
|
||||
static_configs:
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
-- W3C context is part of the durable work intent, not an in-memory cache.
|
||||
-- Retry attempts retain the latest causal span; business payload is unchanged.
|
||||
ALTER TABLE attachments ADD COLUMN traceparent TEXT, ADD COLUMN tracestate TEXT;
|
||||
ALTER TABLE outbox ADD COLUMN traceparent TEXT, ADD COLUMN tracestate TEXT;
|
||||
@@ -0,0 +1,680 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Capture real lab6 observations. Standard library; local synthetic stand only.
|
||||
|
||||
Start Compose first. Commands: --baseline, --chain-with-retry, --incidents.
|
||||
Evidence contains actual backend responses, never credentials or message bodies.
|
||||
Every stopped service is restarted in finally, including failed assertions.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import base64
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from datetime import datetime, timezone
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import subprocess
|
||||
import threading
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
import uuid
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
EVIDENCE = ROOT / "evidence"
|
||||
PROMETHEUS = os.environ.get("PROMETHEUS_URL", "http://localhost:9090").rstrip("/")
|
||||
LOKI = os.environ.get("LOKI_URL", "http://localhost:3100").rstrip("/")
|
||||
TEMPO = os.environ.get("TEMPO_URL", "http://localhost:3200").rstrip("/")
|
||||
GRAFANA = os.environ.get("GRAFANA_URL", "http://localhost:3000").rstrip("/")
|
||||
BASE = os.environ.get("BASE_URL", "http://localhost:8080").rstrip("/")
|
||||
START_NS = int((time.time() - 5) * 1e9)
|
||||
SECRETS: list[str] = []
|
||||
|
||||
|
||||
def now() -> str:
|
||||
return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
|
||||
|
||||
|
||||
def check(condition, message: str) -> None:
|
||||
if not condition:
|
||||
raise AssertionError(message)
|
||||
|
||||
|
||||
def http(method: str, url: str, body=None, headers=None, raw=None, timeout=15):
|
||||
request_headers = {"Accept": "application/json", **(headers or {})}
|
||||
if body is not None:
|
||||
raw = json.dumps(body, ensure_ascii=False).encode("utf-8")
|
||||
request_headers["Content-Type"] = "application/json"
|
||||
request = urllib.request.Request(url, data=raw, headers=request_headers, method=method)
|
||||
try:
|
||||
response = urllib.request.urlopen(request, timeout=timeout)
|
||||
except urllib.error.HTTPError as exc:
|
||||
response = exc
|
||||
with response:
|
||||
data = response.read(16 * 1024 * 1024 + 1)
|
||||
check(len(data) <= 16 * 1024 * 1024, "Backend response exceeds safety limit")
|
||||
try:
|
||||
payload = json.loads(data)
|
||||
except (ValueError, UnicodeError):
|
||||
payload = None
|
||||
return response.status, payload, dict(response.headers), data
|
||||
|
||||
|
||||
def header(headers, name):
|
||||
return next((value for key, value in headers.items() if key.lower() == name.lower()), None)
|
||||
|
||||
|
||||
def get(url):
|
||||
status, data, _, _ = http("GET", url)
|
||||
check(status == 200 and data is not None, "Telemetry query failed: HTTP " + str(status))
|
||||
return data
|
||||
|
||||
|
||||
def query(expression):
|
||||
result = get(PROMETHEUS + "/api/v1/query?" + urllib.parse.urlencode({"query": expression}))
|
||||
check(result.get("status") == "success", "Prometheus query is unsuccessful")
|
||||
return result["data"]["result"]
|
||||
|
||||
|
||||
def number(expression):
|
||||
rows = query(expression)
|
||||
return float(rows[0]["value"][1]) if rows else None
|
||||
|
||||
|
||||
def wait(label, predicate, timeout=150, interval=2):
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
result = predicate()
|
||||
except (urllib.error.URLError, TimeoutError, ConnectionError):
|
||||
# Socket errors are expected during startup/recovery. Assertions
|
||||
# still propagate immediately: bad telemetry must never be masked.
|
||||
result = None
|
||||
if result:
|
||||
return result
|
||||
time.sleep(interval)
|
||||
raise AssertionError("Timed out: " + label)
|
||||
|
||||
|
||||
def wait_backends():
|
||||
def ready():
|
||||
endpoints = ((PROMETHEUS, "/-/ready"), (LOKI, "/ready"), (TEMPO, "/ready"))
|
||||
for base, path in endpoints:
|
||||
if http("GET", base + path, timeout=3)[0] != 200:
|
||||
return False
|
||||
status, payload, _, _ = http("GET", GRAFANA + "/api/health", timeout=3)
|
||||
return status == 200 and isinstance(payload, dict) and payload.get("database") == "ok"
|
||||
wait("Prometheus, Loki, Tempo and Grafana readiness", ready, timeout=150)
|
||||
|
||||
|
||||
def safe_json(value):
|
||||
encoded = json.dumps(value, ensure_ascii=False)
|
||||
for secret in SECRETS:
|
||||
check(secret not in encoded, "Sensitive synthetic marker found in exported telemetry")
|
||||
# These keys are unnecessary for this stand and can expose request data.
|
||||
forbidden = {"db.statement", "db.query.text", "exception.message", "exception.stacktrace",
|
||||
"url.full", "http.url", "authorization", "http.request.body", "password"}
|
||||
def walk(item):
|
||||
if isinstance(item, dict):
|
||||
check(not forbidden.intersection(key.lower() for key in item), "Forbidden telemetry key")
|
||||
if item.get("key", "").lower() in forbidden:
|
||||
raise AssertionError("Forbidden telemetry attribute")
|
||||
for child in item.values():
|
||||
walk(child)
|
||||
elif isinstance(item, list):
|
||||
for child in item:
|
||||
walk(child)
|
||||
walk(value)
|
||||
return encoded
|
||||
|
||||
|
||||
def save(name, value):
|
||||
safe_json(value)
|
||||
EVIDENCE.mkdir(exist_ok=True)
|
||||
(EVIDENCE / name).write_text(json.dumps(value, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def compose(*arguments, timeout=90):
|
||||
result = subprocess.run(["docker", "compose", *arguments], cwd=ROOT, capture_output=True,
|
||||
text=True, encoding="utf-8", timeout=timeout)
|
||||
check(result.returncode == 0, "docker compose " + " ".join(arguments[:2]) + " failed")
|
||||
return result.stdout.strip()
|
||||
|
||||
|
||||
class Stand:
|
||||
def __init__(self):
|
||||
self.stopped = set()
|
||||
|
||||
def stop(self, *services):
|
||||
# Register before the command: even a partial stop must be undone.
|
||||
self.stopped.update(services)
|
||||
compose("stop", "-t", "3", *services)
|
||||
|
||||
def start(self, *services):
|
||||
# Compose start --wait may also start dependencies (including S3 while
|
||||
# testing its outage). Restart only these exact existing containers.
|
||||
ids = compose("ps", "-a", "-q", *services).splitlines()
|
||||
check(bool(ids), "No existing containers for requested services")
|
||||
result = subprocess.run(["docker", "start", *ids], cwd=ROOT, capture_output=True, timeout=30)
|
||||
check(result.returncode == 0, "docker start failed")
|
||||
def healthy():
|
||||
status = subprocess.run(["docker", "inspect", "--format",
|
||||
"{{.State.Running}} {{if .State.Health}}{{.State.Health.Status}}{{else}}healthy{{end}}", *ids],
|
||||
cwd=ROOT, capture_output=True, text=True, timeout=10)
|
||||
check(status.returncode == 0, "Container state lookup failed")
|
||||
return all(line.strip() == "true healthy" for line in status.stdout.splitlines())
|
||||
wait("requested containers healthy", healthy, timeout=120)
|
||||
self.stopped.difference_update(services)
|
||||
|
||||
def restore(self):
|
||||
if self.stopped:
|
||||
self.start(*sorted(self.stopped))
|
||||
|
||||
|
||||
class Business:
|
||||
def __init__(self):
|
||||
self.token = None
|
||||
self.chat = None
|
||||
|
||||
def call(self, method, path, expected=200, body=None, headers=None, raw=None):
|
||||
request_headers = dict(headers or {})
|
||||
if self.token:
|
||||
request_headers["Authorization"] = "Bearer " + self.token
|
||||
status, data, response_headers, content = http(method, BASE + path, body, request_headers, raw)
|
||||
check(status == expected, "Business request returned " + str(status) + ", expected " + str(expected))
|
||||
check(bool(header(response_headers, "X-Request-Id")), "Missing request ID")
|
||||
check(bool(header(response_headers, "X-Instance-Id")), "Missing API instance ID")
|
||||
return data, response_headers, content
|
||||
|
||||
def bootstrap(self):
|
||||
username = "obs_" + uuid.uuid4().hex[:20]
|
||||
password = "Never-export-" + uuid.uuid4().hex
|
||||
SECRETS.append(password)
|
||||
user, _, _ = self.call("POST", "/api/v1/users", 201,
|
||||
{"username": username, "display_name": "Observability", "password": password})
|
||||
credential = base64.b64encode((username + ":" + password).encode()).decode()
|
||||
SECRETS.append(credential)
|
||||
session, _, _ = self.call("POST", "/api/v1/auth/sessions", 201,
|
||||
headers={"Authorization": "Basic " + credential})
|
||||
self.token = session["token"]
|
||||
SECRETS.append(self.token)
|
||||
self.chat, _, _ = self.call("POST", "/api/v1/chats", 201,
|
||||
{"title": "Synthetic observations", "member_ids": [user["id"]]})
|
||||
|
||||
def upload(self):
|
||||
rid = "observe-" + uuid.uuid4().hex
|
||||
filename = "never-export-name-" + uuid.uuid4().hex + ".png"
|
||||
SECRETS.append(filename)
|
||||
data, headers, _ = self.call("POST", "/api/v1/chats/" + self.chat["id"] +
|
||||
"/attachments?" + urllib.parse.urlencode({"filename": filename}), 202,
|
||||
headers={"Content-Type": "image/png", "X-Request-Id": rid},
|
||||
raw=(ROOT / "tests/fixtures/sample.png").read_bytes())
|
||||
check(data["status"] == "queued", "202 must contain queued snapshot")
|
||||
check(header(headers, "X-Request-Id") == rid, "Request ID was not preserved")
|
||||
return {"attachment_id": data["id"], "request_id": rid,
|
||||
"api_trace_id": header(headers, "X-Trace-Id"), "accepted_at": now()}
|
||||
|
||||
def ready(self, upload):
|
||||
data, _, _ = self.call("GET", "/api/v1/attachments/" + upload["attachment_id"])
|
||||
check(data["status"] != "failed", "Synthetic attachment reached failed state")
|
||||
return data["status"] == "ready"
|
||||
|
||||
def verify_content(self, upload):
|
||||
import hashlib
|
||||
attachment_id = upload["attachment_id"]
|
||||
data, _, _ = self.call("GET", "/api/v1/attachments/" + attachment_id)
|
||||
_, _, original = self.call("GET", "/api/v1/attachments/" + attachment_id + "/content")
|
||||
fixture = (ROOT / "tests/fixtures/sample.png").read_bytes()
|
||||
check(original == fixture, "Recovered original differs from accepted bytes")
|
||||
check(data["sha256"] == hashlib.sha256(original).hexdigest(), "Original checksum mismatch")
|
||||
thumbnail = next((v for v in data["variants"] if v["name"] == "thumbnail"), None)
|
||||
check(thumbnail and (thumbnail["width"], thumbnail["height"]) == (256, 160),
|
||||
"Recovered thumbnail dimensions are incorrect")
|
||||
return {"attachment_id": attachment_id, "original_sha256": data["sha256"],
|
||||
"original_bytes": len(original), "thumbnail_size": [256, 160]}
|
||||
|
||||
def message(self, prefix="observe-message"):
|
||||
text = "Never-export-message-" + uuid.uuid4().hex
|
||||
SECRETS.append(text)
|
||||
rid = prefix + "-" + uuid.uuid4().hex
|
||||
_, headers, _ = self.call("POST", "/api/v1/chats/" + self.chat["id"] + "/messages", 201,
|
||||
{"text": text}, {"Idempotency-Key": uuid.uuid4().hex, "X-Request-Id": rid})
|
||||
return {"request_id": rid, "trace_id": header(headers, "X-Trace-Id")}
|
||||
|
||||
|
||||
def logs(service, marker):
|
||||
field = "request_id" if service == "messenger-api" else "attachment_id"
|
||||
# OTel attributes become Loki structured metadata. Body is the source-defined
|
||||
# event name; no request body or arbitrary message text is exported.
|
||||
expression = '{service_name="' + service + '"} | ' + field + "=" + json.dumps(marker)
|
||||
result = get(LOKI + "/loki/api/v1/query_range?" + urllib.parse.urlencode(
|
||||
{"query": expression, "start": START_NS, "end": time.time_ns(), "limit": 1000, "direction": "forward"}))
|
||||
check(result.get("status") == "success", "Loki query failed")
|
||||
records = []
|
||||
for stream in result["data"]["result"]:
|
||||
for sample in stream.get("values", []):
|
||||
try:
|
||||
record = json.loads(sample[1])
|
||||
except (ValueError, TypeError):
|
||||
record = {"event": sample[1]}
|
||||
if not isinstance(record, dict):
|
||||
record = {"event": sample[1]}
|
||||
if len(sample) > 2 and isinstance(sample[2], dict):
|
||||
record.update(sample[2])
|
||||
# Some Loki versions put structured metadata in stream labels.
|
||||
record = {**stream.get("stream", {}), **record}
|
||||
records.append(record)
|
||||
safe_json(result)
|
||||
return result, records
|
||||
|
||||
|
||||
def hex_id(value):
|
||||
if not value:
|
||||
return ""
|
||||
if len(value) in (16, 32) and all(char in "0123456789abcdefABCDEF" for char in value):
|
||||
return value.lower()
|
||||
try:
|
||||
return base64.b64decode(value, validate=True).hex()
|
||||
except (ValueError, TypeError):
|
||||
return value
|
||||
|
||||
|
||||
def wait_logs(service, marker, predicate, label):
|
||||
def matching():
|
||||
result = logs(service, marker)
|
||||
return result if any(predicate(record) for record in result[1]) else None
|
||||
return wait(label, matching)
|
||||
|
||||
|
||||
def trace(trace_id):
|
||||
status, data, _, _ = http("GET", TEMPO + "/api/traces/" + trace_id)
|
||||
if status == 404:
|
||||
return None
|
||||
check(status == 200 and data is not None, "Tempo trace lookup failed")
|
||||
safe_json(data)
|
||||
return data
|
||||
|
||||
|
||||
def attribute_value(value):
|
||||
if isinstance(value, dict):
|
||||
return next(iter(value.values()), None)
|
||||
return value
|
||||
|
||||
|
||||
def spans(data):
|
||||
found = []
|
||||
def walk(value):
|
||||
if isinstance(value, dict):
|
||||
if "resource" in value and ("scopeSpans" in value or "instrumentationLibrarySpans" in value):
|
||||
attributes = {item["key"]: attribute_value(item["value"])
|
||||
for item in value.get("resource", {}).get("attributes", [])}
|
||||
for scope in value.get("scopeSpans", value.get("instrumentationLibrarySpans", [])):
|
||||
for span in scope.get("spans", []):
|
||||
found.append({"service": attributes.get("service.name"), "span": span})
|
||||
else:
|
||||
for child in value.values():
|
||||
walk(child)
|
||||
elif isinstance(value, list):
|
||||
for child in value:
|
||||
walk(child)
|
||||
walk(data)
|
||||
return found
|
||||
|
||||
|
||||
def correlated(upload, prefix="baseline", require_chain=False, require_retry=False):
|
||||
api_result = wait_logs("messenger-api", upload["request_id"], lambda record:
|
||||
record.get("event") == "http.request.complete" and
|
||||
record.get("request_id") == upload["request_id"], "API request log")
|
||||
api_log = next(record for record in api_result[1]
|
||||
if record.get("event") == "http.request.complete" and record.get("request_id") == upload["request_id"])
|
||||
api_tid = hex_id(api_log.get("trace_id"))
|
||||
check(len(api_tid) == 32 and int(api_tid, 16) != 0, "API log has no valid trace ID")
|
||||
if upload["api_trace_id"]:
|
||||
check(api_tid == upload["api_trace_id"], "API response and log disagree on trace ID")
|
||||
api_trace = wait("API trace", lambda: trace(api_tid))
|
||||
matching_api = [entry for entry in spans(api_trace) if entry["service"] == "messenger-api"
|
||||
and hex_id(entry["span"].get("spanId")) == hex_id(api_log.get("span_id"))]
|
||||
check(bool(matching_api), "API log span is absent from Tempo trace")
|
||||
check(str(matching_api[0]["span"].get("kind")) in ("2", "SPAN_KIND_SERVER"), "API span is not SERVER")
|
||||
worker_result = wait_logs("messenger-worker", upload["attachment_id"], lambda record:
|
||||
record.get("event") == "job.complete" and record.get("outcome") == "success",
|
||||
"worker completion log")
|
||||
job_logs = [record for record in worker_result[1] if record.get("event") == "job.complete"]
|
||||
worker_log = next(record for record in job_logs if record.get("outcome") == "success")
|
||||
check(bool(worker_log.get("job_id")), "Worker log does not identify logical job")
|
||||
worker_tid = hex_id(worker_log.get("trace_id"))
|
||||
worker_trace = wait("worker trace", lambda: trace(worker_tid))
|
||||
matching_worker = [entry for entry in spans(worker_trace) if entry["service"] == "messenger-worker"
|
||||
and hex_id(entry["span"].get("spanId")) == hex_id(worker_log.get("span_id"))]
|
||||
check(bool(matching_worker), "Worker log span is absent from Tempo trace")
|
||||
check(str(matching_worker[0]["span"].get("kind")) in ("5", "SPAN_KIND_CONSUMER"), "Worker span is not CONSUMER")
|
||||
check(all(int(entry["span"].get("endTimeUnixNano", 0)) > int(entry["span"].get("startTimeUnixNano", 0))
|
||||
for entry in matching_api + matching_worker), "Trace contains zero-duration synthetic span")
|
||||
if require_chain:
|
||||
check(api_tid == worker_tid, "Upload and worker are not in the same propagated trace")
|
||||
def complete_chain():
|
||||
value = trace(api_tid)
|
||||
return value if value and any(str(entry["span"].get("kind")) in ("4", "SPAN_KIND_PRODUCER")
|
||||
for entry in spans(value)) else None
|
||||
chain = wait("complete producer/consumer trace", complete_chain)
|
||||
names = [entry["span"].get("name", "") for entry in spans(chain)]
|
||||
check(any(name.startswith("s3.") for name in names), "S3 operation is absent from upload trace")
|
||||
check(any(name.startswith("db.") for name in names), "DB operation is absent from upload trace")
|
||||
def has_parent_chain(value):
|
||||
nodes = {hex_id(entry["span"].get("spanId")): entry["span"] for entry in spans(value)}
|
||||
server_id = hex_id(api_log.get("span_id"))
|
||||
for record in job_logs:
|
||||
consumer_id = hex_id(record.get("span_id"))
|
||||
consumer = nodes.get(consumer_id)
|
||||
if consumer is None:
|
||||
return False
|
||||
producer = nodes.get(hex_id(consumer.get("parentSpanId")))
|
||||
if producer is None or str(producer.get("kind")) not in ("4", "SPAN_KIND_PRODUCER"):
|
||||
return False
|
||||
current, visited = consumer_id, set()
|
||||
while current != server_id:
|
||||
if not current or current in visited or current not in nodes:
|
||||
return False
|
||||
visited.add(current)
|
||||
current = hex_id(nodes[current].get("parentSpanId"))
|
||||
return True
|
||||
chain = wait("actual producer/consumer parents lead to upload SERVER span", lambda:
|
||||
value if (value := trace(api_tid)) and has_parent_chain(value) else None)
|
||||
if require_retry:
|
||||
check(any(record.get("outcome") == "error" for record in job_logs), "No failed worker attempt was recorded")
|
||||
check(len({record.get("attempt") for record in job_logs}) >= 2, "Retry did not create a later attempt")
|
||||
check(all(hex_id(record.get("trace_id")) == api_tid for record in job_logs), "Retry broke propagated trace")
|
||||
api_trace = worker_trace = chain
|
||||
save(prefix + "-api-logs.json", api_result[0])
|
||||
save(prefix + "-worker-logs.json", worker_result[0])
|
||||
save(prefix + "-api-trace.json", api_trace)
|
||||
save(prefix + "-worker-trace.json", worker_trace)
|
||||
return {**upload, "api_trace_id": api_tid, "worker_trace_id": worker_tid,
|
||||
"job_id": worker_log["job_id"], "worker_attempts": [record.get("attempt") for record in job_logs],
|
||||
"same_trace": api_tid == worker_tid, "correlated_at": now()}
|
||||
|
||||
|
||||
def metric_snapshot():
|
||||
expressions = {
|
||||
"requests": "messenger_http_requests_total",
|
||||
"http_histogram": "messenger_http_duration_seconds_bucket",
|
||||
"jobs": "messenger_jobs_processed_total",
|
||||
"job_histogram": "messenger_jobs_duration_seconds_bucket",
|
||||
}
|
||||
result = {name: query(expression) for name, expression in expressions.items()}
|
||||
forbidden_labels = {"user_id", "chat_id", "message_id", "attachment_id", "request_id", "filename", "url"}
|
||||
for rows in result.values():
|
||||
for row in rows:
|
||||
check(not forbidden_labels.intersection(row["metric"]), "High-cardinality metric labels")
|
||||
route = row["metric"].get("route", "")
|
||||
check("?" not in route and not re.search(r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}", route),
|
||||
"Raw URL label")
|
||||
# SDK export and Prometheus scrape run independently. An empty vector is
|
||||
# temporarily missing evidence, while invalid labels above are a failure.
|
||||
if not all(result.values()):
|
||||
return None
|
||||
if not any(float(row["value"][1]) > 0 for row in result["requests"]):
|
||||
return None
|
||||
if not any(row["metric"].get("outcome") == "success" and float(row["value"][1]) > 0
|
||||
for row in result["jobs"]):
|
||||
return None
|
||||
return {"captured_at": now(), "queries": expressions, "responses": result}
|
||||
|
||||
|
||||
def baseline(client):
|
||||
check(get(GRAFANA + "/api/health").get("database") == "ok", "Grafana database is unhealthy")
|
||||
upload = client.upload()
|
||||
wait("baseline attachment ready", lambda: client.ready(upload), timeout=90)
|
||||
correlation = correlated(upload)
|
||||
metrics = wait("all application signals", lambda: metric_snapshot())
|
||||
result = {"started_at": datetime.fromtimestamp(START_NS / 1e9, timezone.utc).isoformat(),
|
||||
"finished_at": now(), "scenario": "real control upload", "correlation": correlation, "metrics": metrics}
|
||||
save("observability-baseline.json", result)
|
||||
print("PASS: real API/worker logs, traces and metrics; evidence/observability-baseline.json", flush=True)
|
||||
|
||||
|
||||
def chain_with_retry(client, stand):
|
||||
stand.stop("worker")
|
||||
upload = client.upload()
|
||||
try:
|
||||
stand.stop("s3")
|
||||
stand.start("worker")
|
||||
wait("failed attempt during S3 outage", lambda: any(
|
||||
record.get("event") == "job.complete" and record.get("outcome") == "error"
|
||||
for record in logs("messenger-worker", upload["attachment_id"])[1]), timeout=120)
|
||||
finally:
|
||||
if "s3" in stand.stopped:
|
||||
stand.start("s3")
|
||||
if "worker" in stand.stopped:
|
||||
stand.start("worker")
|
||||
wait("retry attachment ready", lambda: client.ready(upload), timeout=180)
|
||||
result = correlated(upload, "retry", require_chain=True, require_retry=True)
|
||||
save("observability-retry.json", {"scenario": "S3 unavailable during first attempt, then restored", **result,
|
||||
"content": client.verify_content(upload)})
|
||||
print("PASS: propagated upload/publish/consume/S3/DB trace including retry", flush=True)
|
||||
|
||||
|
||||
def alert(name):
|
||||
payload = get(PROMETHEUS + "/api/v1/alerts")
|
||||
return [item for item in payload["data"]["alerts"] if item["labels"].get("alertname") == name]
|
||||
|
||||
|
||||
def alert_state(name, state):
|
||||
current = alert(name)
|
||||
return current if state == "firing" and any(item["state"] == "firing" for item in current) else (
|
||||
{"alertname": name, "state": "resolved", "observed_at": now()} if state == "resolved" and not current else None)
|
||||
|
||||
|
||||
def lag_incident(client, stand, name):
|
||||
wait("clean queue and inactive lag alert before injection", lambda:
|
||||
number('max(messenger_queue_depth{service_name="messenger-publisher"})') == 0 and
|
||||
alert_state(name, "resolved"), timeout=180)
|
||||
started = now()
|
||||
stand.stop("worker")
|
||||
try:
|
||||
uploads = [client.upload() for _ in range(4)]
|
||||
firing = wait("queue lag alert firing", lambda: alert_state(name, "firing"), timeout=180)
|
||||
backlog = query('{__name__=~"messenger_queue_(depth|oldest_age.*|sample_age.*)"}')
|
||||
check(bool(backlog), "Queue gauges are absent during worker outage")
|
||||
finally:
|
||||
stand.start("worker")
|
||||
wait("queue drain", lambda: all(client.ready(upload) for upload in uploads), timeout=180)
|
||||
resolved = wait("queue lag alert resolved", lambda: alert_state(name, "resolved"), timeout=180)
|
||||
return {"incident": "worker stopped", "started_at": started, "finished_at": now(),
|
||||
"accepted_attachments": [item["attachment_id"] for item in uploads], "firing": firing,
|
||||
"backlog": backlog, "resolved": resolved,
|
||||
"content": [client.verify_content(upload) for upload in uploads]}
|
||||
|
||||
|
||||
def direct_db_failure(container_id):
|
||||
# Runs inside the existing API container so HAProxy readiness cannot mask
|
||||
# the API's real dependency error. It prints only status and correlation IDs.
|
||||
code = '''import json,urllib.request,urllib.error,uuid
|
||||
body=json.dumps({"username":"incident_"+uuid.uuid4().hex[:16],"display_name":"Synthetic DB incident","password":"SyntheticPasswordForLocalIncidentOnly"}).encode()
|
||||
req=urllib.request.Request("http://127.0.0.1:8080/api/v1/users",data=body,headers={"Content-Type":"application/json"},method="POST")
|
||||
try:r=urllib.request.urlopen(req,timeout=10)
|
||||
except urllib.error.HTTPError as e:r=e
|
||||
with r:
|
||||
p=json.loads(r.read());print(json.dumps({"status":r.status,"request_id":r.headers.get("X-Request-Id"),"trace_id":r.headers.get("X-Trace-Id"),"error_code":p.get("error",{}).get("code")}))
|
||||
'''
|
||||
result = subprocess.run(["docker", "exec", container_id, "python", "-c", code], cwd=ROOT,
|
||||
capture_output=True, text=True, timeout=20)
|
||||
check(result.returncode == 0, "Direct API request during DB outage failed before receiving HTTP")
|
||||
value = json.loads(result.stdout)
|
||||
check(value["status"] == 503 and value["error_code"] == "unavailable", "DB outage must produce API 503")
|
||||
check(bool(value["request_id"]), "DB error has no request ID")
|
||||
return value
|
||||
|
||||
|
||||
def db_incident(client, stand, name):
|
||||
wait("inactive HTTP error alert before injection", lambda: alert_state(name, "resolved"), timeout=120)
|
||||
started = now()
|
||||
containers = compose("ps", "-q", "api").splitlines()
|
||||
check(len(containers) >= 2, "DB incident requires two actual API containers")
|
||||
samples = []
|
||||
stopped = threading.Event()
|
||||
def flood(container):
|
||||
while not stopped.is_set():
|
||||
samples.append(direct_db_failure(container))
|
||||
stopped.wait(0.5)
|
||||
stand.stop("db")
|
||||
try:
|
||||
with ThreadPoolExecutor(max_workers=len(containers)) as pool:
|
||||
futures = [pool.submit(flood, container) for container in containers]
|
||||
try:
|
||||
def firing_or_error():
|
||||
for future in futures:
|
||||
if future.done():
|
||||
future.result()
|
||||
return alert_state(name, "firing")
|
||||
firing = wait("HTTP error alert firing", firing_or_error, timeout=180)
|
||||
database_metrics = query('{__name__=~"messenger_db_pool_.*"}')
|
||||
check(bool(database_metrics), "DB pool metrics are absent")
|
||||
finally:
|
||||
stopped.set()
|
||||
finally:
|
||||
stand.start("db")
|
||||
wait("API ready after DB recovery", lambda: http("GET", BASE + "/health/ready")[0] == 200, timeout=120)
|
||||
client.message()
|
||||
resolved = wait("HTTP error alert resolved", lambda: alert_state(name, "resolved"), timeout=180)
|
||||
check(bool(samples), "No API dependency errors captured")
|
||||
first = samples[0]
|
||||
raw_logs, records = wait_logs("messenger-api", first["request_id"], lambda _record: True,
|
||||
"DB error correlation log")
|
||||
trace_id = first.get("trace_id") or next((row.get("trace_id") for row in records if row.get("trace_id")), None)
|
||||
check(bool(trace_id), "DB error log has no trace ID")
|
||||
data = wait("DB error trace", lambda: trace(hex_id(trace_id)))
|
||||
check(any(entry["span"].get("name", "").startswith("db.") and
|
||||
str(entry["span"].get("status", {}).get("code")) in ("2", "STATUS_CODE_ERROR")
|
||||
for entry in spans(data)), "No failing DB operation in dependency-error trace")
|
||||
save("incident-db-logs.json", raw_logs)
|
||||
save("incident-db-trace.json", data)
|
||||
return {"incident": "PostgreSQL unavailable", "started_at": started, "finished_at": now(),
|
||||
"api_error_responses": samples, "firing": firing, "db_pool": database_metrics, "resolved": resolved}
|
||||
|
||||
|
||||
def collector_snapshot():
|
||||
return query('{__name__=~"otelcol_(exporter_(send_failed|enqueue_failed|queue).*|process_memory_rss.*)"}')
|
||||
|
||||
|
||||
def telemetry_incident(client, stand):
|
||||
before = collector_snapshot()
|
||||
check(bool(before), "Collector self metrics are absent")
|
||||
started = now()
|
||||
samples = []
|
||||
controls = []
|
||||
successes = 0
|
||||
stand.stop("tempo", "loki")
|
||||
try:
|
||||
deadline = time.monotonic() + 65
|
||||
while time.monotonic() < deadline:
|
||||
controls.append(client.message("telemetry-loss"))
|
||||
successes += 1
|
||||
if not samples or time.time() - samples[-1]["time"] >= 5:
|
||||
samples.append({"time": time.time(), "observed_at": now(), "metrics": collector_snapshot()})
|
||||
time.sleep(0.15)
|
||||
after = collector_snapshot()
|
||||
def failure_count(rows):
|
||||
return sum(float(row["value"][1]) for row in rows
|
||||
if "send_failed" in row["metric"].get("__name__", ""))
|
||||
check(failure_count(after) > failure_count(before), "No backend export failures observed")
|
||||
rss = [float(row["value"][1]) for sample in samples for row in sample["metrics"]
|
||||
if "process_memory_rss" in row["metric"].get("__name__", "")]
|
||||
check(bool(rss) and max(rss) < 256 * 1024 * 1024, "Collector RSS missing or exceeds 256 MiB limit")
|
||||
for sample in samples:
|
||||
capacity = {row["metric"].get("exporter"): float(row["value"][1]) for row in sample["metrics"]
|
||||
if "queue_capacity" in row["metric"].get("__name__", "")}
|
||||
queue_sizes = [row for row in sample["metrics"] if "queue_size" in row["metric"].get("__name__", "")]
|
||||
check(bool(capacity) and bool(queue_sizes), "Bounded exporter queue metrics are absent")
|
||||
for row in queue_sizes:
|
||||
check(float(row["value"][1]) <= capacity[row["metric"].get("exporter")], "Exporter queue exceeds configured capacity")
|
||||
finally:
|
||||
stand.start("tempo", "loki")
|
||||
wait_backends()
|
||||
client.message()
|
||||
upload = client.upload()
|
||||
wait("job remains functional after telemetry recovery", lambda: client.ready(upload), timeout=90)
|
||||
correlation = correlated(upload, "telemetry-recovery", require_chain=True)
|
||||
def queues_drained():
|
||||
value = number('sum(otelcol_exporter_queue_size{exporter=~"otlphttp/.*"})')
|
||||
return value is not None and value == 0
|
||||
wait("Collector export queues drained after recovery", queues_drained)
|
||||
# Queued metrics can be zero while the last two batches are in flight.
|
||||
time.sleep(5)
|
||||
availability = []
|
||||
for control in controls:
|
||||
data = wait("control trace lookup", lambda: (trace(control["trace_id"]),), timeout=60)[0]
|
||||
server_present = bool(data) and any(
|
||||
entry["service"] == "messenger-api" and
|
||||
str(entry["span"].get("kind")) in ("2", "SPAN_KIND_SERVER")
|
||||
for entry in spans(data))
|
||||
availability.append({**control, "trace_available": data is not None,
|
||||
"server_span_available": server_present})
|
||||
log_query = '{service_name="messenger-api"} | request_id=~"telemetry-loss-.*"'
|
||||
raw_logs = get(LOKI + "/loki/api/v1/query_range?" + urllib.parse.urlencode({
|
||||
"query": log_query, "start": START_NS, "end": time.time_ns(), "limit": 5000}))
|
||||
present_ids = set()
|
||||
for stream in raw_logs["data"]["result"]:
|
||||
for sample in stream.get("values", []):
|
||||
metadata = {**stream.get("stream", {}), **(sample[2] if len(sample) > 2 else {})}
|
||||
if sample[1] == "http.request.complete":
|
||||
present_ids.add(metadata.get("request_id"))
|
||||
for control in availability:
|
||||
control["log_available"] = control["request_id"] in present_ids
|
||||
save("telemetry-loss-control-logs.json", raw_logs)
|
||||
losses = {"expected_control_requests": len(availability),
|
||||
"missing_control_traces": sum(not x["trace_available"] for x in availability),
|
||||
"missing_control_server_spans": sum(not x["server_span_available"] for x in availability),
|
||||
"missing_control_logs": sum(not x["log_available"] for x in availability),
|
||||
"checked_at": now(), "controls": availability}
|
||||
return {"incident": "Tempo and Loki unavailable", "started_at": started, "finished_at": now(),
|
||||
"successful_message_writes_during_outage": successes, "before": before, "samples": samples,
|
||||
"after": after, "max_collector_rss_bytes": max(rss), "recovery": correlation,
|
||||
"loss_measurement": losses,
|
||||
"interpretation": "send_failed counts failed export attempts, not unique lost spans/log records; "
|
||||
"loss_measurement counts absent control request traces/logs after recovery and queue drain, "
|
||||
"not all internal spans. Business writes stayed successful."}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--baseline", action="store_true")
|
||||
parser.add_argument("--chain-with-retry", action="store_true")
|
||||
parser.add_argument("--incidents", action="store_true")
|
||||
parser.add_argument("--queue-alert", default="MessengerQueueLag")
|
||||
parser.add_argument("--http-alert", default="MessengerHighErrorRate")
|
||||
args = parser.parse_args()
|
||||
check(args.baseline or args.chain_with_retry or args.incidents, "Choose at least one scenario")
|
||||
client, stand = Business(), Stand()
|
||||
try:
|
||||
wait_backends()
|
||||
wait("business API readiness", lambda: http("GET", BASE + "/health/ready", timeout=3)[0] == 200,
|
||||
timeout=120)
|
||||
client.bootstrap()
|
||||
if args.baseline:
|
||||
baseline(client)
|
||||
if args.chain_with_retry:
|
||||
chain_with_retry(client, stand)
|
||||
if args.incidents:
|
||||
incidents = []
|
||||
for scenario in (lambda: lag_incident(client, stand, args.queue_alert),
|
||||
lambda: db_incident(client, stand, args.http_alert),
|
||||
lambda: telemetry_incident(client, stand)):
|
||||
incidents.append(scenario())
|
||||
save("observability-incidents.json", {"captured_at": now(), "incidents": incidents})
|
||||
print("PASS: " + incidents[-1]["incident"], flush=True)
|
||||
finally:
|
||||
stand.restore()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except (AssertionError, OSError, ValueError, subprocess.SubprocessError, urllib.error.URLError) as exc:
|
||||
# Never print backend bodies, exception URLs or subprocess output.
|
||||
print("FAIL: " + (str(exc) if isinstance(exc, AssertionError) else type(exc).__name__), flush=True)
|
||||
raise SystemExit(1)
|
||||
@@ -0,0 +1,42 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run checks sequentially and save UTF-8 evidence on Windows/Linux.
|
||||
|
||||
Start Compose first. --incidents also stops/restores worker, DB and telemetry
|
||||
backends. Run only against the local synthetic stand, without concurrent tests.
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
from test_lab5 import ROOT, run, snapshot, tester
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--incidents", action="store_true")
|
||||
args = parser.parse_args()
|
||||
os.chdir(ROOT)
|
||||
for stream in (sys.stdout, sys.stderr):
|
||||
if hasattr(stream, "reconfigure"):
|
||||
stream.reconfigure(encoding="utf-8", errors="replace")
|
||||
run(["docker", "compose", "config", "--quiet"], "compose-validation.txt")
|
||||
run([sys.executable, "-X", "utf8", "scripts/check_contract.py"], "check.txt")
|
||||
run(tester("make", "check", "test"), "make-check-test.txt")
|
||||
run(["docker", "compose", "exec", "-T", "api", "python", "-"], "observability-unit.txt",
|
||||
stdin_file=ROOT / "student-tests/observability_unit.py")
|
||||
run([sys.executable, "-X", "utf8", "scripts/observe_lab6.py", "--baseline"], "observability-baseline.txt")
|
||||
if args.incidents:
|
||||
snapshot("Before lab6 incidents", append=False)
|
||||
try:
|
||||
run([sys.executable, "-X", "utf8", "scripts/observe_lab6.py", "--chain-with-retry"],
|
||||
"observability-retry.txt")
|
||||
run([sys.executable, "-X", "utf8", "scripts/observe_lab6.py", "--incidents"],
|
||||
"observability-incidents.txt")
|
||||
finally:
|
||||
snapshot("After lab6 incidents")
|
||||
print("PASS: lab6 checks; see evidence/.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -106,9 +106,14 @@ def main():
|
||||
|
||||
path = f"/api/v1/chats/{chat['id']}/attachments?filename=bad.png"
|
||||
for data, mime, expected in ((b"this is not a PNG", "image/png", 415),
|
||||
(source, "image/png", 415),
|
||||
(b"x" * (10485760 + 1), "image/png", 413)):
|
||||
(source, "image/png", 415)):
|
||||
client.call("POST", path, expected, actor=alice, raw=data, headers={"Content-Type": mime})
|
||||
# Verify early rejection from Content-Length without writing 10 MiB after
|
||||
# the server has already returned 413 and closed the connection. urllib
|
||||
# writes the whole body before reading and can otherwise lose the response
|
||||
# to BrokenPipe. No upload is retried, and public tests are unchanged.
|
||||
client.call("POST", path, 413, actor=alice, headers={
|
||||
"Content-Type": "image/png", "Content-Length": str(10485760 + 1)})
|
||||
def png_chunk(kind, data):
|
||||
return struct.pack(">I", len(data)) + kind + data + struct.pack(">I", zlib.crc32(kind + data))
|
||||
bad_filter = (b"\x89PNG\r\n\x1a\n"
|
||||
|
||||
@@ -0,0 +1,222 @@
|
||||
"""OTel contract/security checks without a running DB, broker or backends.
|
||||
|
||||
Run in the application image, where the pinned SDK/FastAPI are installed:
|
||||
docker compose run --rm --no-deps -v ./:/work -w /work api python student-tests/observability_unit.py
|
||||
Only synthetic markers are used; no external exporter is configured.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
import sys
|
||||
from types import SimpleNamespace
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
# app.errors reads configuration to build response headers; no connection is made.
|
||||
os.environ.setdefault("DATABASE_URL", "postgresql://unit:unit@unused/unit")
|
||||
os.environ.setdefault("S3_ACCESS_KEY_ID", "synthetic-unit-key")
|
||||
os.environ.setdefault("S3_SECRET_ACCESS_KEY", "synthetic-unit-secret")
|
||||
|
||||
from opentelemetry import metrics, trace
|
||||
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
|
||||
from opentelemetry.sdk._logs.export import InMemoryLogExporter, SimpleLogRecordProcessor
|
||||
from opentelemetry.sdk.metrics import MeterProvider
|
||||
from opentelemetry.sdk.metrics.export import InMemoryMetricReader
|
||||
from opentelemetry.sdk.resources import Resource
|
||||
from opentelemetry.sdk.trace import TracerProvider
|
||||
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
|
||||
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
|
||||
from opentelemetry.trace import StatusCode
|
||||
|
||||
from app import telemetry
|
||||
from app.trace_context import carrier, parent_context
|
||||
|
||||
|
||||
class ObservabilityTests(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
resource = Resource({"service.name": "messenger-unit"})
|
||||
cls.spans = InMemorySpanExporter()
|
||||
cls.traces = TracerProvider(resource=resource)
|
||||
cls.traces.add_span_processor(SimpleSpanProcessor(cls.spans))
|
||||
trace.set_tracer_provider(cls.traces)
|
||||
cls.reader = InMemoryMetricReader()
|
||||
cls.meters = MeterProvider(resource=resource, metric_readers=[cls.reader])
|
||||
metrics.set_meter_provider(cls.meters)
|
||||
cls.logs = InMemoryLogExporter()
|
||||
cls.log_provider = LoggerProvider(resource=resource)
|
||||
cls.log_provider.add_log_record_processor(SimpleLogRecordProcessor(cls.logs))
|
||||
cls.logger = logging.getLogger("messenger.telemetry.events")
|
||||
cls.handler = LoggingHandler(logger_provider=cls.log_provider)
|
||||
cls.logger.addHandler(cls.handler)
|
||||
|
||||
@classmethod
|
||||
def tearDownClass(cls):
|
||||
cls.logger.removeHandler(cls.handler)
|
||||
cls.meters.shutdown()
|
||||
cls.log_provider.shutdown()
|
||||
cls.traces.shutdown()
|
||||
|
||||
def setUp(self):
|
||||
self.spans.clear()
|
||||
self.logs.clear()
|
||||
|
||||
def test_w3c_parent_child_and_durable_retry_preserve_trace(self):
|
||||
initial = {
|
||||
"traceparent": "00-12345678901234567890123456789012-1234567890123456-01",
|
||||
"tracestate": "vendor=unit",
|
||||
"Authorization": "Bearer MUST_NOT_PROPAGATE",
|
||||
}
|
||||
with telemetry.span("upload", parent=parent_context(initial)) as upload:
|
||||
durable = carrier()
|
||||
self.assertEqual(set(durable), {"traceparent", "tracestate"})
|
||||
self.assertEqual(durable["tracestate"], "vendor=unit")
|
||||
with telemetry.span("publish", parent=parent_context(durable)) as producer:
|
||||
envelope = carrier()
|
||||
self.assertEqual(producer.get_span_context().trace_id, upload.get_span_context().trace_id)
|
||||
with telemetry.span("consume", parent=parent_context(envelope)) as consumer:
|
||||
retry = carrier()
|
||||
with telemetry.span("retry publish", parent=parent_context(retry)) as retried:
|
||||
self.assertEqual(retried.get_span_context().trace_id, upload.get_span_context().trace_id)
|
||||
by_name = {item.name: item for item in self.spans.get_finished_spans()}
|
||||
self.assertEqual(by_name["publish"].parent.span_id, upload.get_span_context().span_id)
|
||||
self.assertEqual(by_name["consume"].parent.span_id, producer.get_span_context().span_id)
|
||||
self.assertEqual(by_name["retry publish"].parent.span_id, consumer.get_span_context().span_id)
|
||||
self.assertTrue(envelope["traceparent"].endswith("-01"))
|
||||
|
||||
def test_unsampled_trace_flags_and_tracestate_survive(self):
|
||||
initial = {
|
||||
"traceparent": "00-12345678901234567890123456789012-1234567890123456-00",
|
||||
"tracestate": "vendor=unit",
|
||||
}
|
||||
with telemetry.span("unsampled", parent=parent_context(initial)) as active:
|
||||
propagated = carrier()
|
||||
self.assertEqual(active.get_span_context().trace_id, int(initial["traceparent"].split("-")[1], 16))
|
||||
self.assertEqual(propagated["tracestate"], "vendor=unit")
|
||||
self.assertTrue(propagated["traceparent"].endswith("-00"))
|
||||
self.assertEqual(self.spans.get_finished_spans(), ())
|
||||
|
||||
def test_missing_or_invalid_context_starts_new_root_even_inside_span(self):
|
||||
with telemetry.span("unrelated maintenance") as ambient:
|
||||
for candidate in (None, {}, {"traceparent": None}, {"traceparent": "broken"},
|
||||
{"traceparent": "00-" + "0" * 32 + "-1234567890123456-01"}):
|
||||
with telemetry.span("new root", parent=parent_context(candidate)) as active:
|
||||
self.assertNotEqual(active.get_span_context().trace_id, ambient.get_span_context().trace_id)
|
||||
roots = [item for item in self.spans.get_finished_spans() if item.name == "new root"]
|
||||
self.assertEqual(len(roots), 5)
|
||||
self.assertTrue(all(item.parent is None for item in roots))
|
||||
self.assertEqual(carrier(), {})
|
||||
|
||||
def _request(self, *, template=None, status=202, failure=None):
|
||||
secrets = {
|
||||
"raw_id": "3d7b756a-cb54-4ce5-a7a8-83aa92352cee",
|
||||
"filename": "PRIVATE_FILENAME.png",
|
||||
"token": "PRIVATE_BEARER_TOKEN",
|
||||
"body": "PRIVATE_MESSAGE_BODY",
|
||||
"exception": "PRIVATE_EXCEPTION_CONNECTION_URL",
|
||||
}
|
||||
received = []
|
||||
responses = []
|
||||
scope = {
|
||||
"type": "http", "asgi": {"version": "3.0"}, "http_version": "1.1",
|
||||
"method": "POST", "scheme": "http", "root_path": "",
|
||||
"path": f"/api/v1/chats/{secrets['raw_id']}/attachments",
|
||||
"raw_path": f"/api/v1/chats/{secrets['raw_id']}/attachments".encode(),
|
||||
"query_string": f"filename={secrets['filename']}".encode(),
|
||||
"headers": [(b"authorization", f"Bearer {secrets['token']}".encode()),
|
||||
(b"content-type", b"application/json"),
|
||||
(b"x-request-id", b"unit-request-correlation")],
|
||||
"client": ("127.0.0.1", 1234), "server": ("unit", 80),
|
||||
}
|
||||
|
||||
async def receive():
|
||||
received.append(True)
|
||||
return {"type": "http.request", "body": secrets["body"].encode(), "more_body": False}
|
||||
|
||||
async def send(message):
|
||||
responses.append(message)
|
||||
|
||||
async def application(scope, receive, send):
|
||||
if template:
|
||||
scope["route"] = SimpleNamespace(path=template)
|
||||
# Prove the same secret body is available to the handler, but absent
|
||||
# from the independently exported observations.
|
||||
message = await receive()
|
||||
assert message["body"] == secrets["body"].encode()
|
||||
if failure:
|
||||
raise RuntimeError(secrets["exception"])
|
||||
await send({"type": "http.response.start", "status": status, "headers": []})
|
||||
await send({"type": "http.response.body", "body": b"ok", "more_body": False})
|
||||
|
||||
run = telemetry.HttpTelemetryMiddleware(application)(scope, receive, send)
|
||||
if failure:
|
||||
with self.assertRaisesRegex(RuntimeError, secrets["exception"]):
|
||||
asyncio.run(run)
|
||||
else:
|
||||
asyncio.run(run)
|
||||
self.assertEqual(len(received), 1)
|
||||
return secrets, responses
|
||||
|
||||
def _exported(self):
|
||||
spans = self.spans.get_finished_spans()
|
||||
logs = [entry.log_record for entry in self.logs.get_finished_logs()]
|
||||
data = self.reader.get_metrics_data()
|
||||
point_attrs = []
|
||||
for resource_metrics in data.resource_metrics:
|
||||
for scope_metrics in resource_metrics.scope_metrics:
|
||||
for metric in scope_metrics.metrics:
|
||||
for point in metric.data.data_points:
|
||||
point_attrs.append(dict(point.attributes))
|
||||
value = {
|
||||
"spans": [{"name": item.name, "attrs": dict(item.attributes),
|
||||
"events": [{"name": event.name, "attrs": dict(event.attributes)}
|
||||
for event in item.events]} for item in spans],
|
||||
"logs": [{"body": item.body, "attrs": dict(item.attributes or {})} for item in logs],
|
||||
"metrics": point_attrs,
|
||||
}
|
||||
return spans, logs, point_attrs, json.dumps(value, default=str)
|
||||
|
||||
def test_http_route_template_and_request_trace_log_correlation_without_secrets(self):
|
||||
template = "/api/v1/chats/{chat_id}/attachments"
|
||||
secrets, responses = self._request(template=template)
|
||||
spans, logs, metric_attrs, exported = self._exported()
|
||||
self.assertEqual(len(spans), 1)
|
||||
self.assertEqual(spans[0].name, "POST " + template)
|
||||
self.assertTrue(metric_attrs)
|
||||
self.assertTrue(all(item["route"] in {template, "unmatched"} for item in metric_attrs))
|
||||
self.assertTrue(all(set(item) <= {"method", "route", "status_class"} for item in metric_attrs))
|
||||
self.assertEqual(len(logs), 1)
|
||||
self.assertEqual(logs[0].attributes["request_id"], "unit-request-correlation")
|
||||
self.assertEqual(logs[0].trace_id, spans[0].context.trace_id)
|
||||
self.assertEqual(logs[0].span_id, spans[0].context.span_id)
|
||||
headers = dict(responses[0]["headers"])
|
||||
self.assertEqual(headers[b"x-request-id"], b"unit-request-correlation")
|
||||
self.assertEqual(headers[b"x-trace-id"].decode(), f"{spans[0].context.trace_id:032x}")
|
||||
for value in secrets.values():
|
||||
self.assertNotIn(value, exported)
|
||||
|
||||
def test_unknown_route_and_exception_message_do_not_escape_into_telemetry(self):
|
||||
secrets, _ = self._request(status=404)
|
||||
_, _, _, exported = self._exported()
|
||||
self.assertEqual(self.spans.get_finished_spans()[0].name, "POST unmatched")
|
||||
for value in secrets.values():
|
||||
self.assertNotIn(value, exported)
|
||||
self.spans.clear()
|
||||
self.logs.clear()
|
||||
secrets, _ = self._request(failure=True)
|
||||
spans, logs, metric_attrs, exported = self._exported()
|
||||
self.assertEqual(spans[0].status.status_code, StatusCode.ERROR)
|
||||
self.assertEqual(spans[0].attributes["exception_type"], "RuntimeError")
|
||||
self.assertFalse(spans[0].events, "No exception stack/message should be auto-recorded")
|
||||
self.assertEqual(logs[0].attributes["status_code"], 500)
|
||||
self.assertTrue(any(item["status_class"] == "5xx" for item in metric_attrs))
|
||||
for value in secrets.values():
|
||||
self.assertNotIn(value, exported)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
Reference in New Issue
Block a user