forked from mtusi2026/mtusi2026lab6
Реализовать лабораторную 6 на 5: SLO и обнаружение инцидентов
This commit is contained in:
+11
-20
@@ -1,20 +1,11 @@
|
||||
.git
|
||||
.github
|
||||
.env
|
||||
.env.*
|
||||
!.env.example
|
||||
secrets
|
||||
certs
|
||||
volumes
|
||||
data
|
||||
backups
|
||||
*.key
|
||||
*.pem
|
||||
*.p12
|
||||
*.pfx
|
||||
*.jks
|
||||
.venv
|
||||
__pycache__
|
||||
node_modules
|
||||
evidence
|
||||
.DS_Store
|
||||
*
|
||||
!Dockerfile
|
||||
!requirements.txt
|
||||
!src/
|
||||
!src/**
|
||||
!migrations/
|
||||
!migrations/**
|
||||
!research/
|
||||
!research/**
|
||||
**/__pycache__/
|
||||
**/*.py[cod]
|
||||
|
||||
+10
-2
@@ -3,11 +3,12 @@
|
||||
APP_HOST=0.0.0.0
|
||||
APP_PORT=8080
|
||||
APP_ENV=lab
|
||||
HOST_PORT=8080
|
||||
SESSION_TTL_SECONDS=3600
|
||||
# Choose credentials locally. File paths contain no secret values.
|
||||
DATABASE_URL_FILE=/run/secrets/database_url
|
||||
# Optional on 3, required use on 4 in lab2; Redis OR Valkey:
|
||||
SESSION_STORE=postgres
|
||||
# Shared sessions carried over from lab4 use Valkey.
|
||||
SESSION_STORE=valkey
|
||||
CACHE_URL_FILE=/run/secrets/cache_url
|
||||
API_REPLICAS=2
|
||||
IDEMPOTENCY_TTL_SECONDS=86400
|
||||
@@ -18,13 +19,20 @@ S3_ACCESS_KEY_ID_FILE=/run/secrets/s3_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE=/run/secrets/s3_secret_access_key
|
||||
MAX_UPLOAD_BYTES=10485760
|
||||
MAX_IMAGE_PIXELS=40000000
|
||||
UPLOAD_CLEANUP_GRACE_SECONDS=3600
|
||||
BROKER_URL_FILE=/run/secrets/broker_url
|
||||
WORKER_CONCURRENCY=2
|
||||
JOB_MAX_ATTEMPTS=5
|
||||
JOB_TIMEOUT_SECONDS=60
|
||||
STAGING_RETENTION_SECONDS=86400
|
||||
JOB_LEASE_SECONDS=90
|
||||
JOB_BACKOFF_SECONDS=2
|
||||
WORKER_TASK_MEMORY_MB=640
|
||||
OTEL_SERVICE_NAME=messenger-api
|
||||
# Worker must set a distinct service.name.
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
|
||||
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
|
||||
OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=lab
|
||||
GRAFANA_PORT=3000
|
||||
OTEL_SDK_DISABLED=false
|
||||
OTEL_PYTHON_SDK_INTERNAL_METRICS_ENABLED=true
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
## Лабораторная 5
|
||||
|
||||
Автор / группа: Савельев Иван / БВТ2304
|
||||
Целевая оценка (3/4/5): **4** — выполнены требования уровней 3 и 4.
|
||||
Трек 5, если заявлен: не заявлен.
|
||||
|
||||
### Что реализовано и что перенесено
|
||||
|
||||
Из lab4 перенесены API мессенджера, авторизация и сессии, ACL, идемпотентность сообщений, миграции, приватный S3, две API-реплики за HAProxy, Docker-конфигурация и собственные тесты.
|
||||
|
||||
- Загрузка возвращает `202 queued` после устойчивого сохранения оригинала в S3 staging и транзакции metadata + job + outbox. Статус доступен через GET; content до готовности возвращает 409.
|
||||
- Добавлены отдельные Valkey Streams broker с AOF и volume, outbox publisher и worker. Доставка at least once, ACK после устойчивого результата; дубли не создают второй набор объектов.
|
||||
- Worker сохраняет неизменённый оригинал, JPEG thumbnail, центрированный crop 128×128 и optimized WebP по контракту, с учётом EXIF orientation и прозрачности.
|
||||
- Реализованы пять автоматических попыток, backoff с jitter, lease/recovery, безопасные коды ошибок, реестр failed и явный replay с audit. Ready-задания повторно не запускаются.
|
||||
- Добавлены ограничения bytes/pixels/времени/памяти, ограниченная конкуренция и очистка staging после фиксации результата либо истечения retention failed.
|
||||
|
||||
### Запуск из чистого clone
|
||||
|
||||
Нужны Docker Engine с Compose v2, Python 3.11, Bash и make. Команды из корня репозитория:
|
||||
|
||||
```sh
|
||||
python3 -m venv .venv
|
||||
.venv/bin/python -m pip install -r requirements.txt -r requirements-dev.txt
|
||||
make check validate PYTHON=.venv/bin/python
|
||||
./scripts/start.sh
|
||||
./scripts/check.sh
|
||||
|
||||
# Полная проверка на отдельном временном Docker Compose project:
|
||||
./scripts/check-docker.sh
|
||||
|
||||
# Остановка обычного стенда с сохранением volumes:
|
||||
./scripts/stop.sh
|
||||
```
|
||||
|
||||
`start.sh` создаёт локальные `.env` и secrets, собирает приложение, выполняет миграции и bootstrap, ждёт готовности обеих API-реплик. API доступен на `http://localhost:8080`.
|
||||
|
||||
### Результаты проверок
|
||||
|
||||
Проверено 12.11.2026: `make check validate test`, 39 собственных API-тестов и все 17 групп полного сценария прошли. Воспроизведены остановка worker и рестарты API/broker, оба разрыва outbox, crash worker после PUT и перед ACK, дубли, исчерпание retry/replay, постоянная ошибка, timeout/MemoryError, очистка staging, недоступность broker, persistence и failover API. Отдельно подтверждён фактический restart всех 11 постоянных контейнеров с сохранением их ID и volumes.
|
||||
|
||||
- [Отчёт, архитектура, команды и ограничения](../REPORT.md).
|
||||
- [Полный вывод проверок](../evidence/lab5-checks.txt) и [JSON результатов](../evidence/lab5-checks.json).
|
||||
- [Доказательство рестартов](../evidence/lab5-restart.json) и [вывод persistence](../evidence/lab5-restart.txt).
|
||||
|
||||
### Ограничения / известные ошибки
|
||||
|
||||
Стенд учебный, на одном Docker-host; потеря host/disk/volumes и ручное удаление объектов не покрываются. Поддерживается pipeline_version=1; исследовательские треки на 5 не реализованы. При длительном простое worker outbox и staging требуют свободного диска; общей квоты принятой работы нет. Failed staging хранится 24 часа, после удаления replay запрещён. Превышение ресурсов обработки заканчивается failed после ограниченного числа попыток. Старые ready-вложения lab4 остаются доступны без автоматического создания вариантов.
|
||||
|
||||
### Источники и использование ИИ
|
||||
|
||||
Основа — ТЗ, OpenAPI, `attachment-states.md` и `jobs.schema.json` этого репозитория; официальная документация Redis Streams/XAUTOCLAIM, Valkey persistence, Pillow ImageOps и Python resource. Ссылки приведены в [REPORT.md](../REPORT.md#первичные-источники).
|
||||
|
||||
- [x] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
|
||||
- [x] Контракт и публичные тесты не ослаблены.
|
||||
- [x] make check и make test выполнены; приложен реальный вывод.
|
||||
- [x] Сценарии отказов/безопасности своего уровня воспроизведены.
|
||||
- [x] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
|
||||
- [x] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
|
||||
@@ -1,18 +1,55 @@
|
||||
## Лабораторная 6
|
||||
## Лабораторная 6 — наблюдаемость, оценка 5, трек 2
|
||||
|
||||
Автор / группа: TODO
|
||||
Целевая оценка (3/4/5): TODO
|
||||
Трек 5, если заявлен: TODO
|
||||
Автор / группа: Савельев Иван / БВТ2304.
|
||||
|
||||
Что реализовано и что перенесено: TODO
|
||||
Команды запуска из чистого clone: TODO
|
||||
Результаты проверок и ссылка на REPORT.md: TODO
|
||||
Ограничения / известные ошибки: TODO
|
||||
Источники и использование ИИ: TODO
|
||||
### Результат
|
||||
|
||||
- [ ] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
|
||||
- [ ] Контракт и публичные тесты не ослаблены.
|
||||
- [ ] make check и make test выполнены; приложен реальный вывод.
|
||||
- [ ] Сценарии отказов/безопасности своего уровня воспроизведены.
|
||||
- [ ] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
|
||||
- [ ] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
|
||||
Сохранён работающий pipeline lab5. Добавлены явный OTel SDK для API/worker/publisher,
|
||||
Collector, Prometheus, Loki, Tempo и Grafana с provisioning трёх источников и
|
||||
импортируемым dashboard на 19 панелей. W3C context хранится в durable job/outbox,
|
||||
проходит через broker и сохраняет цепочку при retry до S3/DB ready.
|
||||
|
||||
Реализованы SLO доступности API 99%/1h и готовности изображения 95% за 30s/1h.
|
||||
SLI изображений учитывает failed, late и не завершившиеся jobs. Fast/slow burn
|
||||
alerts используют пары 2m/30s и 6m/1m, guards и for; есть high error rate и queue lag.
|
||||
Исследование сравнивает короткий всплеск с длительными отказами DB/worker,
|
||||
измеряет detection/resolution и ложные срабатывания. Разбор причин использует
|
||||
сохранённые метрики, логи и реальные traces; запросы собраны в runbook.
|
||||
|
||||
### Запуск и проверки
|
||||
|
||||
```sh
|
||||
python3 -m venv .venv
|
||||
.venv/bin/python -m pip install -r requirements.txt -r requirements-dev.txt
|
||||
./scripts/start.sh
|
||||
./scripts/check-lab6.sh
|
||||
```
|
||||
|
||||
- API: http://localhost:8080.
|
||||
- Dashboard: http://localhost:3000/d/messenger-lab6, локальный Viewer.
|
||||
- [Отчёт, результаты, версии и ограничения](../REPORT.md).
|
||||
- [Runbook](../docs/runbook.md), [методика](../research/README.md).
|
||||
- API/39 собственных tests: `evidence/lab6-api-checks.txt`.
|
||||
- 17 групп сохранённых критериев lab5: `evidence/lab6-lab5-regression.json`.
|
||||
- Trace с retry и логи: `evidence/lab6-trace-retry.json`, `evidence/lab6-correlated-logs.json`.
|
||||
- Loki outage/ограниченная RAM/измеренные потери: `evidence/lab6-checks.json`.
|
||||
- Реальные restart/volumes: `evidence/lab6-restart.json`, `evidence/lab6-telemetry-persistence.json`.
|
||||
- Трек 2: `evidence/lab6-slo-research.json`, `evidence/lab6-slo-series.json`.
|
||||
- Проверки правил: `evidence/lab6-promtool.txt`.
|
||||
|
||||
### Ограничения
|
||||
|
||||
Один host; потеря диска/volumes не покрыта. Telemetry buffer ограничен и допускает
|
||||
измеряемые потери при недоступном backend. SLO периода час задан, но один короткий
|
||||
live эксперимент проверяет детектор, а не долгосрочную надёжность. Отказы LB/сети
|
||||
до API требуют отдельного внешнего SLI. Внешняя доставка alerts не требуется.
|
||||
|
||||
Источники — ТЗ и официальная документация OTel, W3C, Grafana/Prometheus и Google SRE
|
||||
Workbook; ссылки в REPORT.md. Реализация и проверки подготовлены с помощью Codex.
|
||||
|
||||
- [x] Минимум предыдущей работы сохранён.
|
||||
- [x] Обязательные контракты и публичные tests не изменены.
|
||||
- [x] Три сигнала поступают в реальные persistent stores; dashboard provisioned из Git.
|
||||
- [x] Секреты, тела сообщений/изображений и SQL/params не экспортируются.
|
||||
- [x] Локальные .env/secrets и runtime artifacts исключены из Git.
|
||||
- [x] Условия alerts, окна, for и runbook сохранены в Git.
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
name: Application
|
||||
on: [push, pull_request]
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
docker:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 35
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.11'
|
||||
- run: python -m venv .venv
|
||||
- run: .venv/bin/python -m pip install -r requirements.txt -r requirements-dev.txt
|
||||
- run: ./scripts/check-lab6.sh
|
||||
- uses: actions/upload-artifact@v4
|
||||
if: always()
|
||||
with:
|
||||
name: lab6-evidence
|
||||
path: |
|
||||
evidence/lab6-*.txt
|
||||
evidence/lab6-*.json
|
||||
+28
@@ -0,0 +1,28 @@
|
||||
FROM python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
|
||||
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
APP_HOST=0.0.0.0 \
|
||||
APP_PORT=8080
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY requirements.txt .
|
||||
RUN python -m pip install --no-cache-dir -r requirements.txt \
|
||||
&& python -m pip check
|
||||
|
||||
RUN groupadd --gid 10001 app \
|
||||
&& useradd --uid 10001 --gid app --no-create-home app
|
||||
|
||||
COPY src/ ./src/
|
||||
COPY migrations/ ./migrations/
|
||||
COPY research/ ./research/
|
||||
|
||||
USER 10001:10001
|
||||
EXPOSE 8080
|
||||
STOPSIGNAL SIGTERM
|
||||
|
||||
HEALTHCHECK --interval=2s --timeout=3s --start-period=5s --retries=3 \
|
||||
CMD ["python", "-m", "src.healthcheck"]
|
||||
|
||||
CMD ["python", "-m", "src"]
|
||||
+184
@@ -0,0 +1,184 @@
|
||||
# Отчёт — лабораторная 5
|
||||
|
||||
- Автор, группа: Савельев Иван, БВТ2304.
|
||||
- Целевая оценка: **4** — все требования уровней 3 и 4. Исследовательский трек оценки 5 не заявлен.
|
||||
- Стенд: Docker Compose; HAProxy, две реплики FastAPI, PostgreSQL, Valkey для сессий, SeaweedFS S3; отдельные Valkey Streams broker, outbox publisher и worker.
|
||||
- Commit/PR: после публикации. Evidence содержит source_commit и working_tree=true: проверялось рабочее дерево поверх указанного commit.
|
||||
- Использование ИИ: перенос базы, код, проверки и отчёт подготовлены с помощью Codex. Результаты проверок получены реальным запуском Docker, а не предполагаются по коду.
|
||||
|
||||
## Запуск из чистого clone
|
||||
|
||||
Нужны Docker Engine с Compose v2, Python 3.11, Bash и make. Все команды выполняются из корня репозитория. Внешние заранее установленные БД/брокер/S3 не нужны.
|
||||
|
||||
```sh
|
||||
python3 -m venv .venv
|
||||
.venv/bin/python -m pip install -r requirements.txt -r requirements-dev.txt
|
||||
make check validate PYTHON=.venv/bin/python
|
||||
./scripts/start.sh
|
||||
./scripts/check.sh
|
||||
```
|
||||
|
||||
`start.sh` создаёт локальные `.env` и `secrets/`, собирает образ, выполняет миграции и bootstrap приватного bucket, поднимает сервисы и ждёт обе API-реплики через LB. Существующие настройки и пароли сохраняются. `.env`, secrets, виртуальное окружение и runtime data исключены из Git; Docker context содержит только исходники, зависимости и миграции.
|
||||
|
||||
API: http://localhost:8080. Если порт занят, задайте `HOST_PORT=8081` в `.env`; HTTP-проверки запускайте с `BASE_URL=http://localhost:8081`. Compose project `mtusi-lab5`, образы `mtusi-messenger:lab5` и volumes отделены от предыдущих лабораторных. Локальные credentials генерируются заново, файлы из lab4 не копируются.
|
||||
|
||||
```sh
|
||||
# Полный автоматический сценарий на отдельном временном project и свободном порту:
|
||||
./scripts/check-docker.sh
|
||||
|
||||
# Обычный стенд, остановка без потери данных:
|
||||
./scripts/stop.sh
|
||||
./scripts/start.sh
|
||||
|
||||
# Перезапуск всех постоянных процессов и хранилищ с сохранением volumes:
|
||||
./scripts/restart.sh
|
||||
|
||||
# Только для намеренного удаления учебных данных ТЕКУЩЕГО Compose project:
|
||||
docker compose down --volumes
|
||||
```
|
||||
|
||||
Полный сценарий создаёт синтетические данные в отдельном `lab5-check-<random>` и удаляет только его контейнеры/volumes. Пишет `evidence/lab5-checks.txt` и `evidence/lab5-checks.json`. Для диагностики сбоя можно явно задать `KEEP_TEST_STACK=1`; это оставит только неуспешный временный стенд. Основной стенд не затрагивается.
|
||||
|
||||
## Что перенесено и изменено
|
||||
|
||||
Из lab4 сохранены регистрация/Argon2id, Basic-вход и Bearer-сессии в Valkey, ACL чатов и вложений, сообщения и keyset-пагинация, X-Request-Id/X-Instance-Id, общая идемпотентность сообщений, две API-реплики за LB, приватный S3, лимиты загрузки, миграции и эксплуатационные скрипты. Предыдущие отчёты — `REPORT.lab2.md`, `REPORT.lab3.md`, `REPORT.lab4.md`; старые evidence относятся к соответствующим работам.
|
||||
|
||||
Объявленный переход API: upload возвращает **202 со снимком queued**, а клиент опрашивает GET metadata. Queued/processing уже можно прикрепить к сообщению в том же чате. Для failed новое прикрепление даёт 409. Content любого не-ready вложения даёт 409, отсутствующий вариант ready — 404. Оригинал скачивается побайтно без перекодирования и redirect; API не раскрывает storage keys/credentials. Контракт, README задания и публичные tests не изменялись.
|
||||
|
||||
Миграции 001–003 сохранены. Миграция 004 добавляет состояния/metadata вариантов и таблицы jobs, outbox, audit событий. Старые ready-вложения сохраняют свои ключи и читаются после обновления; варианты у старых записей не создаются автоматически. Схема описана в `docs/schema.md`.
|
||||
|
||||
## Архитектура и границы устойчивости
|
||||
|
||||
```text
|
||||
Клиент → HAProxy → API ×2 → PostgreSQL: metadata + job + outbox
|
||||
→ приватный S3 staging
|
||||
|
||||
PostgreSQL outbox → publisher → Valkey Streams (AOF, volume)
|
||||
↓ XREADGROUP, prefetch=1
|
||||
worker (2 процесса)
|
||||
↓
|
||||
отдельный resize subprocess
|
||||
↓
|
||||
S3 original + thumbnail + crop + optimized
|
||||
↓
|
||||
PostgreSQL COMMIT ready → broker ACK
|
||||
|
||||
cleanup → orphan PUT journal + staging только ready/просроченных failed
|
||||
```
|
||||
|
||||
PostgreSQL — источник истины для намерения обработки и состояния. Valkey Streams обеспечивает at-least-once доставку. Это отдельный service `broker`, не Pub/Sub и не тот же экземпляр, что сессии. Stream `attachments:v1`, группа `image-workers`, начало чтения `0`; topology создаётся идемпотентным XGROUP CREATE MKSTREAM. Сообщения не trim-ятся до ACK. После XACK запись удаляется XDEL в одной broker transaction; используется одна группа.
|
||||
|
||||
1. API принимает все bytes с ограничением размера, проверяет имя, MIME и настоящий PNG/JPEG. Resize/генерация вариантов в API не выполняются.
|
||||
2. Перед PUT фиксируется журнал `attachment_uploads`. Полный проверенный оригинал записывается в S3 `attachments/staging/<UUID>`. Это тот же приватный S3, но отдельный staging prefix; на этом этапе конечного original/вариантов ещё нет.
|
||||
3. Одна транзакция PostgreSQL вставляет queued metadata, job и outbox и удаляет запись журнала. **202 отправляется после PUT и успешного COMMIT.** Локальный TemporaryFile нужен только во время HTTP-приёма, устойчивым staging не является.
|
||||
4. Отдельный publisher читает due outbox с блокировкой SKIP LOCKED и делает XADD. Broker настроен AOF `appendfsync always`, volume, `noeviction`. Успешный XADD подтверждает приём брокером с этой настройкой persistence; это не consumer ACK. После ответа dispatcher отмечает published_at.
|
||||
5. Worker захватывает job транзакционно: status=processing, случайный lease_token и deadline. Данные для скачивания берёт из БД, а не из сообщения/filename. Проверяет размер и SHA-256 staging. Записывает original и все три варианта, затем в одной транзакции ставит ready и metadata/checksums. Только после COMMIT подтверждает доставку.
|
||||
|
||||
В сообщении только поля `jobs.schema.json`: schema_version, job_id, type, attachment_id, pipeline_version=1, attempt, created_at. Bytes, URL, credentials, filename и storage key отсутствуют. job_id постоянен при redelivery и managed retry; attempt начинается с 1 и меняется при новой управляемой попытке. Native stream ID — отдельный идентификатор доставки.
|
||||
|
||||
## Идемпотентность, retry и replay
|
||||
|
||||
Ключ эффекта `(attachment_id, pipeline_version, variant)` реализован детерминированными объектами:
|
||||
|
||||
```text
|
||||
attachments/<UUID>/v1/original
|
||||
attachments/<UUID>/v1/thumbnail
|
||||
attachments/<UUID>/v1/crop
|
||||
attachments/<UUID>/v1/optimized
|
||||
```
|
||||
|
||||
Число объектов конечного результата — четыре. Дубль queued/processing не захватывает уже арендованную job. Устаревшая попытка не меняет актуальное состояние. Перед PUT и COMMIT worker блокирует строки job/attachment и сверяет lease_token и deadline. При повторе после незавершённого PUT те же детерминированные объекты могут записаться ещё раз, второй набор не создаётся. Дубль ready проверяет полную metadata и размеры объектов через S3 HEAD, затем ACK, не меняя status/updated_at и не выполняя resize. В этой работе поддерживается только pipeline_version=1; трек versioned pipeline на 5 не заявлен.
|
||||
|
||||
Временные ошибки S3, resource limit и timeout дают queued с новым due outbox. Backoff: `uniform(0.5,1.5) × min(60, JOB_BACKOFF_SECONDS × 2^(attempt-1))`; интервал сохраняется в PostgreSQL. Default base=2 s, jitter присутствует. Автоматический бюджет — `JOB_MAX_ATTEMPTS=5`. При исчерпании — failed/error_code=retry_exhausted; последняя безопасная причина сохраняется в job. Нарушение checksum/размера staging или невалидное изображение — постоянный invalid_image, сразу failed без повторов. Тексты исключений и stack traces не передаются в API/registry.
|
||||
|
||||
Publisher также восстанавливает истёкший processing lease: записывает worker_lost и создаёт следующую попытку либо failed при исчерпании бюджета. Это не бесконечный retry. Pending-доставки умершего consumer забираются XAUTOCLAIM после lease timeout. Если recovery уже создал новую попытку, старое сообщение становится obsolete и получает ACK; намерение следующей попытки уже устойчиво сохранено.
|
||||
|
||||
Реестр failed и явный replay:
|
||||
|
||||
```sh
|
||||
docker compose exec -T worker python -m src.job_admin failed
|
||||
docker compose exec -T worker python -m src.job_admin replay <job_id>
|
||||
```
|
||||
|
||||
Replay разрешён только для failed с ещё сохранённым staging. Он сохраняет job_id/ключ эффекта, записывает operator_replay в audit, увеличивает глобальный attempt и выделяет новый ограниченный бюджет из пяти попыток. Ready/queued/processing повторять этой командой нельзя; повтор ready завершается ошибкой без изменения БД/объектов. Глобальный attempt монотонен, поэтому старые сообщения не запускают replay как новую работу. После удаления failed staging replay запрещён: исходник надо загрузить заново. HTTP endpoint управления заданиями не добавлен.
|
||||
|
||||
## Правила изображений и ограничения ресурсов
|
||||
|
||||
- Оригинал остаётся исходными bytes, MIME, size_bytes и SHA-256 относятся к нему.
|
||||
- EXIF orientation применяется к производным изображениям. Прозрачность компонуется на белый; производные RGB, без EXIF/ICC/comment metadata.
|
||||
- Thumbnail: JPEG quality=85, вписывание в 256×256 без увеличения, целочисленное округление вниз с минимумом 1 px. Fixture 320×200 → 256×160.
|
||||
- Crop: JPEG quality=85, центрированный квадрат 128×128, LANCZOS; маленький исходник может увеличиваться.
|
||||
- Optimized: WebP quality=80, method=4; вписывание в 1280×1280 без увеличения. Уменьшение байтов каждого маленького файла не обещается.
|
||||
- Загрузка ≤10 MiB и ≤40 млн pixels. Limit проверяется до полной распаковки; Content-Length и фактический поток проверяются независимо. Corrupt/MIME spoof — 415, bytes/pixels overflow — 413. API допускает одну проверяемую загрузку на реплику, таймаут тела 30 s и ожидания слота 20 s.
|
||||
- Worker concurrency=2, каждый consumer получает максимум одну доставку за раз. Для заданного Compose memory budget разрешено concurrency 1 или 2.
|
||||
- Каждый resize выполняется отдельным subprocess: wall timeout 60 s, CPU limit 60/61 s, address-space limit 640 MiB (`RLIMIT_AS`). При timeout subprocess убивается; MemoryError/resource termination не убивает основной consumer и проходит bounded retry. Worker container ограничен 1536 MiB, 2 CPU и 96 PID. Lease по умолчанию 90 s и должен превышать resize timeout минимум на 5 s.
|
||||
- S3 операции имеют connect timeout 1 s/read timeout 3 s, SDK retries отключены. PostgreSQL I/O, pool wait и блокировки ограничены. CPU/resize выполняется вне DB-транзакции; транзакция публикации результата включает только fencing, четыре ограниченных PUT и COMMIT.
|
||||
|
||||
`WORKER_TASK_MEMORY_MB` ограничивает адресное пространство процесса, а Compose mem_limit — весь контейнер. Это разные ограничения; поддержка 40 млн pixels не обещает, что любой формат/ориентация уложится в память: превышение ресурсов заканчивается наблюдаемым failed после ограниченного бюджета, а не бесконечной обработкой.
|
||||
|
||||
## Staging retention и очистка
|
||||
|
||||
`cleanup` запускается раз в 30 s. Журнал orphan PUT старше `UPLOAD_CLEANUP_GRACE_SECONDS=3600` очищается под advisory lock: свежая/активная загрузка не удаляется. Crash после staging, но до metadata/job не дал 202; его журнал позволяет удалить orphan без обхода всего bucket.
|
||||
|
||||
Staging queued/processing не удаляется независимо от возраста. После безопасного COMMIT ready staging удаляется при ближайшем проходе; конечные четыре объекта сохраняются. Failed staging и частичные конечные объекты хранятся `STAGING_RETENTION_SECONDS=86400` после последнего перехода в failed, затем удаляются; metadata, failed registry и audit остаются. Операции DELETE идемпотентны, отметка staging_deleted_at фиксируется после удаления. Cleaner и replay блокируют те же строки, поэтому не удаляют staging текущей replay-работы.
|
||||
|
||||
## Сервисы, настройки и bootstrap
|
||||
|
||||
| Роль / service | Версия | Readiness / persistence / credentials |
|
||||
| --- | --- | --- |
|
||||
| API / api ×2 | Python 3.11.15; FastAPI 0.142.4 | non-root UID 10001, health live/ready; 384 MiB/1 CPU на реплику; Docker secrets |
|
||||
| БД / postgres | PostgreSQL 17.6-bookworm | pg_isready, postgres_data; отдельный migrate с checksum/advisory lock |
|
||||
| Сессии / valkey | Valkey 8.1.3 | PING, AOF always, valkey_data, внутренний порт |
|
||||
| Брокер / broker | Valkey 8.1.3 | авторизованный PING, AOF always, broker_data, отдельный пароль через secret |
|
||||
| Outbox / publisher | тот же собранный Python image | restart unless-stopped; DB+broker secrets; topology bootstrap идемпотентен |
|
||||
| Worker / worker | тот же Python image; Pillow 13.0.0 | restart supervisor, 2 consumer-процесса, leases; DB+broker+S3 secrets |
|
||||
| S3 / s3 + s3-storage | SeaweedFS 3.85 с digest | gateway probe + filer HTTP; s3_data; storage только в s3-internal |
|
||||
| S3 bootstrap / s3-bootstrap | MinIO SDK 7.2.20 | root secrets только для создания приватного bucket; повтор безопасен |
|
||||
| LB / lb | HAProxy 3.2.6-alpine | readiness backend; единственный порт 127.0.0.1:8080 |
|
||||
| Cleanup / cleanup | тот же Python image | restart unless-stopped, DB+S3 secrets; без broker credentials |
|
||||
|
||||
Python base image закреплён digest в Dockerfile; S3 image — digest в Compose. Все Python-зависимости закреплены в requirements.txt. Административные интерфейсы S3 и внутренние API/БД/broker ports наружу не опубликованы. S3 application credentials разрешают работу только своего bucket/prefix attachments/*; root используется только bootstrap. API/cleaner не получают broker password. `.env.example` содержит пути secret-файлов, а не их значения.
|
||||
|
||||
Соответствие настроек: APP_HOST/APP_PORT — адрес API внутри контейнера; HOST_PORT — порт LB; APP_ENV — учебная метка. SESSION_STORE=valkey и SESSION_TTL_SECONDS=3600 — общее session storage; CACHE_URL_FILE/DATABASE_URL_FILE — Docker secrets. API_REPLICAS=2; IDEMPOTENCY_TTL_SECONDS≥86400. S3_ENDPOINT/REGION/BUCKET и S3_*_FILE — приватный bucket. MAX_UPLOAD_BYTES/MAX_IMAGE_PIXELS и UPLOAD_CLEANUP_GRACE_SECONDS — приём и orphan cleanup. BROKER_URL_FILE — авторизованный Redis-compatible URL отдельного broker. WORKER_CONCURRENCY, WORKER_TASK_MEMORY_MB, JOB_TIMEOUT_SECONDS, JOB_LEASE_SECONDS, JOB_MAX_ATTEMPTS, JOB_BACKOFF_SECONDS — ограничения/повторы. STAGING_RETENTION_SECONDS — срок хранения failed исходников.
|
||||
|
||||
/health/ready API зависит от PostgreSQL, сессий и S3: они нужны синхронному пути. Broker/worker/publisher не являются зависимостями readiness API: при их остановке original+outbox надёжно принимаются, текстовые сообщения работают, resize синхронно не запускается. У фоновых процессов нет HTTP health endpoint; состояние jobs/pending/failed наблюдается через БД, broker и операторскую CLI.
|
||||
|
||||
## Доказательства критериев
|
||||
|
||||
Проверено 12.11.2026: все **17 групп проверок** и **39 API regression tests** прошли. Docker 29.0.1, aarch64, 10 CPU, 8 217 448 448 bytes RAM Docker VM.
|
||||
|
||||
Полный воспроизводимый запуск: `./scripts/check-docker.sh`. Исходник сценария — `student-tests/lab5_pipeline.py`. Результаты и реальные job/attachment IDs — `evidence/lab5-checks.json`, консольный вывод — `evidence/lab5-checks.txt`. Дополнительное доказательство реального restart обычного стенда с прежними container IDs и volumes — `evidence/lab5-restart.json`; воспроизводится `.venv/bin/python student-tests/verify_restart.py`.
|
||||
|
||||
| Критерий README | Команда / ключ проверки evidence | Наблюдаемый результат |
|
||||
| --- | --- | --- |
|
||||
| Сохранён lab4, переход 201→202, smoke | make check validate test; public_smoke_and_schema, student_api_regressions | Публичный lab5 сценарий и собственные API regression tests |
|
||||
| Отдельные broker/worker, persistence, лимиты | topology_and_limits | Реальные контейнеры/volumes, две API, broker AOF always, worker concurrency=2/prefetch=1 |
|
||||
| Worker остановлен → 202 → сообщение → рестарты → ready | worker_offline_api_broker_restart | Тот же attachment/job после рестарта API и broker; 409 до ready; original/три варианта доступны |
|
||||
| Original/thumbnail/crop/WebP по контракту | image_semantics | Bytes/checksums/MIME и реальные размеры; small/alpha/EXIF/floor cases |
|
||||
| Outbox, crash COMMIT→publish | outbox_crash_before_publish | Устойчивое намерение доставляется после автоматического restart publisher |
|
||||
| Неопределённый XADD, дубли | outbox_crash_after_publish | Повтор publish не создаёт второго результата |
|
||||
| Crash после side effect до COMMIT/ACK | worker_crash_after_objects, worker_crash_before_ack | Lease/reclaim восстанавливают работу; четыре canonical объекта; ready дубль — no-op |
|
||||
| Retry/backoff/jitter, failed registry, replay | bounded_retry_failed_registry_replay | 5 автоматических неудач → failed; audit содержит время/попытки; явный replay → ready; ready replay отклонён |
|
||||
| Постоянная ошибка | permanent_error | invalid_image с первой попытки, безопасный код |
|
||||
| Ограничение времени и памяти обработки | image_subprocess_timeout, image_subprocess_memory | Реальное убийство зависшего subprocess и MemoryError при выделении 1 GiB под лимитом 640 MiB; следующий retry завершает тот же job |
|
||||
| Безопасная очистка staging | staging_cleanup | Active staging сохранён; ready/expired failed удалены; replay без исходника запрещён |
|
||||
| Broker недоступен | broker_offline_outbox | API принимает 202 и текст; outbox доставляет после восстановления |
|
||||
| Сохранность после всех рестартов | make test-persistence; public_persistence | Сессия, история и original переживают рестарты с volumes |
|
||||
| Отказ API | make test-failover; public_api_failover | Другая реплика продолжает обслуживать подтверждённые сообщения без дублей |
|
||||
|
||||
Failpoints включаются только явным `ENABLE_FAULT_INJECTION=1` во временном тестовом project; по умолчанию отключены. Файлы управления доступны только через docker exec внутри контейнера, HTTP управления ими нет. `before_publish` завершает publisher после DB COMMIT намерения до XADD; `after_publish` — после XADD до фиксации outbox. `after_objects` завершает consumer после PUT четырёх объектов до COMMIT ready; `before_ack` — после COMMIT ready до XACK. Маркеры срабатывания проверяются тестом, поэтому простой быстрый успех без внедрённого сбоя не засчитывается. Отдельный timeout failpoint усыпляет именно ограниченный resize subprocess, не подменяет результат mock-объектом.
|
||||
|
||||
## Эксперименты и ограничения
|
||||
|
||||
Это проверка критериев на 4; исследование максимального throughput, сравнение concurrency/prefetch и versioned pipeline не выполнялись. Полный сценарий использует concurrency=2/prefetch=1, max_attempts=5; для быстрых fault checks timeout=5 s, lease=15 s, base backoff=0.15 s. Production-подобные default настройки остаются 60/90/2 s в .env.example. Дата, Docker/hardware и лимиты конкретного запуска записаны в evidence. Измерения latency/RSS этой проверки не трактуются как нагрузочный benchmark.
|
||||
|
||||
Обработка at-least-once, не exactly-once: XADD и PUT могут повториться, идемпотентен конечный эффект и его публикация. Сбой host/disk с потерей volumes, потеря единственной PostgreSQL/S3 или ручное удаление объектов не покрываются. Backup/replication этих хранилищ в lab5 не добавлялись. Очередь ограничена maxmemory/noeviction broker; outbox и staging во время длительного простоя требуют диска, общей квоты принятой работы эта оценка не требует. Повторы самого HTTP upload могут создать отдельные вложения по контракту.
|
||||
|
||||
## Первичные источники
|
||||
|
||||
- [Задание и объявленные переходы](README.md), [состояния](contracts/attachment-states.md), [схема job](contracts/jobs.schema.json).
|
||||
- [Redis Streams: consumer groups, pending, acknowledgements](https://redis.io/docs/latest/develop/data-types/streams/).
|
||||
- [XAUTOCLAIM: восстановление pending](https://redis.io/docs/latest/commands/xautoclaim/).
|
||||
- [Valkey persistence: AOF и fsync](https://valkey.io/topics/persistence/).
|
||||
- [Pillow ImageOps: EXIF transpose и центрированный fit](https://pillow.readthedocs.io/en/stable/reference/ImageOps.html).
|
||||
- [Python resource limits](https://docs.python.org/3/library/resource.html).
|
||||
@@ -1,37 +1,221 @@
|
||||
# Отчёт — лабораторная 6
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
- Автор, группа: Савельев Иван, БВТ2304.
|
||||
- Целевая оценка: **5**, трек 2 — **SLO и обнаружение инцидентов**. Реализованы уровни 3 и 4.
|
||||
- Основа: готовая lab5, включая асинхронный pipeline и все её проверки на 4. Исторический отчёт — [REPORT.lab5.md](REPORT.lab5.md).
|
||||
- Commit/PR: рабочее дерево поверх исходного skeleton; публикация выполняется отдельно. Проверенные файлы перечислены с SHA-256 в `evidence/lab6-source-manifest.json`.
|
||||
- Помощь ИИ: Codex использован для переноса, реализации, проверок и подготовки отчёта. Evidence получен реальными запусками Docker и запросами к хранилищам.
|
||||
- Проверено 19.11.2026 на macOS arm64, Docker Desktop 29.0.1: 10 CPU, 8217448448 bytes RAM (≈7.65 GiB) доступны Docker VM. Это учебный стенд на одном host.
|
||||
|
||||
## Запуск из чистого clone
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
Нужны Docker Engine/Desktop с Compose v2, Python 3.11, Bash и make. Интернет нужен для первой установки зависимостей и загрузки закреплённых образов. Все команды из корня:
|
||||
|
||||
## Архитектура и решения
|
||||
```sh
|
||||
python3 -m venv .venv
|
||||
.venv/bin/python -m pip install -r requirements.txt -r requirements-dev.txt
|
||||
make check validate PYTHON=.venv/bin/python
|
||||
./scripts/start.sh
|
||||
./scripts/check.sh
|
||||
```
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab5, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
`start.sh` создаёт локальные `.env` и `secrets/`, собирает приложение, применяет миграции и bootstrap приватного S3 bucket, поднимает API/worker и наблюдаемость. Затем ожидает две API-реплики через LB, scrape Collector, все хранилища и provisioning Grafana. Ручной импорт dashboard не нужен.
|
||||
|
||||
- API: http://localhost:8080; health: `/health/live`, `/health/ready`.
|
||||
- [Grafana dashboard](http://localhost:3000/d/messenger-lab6): локальный anonymous Viewer; пароли приложения в Grafana не вводятся.
|
||||
- Порты меняются через `HOST_PORT` и `GRAFANA_PORT` в `.env`; для API-тестов передать соответствующий `BASE_URL`. Драйвер наблюдаемости определяет опубликованный порт Grafana через Compose; можно задать `GRAFANA_URL`.
|
||||
- Compose project `mtusi-lab6`, приложение `mtusi-messenger:lab6`; volumes отделены от lab5. Локальные `.env`/secrets не входят в Git и Docker build context. В новом clone credentials генерируются заново.
|
||||
|
||||
```sh
|
||||
# Весь минимум lab5 + API/telemetry/persistence + исследование lab6:
|
||||
./scripts/check-lab6.sh
|
||||
|
||||
# Только исследование трека 2 на уже запущенном синтетическом стенде (~11 минут):
|
||||
./scripts/research.sh
|
||||
|
||||
# Остановка с сохранением данных / повторный запуск:
|
||||
./scripts/stop.sh
|
||||
./scripts/start.sh
|
||||
|
||||
# Перезапуск процессов и хранилищ без удаления volumes:
|
||||
./scripts/restart.sh
|
||||
|
||||
# Только намеренное удаление данных СВОЕГО учебного Compose project:
|
||||
docker compose down --volumes
|
||||
```
|
||||
|
||||
Полный script сначала проверяет lab5 в отдельном временном project на свободных портах и новых volumes, затем удаляет только этот временный стенд. Исследование lab6 использует текущий локальный синтетический стенд и намеренно создаёт ограниченные отказы. Стандартная работа приложения не требует `ENABLE_FAULT_INJECTION`; проверка retry временно включает его только для worker и затем выключает.
|
||||
|
||||
## Архитектура и версии
|
||||
|
||||
```text
|
||||
HTTP → HAProxy → API ×2 → PostgreSQL / Valkey sessions / private S3
|
||||
└ durable metadata + job + outbox (W3C context)
|
||||
→ publisher → Valkey Streams → worker ×2
|
||||
→ S3 GET/PUT → DB ready → ACK
|
||||
|
||||
API / publisher / worker → OTLP HTTP → Collector
|
||||
├ Prometheus exporter → Prometheus ← scrape
|
||||
├ OTLP HTTP → Loki
|
||||
└ OTLP gRPC → Tempo
|
||||
↑
|
||||
Grafana: metrics / logs / traces
|
||||
```
|
||||
|
||||
API, ACL, сессии, идемпотентность, ограничения изображений, at-least-once/retry/replay и очистка staging перенесены из lab5. Обязательные контракты и публичные tests сохранены. Миграции 001–004 не изменены; 005 добавляет сохранённый trace context и `ready_at`. Новые поля не меняют бизнес-API. Источник истины — PostgreSQL; ACK по-прежнему выполняется после устойчивого результата/retry.
|
||||
|
||||
| Роль / service | Закреплённая версия | Persistence / readiness / credentials |
|
||||
| --- | --- | --- |
|
||||
| API / worker / publisher / cleanup / migrate | Python 3.11.15, FastAPI 0.142.4; image `mtusi-messenger:lab6` | Исходники и lock версий в requirements; API `/health/ready`, worker читает broker; отдельный migrate проверяет checksum. Docker secrets, non-root UID 10001 |
|
||||
| PostgreSQL / postgres | 17.6-bookworm | `postgres_data`, pg_isready; password/DSN из secrets |
|
||||
| Сессии / valkey | 8.1.3 | `valkey_data`, AOF fsync always; ping; только внутренняя сеть |
|
||||
| Очередь / broker | Valkey 8.1.3 | `broker_data`, AOF fsync always; authenticated ping; password/URL из secrets |
|
||||
| S3 / s3, s3-storage | SeaweedFS 3.85 с digest | `s3_data`, внутренний TCP/HTTP readiness; private bucket и service account создаёт bootstrap; secrets |
|
||||
| LB / lb | HAProxy 3.2.6-alpine | `/health/ready`; только `127.0.0.1:8080`; API-порты напрямую не опубликованы |
|
||||
| Collector / otel-collector | contrib 0.123.0 | Config `infra/otel/collector.yaml`, health 13133 и self metrics 8888 внутри сети; долговременной БД не является |
|
||||
| Метрики / prometheus | 3.4.1 | `prometheus_data`, retention 7d, scrape/evaluation 5s; rules в Git |
|
||||
| Логи / loki | 3.5.1 | `loki_data`, TSDB v13/filesystem, retention 7d, native OTLP со structured metadata |
|
||||
| Трассы / tempo | 2.8.1 | `tempo_data`, WAL + local blocks, retention 7d; OTLP receiver только внутри сети |
|
||||
| Grafana / grafana | 12.0.2 | `grafana_data`, `/api/health`; 3 datasources + dashboard provisioned из Git; localhost:3000, Viewer |
|
||||
|
||||
Grafana запрашивает хранилища, сама не хранит три сигнала. Collector принимает/буферизует/экспортирует, долгую историю хранят backends. У всех контейнеров заданы CPU/RAM limits; точные значения, volumes и опубликованные порты — `lab6-runtime.json`. Collector: 1 CPU/256 MiB; Prometheus/Loki/Grafana: по 1 CPU/384 MiB; Tempo: 1 CPU/512 MiB с GOMEMLIMIT 384 MiB. Приложение сохранило ограничения lab5.
|
||||
|
||||
SDK/API/exporters OTel **1.45.1**, semantic conventions **0.66b1**; полный список зависимостей закреплён в `requirements.txt`. Используются явные span/metric имена, а не автосбор SQL/HTTP headers. Встроенная автоматическая telemetry FastAPI отключена: один SDK и один набор инструментов без дублирования сигналов. Counter series для известных маршрутов и status classes инициализированы нулём до первого scrape: короткий первый всплеск ошибок не теряется из rate из-за отсутствия исходного нулевого sample. Фиктивные запросы или histogram observations для этого не создаются.
|
||||
|
||||
`.env.example` соответствует настройкам: `DATABASE_URL_FILE`, `CACHE_URL_FILE`, S3 и `BROKER_URL_FILE` читают Docker secrets; worker concurrency/retry/lease/time/memory — перенесённые настройки pipeline; `OTEL_EXPORTER_OTLP_ENDPOINT` — общий Collector, protocol HTTP/protobuf. Resource `service.name` устанавливается отдельно для API, worker и publisher; `service.version=6.0.0`, instance — hostname:pid, environment — lab. `OTEL_PYTHON_SDK_INTERNAL_METRICS_ENABLED=true` включает контроль очередей SDK. Перезапуск сохраняет volumes всех signal stores.
|
||||
|
||||
## Сигналы и запросы
|
||||
|
||||
[Dashboard JSON](infra/grafana/dashboards/messenger.json) содержит 19 панелей; datasources/provider рядом в `infra/grafana/provisioning/`. `lab6-dashboard-checks.json` подтверждает выполнение запросов ко всем трём источникам.
|
||||
|
||||
| Реальное имя в Prometheus | Тип / единицы / labels |
|
||||
| --- | --- |
|
||||
| `messenger_http_requests_total` | counter запросов; method, route template, status_class; health исключён |
|
||||
| `messenger_http_duration_seconds_*` | histogram, секунды до response headers; те же labels |
|
||||
| `messenger_job_attempts_total` | counter попыток; job_type, outcome (ready/retry/failed/error/duplicate/busy/obsolete/invalid) |
|
||||
| `messenger_job_duration_seconds_*` | histogram длительности consumer attempt, секунды; job_type/outcome |
|
||||
| `messenger_attachment_ready_duration_seconds_*` | histogram acceptance → ready успешных jobs, секунды; job_type |
|
||||
| `messenger_queue_depth` / `messenger_queue_oldest_age_seconds` | gauges: незавершённые durable jobs / возраст самой старой, секунды; queue=attachments |
|
||||
| `messenger_db_pool` | gauge соединений; state=used/waiting |
|
||||
| `messenger_worker_busy` / `messenger_worker_capacity` | gauges занятых/доступных worker процессов |
|
||||
| `messenger_process_cpu_time_seconds_total` / `messenger_process_memory_bytes` | process CPU seconds / RSS bytes; группировка по service_name |
|
||||
| `messenger_ready_deadlines_total` | durable cohort counter; outcome=good/bad, включая failed/stuck/late jobs |
|
||||
|
||||
`_bucket`, `_sum`, `_count` — части одного histogram. Resource labels добавляются Collector; технический instance ID нужен для суммирования процессов и различения перезапусков. User/chat/message/attachment/request IDs, raw URL, filename и текст labels не являются. Лимит экспирации неактивных metric series Collector — 30s.
|
||||
|
||||
```promql
|
||||
# RPS (без health), ошибки 5xx / все business requests:
|
||||
sum(rate(messenger_http_requests_total[1m]))
|
||||
lab6:api_bad_ratio:1m
|
||||
|
||||
# p95 по всем API-репликам; для p50/p99 меняется quantile:
|
||||
histogram_quantile(0.95,
|
||||
sum by (le) (rate(messenger_http_duration_seconds_bucket[1m])))
|
||||
|
||||
max(messenger_queue_depth)
|
||||
max(messenger_queue_oldest_age_seconds)
|
||||
sum by (state, service_name) (messenger_db_pool)
|
||||
sum(messenger_worker_busy) / sum(messenger_worker_capacity)
|
||||
```
|
||||
|
||||
Buckets в секундах: .005, .01, .025, .05, .1, .25, .5, 1, 2, 5, 10, 20, **30**, 60, 120, 300, 600, +Inf. Percentiles считаются из суммарных buckets, не средних latency/p95 реплик. Backlog не подменяет processing duration. `s3.get_object` измеряет получение ответа S3; чтение stream входит в общий consumer span.
|
||||
|
||||
Лог — структурный JSON с timestamp, severity, service.name, event, trace_id/span_id; HTTP содержит также `request_id`, совпадающий с X-Request-Id. В ответе есть X-Trace-Id. Только job/attachment IDs разрешены в явных диагностических событиях для синтетических заданий; в labels их нет. Пароли, Authorization/session tokens, bodies, изображения, filenames/storage keys/signed URLs и SQL/params не собираются. Exception events с текстом/stack trace отключены; сохраняется только класс ошибки. Loki индексирует service/environment, instance — structured metadata.
|
||||
|
||||
W3C traceparent/tracestate сохраняются в job той же транзакцией, что metadata/outbox. Producer восстанавливает parent, инжектирует свой context в broker payload. Consumer извлекает его; retry сохраняет актуальный context для следующей публикации. После перезапуска процессов связь сохраняется. [Сохранённый trace с retry](evidence/lab6-trace-retry.json) и [логи](evidence/lab6-correlated-logs.json) показывают реальную цепочку API → producer → consumer → retry → producer → consumer → S3/DB. Sampling 100% для учебного стенда.
|
||||
|
||||
## SLI/SLO и alerts
|
||||
|
||||
- **API availability ≥99% за скользящий час**. Bad — instrumented business responses 5xx; denominator — все business requests, включая 4xx; health исключён. Неверный пароль не является серверной ошибкой. Нужны внешние probes для отказов LB/сети до API; в этот SLI они не входят.
|
||||
- **Ready ≥95% за ≤30s, окно час**. Acceptance timestamp — `job.created_at` в durable metadata transaction; запись становится видна после COMMIT. По `ready_at` оценивается каждый cohort после истечения deadline. Позднее завершение, failed и незавершившееся задание считаются bad. Часовое окно относится к созревшим cohorts, acceptance сдвинута на 30s. В отличие от histogram только successful jobs, SLI не скрывает остановку worker.
|
||||
|
||||
Обоснование: 99% availability задаёт заметный бюджет серверных ошибок для учебного single-host стенда с единственной БД; это проверяемая цель, а не обещание production SLA. Для асинхронного изображения 30s оставляют запас на S3, ожидание публикации и краткий retry; минуты без обработки уже нарушают пользовательское ожидание. Цель 95% допускает редкие тяжёлые изображения/повторы, сохраняя чувствительность к остановке worker. При нашем постоянном профиле час содержит около 9000 HTTP requests и 1800 jobs: budgets — 90 ошибок и 90 просроченных jobs. Четыре ошибки короткого всплеска расходуют около 4.4% часового API budget, ниже порогов эскалации 20%/30%. Час выбран для воспроизводимого учебного исследования; production потребует отдельного обоснования периода и целей.
|
||||
|
||||
| Alert policy | Long / short window | Burn rate / порог API / ready | for / guards |
|
||||
| --- | --- | --- | --- |
|
||||
| Fast | 2m AND 30s | 6× / 6% / 30% bad | 15s; long ≥30 API / ≥5 jobs, short ≥2 события |
|
||||
| Slow | 6m AND 1m | 3× / 3% / 15% bad | 30s; те же guards |
|
||||
| APIHighErrorRate (сравнение) | 30s | >5% 5xx | 5s |
|
||||
| AttachmentQueueLag | текущие age/depth | oldest >20s и depth >0 | 10s |
|
||||
|
||||
Fast и slow одновременно проверяют оба окна. Это учебный **1h SLO**: fast соответствует расходу 20% бюджета за 2m, slow — 30% за 6m (`burn × long_window / 1h`). Production окна из примера 30-дневного SLO не копируются механически. Guards снижают шум при малом числе событий. При нулевом denominator ratio — NaN, а не искусственные 0% ошибок/100% availability; отдельно проверяются scrape и трафик. Есть CollectorUnavailable warning.
|
||||
|
||||
Rules/условия/описания/for/runbook находятся в [alerts.yaml](infra/otel/alerts.yaml). Доставка наружу не нужна; firing/inactive (resolved) наблюдаются через Prometheus API и панель ALERTS. [Runbook](docs/runbook.md) содержит запросы и порядок диагностики другим студентом.
|
||||
|
||||
## Эксперимент трека 2 и разбор инцидентов
|
||||
|
||||
Гипотеза, заранее заданная методика, нагрузка и ограничения — [research/README.md](research/README.md). Один real-time прогон: 2 сообщения/s + 0.5 PNG/s, 16 max in-flight, без retries; baseline 120s, spike БД 4s, recovery 60s, lock messages 90s, recovery 100s, worker stopped 120s, recovery 120s. SQL-lock ограничен pg_sleep; worker восстанавливается в finally. Каждые 5s читаются сохранённые alert states/метрики. Исследование выполняется отдельно от полного lab5 regression, чтобы они не конкурировали за ресурсы.
|
||||
|
||||
Финальный прогон **19.11.2026, 18:26:14–18:36:34 МСК**, 619.4s. 1544 ответов, пропущенных arrivals: 0. p50/p95/p99 ниже рассчитаны по всем ответам, включая 5xx; successful RPS — только 2xx.
|
||||
[Открыть этот интервал в Grafana](http://localhost:3000/d/messenger-lab6?from=1795101974987&to=1795102594377).
|
||||
|
||||
| Фаза | Запросы / 5xx | Successful RPS | p50 / p95 / p99, ms |
|
||||
| --- | --- | --- | --- |
|
||||
| Baseline | 301 / 0 | 2.51 | 16.6 / 31.3 / 35.2 |
|
||||
| Короткий DB spike, 4s | 10 / 4 | 1.50 | 1670.7 / 2017.2 / 2017.2 |
|
||||
| После короткого всплеска | 150 / 0 | 2.50 | 16.5 / 30.7 / 32.8 |
|
||||
| Длительное ожидание DB, 90s | 225 / 176 | 0.54 | 2011.8 / 2022.5 / 2025.8 |
|
||||
| После восстановления DB | 250 / 0 | 2.50 | 16.3 / 31.0 / 33.4 |
|
||||
| Worker stopped, 120s | 300 / 0 | 2.50 | 16.2 / 31.3 / 35.5 |
|
||||
| После восстановления worker | 300 / 0 | 2.50 | 13.1 / 30.7 / 35.3 |
|
||||
|
||||
| Alert | Detection после воздействия, s | Resolved после восстановления, s |
|
||||
| --- | --- | --- |
|
||||
| APIAvailabilityFastBurn | 40.2 | 35.1 |
|
||||
| APIAvailabilitySlowBurn | 60.3 | 65.3 |
|
||||
| APIHighErrorRate | 25.1 | 35.1 |
|
||||
| AttachmentQueueLag | 45.2 | 30.1 |
|
||||
| AttachmentReadyFastBurn | 95.5 | 45.2 |
|
||||
| AttachmentReadySlowBurn | 131.6 | 75.3 |
|
||||
|
||||
Baseline: **0** firing alerts. Короткий реальный всплеск виден в сохранённой 30s error ratio (peak **6.37%**); burn alerts на него сработали **0** раз. Single-window comparator: **APIHighErrorRate**.
|
||||
В исследовании false positive означает эскалацию короткого всплеска, заранее принятого допустимым для выбранной SLO policy; предупреждение APIHighErrorRate при этом корректно соответствует собственному порогу.
|
||||
Все заявленные alerts наблюдались firing и затем inactive/resolved. Сохранённый worker trace подтверждает consumer спустя **94.3s** после upload, уже после восстановления worker.
|
||||
|
||||
| Процесс | Max 1m CPU rate, cores | Max RSS, MiB |
|
||||
| --- | --- | --- |
|
||||
| messenger-api | 0.149 | 603.75 |
|
||||
| messenger-publisher | 0.012 | 147.05 |
|
||||
| messenger-worker | 0.013 | 319.40 |
|
||||
|
||||
Это maxima точек с шагом 5s; CPU усреднён по минуте, RSS суммирован по процессам сервиса. Два consumer процесса worker показаны суммарно. Исходные samples, timestamps и все запросы без credentials сохранены в `lab6-slo-research.json`, ряды — в `lab6-slo-series.json`.
|
||||
|
||||
Причина деградации API локализуется по сохранённым наблюдениям: 5xx и HTTP latency растут, внутри server trace `postgresql.SELECT` занимает около 2s и заканчивается `_WaitTimeout`, тогда как acceptance и worker продолжаются. Это I/O ожидание PostgreSQL, а не искусственный sleep HTTP handler. Конкретные spans/durations — `lab6-db-incident-trace.json`, коррелированные request logs — `lab6-db-incident-logs.json`. Методика внедряет SQL lock, но по одному сохранённому timeout нельзя доказать именно блокировку вместо другой причины медленного ответа БД; этот предел диагностики явно учитывается.
|
||||
|
||||
При остановке worker acceptance/producer продолжаются, successful processing прекращается, capacity исчезает после истечения series, backlog/oldest age растут. В trace принятого upload до восстановления отсутствует consumer; после восстановления очередь дренируется, появляются S3/DB ready spans, но просроченные cohorts сохраняют bad исход. Это проверено по timestamps consumer в `lab6-worker-incident-trace.json` и связанным `lab6-worker-incident-logs.json`. Подробные ряды, CPU/RSS, pool и saturation — `lab6-slo-series.json`.
|
||||
|
||||
## Отказ telemetry backend и границы буферизации
|
||||
|
||||
Collector memory limiter: 192 MiB со spike 32 MiB, check 1s; hard container limit 256 MiB. Batch максимум 128 записей; отдельные exporter queues по 64 batch, 2 consumers, timeout 2s, retry максимум 15s. SDK trace/log queues по 256 записей, batch 64, фоновый export; metric export каждые 5s. Internal SDK queue capacity/size и Collector exporter metrics доступны в Prometheus. Бесконечной буферизации нет: при недоступном backend telemetry теряется, бизнес-запрос не ждёт экспорт.
|
||||
|
||||
Контролируемая остановка Loki проверена `student-tests/lab6_observability.py`: отказ около **51.39s**, **24/24 сообщения записаны**, история читается. После восстановления не найдены **6 из 24** соответствующих request logs; это фактически измеренная потеря по известным request IDs. RAM Collector в четырёх samples — **70.46–72.80 MiB** при limit 256 MiB, OOM=false. Отдельный счётчик failed exports вырос на 6; вообще его нельзя считать числом потерянных записей без проверки из-за retries. Raw результаты — `lab6-checks.json`. Grafana/Prometheus/Tempo сохраняли доступность; Loki после запуска снова принимает OTLP. Потери не маскируются debug exporter или stdout.
|
||||
|
||||
## Доказательства критериев
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
|
||||
## Эксперименты и отказы
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
| Критерий | Команда/сценарий | Наблюдение / evidence |
|
||||
| --- | --- | --- |
|
||||
| Весь minimum lab5 и сохранённые переходы | `.venv/bin/python -u student-tests/lab5_pipeline.py` (тот же stage в check-lab6) | 17 групп PASS, включая outbox/worker crash, retry/replay, processing limits, cleanup, broker outage, persistence/failover; `lab6-lab5-regression.json/.txt` |
|
||||
| Контракт, публичный smoke, свои API-тесты с telemetry | make check validate test; unittest discover | PASS и 39 tests OK; `lab6-api-checks.txt` |
|
||||
| Collector + три persistent stores + Grafana | start.sh; readiness/provisioning | 2 API, CPU/RAM limits, localhost порты, volumes; `lab6-runtime.json` |
|
||||
| Все обязательные metrics и dashboard queries | lab6_observability; check_dashboard | Реальные series и 19 панелей трёх источников; `lab6-checks.json`, `lab6-dashboard-checks.json` |
|
||||
| API/worker logs + traces, W3C, retry → S3/DB | Deterministic transient worker failpoint | 59 связанных spans; `lab6-trace-retry.json`, `lab6-correlated-logs.json` |
|
||||
| Отсутствие секретов/body/ID labels | Canary body/filename/password/token + metric label check | PASS; `lab6-checks.json` |
|
||||
| Telemetry persistence | Restart всех постоянных контейнеров и запрос старого trace | IDs/volumes сохранены; `lab6-restart.json/.txt`, `lab6-telemetry-persistence.json` |
|
||||
| Ограниченный отказ Loki | Stop/start Loki, 24 записи под отказом, memory samples | API работает, потери logs измерены, OOM=false; `lab6-checks.json/.txt` |
|
||||
| SLO/rules и пограничные сценарии | promtool test rules | Spike/sustained/recovery/stalled/no traffic PASS; `lab6-promtool.txt` |
|
||||
| Два инцидента + трек 2 | research.sh, runbook | Реальные firing/resolved, detection/recovery/false positives, stored observations; `lab6-slo-research.json/.txt`, `lab6-slo-series.json` |
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
Один host, единственные PostgreSQL/S3/telemetry stores: потеря host/disk/volumes не покрыта. Persistent volumes и проверенный restart не означают HA или backup. Срок хранения telemetry 7d; heap/queues ограничены, при длительном отказе часть наблюдений теряется. Нет внешней доставки alerts и внешнего black-box SLI LB/сети: это не требуется выбранным минимумом. Histogram HTTP измеряет время до headers; полный stream download отдельно не профилируется. Queue depth берётся из durable state БД, а не только broker.
|
||||
|
||||
Эксперимент короче часа и выполнен по одному live прогону каждого отказа на небольшой синтетической нагрузке. Он проверяет работу детектора и воспроизводимость диагностики; не доказывает часовую/месячную надёжность, предел RPS или универсальную частоту ложных срабатываний. Точность detection/resolution ограничена шагом 5s плюс SDK export/scrape. Часовые ratio при неполной истории — только доступная часть окна. CPU/RSS относятся к долгоживущим instrumented процессам; resize subprocess ограничен общим Docker limit и отдельные resource series не экспортирует. SQL cohort scan достаточен для учебного объёма; на больших объёмах понадобится incremental aggregation с сохранением состояния.
|
||||
|
||||
## Первичные источники
|
||||
|
||||
- [OpenTelemetry Collector configuration](https://opentelemetry.io/docs/collector/configuration/) и [internal telemetry](https://opentelemetry.io/docs/collector/internal-telemetry/).
|
||||
- [OTel Python SDK](https://opentelemetry.io/docs/languages/python/) и [W3C Trace Context](https://www.w3.org/TR/trace-context/).
|
||||
- [Native OTLP в Loki](https://grafana.com/docs/loki/latest/send-data/otel/).
|
||||
- [Prometheus histograms](https://prometheus.io/docs/practices/histograms/) и [configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/).
|
||||
- [Google SRE Workbook: multiwindow, multi-burn-rate alerts](https://sre.google/workbook/alerting-on-slos/) — принцип двух окон, адаптированный к явно выбранному учебному периоду.
|
||||
- [Grafana provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/); compatibility TraceQL query проверена по source Grafana v12.0.2 и реальному Tempo search API.
|
||||
|
||||
+289
@@ -0,0 +1,289 @@
|
||||
name: mtusi-lab6
|
||||
|
||||
x-application: &application
|
||||
image: mtusi-messenger:lab6
|
||||
env_file: .env
|
||||
environment: &pipeline-environment
|
||||
OTEL_PYTHON_SDK_INTERNAL_METRICS_ENABLED: "true"
|
||||
JOB_MAX_ATTEMPTS: ${JOB_MAX_ATTEMPTS:-5}
|
||||
JOB_TIMEOUT_SECONDS: ${JOB_TIMEOUT_SECONDS:-60}
|
||||
JOB_LEASE_SECONDS: ${JOB_LEASE_SECONDS:-90}
|
||||
JOB_BACKOFF_SECONDS: ${JOB_BACKOFF_SECONDS:-2}
|
||||
WORKER_CONCURRENCY: ${WORKER_CONCURRENCY:-2}
|
||||
WORKER_TASK_MEMORY_MB: ${WORKER_TASK_MEMORY_MB:-640}
|
||||
ENABLE_FAULT_INJECTION: ${ENABLE_FAULT_INJECTION:-0}
|
||||
cpus: 1
|
||||
mem_limit: 384m
|
||||
|
||||
x-postgres: &postgres
|
||||
image: postgres:17.6-bookworm
|
||||
environment:
|
||||
POSTGRES_USER: messenger
|
||||
POSTGRES_DB: messenger
|
||||
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
|
||||
secrets: [postgres_password]
|
||||
cpus: 1
|
||||
mem_limit: 512m
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U messenger -d messenger"]
|
||||
interval: 2s
|
||||
timeout: 3s
|
||||
retries: 30
|
||||
|
||||
services:
|
||||
postgres:
|
||||
<<: *postgres
|
||||
volumes: [postgres_data:/var/lib/postgresql/data]
|
||||
|
||||
valkey:
|
||||
image: valkey/valkey:8.1.3
|
||||
command: ["valkey-server", "--appendonly", "yes", "--appendfsync", "always", "--save", "", "--maxmemory", "64mb", "--maxmemory-policy", "noeviction"]
|
||||
volumes: [valkey_data:/data]
|
||||
cpus: 0.5
|
||||
mem_limit: 128m
|
||||
healthcheck:
|
||||
test: ["CMD", "valkey-cli", "ping"]
|
||||
interval: 2s
|
||||
timeout: 3s
|
||||
retries: 30
|
||||
|
||||
broker:
|
||||
image: valkey/valkey:8.1.3
|
||||
command: ["sh", "-c", 'exec valkey-server --appendonly yes --appendfsync always --save "" --maxmemory 128mb --maxmemory-policy noeviction --requirepass "$$(cat /run/secrets/broker_password)"']
|
||||
secrets: [broker_password]
|
||||
volumes: [broker_data:/data]
|
||||
cpus: 0.5
|
||||
mem_limit: 256m
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", 'REDISCLI_AUTH="$$(cat /run/secrets/broker_password)" valkey-cli ping | grep -q PONG']
|
||||
interval: 2s
|
||||
timeout: 3s
|
||||
retries: 30
|
||||
|
||||
publisher:
|
||||
<<: *application
|
||||
command: ["python", "-m", "src.publisher"]
|
||||
secrets: [database_url, broker_url]
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
disable: true
|
||||
depends_on:
|
||||
migrate: {condition: service_completed_successfully}
|
||||
broker: {condition: service_healthy}
|
||||
|
||||
worker:
|
||||
<<: *application
|
||||
command: ["python", "-m", "src.worker"]
|
||||
secrets: [database_url, broker_url, s3_access_key_id, s3_secret_access_key]
|
||||
restart: unless-stopped
|
||||
mem_limit: 1536m
|
||||
cpus: 2
|
||||
pids_limit: 96
|
||||
healthcheck:
|
||||
disable: true
|
||||
depends_on:
|
||||
migrate: {condition: service_completed_successfully}
|
||||
broker: {condition: service_healthy}
|
||||
s3-bootstrap: {condition: service_completed_successfully}
|
||||
stop_grace_period: 8s
|
||||
|
||||
api:
|
||||
<<: *application
|
||||
build: .
|
||||
deploy:
|
||||
replicas: ${API_REPLICAS:-2}
|
||||
environment:
|
||||
<<: *pipeline-environment
|
||||
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
|
||||
secrets: [database_url, cache_url, s3_access_key_id, s3_secret_access_key]
|
||||
depends_on:
|
||||
postgres: {condition: service_healthy}
|
||||
valkey: {condition: service_healthy}
|
||||
migrate: {condition: service_completed_successfully}
|
||||
s3: {condition: service_healthy}
|
||||
s3-bootstrap: {condition: service_completed_successfully}
|
||||
stop_grace_period: 8s
|
||||
|
||||
migrate:
|
||||
<<: *application
|
||||
command: ["python", "-m", "src.migrate"]
|
||||
secrets: [database_url]
|
||||
healthcheck:
|
||||
disable: true
|
||||
depends_on:
|
||||
postgres: {condition: service_healthy}
|
||||
|
||||
s3:
|
||||
image: chrislusf/seaweedfs:3.85@sha256:49312939c00c01e5ee6afbd7d728b18027821d3764c35a797a72acd4fdf3296a
|
||||
command: ["s3", "-filer=s3-storage:8888", "-ip.bind=0.0.0.0", "-port=9000", "-config=/run/secrets/s3_config"]
|
||||
secrets: [s3_config]
|
||||
networks: [default, s3-internal]
|
||||
depends_on:
|
||||
s3-storage: {condition: service_healthy}
|
||||
cpus: 0.5
|
||||
mem_limit: 256m
|
||||
healthcheck:
|
||||
test: ["CMD", "nc", "-z", "127.0.0.1", "9000"]
|
||||
interval: 2s
|
||||
timeout: 2s
|
||||
retries: 30
|
||||
|
||||
s3-storage:
|
||||
image: chrislusf/seaweedfs:3.85@sha256:49312939c00c01e5ee6afbd7d728b18027821d3764c35a797a72acd4fdf3296a
|
||||
command: ["server", "-dir=/data", "-ip=s3-storage", "-ip.bind=0.0.0.0", "-filer", "-master.volumeSizeLimitMB=64", "-volume.max=64", "-volume.preStopSeconds=0"]
|
||||
volumes: [s3_data:/data]
|
||||
networks: [s3-internal]
|
||||
cpus: 1
|
||||
mem_limit: 512m
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:8888/"]
|
||||
interval: 2s
|
||||
timeout: 2s
|
||||
retries: 30
|
||||
|
||||
s3-bootstrap:
|
||||
<<: *application
|
||||
command: ["python", "-m", "src.bootstrap_s3"]
|
||||
environment:
|
||||
S3_ROOT_USER_FILE: /run/secrets/s3_root_user
|
||||
S3_ROOT_PASSWORD_FILE: /run/secrets/s3_root_password
|
||||
secrets: [s3_root_user, s3_root_password]
|
||||
cpus: 0.5
|
||||
mem_limit: 128m
|
||||
healthcheck:
|
||||
disable: true
|
||||
depends_on:
|
||||
s3: {condition: service_healthy}
|
||||
|
||||
cleanup:
|
||||
<<: *application
|
||||
command: ["python", "-m", "src.cleanup"]
|
||||
secrets: [database_url, s3_access_key_id, s3_secret_access_key]
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
disable: true
|
||||
depends_on:
|
||||
migrate: {condition: service_completed_successfully}
|
||||
s3-bootstrap: {condition: service_completed_successfully}
|
||||
|
||||
lb:
|
||||
image: haproxy:3.2.6-alpine
|
||||
ports: ["127.0.0.1:${HOST_PORT:-8080}:8080"]
|
||||
volumes: [./infra/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro]
|
||||
cpus: 0.5
|
||||
mem_limit: 64m
|
||||
depends_on:
|
||||
api: {condition: service_healthy}
|
||||
stop_grace_period: 10s
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:8080/health/ready"]
|
||||
interval: 2s
|
||||
timeout: 2s
|
||||
retries: 3
|
||||
|
||||
otel-collector:
|
||||
image: otel/opentelemetry-collector-contrib:0.123.0
|
||||
command: ["--config=/etc/otelcol/config.yaml"]
|
||||
volumes: [./infra/otel/collector.yaml:/etc/otelcol/config.yaml:ro]
|
||||
depends_on: [loki, tempo]
|
||||
cpus: 1
|
||||
mem_limit: 256m
|
||||
restart: unless-stopped
|
||||
|
||||
prometheus:
|
||||
image: prom/prometheus:v3.4.1
|
||||
command: ["--config.file=/etc/prometheus/prometheus.yaml", "--storage.tsdb.path=/prometheus", "--storage.tsdb.retention.time=7d"]
|
||||
volumes:
|
||||
- ./infra/otel/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro
|
||||
- ./infra/otel/alerts.yaml:/etc/prometheus/alerts.yaml:ro
|
||||
- prometheus_data:/prometheus
|
||||
depends_on: [otel-collector]
|
||||
cpus: 1
|
||||
mem_limit: 384m
|
||||
restart: unless-stopped
|
||||
|
||||
loki:
|
||||
image: grafana/loki:3.5.1
|
||||
command: ["-config.file=/etc/loki/config.yaml"]
|
||||
volumes:
|
||||
- ./infra/otel/loki.yaml:/etc/loki/config.yaml:ro
|
||||
- loki_data:/loki
|
||||
cpus: 1
|
||||
mem_limit: 384m
|
||||
restart: unless-stopped
|
||||
|
||||
tempo:
|
||||
image: grafana/tempo:2.8.1
|
||||
environment:
|
||||
GOMEMLIMIT: 384MiB
|
||||
command: ["-config.file=/etc/tempo/config.yaml"]
|
||||
volumes:
|
||||
- ./infra/otel/tempo.yaml:/etc/tempo/config.yaml:ro
|
||||
- tempo_data:/var/tempo
|
||||
cpus: 1
|
||||
mem_limit: 512m
|
||||
restart: unless-stopped
|
||||
|
||||
grafana:
|
||||
image: grafana/grafana:12.0.2
|
||||
environment:
|
||||
GF_AUTH_ANONYMOUS_ENABLED: "true"
|
||||
GF_AUTH_ANONYMOUS_ORG_ROLE: Viewer
|
||||
GF_AUTH_BASIC_ENABLED: "false"
|
||||
GF_AUTH_DISABLE_LOGIN_FORM: "true"
|
||||
GF_ANALYTICS_REPORTING_ENABLED: "false"
|
||||
GF_ANALYTICS_CHECK_FOR_UPDATES: "false"
|
||||
GF_PLUGINS_PREINSTALL_DISABLED: "true"
|
||||
GF_PLUGINS_CHECK_FOR_UPDATES: "false"
|
||||
GF_NEWS_NEWS_FEED_ENABLED: "false"
|
||||
ports: ["127.0.0.1:${GRAFANA_PORT:-3000}:3000"]
|
||||
volumes:
|
||||
- ./infra/grafana/provisioning:/etc/grafana/provisioning:ro
|
||||
- ./infra/grafana/dashboards:/var/lib/grafana/dashboards:ro
|
||||
- grafana_data:/var/lib/grafana
|
||||
depends_on: [prometheus, loki, tempo]
|
||||
cpus: 1
|
||||
mem_limit: 384m
|
||||
restart: unless-stopped
|
||||
|
||||
postgres-restore:
|
||||
<<: *postgres
|
||||
profiles: [research]
|
||||
volumes: [restore_data:/var/lib/postgresql/data]
|
||||
|
||||
volumes:
|
||||
prometheus_data:
|
||||
loki_data:
|
||||
tempo_data:
|
||||
grafana_data:
|
||||
postgres_data:
|
||||
valkey_data:
|
||||
restore_data:
|
||||
s3_data:
|
||||
broker_data:
|
||||
|
||||
networks:
|
||||
s3-internal:
|
||||
internal: true
|
||||
|
||||
secrets:
|
||||
broker_password:
|
||||
file: ./secrets/broker_password
|
||||
broker_url:
|
||||
file: ./secrets/broker_url
|
||||
postgres_password:
|
||||
file: ./secrets/postgres_password
|
||||
database_url:
|
||||
file: ./secrets/database_url
|
||||
cache_url:
|
||||
file: ./secrets/cache_url
|
||||
s3_root_user:
|
||||
file: ./secrets/s3_root_user
|
||||
s3_root_password:
|
||||
file: ./secrets/s3_root_password
|
||||
s3_access_key_id:
|
||||
file: ./secrets/s3_access_key_id
|
||||
s3_secret_access_key:
|
||||
file: ./secrets/s3_secret_access_key
|
||||
s3_config:
|
||||
file: ./secrets/s3_config
|
||||
@@ -0,0 +1,91 @@
|
||||
# Runbook: лабораторная 6
|
||||
|
||||
Открыть [dashboard](http://localhost:3000/d/messenger-lab6). Выбрать время инцидента,
|
||||
а не только последние пять минут. Все шаги диагностики ниже используют сохранённые
|
||||
Prometheus/Loki/Tempo данные. Команды управления выполняются **после** диагноза.
|
||||
Внешняя доставка alerts не настроена: состояние проверяется в Prometheus через
|
||||
Grafana datasource proxy (`research/observations.py`).
|
||||
|
||||
## api
|
||||
|
||||
1. Проверить `API 5xx / all business requests`, RPS и p95/p99. Health и ошибочный
|
||||
пароль не являются серверными 5xx. Посмотреть обе пары burn rate: 2m/30s и 6m/1m.
|
||||
2. В Explore → Loki выполнить
|
||||
`{service_name="messenger-api"} | json | status >= 500`.
|
||||
По `request_id` найти ответ пользователя; скопировать `trace_id` или нажать TraceID.
|
||||
3. В Tempo открыть trace. Развернуть server span и `postgresql.transaction`:
|
||||
медленный `postgresql.SELECT/INSERT` с `error.type=_WaitTimeout` и длительностью
|
||||
около 2s указывает на срабатывание I/O timeout ожидания PostgreSQL. Если сервер
|
||||
успел вернуть `LockNotAvailable`, дополнительно подтверждена блокировка.
|
||||
По одному `_WaitTimeout` нельзя отличить SQL lock от других причин медленной БД.
|
||||
Это отличается от медленного handler:
|
||||
при нём server span длинный, а DB spans короткие. `db.pool{state="used"}`
|
||||
показывает занятые соединения, `waiting` — ожидание свободного соединения.
|
||||
Наш эксперимент блокирует только messages, поэтому health остаётся успешным.
|
||||
4. После определения причины дождаться окончания ограниченного failpoint.
|
||||
Для досрочного восстановления своего учебного эксперимента:
|
||||
|
||||
```sh
|
||||
docker compose exec -T postgres psql -U messenger -d messenger -c \
|
||||
"SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE application_name='lab6-incident'"
|
||||
```
|
||||
|
||||
5. Проверить успешные записи, короткие DB spans и спад error ratio.
|
||||
Alert должен перейти firing → inactive (resolved); long-window ratio может
|
||||
оставаться высоким, но short window уже должен очищаться. Состояния измеряются
|
||||
раз в 5s; не объявлять recovery по одному успешному запросу.
|
||||
|
||||
## ready
|
||||
|
||||
1. Проверить `Outstanding jobs` и `Oldest unfinished job age`. Это очередь
|
||||
незавершённых **durable jobs в БД**, включая outbox/retry/processing;
|
||||
длина Redis Stream сама по себе не отражает всю принятую работу.
|
||||
2. Сопоставить worker capacity/busy, process CPU/RSS и rate успешных jobs.
|
||||
Если capacity исчезла, successful jobs остановились, а acceptance продолжается
|
||||
и backlog/age растут — worker не обрабатывает очередь. После 30s устаревшие
|
||||
gauge series удаляются Collector; краткое сохранение старого значения допустимо.
|
||||
3. В Loki:
|
||||
`{service_name="messenger-api"} | json | event="attachment.accepted"`.
|
||||
Открыть trace: виден API → `attachments publish`, но в интервале простоя
|
||||
нет нового `attachments process`. После восстановления в том же trace появятся
|
||||
consumer → S3 GET/PUT → PostgreSQL UPDATE. Retry — отдельные producer/consumer
|
||||
spans с возрастающим `job.attempt` и логом `job.retry`.
|
||||
4. Восстановить worker: `docker compose start worker`.
|
||||
5. Проверить уменьшение backlog до 0, age до 0, возврат successful jobs и capacity.
|
||||
Ready SLO остаётся нарушенным для опоздавших заданий: позднее ready не превращает
|
||||
уже нарушенный 30s deadline в успешный. Дождаться resolved в обеих парах окон.
|
||||
|
||||
## telemetry
|
||||
|
||||
1. Проверить `up{job="collector"}`, `otelcol_exporter_queue_size` и
|
||||
`otelcol_exporter_send_failed_log_records_total`. Ошибки экспортёра сами по себе
|
||||
не равны числу окончательно потерянных записей: возможны retries.
|
||||
2. При живом Collector и ошибках только `otlphttp/loki` проверить доступность
|
||||
datasource Loki. Если API принимает сообщения и метрики/traces приходят,
|
||||
это отдельный отказ хранилища логов, а не отказ бизнес-API.
|
||||
3. Восстановить `docker compose start loki`. Collector ограничен 256 MiB,
|
||||
memory limiter 192 MiB, очередь каждого экспортёра — 64 batch, максимальный
|
||||
batch 128 records, retry не дольше 15s. SDK очереди traces/logs — по 256 записей;
|
||||
сеть работает в фоновых потоках. При исчерпании лимитов telemetry теряется.
|
||||
4. Проверить возобновление Loki ingestion и опустошение exporter queue.
|
||||
В `lab6-checks.json` число потерянных request logs считается по известным
|
||||
request IDs после восстановления, отдельно от счётчиков неудачных экспортов.
|
||||
|
||||
## SLO и интерпретация
|
||||
|
||||
- API: ≥99% business responses за скользящий час без 5xx. 4xx входят в
|
||||
denominator, но не numerator. Отказы LB/сети до API требуют внешнего probe
|
||||
и этим SLI не покрываются.
|
||||
- Изображения: ≥95% accepted jobs готовы за ≤30s; cohort оценивается после
|
||||
истечения deadline. Failed и не завершившиеся задания входят в bad.
|
||||
Скользящее окно — час **созревших** cohorts (acceptance сдвинута на 30s).
|
||||
- Fast burn: 6× бюджета одновременно за 2m и 30s, `for: 15s`.
|
||||
Slow burn: 3× за 6m и 1m, `for: 30s`. Это учебные окна для 1h SLO,
|
||||
а не механически перенесённые production окна для 30 дней.
|
||||
- Guards: ≥30 API requests / ≥5 mature jobs в long window и ≥2 события в short.
|
||||
Нет данных — нет утверждения об успешном SLO; проверять scrape и traffic.
|
||||
- `APIHighErrorRate` — single-window comparator (5%/30s, for 5s),
|
||||
`AttachmentQueueLag` — отдельный эксплуатационный warning (>20s, for 10s).
|
||||
Сравнивать их с burn alerts, а не считать любой warning доказательством
|
||||
исчерпания бюджета. Данные одного короткого эксперимента не доказывают
|
||||
соблюдение SLO в течение полного часа или отсутствие ложных срабатываний вообще.
|
||||
@@ -0,0 +1,40 @@
|
||||
# Схема и транзакции
|
||||
|
||||
Лабораторная 5, оценка 4: клиент → HAProxy → FastAPI × 2 → PostgreSQL + Valkey sessions + приватный S3; outbox publisher → Valkey Streams broker → worker.
|
||||
|
||||
| Таблица PostgreSQL | Назначение и ограничения |
|
||||
| --- | --- |
|
||||
| users | UUID PK; username UNIQUE; password_hash Argon2id; display_name; created_at |
|
||||
| chats | UUID PK; title; created_at |
|
||||
| chat_members | PK (chat_id, user_id); FK на chats и users |
|
||||
| messages | UUID PK; FK на чат и членство отправителя; text до 4000 code points; created_at |
|
||||
| message_attachments | FK на messages и attachments; до 10 позиций; UNIQUE(message_id, position); PK(message_id, attachment_id) |
|
||||
| attachments | UUID PK; FK chat_id и (chat_id, owner_id); UNIQUE object_key; original metadata; status, variants, error_code, created_at/updated_at |
|
||||
| attachment_jobs | UUID job_id PK; UNIQUE attachment_id/staging_key; version, attempt/budget, lease, retention |
|
||||
| attachment_outbox | PK (job_id, attempt); available_at, published_at; индекс неопубликованных |
|
||||
| attachment_job_events | Audit переходов, safe error_code, attempt, время |
|
||||
| attachment_uploads | UUID PK, UNIQUE object_key, created_at; долговечный журнал незавершённых PUT |
|
||||
| message_idempotency | PK (user_id, chat_id, key); request_text, request_attachment_ids, message_id FK, expires_at |
|
||||
| schema_migrations | Имя и SHA-256 применённой миграции; applied_at |
|
||||
|
||||
В PostgreSQL нет bytes оригинала. `object_key=attachments/<UUID>/v1/original` назначает сервер; filename не участвует в ключе. До ответа 202 в таблице attachments есть status=queued, variants=[], error_code=null. Миграция 004 добавляет status, variants, error_code, updated_at; уже существующие ready-вложения lab4 не меняют ключи и остаются доступны.
|
||||
|
||||
`attachment_jobs` хранит один job_id на attachment_id, staging_key, pipeline_version=1, текущую попытку, бюджет, lease_token/lease_until и отметку удаления staging. `attachment_outbox(job_id,attempt)` создаётся в той же транзакции, что metadata/намерение. Dispatcher отмечает published_at после XADD; потеря ответа вызывает безопасный дубль. Отложенный retry получает отдельную строку outbox с available_at. `attachment_job_events` — audit переходов и явных replay без credentials и текста исключений.
|
||||
|
||||
Данные при 202: приватный S3 `attachments/staging/<UUID>` и COMMIT metadata+job+outbox. Готовность: worker сохраняет неизменённый original и три варианта по детерминированным ключам, затем атомарно фиксирует status=ready и JSON metadata. До ready content отвечает 409. Worker проверяет lease_token под блокировкой строк перед записью результатов. Cleaner не удаляет staging queued/processing; после ready удаляет staging, а failed хранит до STAGING_RETENTION_SECONDS.
|
||||
|
||||
Индексы: UNIQUE users(username); chat_members(user_id, chat_id); messages(chat_id, created_at, id); attachments(chat_id); attachment_uploads(created_at). Первичные/уникальные ключи индексируют поиск ID и object_key.
|
||||
|
||||
Создание чата и его участников атомарно. Неизвестный участник откатывает весь чат. Сообщение проверяет членство и все attachment_ids в том же чате, затем одной транзакцией записывает messages, message_attachments и результат идемпотентности. Составной FK дополнительно запрещает отправителя вне чата. Пустой/пробельный text разрешён только с вложением: это проверяет модель запроса; FK таблицы message_attachments запрещают несуществующие ссылки. Удаления вложений в HTTP API нет.
|
||||
|
||||
Общая advisory transaction lock сериализует область пользователь + POST + путь чата + Idempotency-Key. Сравниваются text и упорядоченный нормализованный список UUID; отсутствие attachment_ids эквивалентно []. Совпадение возвращает исходный 201 и Message, отличие — 409. TTL не меньше 24 часов. Чтение истории использует keyset (created_at,id), без кэша; на следующем запросе видны COMMIT любой реплики. При конкурентных вставках snapshot между страницами нет; запись с более ранними часами может оказаться до уже пройденного cursor.
|
||||
|
||||
Upload сначала полностью принимает ограниченный поток во временный файл и проверяет PNG/JPEG. Затем отдельной транзакцией фиксирует attachment_uploads. В следующей транзакции берётся advisory lock object_key, выполняется S3 PUT, вставляются queued metadata, job и outbox, удаляется журнал. 202 отправляется после COMMIT. S3 и PostgreSQL не образуют распределённую транзакцию.
|
||||
|
||||
Если PUT прошёл, а COMMIT нет, остаётся журнал. Если исход COMMIT неизвестен, немедленного удаления нет: metadata/job могли сохраниться, а удаление журнала атомарно с ними. Cleaner раз в 30 секунд берёт до 100 записей старше grace period (по умолчанию 3600 s, минимум 60 s), пробует ту же advisory lock и повторно проверяет запись. Активная загрузка пропускается; при наличии ready-метаданных удаляется только запись журнала; иначе удаляется S3 object и затем журнал. DELETE объекта идемпотентен. Сбой S3/БД откладывает обработку; следующий проход повторяет её. Это компенсация журналируемых загрузок уровня 4, без полного обхода bucket.
|
||||
|
||||
Миграции 001 и 002 сохранены. Миграция 003 добавляет вложения, связи, журнал и список вложений в ключе повтора; старые сообщения имеют пустой список. DDL выполняет один migrate-service под общей advisory lock; API проверяют версии и checksum.
|
||||
|
||||
Сервис valkey хранит только серверные сессии; отдельный сервис broker хранит Stream. Сессии: session:SHA256(token) → user_id, expires_at, TTL. Включены AOF appendfsync always и постоянный volume. Сырой токен не записывается в evidence. PostgreSQL, Valkey и S3 bucket проверяются /health/ready; /health/live не обращается к зависимостям.
|
||||
|
||||
Основа схемы перенесена из lab4; команды, измерения и ограничения прежней версии — в [REPORT.lab4.md](../REPORT.lab4.md). Исходные отчёты и результаты lab2/lab3 сохранены отдельно. Текущая лабораторная описывается в [REPORT.md](../REPORT.md).
|
||||
@@ -0,0 +1,66 @@
|
||||
UTC: 2026-10-22T11:17:22.562634+00:00
|
||||
Host: Darwin arm64 Python 3.11.9
|
||||
Docker: 29.0.1 aarch64 CPUs=10 RAM=8217448448
|
||||
Commit: 9961362 + working tree
|
||||
Command: ./scripts/check-docker.sh; project: lab2-check-ae280d00db
|
||||
PASS: start.sh built and started the stack, readiness reached
|
||||
api image=sha256:0b3dc5056cced642eea5ade11d235dce9f69f0d9e6d63956ce9479f2deaa10a0 started=2026-10-22T11:17:27.057932632Z memory=268435456 nano_cpus=1000000000
|
||||
postgres image=sha256:f3bd19c606e442c3d7bdfa8002e03fe260a1023351e0ea4598032022b68dd6e3 started=2026-10-22T11:17:24.411773881Z memory=536870912 nano_cpus=1000000000
|
||||
valkey image=sha256:fea8b3e67b15729d4bb70589eb03367bab9ad1ee89c876f54327fc7c6e618571 started=2026-10-22T11:17:24.406509756Z memory=134217728 nano_cpus=500000000
|
||||
PASS: non-root UID 10001; Python PID 1; healthcheck; named volumes; no API data volume
|
||||
Проверяем сервер: http://127.0.0.1:61900
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab2/mtusi2026lab2/.venv/bin/python scripts/check_contract.py
|
||||
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab2/mtusi2026lab2/.venv/bin/python tests/smoke.py --base-url "http://127.0.0.1:61900"
|
||||
PASS: lab2 public API scenario; infrastructure and grading evidence remain separate.
|
||||
test_chat_access (test_api.ApiTests.test_chat_access) ... ok
|
||||
test_chat_pagination_and_cursor_owner (test_api.ApiTests.test_chat_pagination_and_cursor_owner) ... ok
|
||||
test_cursor_cannot_be_used_for_another_chat_or_changed (test_api.ApiTests.test_cursor_cannot_be_used_for_another_chat_or_changed) ... ok
|
||||
test_default_page_size (test_api.ApiTests.test_default_page_size) ... ok
|
||||
test_duplicate_username (test_api.ApiTests.test_duplicate_username) ... ok
|
||||
test_framework_errors_and_request_ids (test_api.ApiTests.test_framework_errors_and_request_ids) ... ok
|
||||
test_identification_and_authentication_order (test_api.ApiTests.test_identification_and_authentication_order) ... ok
|
||||
test_invalid_chat_members (test_api.ApiTests.test_invalid_chat_members) ... ok
|
||||
test_invalid_chat_title (test_api.ApiTests.test_invalid_chat_title) ... ok
|
||||
test_invalid_limits_and_cursors (test_api.ApiTests.test_invalid_limits_and_cursors) ... ok
|
||||
test_invalid_message_fields (test_api.ApiTests.test_invalid_message_fields) ... ok
|
||||
test_invalid_user_fields (test_api.ApiTests.test_invalid_user_fields) ... ok
|
||||
test_malformed_json_and_encoding (test_api.ApiTests.test_malformed_json_and_encoding) ... ok
|
||||
test_message_pagination_and_changing_limit (test_api.ApiTests.test_message_pagination_and_changing_limit) ... ok
|
||||
test_unicode_and_text_preservation (test_api.ApiTests.test_unicode_and_text_preservation) ... ok
|
||||
test_wrong_content_type (test_api.ApiTests.test_wrong_content_type) ... ok
|
||||
test_simultaneous_registration_of_one_username (test_concurrency.ConcurrencyTests.test_simultaneous_registration_of_one_username) ... ok
|
||||
test_ten_clients_send_one_hundred_messages (test_concurrency.ConcurrencyTests.test_ten_clients_send_one_hundred_messages) ... ok
|
||||
test_logout_revokes_only_one_session (test_sessions.SessionTests.test_logout_revokes_only_one_session) ... ok
|
||||
test_password_validation (test_sessions.SessionTests.test_password_validation) ... ok
|
||||
test_sql_like_message_is_plain_text (test_sessions.SessionTests.test_sql_like_message_is_plain_text) ... ok
|
||||
test_unicode_password_with_colon_and_private_response (test_sessions.SessionTests.test_unicode_password_with_colon_and_private_response) ... ok
|
||||
test_wrong_password_and_basic_only_for_login (test_sessions.SessionTests.test_wrong_password_and_basic_only_for_login) ... ok
|
||||
|
||||
----------------------------------------------------------------------
|
||||
Ran 23 tests in 5.743s
|
||||
|
||||
OK
|
||||
PASS: 10 clients sent 100 messages; history contains exactly 100 unique messages.
|
||||
PASS: demo.sh registered users, logged in and read two messages
|
||||
PASS: Argon2id in PostgreSQL; logs do not contain message text, passwords or session tokens
|
||||
PASS: failed membership insert rolled back chat; migrations run twice without duplicate data
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab2/mtusi2026lab2/.venv/bin/python tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://127.0.0.1:61900"
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
Restart api 2026-10-22T11:17:27.057932632Z -> 2026-10-22T11:17:45.460071626Z
|
||||
Restart postgres 2026-10-22T11:17:24.411773881Z -> 2026-10-22T11:17:44.637353001Z
|
||||
Restart valkey 2026-10-22T11:17:24.406509756Z -> 2026-10-22T11:17:44.670095293Z
|
||||
PASS: API, PostgreSQL and Valkey really restarted; previous token still works
|
||||
PASS: valkey unavailable -> 503 in 0.11s; anonymous -> 401; readiness 503, liveness 200
|
||||
API ready
|
||||
PASS: valkey recovered without a new login
|
||||
PASS: postgres unavailable -> 503 in 2.01s; anonymous -> 401; readiness 503, liveness 200
|
||||
API ready
|
||||
PASS: postgres recovered without a new login
|
||||
API ready
|
||||
PASS: SESSION_TTL_SECONDS=2; valid token expires without logout
|
||||
PASS: SIGTERM with unfinished request: 5.54s; new requests refused after 0.16s; exit=0; OOM=false
|
||||
PASS: all lab2 Docker checks completed
|
||||
PASS: stop.sh removed containers and preserved named volumes
|
||||
Removed only temporary Compose project and its test volumes: lab2-check-ae280d00db
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,403 @@
|
||||
Command: ./scripts/check-docker.sh; project: lab3-check-a31f544685
|
||||
Docker: 29.0.1 aarch64 CPUs=10 RAM=8217448448
|
||||
Local .env and secrets are ready; values are not printed
|
||||
#1 [internal] load local bake definitions
|
||||
#1 reading from stdin 1.05kB done
|
||||
#1 DONE 0.0s
|
||||
|
||||
#2 [migrate internal] load build definition from Dockerfile
|
||||
#2 transferring dockerfile: 733B done
|
||||
#2 DONE 0.0s
|
||||
|
||||
#3 [api internal] load metadata for docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
|
||||
#3 DONE 0.0s
|
||||
|
||||
#4 [api internal] load .dockerignore
|
||||
#4 transferring context: 167B done
|
||||
#4 DONE 0.0s
|
||||
|
||||
#5 [migrate internal] load build context
|
||||
#5 transferring context: 809B done
|
||||
#5 DONE 0.0s
|
||||
|
||||
#6 [migrate 1/8] FROM docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
|
||||
#6 resolve docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3 done
|
||||
#6 DONE 0.0s
|
||||
|
||||
#7 [migrate 2/8] WORKDIR /app
|
||||
#7 CACHED
|
||||
|
||||
#8 [migrate 3/8] COPY requirements.txt .
|
||||
#8 CACHED
|
||||
|
||||
#9 [migrate 4/8] RUN python -m pip install --no-cache-dir -r requirements.txt && python -m pip check
|
||||
#9 CACHED
|
||||
|
||||
#10 [migrate 6/8] COPY src/ ./src/
|
||||
#10 CACHED
|
||||
|
||||
#11 [migrate 7/8] COPY migrations/ ./migrations/
|
||||
#11 CACHED
|
||||
|
||||
#12 [migrate 5/8] RUN groupadd --gid 10001 app && useradd --uid 10001 --gid app --no-create-home app
|
||||
#12 CACHED
|
||||
|
||||
#13 [migrate 8/8] COPY research/ ./research/
|
||||
#13 CACHED
|
||||
|
||||
#14 [api] exporting to image
|
||||
#14 exporting layers done
|
||||
#14 exporting manifest sha256:06163e45d3676c7cf4f7a1c7bfe108ed3b98559ebb485c7a09d1e87e6f418d17 done
|
||||
#14 exporting config sha256:3fed7460e9510eef7b2928f8d600d06d5d65329f74d820beac13809476b11398 done
|
||||
#14 exporting attestation manifest sha256:352319cde788733f2f237a2a78204630c542e7ca906cff14c902046e8c1d2e5b 0.0s done
|
||||
#14 exporting manifest list sha256:848e0c76f1678f35dd1b483e7c54552466999551ba7e77c0062bf346671e2902 done
|
||||
#14 naming to docker.io/library/mtusi-messenger:lab3 done
|
||||
#14 unpacking to docker.io/library/mtusi-messenger:lab3 done
|
||||
#14 DONE 0.1s
|
||||
|
||||
#15 [migrate] exporting to image
|
||||
#15 exporting layers done
|
||||
#15 exporting manifest sha256:a59568d34e2c5e8b4d23308342601fdaf8c07e78ec3065878679a067c082e952 done
|
||||
#15 exporting config sha256:b67b805ba1b7dd90f0a3e8444455488fe8c7f4118f14d671501dd63e85b59c97 done
|
||||
#15 exporting attestation manifest sha256:b1853c07ddac46c4b398b475052f334aa2753c2fe971e0e4fb2517b0e3e13656 0.0s done
|
||||
#15 exporting manifest list sha256:eff1abdd662eb60e2741e6ce6b8f364467d2817dafeef9754518247124a780eb done
|
||||
#15 naming to docker.io/library/mtusi-messenger:lab3 done
|
||||
#15 unpacking to docker.io/library/mtusi-messenger:lab3 done
|
||||
#15 DONE 0.1s
|
||||
|
||||
#16 [api] resolving provenance for metadata file
|
||||
#16 DONE 0.0s
|
||||
|
||||
#17 [migrate] resolving provenance for metadata file
|
||||
#17 DONE 0.0s
|
||||
mtusi-messenger:lab3 Built
|
||||
mtusi-messenger:lab3 Built
|
||||
Network lab3-check-a31f544685_default Creating
|
||||
Network lab3-check-a31f544685_default Created
|
||||
Volume lab3-check-a31f544685_postgres_data Creating
|
||||
Volume lab3-check-a31f544685_postgres_data Created
|
||||
Volume lab3-check-a31f544685_valkey_data Creating
|
||||
Volume lab3-check-a31f544685_valkey_data Created
|
||||
Container lab3-check-a31f544685-valkey-1 Creating
|
||||
Container lab3-check-a31f544685-postgres-1 Creating
|
||||
Container lab3-check-a31f544685-valkey-1 Created
|
||||
Container lab3-check-a31f544685-postgres-1 Created
|
||||
Container lab3-check-a31f544685-migrate-1 Creating
|
||||
Container lab3-check-a31f544685-migrate-1 Created
|
||||
Container lab3-check-a31f544685-api-2 Creating
|
||||
Container lab3-check-a31f544685-api-1 Creating
|
||||
Container lab3-check-a31f544685-api-1 Created
|
||||
Container lab3-check-a31f544685-api-2 Created
|
||||
Container lab3-check-a31f544685-lb-1 Creating
|
||||
Container lab3-check-a31f544685-lb-1 Created
|
||||
Container lab3-check-a31f544685-valkey-1 Starting
|
||||
Container lab3-check-a31f544685-postgres-1 Starting
|
||||
Container lab3-check-a31f544685-valkey-1 Started
|
||||
Container lab3-check-a31f544685-postgres-1 Started
|
||||
Container lab3-check-a31f544685-postgres-1 Waiting
|
||||
Container lab3-check-a31f544685-postgres-1 Healthy
|
||||
Container lab3-check-a31f544685-migrate-1 Starting
|
||||
Container lab3-check-a31f544685-migrate-1 Started
|
||||
Container lab3-check-a31f544685-migrate-1 Waiting
|
||||
Container lab3-check-a31f544685-postgres-1 Waiting
|
||||
Container lab3-check-a31f544685-valkey-1 Waiting
|
||||
Container lab3-check-a31f544685-valkey-1 Healthy
|
||||
Container lab3-check-a31f544685-postgres-1 Healthy
|
||||
Container lab3-check-a31f544685-migrate-1 Exited
|
||||
Container lab3-check-a31f544685-api-2 Starting
|
||||
Container lab3-check-a31f544685-api-2 Started
|
||||
Container lab3-check-a31f544685-api-1 Starting
|
||||
Container lab3-check-a31f544685-api-1 Started
|
||||
Container lab3-check-a31f544685-api-2 Waiting
|
||||
Container lab3-check-a31f544685-api-1 Waiting
|
||||
Container lab3-check-a31f544685-api-2 Healthy
|
||||
Container lab3-check-a31f544685-api-1 Healthy
|
||||
Container lab3-check-a31f544685-lb-1 Starting
|
||||
Container lab3-check-a31f544685-lb-1 Started
|
||||
API ready through LB: 7f6a80b69e32, f973eaec4d52
|
||||
PASS: 2 real API containers, CPU/RAM limits, API non-root and no public ports; only LB published
|
||||
Проверяем сервер: http://127.0.0.1:58474
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab3/mtusi2026lab3/.venv/bin/python scripts/check_contract.py
|
||||
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab3/mtusi2026lab3/.venv/bin/python tests/smoke.py --base-url "http://127.0.0.1:58474"
|
||||
PASS: lab3 public API scenario; infrastructure and grading evidence remain separate.
|
||||
test_chat_access (test_api.ApiTests.test_chat_access) ... ok
|
||||
test_chat_pagination_and_cursor_owner (test_api.ApiTests.test_chat_pagination_and_cursor_owner) ... ok
|
||||
test_cursor_cannot_be_used_for_another_chat_or_changed (test_api.ApiTests.test_cursor_cannot_be_used_for_another_chat_or_changed) ... ok
|
||||
test_default_page_size (test_api.ApiTests.test_default_page_size) ... ok
|
||||
test_duplicate_username (test_api.ApiTests.test_duplicate_username) ... ok
|
||||
test_framework_errors_and_request_ids (test_api.ApiTests.test_framework_errors_and_request_ids) ... ok
|
||||
test_identification_and_authentication_order (test_api.ApiTests.test_identification_and_authentication_order) ... ok
|
||||
test_invalid_chat_members (test_api.ApiTests.test_invalid_chat_members) ... ok
|
||||
test_invalid_chat_title (test_api.ApiTests.test_invalid_chat_title) ... ok
|
||||
test_invalid_limits_and_cursors (test_api.ApiTests.test_invalid_limits_and_cursors) ... ok
|
||||
test_invalid_message_fields (test_api.ApiTests.test_invalid_message_fields) ... ok
|
||||
test_invalid_user_fields (test_api.ApiTests.test_invalid_user_fields) ... ok
|
||||
test_malformed_json_and_encoding (test_api.ApiTests.test_malformed_json_and_encoding) ... ok
|
||||
test_message_pagination_and_changing_limit (test_api.ApiTests.test_message_pagination_and_changing_limit) ... ok
|
||||
test_unicode_and_text_preservation (test_api.ApiTests.test_unicode_and_text_preservation) ... ok
|
||||
test_wrong_content_type (test_api.ApiTests.test_wrong_content_type) ... ok
|
||||
test_simultaneous_registration_of_one_username (test_concurrency.ConcurrencyTests.test_simultaneous_registration_of_one_username) ... ok
|
||||
test_ten_clients_send_one_hundred_messages (test_concurrency.ConcurrencyTests.test_ten_clients_send_one_hundred_messages) ... ok
|
||||
test_conflict_and_semantic_json_replay (test_idempotency.IdempotencyTests.test_conflict_and_semantic_json_replay) ... ok
|
||||
test_key_format_and_missing_key (test_idempotency.IdempotencyTests.test_key_format_and_missing_key) ... ok
|
||||
test_parallel_replay_across_replicas (test_idempotency.IdempotencyTests.test_parallel_replay_across_replicas) ... ok
|
||||
test_replay_rechecks_session_and_access (test_idempotency.IdempotencyTests.test_replay_rechecks_session_and_access) ... ok
|
||||
test_scope_is_user_and_chat (test_idempotency.IdempotencyTests.test_scope_is_user_and_chat) ... ok
|
||||
test_logout_revokes_only_one_session (test_sessions.SessionTests.test_logout_revokes_only_one_session) ... ok
|
||||
test_password_validation (test_sessions.SessionTests.test_password_validation) ... ok
|
||||
test_sql_like_message_is_plain_text (test_sessions.SessionTests.test_sql_like_message_is_plain_text) ... ok
|
||||
test_unicode_password_with_colon_and_private_response (test_sessions.SessionTests.test_unicode_password_with_colon_and_private_response) ... ok
|
||||
test_wrong_password_and_basic_only_for_login (test_sessions.SessionTests.test_wrong_password_and_basic_only_for_login) ... ok
|
||||
|
||||
----------------------------------------------------------------------
|
||||
Ran 28 tests in 6.783s
|
||||
|
||||
OK
|
||||
PASS: 10 clients sent 100 messages; history contains exactly 100 unique messages.
|
||||
Container lab3-check-a31f544685-postgres-1 Running
|
||||
Container lab3-check-a31f544685-postgres-1 Running
|
||||
PASS: migrations only in one-shot service; repeated execution preserves data
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab3/mtusi2026lab3/.venv/bin/python tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://127.0.0.1:58474"
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
PASS: both API, LB, PostgreSQL, Valkey actually restarted; session and confirmed data preserved
|
||||
PASS: SIGTERM with unfinished request: {"instance": "7f6a80b69e32", "unfinished_request": true, "shutdown_seconds": 5.393, "new_connections_refused_seconds": 0.167, "exit_code": 0, "oom_killed": false}
|
||||
Container lab3-check-a31f544685-postgres-1 Waiting
|
||||
Container lab3-check-a31f544685-postgres-1 Healthy
|
||||
Container lab3-check-a31f544685-migrate-1 Starting
|
||||
Container lab3-check-a31f544685-migrate-1 Started
|
||||
Container lab3-check-a31f544685-valkey-1 Waiting
|
||||
Container lab3-check-a31f544685-migrate-1 Waiting
|
||||
Container lab3-check-a31f544685-postgres-1 Waiting
|
||||
Container lab3-check-a31f544685-valkey-1 Healthy
|
||||
Container lab3-check-a31f544685-postgres-1 Healthy
|
||||
Container lab3-check-a31f544685-migrate-1 Exited
|
||||
Container lab3-check-a31f544685-api-1 Starting
|
||||
Container lab3-check-a31f544685-api-1 Started
|
||||
API ready through LB: 7f6a80b69e32, f973eaec4d52
|
||||
Container lab3-check-a31f544685-valkey-1 Stopping
|
||||
Container lab3-check-a31f544685-valkey-1 Stopped
|
||||
PASS: valkey outage: API live=200, ready=503, LB ready=503
|
||||
Container lab3-check-a31f544685-valkey-1 Starting
|
||||
Container lab3-check-a31f544685-valkey-1 Started
|
||||
API ready through LB: 7f6a80b69e32, f973eaec4d52
|
||||
Container lab3-check-a31f544685-postgres-1 Stopping
|
||||
Container lab3-check-a31f544685-postgres-1 Stopped
|
||||
PASS: postgres outage: API live=200, ready=503, LB ready=503
|
||||
Container lab3-check-a31f544685-postgres-1 Starting
|
||||
Container lab3-check-a31f544685-postgres-1 Started
|
||||
API ready through LB: 7f6a80b69e32, f973eaec4d52
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab3/mtusi2026lab3/.venv/bin/python tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:58474"
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||
{"observed_instances_before": 2, "successful_writes": 113, "failed_attempts": 1, "first_recovery_seconds": 2.452}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
Container lab3-check-a31f544685-postgres-1 Waiting
|
||||
Container lab3-check-a31f544685-postgres-1 Healthy
|
||||
Container lab3-check-a31f544685-migrate-1 Starting
|
||||
Container lab3-check-a31f544685-migrate-1 Started
|
||||
Container lab3-check-a31f544685-postgres-1 Waiting
|
||||
Container lab3-check-a31f544685-valkey-1 Waiting
|
||||
Container lab3-check-a31f544685-migrate-1 Waiting
|
||||
Container lab3-check-a31f544685-postgres-1 Healthy
|
||||
Container lab3-check-a31f544685-valkey-1 Healthy
|
||||
Container lab3-check-a31f544685-migrate-1 Exited
|
||||
Container lab3-check-a31f544685-api-1 Starting
|
||||
Container lab3-check-a31f544685-api-1 Started
|
||||
API ready through LB: 7f6a80b69e32, f973eaec4d52
|
||||
Stopped API: 7f6a80b69e32 container=7f6a80b69e3297a4de72e756a6c9d867a108930c1ca0e57922b3821c26fd03e6
|
||||
{"label": "stop 7f6a80b69e32", "duration_seconds": 16.009, "successful_requests": 1576, "successful_writes": 788, "errors": 4, "successful_rps": 98.44, "first_success_after_action_seconds": {"POST": 0.014, "GET": 0.007}, "maximum_success_gap_seconds": {"POST": 1.957, "GET": 2.004}, "action_start_seconds": 1.003, "action_end_seconds": 1.666, "p50_ms": 7.813, "p95_ms": 16.393, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["7f6a80b69e32", "f973eaec4d52"]}
|
||||
Container lab3-check-a31f544685-postgres-1 Waiting
|
||||
Container lab3-check-a31f544685-postgres-1 Healthy
|
||||
Container lab3-check-a31f544685-migrate-1 Starting
|
||||
Container lab3-check-a31f544685-migrate-1 Started
|
||||
Container lab3-check-a31f544685-postgres-1 Waiting
|
||||
Container lab3-check-a31f544685-valkey-1 Waiting
|
||||
Container lab3-check-a31f544685-migrate-1 Waiting
|
||||
Container lab3-check-a31f544685-valkey-1 Healthy
|
||||
Container lab3-check-a31f544685-postgres-1 Healthy
|
||||
Container lab3-check-a31f544685-migrate-1 Exited
|
||||
Container lab3-check-a31f544685-api-1 Starting
|
||||
Container lab3-check-a31f544685-api-1 Started
|
||||
API ready through LB: 7f6a80b69e32, f973eaec4d52
|
||||
Stopped API: f973eaec4d52 container=f973eaec4d5290eb73cd96949877ccb8df1213523fce0ec05c6b40c1136bb31b
|
||||
{"label": "stop f973eaec4d52", "duration_seconds": 16.015, "successful_requests": 1574, "successful_writes": 786, "errors": 4, "successful_rps": 98.28, "first_success_after_action_seconds": {"POST": 0.002, "GET": 0.008}, "maximum_success_gap_seconds": {"POST": 2.006, "GET": 2.006}, "action_start_seconds": 1.005, "action_end_seconds": 1.713, "p50_ms": 7.98, "p95_ms": 17.217, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["7f6a80b69e32", "f973eaec4d52"]}
|
||||
Container lab3-check-a31f544685-postgres-1 Waiting
|
||||
Container lab3-check-a31f544685-postgres-1 Healthy
|
||||
Container lab3-check-a31f544685-migrate-1 Starting
|
||||
Container lab3-check-a31f544685-migrate-1 Started
|
||||
Container lab3-check-a31f544685-postgres-1 Waiting
|
||||
Container lab3-check-a31f544685-valkey-1 Waiting
|
||||
Container lab3-check-a31f544685-migrate-1 Waiting
|
||||
Container lab3-check-a31f544685-valkey-1 Healthy
|
||||
Container lab3-check-a31f544685-postgres-1 Healthy
|
||||
Container lab3-check-a31f544685-migrate-1 Exited
|
||||
Container lab3-check-a31f544685-api-2 Starting
|
||||
Container lab3-check-a31f544685-api-2 Started
|
||||
API ready through LB: 7f6a80b69e32, f973eaec4d52
|
||||
Local .env and secrets are ready; values are not printed
|
||||
#1 [internal] load local bake definitions
|
||||
#1 reading from stdin 1.03kB done
|
||||
#1 DONE 0.0s
|
||||
|
||||
#2 [internal] load build definition from Dockerfile
|
||||
#2 transferring dockerfile: 733B done
|
||||
#2 DONE 0.0s
|
||||
|
||||
#3 [internal] load metadata for docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
|
||||
#3 DONE 0.0s
|
||||
|
||||
#4 [internal] load .dockerignore
|
||||
#4 transferring context: 167B done
|
||||
#4 DONE 0.0s
|
||||
|
||||
#5 [internal] load build context
|
||||
#5 transferring context: 809B done
|
||||
#5 DONE 0.0s
|
||||
|
||||
#6 [1/8] FROM docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
|
||||
#6 resolve docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3 0.0s done
|
||||
#6 DONE 0.0s
|
||||
|
||||
#7 [4/8] RUN python -m pip install --no-cache-dir -r requirements.txt && python -m pip check
|
||||
#7 CACHED
|
||||
|
||||
#8 [3/8] COPY requirements.txt .
|
||||
#8 CACHED
|
||||
|
||||
#9 [5/8] RUN groupadd --gid 10001 app && useradd --uid 10001 --gid app --no-create-home app
|
||||
#9 CACHED
|
||||
|
||||
#10 [6/8] COPY src/ ./src/
|
||||
#10 CACHED
|
||||
|
||||
#11 [2/8] WORKDIR /app
|
||||
#11 CACHED
|
||||
|
||||
#12 [7/8] COPY migrations/ ./migrations/
|
||||
#12 CACHED
|
||||
|
||||
#13 [8/8] COPY research/ ./research/
|
||||
#13 CACHED
|
||||
|
||||
#14 exporting to image
|
||||
#14 exporting layers done
|
||||
#14 exporting manifest sha256:06163e45d3676c7cf4f7a1c7bfe108ed3b98559ebb485c7a09d1e87e6f418d17 done
|
||||
#14 exporting config sha256:3fed7460e9510eef7b2928f8d600d06d5d65329f74d820beac13809476b11398 done
|
||||
#14 exporting attestation manifest sha256:3a88ce0975ddcbab53c3ecb185da08ef114e818e9db952983e997cd47add67bf done
|
||||
#14 exporting manifest list sha256:433a8a3293202a92f092574052b474e8373c0d58649cbcf1c7ebc1af17933d3c done
|
||||
#14 naming to docker.io/library/mtusi-messenger:lab3 done
|
||||
#14 unpacking to docker.io/library/mtusi-messenger:lab3 done
|
||||
#14 DONE 0.1s
|
||||
|
||||
#15 resolving provenance for metadata file
|
||||
#15 DONE 0.0s
|
||||
mtusi-messenger:lab3 Built
|
||||
Local .env and secrets are ready; values are not printed
|
||||
Container lab3-check-a31f544685-postgres-1 Running
|
||||
#1 [internal] load local bake definitions
|
||||
#1 reading from stdin 576B done
|
||||
#1 DONE 0.0s
|
||||
|
||||
#2 [internal] load build definition from Dockerfile
|
||||
#2 transferring dockerfile: 733B done
|
||||
#2 DONE 0.0s
|
||||
|
||||
#3 [internal] load metadata for docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
|
||||
#3 DONE 0.0s
|
||||
|
||||
#4 [internal] load .dockerignore
|
||||
#4 transferring context: 167B done
|
||||
#4 DONE 0.0s
|
||||
|
||||
#5 [internal] load build context
|
||||
#5 transferring context: 809B done
|
||||
#5 DONE 0.0s
|
||||
|
||||
#6 [1/8] FROM docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
|
||||
#6 resolve docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3 done
|
||||
#6 DONE 0.0s
|
||||
|
||||
#7 [3/8] COPY requirements.txt .
|
||||
#7 CACHED
|
||||
|
||||
#8 [4/8] RUN python -m pip install --no-cache-dir -r requirements.txt && python -m pip check
|
||||
#8 CACHED
|
||||
|
||||
#9 [5/8] RUN groupadd --gid 10001 app && useradd --uid 10001 --gid app --no-create-home app
|
||||
#9 CACHED
|
||||
|
||||
#10 [6/8] COPY src/ ./src/
|
||||
#10 CACHED
|
||||
|
||||
#11 [7/8] COPY migrations/ ./migrations/
|
||||
#11 CACHED
|
||||
|
||||
#12 [2/8] WORKDIR /app
|
||||
#12 CACHED
|
||||
|
||||
#13 [8/8] COPY research/ ./research/
|
||||
#13 CACHED
|
||||
|
||||
#14 exporting to image
|
||||
#14 exporting layers done
|
||||
#14 exporting manifest sha256:a59568d34e2c5e8b4d23308342601fdaf8c07e78ec3065878679a067c082e952 done
|
||||
#14 exporting config sha256:b67b805ba1b7dd90f0a3e8444455488fe8c7f4118f14d671501dd63e85b59c97 done
|
||||
#14 exporting attestation manifest sha256:25ff7934ca5161da9b6c345af61b6a10b452580e4d349b35939d32feb66664ea done
|
||||
#14 exporting manifest list sha256:3e102cd9a2c1e81e95aed4a0d4de1ac94352370595fced350257813f7a07bef8 done
|
||||
#14 naming to docker.io/library/mtusi-messenger:lab3 done
|
||||
#14 unpacking to docker.io/library/mtusi-messenger:lab3 done
|
||||
#14 DONE 0.0s
|
||||
|
||||
#15 resolving provenance for metadata file
|
||||
#15 DONE 0.0s
|
||||
Migrations: already applied
|
||||
API ready through LB: 7f6a80b69e32, f973eaec4d52
|
||||
Replacing API: 7f6a80b69e32
|
||||
Container lab3-check-a31f544685-api-2 Running
|
||||
Container lab3-check-a31f544685-api-3 Creating
|
||||
Container lab3-check-a31f544685-api-3 Created
|
||||
Container lab3-check-a31f544685-api-3 Starting
|
||||
Container lab3-check-a31f544685-api-3 Started
|
||||
API ready through LB: ca9c7a52da66, f973eaec4d52
|
||||
Replacing API: f973eaec4d52
|
||||
Container lab3-check-a31f544685-api-3 Running
|
||||
Container lab3-check-a31f544685-api-4 Creating
|
||||
Container lab3-check-a31f544685-api-4 Created
|
||||
Container lab3-check-a31f544685-api-4 Starting
|
||||
Container lab3-check-a31f544685-api-4 Started
|
||||
API ready through LB: 244bf0b910eb, ca9c7a52da66
|
||||
PASS: all API containers replaced sequentially; stores and LB kept running
|
||||
{"label": "sequential API replacement", "duration_seconds": 23.774, "successful_requests": 2131, "successful_writes": 1063, "errors": 11, "successful_rps": 89.64, "first_success_after_action_seconds": {"POST": 0.018, "GET": 0.004}, "maximum_success_gap_seconds": {"POST": 1.059, "GET": 2.034}, "action_start_seconds": 1.005, "action_end_seconds": 19.63, "p50_ms": 7.723, "p95_ms": 13.565, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["244bf0b910eb", "7f6a80b69e32", "ca9c7a52da66", "f973eaec4d52"]}
|
||||
PASS: all lab3 grade-4 checks completed
|
||||
Container lab3-check-a31f544685-lb-1 Stopping
|
||||
Container lab3-check-a31f544685-lb-1 Stopped
|
||||
Container lab3-check-a31f544685-lb-1 Removing
|
||||
Container lab3-check-a31f544685-lb-1 Removed
|
||||
Container lab3-check-a31f544685-api-3 Stopping
|
||||
Container lab3-check-a31f544685-api-4 Stopping
|
||||
Container lab3-check-a31f544685-api-4 Stopped
|
||||
Container lab3-check-a31f544685-api-4 Removing
|
||||
Container lab3-check-a31f544685-api-4 Removed
|
||||
Container lab3-check-a31f544685-api-3 Stopped
|
||||
Container lab3-check-a31f544685-api-3 Removing
|
||||
Container lab3-check-a31f544685-api-3 Removed
|
||||
Container lab3-check-a31f544685-migrate-1 Stopping
|
||||
Container lab3-check-a31f544685-valkey-1 Stopping
|
||||
Container lab3-check-a31f544685-migrate-1 Stopped
|
||||
Container lab3-check-a31f544685-migrate-1 Removing
|
||||
Container lab3-check-a31f544685-migrate-1 Removed
|
||||
Container lab3-check-a31f544685-postgres-1 Stopping
|
||||
Container lab3-check-a31f544685-valkey-1 Stopped
|
||||
Container lab3-check-a31f544685-valkey-1 Removing
|
||||
Container lab3-check-a31f544685-valkey-1 Removed
|
||||
Container lab3-check-a31f544685-postgres-1 Stopped
|
||||
Container lab3-check-a31f544685-postgres-1 Removing
|
||||
Container lab3-check-a31f544685-postgres-1 Removed
|
||||
Volume lab3-check-a31f544685_valkey_data Removing
|
||||
Network lab3-check-a31f544685_default Removing
|
||||
Volume lab3-check-a31f544685_postgres_data Removing
|
||||
Volume lab3-check-a31f544685_valkey_data Removed
|
||||
Volume lab3-check-a31f544685_postgres_data Removed
|
||||
Network lab3-check-a31f544685_default Removed
|
||||
Removed only temporary test containers and volumes: lab3-check-a31f544685
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,533 @@
|
||||
Command: ./scripts/check-docker.sh; project: lab4-check-5aad8e0065
|
||||
Docker: 29.0.1 aarch64 CPUs=10 RAM=8217448448
|
||||
Local .env and secrets are ready; values are not printed
|
||||
#1 [internal] load local bake definitions
|
||||
#1 reading from stdin 564B done
|
||||
#1 DONE 0.0s
|
||||
|
||||
#2 [internal] load build definition from Dockerfile
|
||||
#2 transferring dockerfile: 733B done
|
||||
#2 DONE 0.0s
|
||||
|
||||
#3 [internal] load metadata for docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
|
||||
#3 DONE 0.0s
|
||||
|
||||
#4 [internal] load .dockerignore
|
||||
#4 transferring context: 167B done
|
||||
#4 DONE 0.0s
|
||||
|
||||
#5 [internal] load build context
|
||||
#5 transferring context: 1.01kB done
|
||||
#5 DONE 0.0s
|
||||
|
||||
#6 [1/8] FROM docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
|
||||
#6 resolve docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3 done
|
||||
#6 DONE 0.0s
|
||||
|
||||
#7 [4/8] RUN python -m pip install --no-cache-dir -r requirements.txt && python -m pip check
|
||||
#7 CACHED
|
||||
|
||||
#8 [7/8] COPY migrations/ ./migrations/
|
||||
#8 CACHED
|
||||
|
||||
#9 [2/8] WORKDIR /app
|
||||
#9 CACHED
|
||||
|
||||
#10 [6/8] COPY src/ ./src/
|
||||
#10 CACHED
|
||||
|
||||
#11 [3/8] COPY requirements.txt .
|
||||
#11 CACHED
|
||||
|
||||
#12 [5/8] RUN groupadd --gid 10001 app && useradd --uid 10001 --gid app --no-create-home app
|
||||
#12 CACHED
|
||||
|
||||
#13 [8/8] COPY research/ ./research/
|
||||
#13 CACHED
|
||||
|
||||
#14 exporting to image
|
||||
#14 exporting layers done
|
||||
#14 exporting manifest sha256:644f5d408505f5982b75f18404860de692ee837b2b2e18f5c92b0b057081b7ec done
|
||||
#14 exporting config sha256:726011343b32f5b6277348f2f8b38f95b8c31754ae0d21336375124a18aa2e0e done
|
||||
#14 exporting attestation manifest sha256:1146b210a20471d58663fb48a3a24ba22986971d0cce8777f912886768f43306 done
|
||||
#14 exporting manifest list sha256:a04b1bfe2aea8391c8f92dd952499559d766cc4b34091963fe57bcbaf6ec015d done
|
||||
#14 naming to docker.io/library/mtusi-messenger:lab4 done
|
||||
#14 unpacking to docker.io/library/mtusi-messenger:lab4 done
|
||||
#14 DONE 0.1s
|
||||
|
||||
#15 resolving provenance for metadata file
|
||||
#15 DONE 0.0s
|
||||
mtusi-messenger:lab4 Built
|
||||
Network lab4-check-5aad8e0065_default Creating
|
||||
Network lab4-check-5aad8e0065_default Created
|
||||
Network lab4-check-5aad8e0065_s3-internal Creating
|
||||
Network lab4-check-5aad8e0065_s3-internal Created
|
||||
Volume lab4-check-5aad8e0065_valkey_data Creating
|
||||
Volume lab4-check-5aad8e0065_valkey_data Created
|
||||
Volume lab4-check-5aad8e0065_postgres_data Creating
|
||||
Volume lab4-check-5aad8e0065_postgres_data Created
|
||||
Volume lab4-check-5aad8e0065_s3_data Creating
|
||||
Volume lab4-check-5aad8e0065_s3_data Created
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Creating
|
||||
Container lab4-check-5aad8e0065-valkey-1 Creating
|
||||
Container lab4-check-5aad8e0065-postgres-1 Creating
|
||||
Container lab4-check-5aad8e0065-valkey-1 Created
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Created
|
||||
Container lab4-check-5aad8e0065-s3-1 Creating
|
||||
Container lab4-check-5aad8e0065-postgres-1 Created
|
||||
Container lab4-check-5aad8e0065-migrate-1 Creating
|
||||
Container lab4-check-5aad8e0065-s3-1 Created
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Creating
|
||||
Container lab4-check-5aad8e0065-migrate-1 Created
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Created
|
||||
Container lab4-check-5aad8e0065-api-2 Creating
|
||||
Container lab4-check-5aad8e0065-api-1 Creating
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Creating
|
||||
Container lab4-check-5aad8e0065-api-1 Created
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Created
|
||||
Container lab4-check-5aad8e0065-api-2 Created
|
||||
Container lab4-check-5aad8e0065-lb-1 Creating
|
||||
Container lab4-check-5aad8e0065-lb-1 Created
|
||||
Container lab4-check-5aad8e0065-postgres-1 Starting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Started
|
||||
Container lab4-check-5aad8e0065-postgres-1 Started
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Starting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-valkey-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
Container lab4-check-5aad8e0065-api-2 Starting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Starting
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Started
|
||||
Container lab4-check-5aad8e0065-api-2 Started
|
||||
Container lab4-check-5aad8e0065-api-1 Starting
|
||||
Container lab4-check-5aad8e0065-api-1 Started
|
||||
Container lab4-check-5aad8e0065-api-2 Waiting
|
||||
Container lab4-check-5aad8e0065-api-1 Waiting
|
||||
Container lab4-check-5aad8e0065-api-2 Healthy
|
||||
Container lab4-check-5aad8e0065-api-1 Healthy
|
||||
Container lab4-check-5aad8e0065-lb-1 Starting
|
||||
Container lab4-check-5aad8e0065-lb-1 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
PASS: 2 real API containers, CPU/RAM limits, API non-root and no public ports; only LB published
|
||||
Проверяем сервер: http://127.0.0.1:57722
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab4/mtusi2026lab4/.venv/bin/python scripts/check_contract.py
|
||||
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab4/mtusi2026lab4/.venv/bin/python tests/smoke.py --base-url "http://127.0.0.1:57722"
|
||||
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
|
||||
test_chat_access (test_api.ApiTests.test_chat_access) ... ok
|
||||
test_chat_pagination_and_cursor_owner (test_api.ApiTests.test_chat_pagination_and_cursor_owner) ... ok
|
||||
test_cursor_cannot_be_used_for_another_chat_or_changed (test_api.ApiTests.test_cursor_cannot_be_used_for_another_chat_or_changed) ... ok
|
||||
test_default_page_size (test_api.ApiTests.test_default_page_size) ... ok
|
||||
test_duplicate_username (test_api.ApiTests.test_duplicate_username) ... ok
|
||||
test_framework_errors_and_request_ids (test_api.ApiTests.test_framework_errors_and_request_ids) ... ok
|
||||
test_identification_and_authentication_order (test_api.ApiTests.test_identification_and_authentication_order) ... ok
|
||||
test_invalid_chat_members (test_api.ApiTests.test_invalid_chat_members) ... ok
|
||||
test_invalid_chat_title (test_api.ApiTests.test_invalid_chat_title) ... ok
|
||||
test_invalid_limits_and_cursors (test_api.ApiTests.test_invalid_limits_and_cursors) ... ok
|
||||
test_invalid_message_fields (test_api.ApiTests.test_invalid_message_fields) ... ok
|
||||
test_invalid_user_fields (test_api.ApiTests.test_invalid_user_fields) ... ok
|
||||
test_malformed_json_and_encoding (test_api.ApiTests.test_malformed_json_and_encoding) ... ok
|
||||
test_message_pagination_and_changing_limit (test_api.ApiTests.test_message_pagination_and_changing_limit) ... ok
|
||||
test_unicode_and_text_preservation (test_api.ApiTests.test_unicode_and_text_preservation) ... ok
|
||||
test_wrong_content_type (test_api.ApiTests.test_wrong_content_type) ... ok
|
||||
test_access_and_authentication (test_attachments.AttachmentTests.test_access_and_authentication) ... ok
|
||||
test_attachment_only_message_and_other_member (test_attachments.AttachmentTests.test_attachment_only_message_and_other_member) ... ok
|
||||
test_chunked_limit_without_content_length (test_attachments.AttachmentTests.test_chunked_limit_without_content_length) ... ok
|
||||
test_content_length_limit_before_body (test_attachments.AttachmentTests.test_content_length_limit_before_body) ... ok
|
||||
test_corruption_and_mime_spoof (test_attachments.AttachmentTests.test_corruption_and_mime_spoof) ... ok
|
||||
test_filename_rejected_before_storage (test_attachments.AttachmentTests.test_filename_rejected_before_storage) ... ok
|
||||
test_foreign_and_unknown_attachment_ids (test_attachments.AttachmentTests.test_foreign_and_unknown_attachment_ids) ... ok
|
||||
test_idempotency_includes_ordered_attachment_ids (test_attachments.AttachmentTests.test_idempotency_includes_ordered_attachment_ids) ... ok
|
||||
test_invalid_attachment_lists (test_attachments.AttachmentTests.test_invalid_attachment_lists) ... ok
|
||||
test_pixel_limit_before_decode (test_attachments.AttachmentTests.test_pixel_limit_before_decode) ... ok
|
||||
test_png_and_jpeg_originals (test_attachments.AttachmentTests.test_png_and_jpeg_originals) ... ok
|
||||
test_simultaneous_registration_of_one_username (test_concurrency.ConcurrencyTests.test_simultaneous_registration_of_one_username) ... ok
|
||||
test_ten_clients_send_one_hundred_messages (test_concurrency.ConcurrencyTests.test_ten_clients_send_one_hundred_messages) ... ok
|
||||
test_conflict_and_semantic_json_replay (test_idempotency.IdempotencyTests.test_conflict_and_semantic_json_replay) ... ok
|
||||
test_key_format_and_missing_key (test_idempotency.IdempotencyTests.test_key_format_and_missing_key) ... ok
|
||||
test_parallel_replay_across_replicas (test_idempotency.IdempotencyTests.test_parallel_replay_across_replicas) ... ok
|
||||
test_replay_rechecks_session_and_access (test_idempotency.IdempotencyTests.test_replay_rechecks_session_and_access) ... ok
|
||||
test_scope_is_user_and_chat (test_idempotency.IdempotencyTests.test_scope_is_user_and_chat) ... ok
|
||||
test_logout_revokes_only_one_session (test_sessions.SessionTests.test_logout_revokes_only_one_session) ... ok
|
||||
test_password_validation (test_sessions.SessionTests.test_password_validation) ... ok
|
||||
test_sql_like_message_is_plain_text (test_sessions.SessionTests.test_sql_like_message_is_plain_text) ... ok
|
||||
test_unicode_password_with_colon_and_private_response (test_sessions.SessionTests.test_unicode_password_with_colon_and_private_response) ... ok
|
||||
test_wrong_password_and_basic_only_for_login (test_sessions.SessionTests.test_wrong_password_and_basic_only_for_login) ... ok
|
||||
|
||||
----------------------------------------------------------------------
|
||||
Ran 39 tests in 9.593s
|
||||
|
||||
OK
|
||||
PASS: 10 clients sent 100 messages; history contains exactly 100 unique messages.
|
||||
Container lab4-check-5aad8e0065-postgres-1 Running
|
||||
Container lab4-check-5aad8e0065-postgres-1 Running
|
||||
PASS: migrations only in one-shot service; repeated execution preserves data
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Stopping
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Stopped
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Running
|
||||
Container lab4-check-5aad8e0065-s3-1 Running
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Running
|
||||
Container lab4-check-5aad8e0065-s3-1 Running
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
PASS: private S3, limited credentials, isolated administration, repeated bootstrap
|
||||
Stopped API: ca1f62dfc015 container=ca1f62dfc0156fc235dbcd2f52b58eec2282e6ba8d3e056de11ca92758c69df0
|
||||
API ready through LB: 19a474c69210
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-valkey-1 Healthy
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
Container lab4-check-5aad8e0065-api-2 Starting
|
||||
Container lab4-check-5aad8e0065-api-2 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
PASS: original downloaded through another replica after stopping uploader
|
||||
PASS: post-PUT DB failure recorded; cleanup protects fresh, active and ready uploads
|
||||
Container lab4-check-5aad8e0065-s3-1 Stopping
|
||||
Container lab4-check-5aad8e0065-s3-1 Stopped
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-1 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
PASS: S3 outage gives API 503; recovery preserves original and session
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Starting
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Started
|
||||
PASS: 10 concurrent real PNG uploads; all checksums match; memory peaks: [215191552, 103378944]
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
|
||||
Container lab4-check-5aad8e0065-valkey-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-valkey-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab4/mtusi2026lab4/.venv/bin/python tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://127.0.0.1:57722"
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
PASS: both API, LB, PostgreSQL, Valkey, S3 and cleaner actually restarted; data and session preserved
|
||||
PASS: SIGTERM with unfinished request: {"instance": "19a474c69210", "unfinished_request": true, "shutdown_seconds": 5.523, "new_connections_refused_seconds": 0.185, "exit_code": 0, "oom_killed": false}
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
Container lab4-check-5aad8e0065-api-1 Starting
|
||||
Container lab4-check-5aad8e0065-api-1 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
Container lab4-check-5aad8e0065-valkey-1 Stopping
|
||||
Container lab4-check-5aad8e0065-valkey-1 Stopped
|
||||
PASS: valkey outage: API live=200, ready=503, LB ready=503
|
||||
Container lab4-check-5aad8e0065-valkey-1 Starting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
Container lab4-check-5aad8e0065-postgres-1 Stopping
|
||||
Container lab4-check-5aad8e0065-postgres-1 Stopped
|
||||
PASS: postgres outage: API live=200, ready=503, LB ready=503
|
||||
Container lab4-check-5aad8e0065-postgres-1 Starting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
Container lab4-check-5aad8e0065-s3-1 Stopping
|
||||
Container lab4-check-5aad8e0065-s3-1 Stopped
|
||||
PASS: s3 outage: API live=200, ready=503, LB ready=503
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-1 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
Container lab4-check-5aad8e0065-postgres-1 Paused
|
||||
Container lab4-check-5aad8e0065-postgres-1 Unpaused
|
||||
PASS: paused PostgreSQL: SQL and direct readiness bounded; live works; unpause restores API without restart
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab4/mtusi2026lab4/.venv/bin/python tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:57722"
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||
{"observed_instances_before": 2, "successful_writes": 110, "failed_attempts": 1, "first_recovery_seconds": 2.467}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Starting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
Container lab4-check-5aad8e0065-api-1 Starting
|
||||
Container lab4-check-5aad8e0065-api-1 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
Stopped API: 19a474c69210 container=19a474c69210a7be1c6631830e8fa7f251cf6dd3890c7407843090383543ee75
|
||||
{"label": "stop 19a474c69210", "duration_seconds": 16.012, "successful_requests": 1420, "successful_writes": 709, "errors": 4, "successful_rps": 88.68, "first_success_after_action_seconds": {"POST": 0.051, "GET": 0.004}, "maximum_success_gap_seconds": {"POST": 2.002, "GET": 1.995}, "action_start_seconds": 1.006, "action_end_seconds": 1.77, "p50_ms": 10.62, "p95_ms": 24.67, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["19a474c69210", "ca1f62dfc015"]}
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Starting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Healthy
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
Container lab4-check-5aad8e0065-api-1 Starting
|
||||
Container lab4-check-5aad8e0065-api-1 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
Stopped API: ca1f62dfc015 container=ca1f62dfc0156fc235dbcd2f52b58eec2282e6ba8d3e056de11ca92758c69df0
|
||||
{"label": "stop ca1f62dfc015", "duration_seconds": 16.008, "successful_requests": 1466, "successful_writes": 732, "errors": 4, "successful_rps": 91.58, "first_success_after_action_seconds": {"POST": 0.018, "GET": 0.028}, "maximum_success_gap_seconds": {"POST": 2.002, "GET": 1.999}, "action_start_seconds": 1.002, "action_end_seconds": 1.832, "p50_ms": 9.397, "p95_ms": 18.616, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["19a474c69210", "ca1f62dfc015"]}
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Healthy
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
Container lab4-check-5aad8e0065-api-2 Starting
|
||||
Container lab4-check-5aad8e0065-api-2 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
Local .env and secrets are ready; values are not printed
|
||||
#1 [internal] load local bake definitions
|
||||
#1 reading from stdin 564B done
|
||||
#1 DONE 0.0s
|
||||
|
||||
#2 [internal] load build definition from Dockerfile
|
||||
#2 transferring dockerfile: 733B done
|
||||
#2 DONE 0.0s
|
||||
|
||||
#3 [internal] load metadata for docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
|
||||
#3 DONE 0.0s
|
||||
|
||||
#4 [internal] load .dockerignore
|
||||
#4 transferring context: 167B done
|
||||
#4 DONE 0.0s
|
||||
|
||||
#5 [internal] load build context
|
||||
#5 transferring context: 1.01kB done
|
||||
#5 DONE 0.0s
|
||||
|
||||
#6 [1/8] FROM docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
|
||||
#6 resolve docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3 0.0s done
|
||||
#6 DONE 0.0s
|
||||
|
||||
#7 [3/8] COPY requirements.txt .
|
||||
#7 CACHED
|
||||
|
||||
#8 [6/8] COPY src/ ./src/
|
||||
#8 CACHED
|
||||
|
||||
#9 [2/8] WORKDIR /app
|
||||
#9 CACHED
|
||||
|
||||
#10 [7/8] COPY migrations/ ./migrations/
|
||||
#10 CACHED
|
||||
|
||||
#11 [5/8] RUN groupadd --gid 10001 app && useradd --uid 10001 --gid app --no-create-home app
|
||||
#11 CACHED
|
||||
|
||||
#12 [4/8] RUN python -m pip install --no-cache-dir -r requirements.txt && python -m pip check
|
||||
#12 CACHED
|
||||
|
||||
#13 [8/8] COPY research/ ./research/
|
||||
#13 CACHED
|
||||
|
||||
#14 exporting to image
|
||||
#14 exporting layers done
|
||||
#14 exporting manifest sha256:644f5d408505f5982b75f18404860de692ee837b2b2e18f5c92b0b057081b7ec done
|
||||
#14 exporting config sha256:726011343b32f5b6277348f2f8b38f95b8c31754ae0d21336375124a18aa2e0e done
|
||||
#14 exporting attestation manifest sha256:f4a5559518ea1ef0e3747d6084c8ad8dbc8b16dc41cb00062303d0f895b9a235 0.0s done
|
||||
#14 exporting manifest list sha256:93c838d55d79a0001346ecab28f72621d9540ba5c52eaeb1ca6020286dd05fd6 done
|
||||
#14 naming to docker.io/library/mtusi-messenger:lab4 done
|
||||
#14 unpacking to docker.io/library/mtusi-messenger:lab4 done
|
||||
#14 DONE 0.1s
|
||||
|
||||
#15 resolving provenance for metadata file
|
||||
#15 DONE 0.0s
|
||||
mtusi-messenger:lab4 Built
|
||||
Container lab4-check-5aad8e0065-postgres-1 Running
|
||||
Migrations: already applied
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
Replacing API: 19a474c69210
|
||||
Container lab4-check-5aad8e0065-api-2 Running
|
||||
Container lab4-check-5aad8e0065-api-3 Creating
|
||||
Container lab4-check-5aad8e0065-api-3 Created
|
||||
Container lab4-check-5aad8e0065-api-3 Starting
|
||||
Container lab4-check-5aad8e0065-api-3 Started
|
||||
API ready through LB: ca1f62dfc015, ed4f3c64a9f6
|
||||
Replacing API: ca1f62dfc015
|
||||
Container lab4-check-5aad8e0065-api-3 Running
|
||||
Container lab4-check-5aad8e0065-api-4 Creating
|
||||
Container lab4-check-5aad8e0065-api-4 Created
|
||||
Container lab4-check-5aad8e0065-api-4 Starting
|
||||
Container lab4-check-5aad8e0065-api-4 Started
|
||||
API ready through LB: 63549c0d63ba, ed4f3c64a9f6
|
||||
PASS: all API containers replaced sequentially; stores and LB kept running
|
||||
{"label": "sequential API replacement", "duration_seconds": 24.869, "successful_requests": 2254, "successful_writes": 1128, "errors": 8, "successful_rps": 90.63, "first_success_after_action_seconds": {"POST": 0.01, "GET": 0.021}, "maximum_success_gap_seconds": {"POST": 1.04, "GET": 1.04}, "action_start_seconds": 1.009, "action_end_seconds": 21.862, "p50_ms": 8.907, "p95_ms": 16.321, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["19a474c69210", "63549c0d63ba", "ca1f62dfc015", "ed4f3c64a9f6"]}
|
||||
PASS: all lab4 grade-4 checks completed
|
||||
Container lab4-check-5aad8e0065-lb-1 Stopping
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Stopping
|
||||
Container lab4-check-5aad8e0065-lb-1 Stopped
|
||||
Container lab4-check-5aad8e0065-lb-1 Removing
|
||||
Container lab4-check-5aad8e0065-lb-1 Removed
|
||||
Container lab4-check-5aad8e0065-api-3 Stopping
|
||||
Container lab4-check-5aad8e0065-api-4 Stopping
|
||||
Container lab4-check-5aad8e0065-api-3 Stopped
|
||||
Container lab4-check-5aad8e0065-api-3 Removing
|
||||
Container lab4-check-5aad8e0065-api-3 Removed
|
||||
Container lab4-check-5aad8e0065-api-4 Stopped
|
||||
Container lab4-check-5aad8e0065-api-4 Removing
|
||||
Container lab4-check-5aad8e0065-api-4 Removed
|
||||
Container lab4-check-5aad8e0065-valkey-1 Stopping
|
||||
Container lab4-check-5aad8e0065-valkey-1 Stopped
|
||||
Container lab4-check-5aad8e0065-valkey-1 Removing
|
||||
Container lab4-check-5aad8e0065-valkey-1 Removed
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Stopped
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Removing
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Removed
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Stopping
|
||||
Container lab4-check-5aad8e0065-migrate-1 Stopping
|
||||
Container lab4-check-5aad8e0065-migrate-1 Stopped
|
||||
Container lab4-check-5aad8e0065-migrate-1 Removing
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Stopped
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Removing
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Removed
|
||||
Container lab4-check-5aad8e0065-s3-1 Stopping
|
||||
Container lab4-check-5aad8e0065-migrate-1 Removed
|
||||
Container lab4-check-5aad8e0065-postgres-1 Stopping
|
||||
Container lab4-check-5aad8e0065-postgres-1 Stopped
|
||||
Container lab4-check-5aad8e0065-postgres-1 Removing
|
||||
Container lab4-check-5aad8e0065-postgres-1 Removed
|
||||
Container lab4-check-5aad8e0065-s3-1 Stopped
|
||||
Container lab4-check-5aad8e0065-s3-1 Removing
|
||||
Container lab4-check-5aad8e0065-s3-1 Removed
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Stopping
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Stopped
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Removing
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Removed
|
||||
Volume lab4-check-5aad8e0065_valkey_data Removing
|
||||
Network lab4-check-5aad8e0065_s3-internal Removing
|
||||
Volume lab4-check-5aad8e0065_postgres_data Removing
|
||||
Volume lab4-check-5aad8e0065_s3_data Removing
|
||||
Network lab4-check-5aad8e0065_default Removing
|
||||
Volume lab4-check-5aad8e0065_valkey_data Removed
|
||||
Volume lab4-check-5aad8e0065_s3_data Removed
|
||||
Volume lab4-check-5aad8e0065_postgres_data Removed
|
||||
Network lab4-check-5aad8e0065_s3-internal Removed
|
||||
Network lab4-check-5aad8e0065_default Removed
|
||||
Removed only temporary test containers and volumes: lab4-check-5aad8e0065
|
||||
@@ -0,0 +1,502 @@
|
||||
{
|
||||
"utc": "2026-11-12T15:23:27.573351+00:00",
|
||||
"command": "./scripts/check-docker.sh",
|
||||
"project": "lab5-check-d233766d34",
|
||||
"host": "Darwin arm64",
|
||||
"source_commit": "276dbdad8a34da4c5b57fbf1879ca169b581d75b",
|
||||
"working_tree": true,
|
||||
"configuration": {
|
||||
"concurrency": 2,
|
||||
"prefetch": 1,
|
||||
"attempts": 5,
|
||||
"timeout_seconds": 5,
|
||||
"lease_seconds": 15,
|
||||
"backoff_base_seconds": 0.15
|
||||
},
|
||||
"checks": {
|
||||
"topology_and_limits": {
|
||||
"passed": true,
|
||||
"api_replicas": 2,
|
||||
"broker": "Valkey Streams with AOF fsync always",
|
||||
"worker_processes": 2
|
||||
},
|
||||
"public_smoke_and_schema": {
|
||||
"passed": true
|
||||
},
|
||||
"student_api_regressions": {
|
||||
"passed": true
|
||||
},
|
||||
"worker_offline_api_broker_restart": {
|
||||
"passed": true,
|
||||
"attachment_id": "63406ab0-cc3c-4fa2-a0aa-e5e3353d5ae3",
|
||||
"job_id": "7f450ff7-cb2d-4189-a668-f1967dbd94e8",
|
||||
"message_id": "bc37bafb-efe1-4481-b304-c725c6e8368d",
|
||||
"variants": {
|
||||
"thumbnail": {
|
||||
"size": [
|
||||
256,
|
||||
160
|
||||
],
|
||||
"bytes": 5813
|
||||
},
|
||||
"crop": {
|
||||
"size": [
|
||||
128,
|
||||
128
|
||||
],
|
||||
"bytes": 2462
|
||||
},
|
||||
"optimized": {
|
||||
"size": [
|
||||
320,
|
||||
200
|
||||
],
|
||||
"bytes": 2462
|
||||
}
|
||||
}
|
||||
},
|
||||
"outbox_crash_before_publish": {
|
||||
"passed": true,
|
||||
"job_id": "a3973e2b-ff73-4bca-bc41-394f2b875d13",
|
||||
"automatic_recovery": true
|
||||
},
|
||||
"outbox_crash_after_publish": {
|
||||
"passed": true,
|
||||
"job_id": "007cb1d1-4b3b-4d12-9b2b-91a4682e8b08",
|
||||
"automatic_recovery": true
|
||||
},
|
||||
"worker_crash_after_objects": {
|
||||
"passed": true,
|
||||
"job_id": "21129707-7dd0-4a72-ac33-5691dae3194e",
|
||||
"attempt": 2,
|
||||
"canonical_objects": 4,
|
||||
"duplicate_noop": true
|
||||
},
|
||||
"worker_crash_before_ack": {
|
||||
"passed": true,
|
||||
"job_id": "2eeba68e-3fc7-42e0-adf9-13ac01c6c982",
|
||||
"attempt": 1,
|
||||
"canonical_objects": 4,
|
||||
"duplicate_noop": true
|
||||
},
|
||||
"bounded_retry_failed_registry_replay": {
|
||||
"passed": true,
|
||||
"job_id": "bf756bf3-c3ca-45d2-acdd-3e67eb882d4d",
|
||||
"max_automatic_attempts": 5,
|
||||
"audit": [
|
||||
{
|
||||
"event": "processing",
|
||||
"attempt": 1,
|
||||
"error_code": null,
|
||||
"at": "2026-11-12T15:25:18.431558+00:00"
|
||||
},
|
||||
{
|
||||
"event": "retry",
|
||||
"attempt": 1,
|
||||
"error_code": "storage_unavailable",
|
||||
"at": "2026-11-12T15:25:18.432821+00:00"
|
||||
},
|
||||
{
|
||||
"event": "processing",
|
||||
"attempt": 2,
|
||||
"error_code": null,
|
||||
"at": "2026-11-12T15:25:18.692895+00:00"
|
||||
},
|
||||
{
|
||||
"event": "retry",
|
||||
"attempt": 2,
|
||||
"error_code": "storage_unavailable",
|
||||
"at": "2026-11-12T15:25:18.69618+00:00"
|
||||
},
|
||||
{
|
||||
"event": "processing",
|
||||
"attempt": 3,
|
||||
"error_code": null,
|
||||
"at": "2026-11-12T15:25:19.206061+00:00"
|
||||
},
|
||||
{
|
||||
"event": "retry",
|
||||
"attempt": 3,
|
||||
"error_code": "storage_unavailable",
|
||||
"at": "2026-11-12T15:25:19.208065+00:00"
|
||||
},
|
||||
{
|
||||
"event": "processing",
|
||||
"attempt": 4,
|
||||
"error_code": null,
|
||||
"at": "2026-11-12T15:25:19.97778+00:00"
|
||||
},
|
||||
{
|
||||
"event": "retry",
|
||||
"attempt": 4,
|
||||
"error_code": "storage_unavailable",
|
||||
"at": "2026-11-12T15:25:19.978984+00:00"
|
||||
},
|
||||
{
|
||||
"event": "processing",
|
||||
"attempt": 5,
|
||||
"error_code": null,
|
||||
"at": "2026-11-12T15:25:21.777757+00:00"
|
||||
},
|
||||
{
|
||||
"event": "failed",
|
||||
"attempt": 5,
|
||||
"error_code": "retry_exhausted",
|
||||
"at": "2026-11-12T15:25:21.78129+00:00"
|
||||
},
|
||||
{
|
||||
"event": "operator_replay",
|
||||
"attempt": 5,
|
||||
"error_code": null,
|
||||
"at": "2026-11-12T15:25:22.680791+00:00"
|
||||
},
|
||||
{
|
||||
"event": "processing",
|
||||
"attempt": 6,
|
||||
"error_code": null,
|
||||
"at": "2026-11-12T15:25:22.802027+00:00"
|
||||
},
|
||||
{
|
||||
"event": "ready",
|
||||
"attempt": 6,
|
||||
"error_code": null,
|
||||
"at": "2026-11-12T15:25:23.151043+00:00"
|
||||
}
|
||||
],
|
||||
"ready_replay_rejected": true
|
||||
},
|
||||
"permanent_error": {
|
||||
"passed": true,
|
||||
"job_id": "c2056d32-08bf-4c8a-8980-2f8b6c2511c9",
|
||||
"attempt": 1,
|
||||
"error_code": "invalid_image"
|
||||
},
|
||||
"image_subprocess_timeout": {
|
||||
"passed": true,
|
||||
"job_id": "6e8efff5-d82d-45f4-af12-3b05dcdb0076",
|
||||
"timeout_seconds": 5,
|
||||
"task_memory_mb": 640,
|
||||
"recovered": true
|
||||
},
|
||||
"image_subprocess_memory": {
|
||||
"passed": true,
|
||||
"job_id": "ff1e7e76-b82d-4ba1-8ca9-79f3d78a4dfe",
|
||||
"timeout_seconds": 5,
|
||||
"task_memory_mb": 640,
|
||||
"recovered": true
|
||||
},
|
||||
"image_semantics": {
|
||||
"passed": true,
|
||||
"cases": [
|
||||
{
|
||||
"case": "small_alpha_on_white",
|
||||
"variants": {
|
||||
"thumbnail": {
|
||||
"size": [
|
||||
17,
|
||||
9
|
||||
],
|
||||
"bytes": 635
|
||||
},
|
||||
"crop": {
|
||||
"size": [
|
||||
128,
|
||||
128
|
||||
],
|
||||
"bytes": 883
|
||||
},
|
||||
"optimized": {
|
||||
"size": [
|
||||
17,
|
||||
9
|
||||
],
|
||||
"bytes": 56
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"case": "exif_orientation_6_and_floor",
|
||||
"variants": {
|
||||
"thumbnail": {
|
||||
"size": [
|
||||
160,
|
||||
256
|
||||
],
|
||||
"bytes": 1270
|
||||
},
|
||||
"crop": {
|
||||
"size": [
|
||||
128,
|
||||
128
|
||||
],
|
||||
"bytes": 886
|
||||
},
|
||||
"optimized": {
|
||||
"size": [
|
||||
201,
|
||||
321
|
||||
],
|
||||
"bytes": 210
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"staging_cleanup": {
|
||||
"passed": true,
|
||||
"active_preserved": true,
|
||||
"orphan_removed": true,
|
||||
"ready_removed": true,
|
||||
"expired_failed_removed": true,
|
||||
"expired_replay_rejected": true
|
||||
},
|
||||
"broker_offline_outbox": {
|
||||
"passed": true,
|
||||
"attachment_id": "3b3d3600-efcc-4840-ab76-07868501c21d",
|
||||
"recovered": true
|
||||
},
|
||||
"public_persistence": {
|
||||
"passed": true
|
||||
},
|
||||
"public_api_failover": {
|
||||
"passed": true
|
||||
}
|
||||
},
|
||||
"docker": "29.0.1 aarch64 CPUs=10 RAM=8217448448",
|
||||
"containers": [
|
||||
{
|
||||
"id": "9fbe552bf1bd8c2be76b3434cc5662479ca8d328ca698813ac9a785bcafc0cfd",
|
||||
"service": "api",
|
||||
"image": "mtusi-messenger:lab5",
|
||||
"memory_bytes": 402653184,
|
||||
"nano_cpus": 1000000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/run/secrets/s3_secret_access_key",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/database_url",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/cache_url",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/s3_access_key_id",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "ebc82ed28847cbaf6f474088c68005f05d49469010b45ff3767fb0a944ead6e0",
|
||||
"service": "api",
|
||||
"image": "mtusi-messenger:lab5",
|
||||
"memory_bytes": 402653184,
|
||||
"nano_cpus": 1000000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/run/secrets/s3_access_key_id",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/s3_secret_access_key",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/database_url",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/cache_url",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "26a48bd7b3390a4cd99667d4d99d6342edce8e26accaee3b181a4d842203261a",
|
||||
"service": "broker",
|
||||
"image": "valkey/valkey:8.1.3",
|
||||
"memory_bytes": 268435456,
|
||||
"nano_cpus": 500000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/data",
|
||||
"name": "lab5-check-d233766d34_broker_data"
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/broker_password",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "232fdc281b939bf905b4123102a6a6eb609a4ffcb730f6d03707beb07283d69c",
|
||||
"service": "cleanup",
|
||||
"image": "mtusi-messenger:lab5",
|
||||
"memory_bytes": 402653184,
|
||||
"nano_cpus": 1000000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/run/secrets/database_url",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/s3_access_key_id",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/s3_secret_access_key",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "e722801286bb112062f1da59af65dc21eddf5059ca086022e0d52f85a0f26154",
|
||||
"service": "lb",
|
||||
"image": "haproxy:3.2.6-alpine",
|
||||
"memory_bytes": 67108864,
|
||||
"nano_cpus": 500000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/usr/local/etc/haproxy/haproxy.cfg",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "ba806a4ce014a516ea867a516743de64bef0d449726bb544d265bf31b4055b1b",
|
||||
"service": "migrate",
|
||||
"image": "mtusi-messenger:lab5",
|
||||
"memory_bytes": 402653184,
|
||||
"nano_cpus": 1000000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/run/secrets/database_url",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "0fd62d960020290ba4089d05fdcc8da2bb0e2faf7528fc6ed3384e35236d4290",
|
||||
"service": "postgres",
|
||||
"image": "postgres:17.6-bookworm",
|
||||
"memory_bytes": 536870912,
|
||||
"nano_cpus": 1000000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/var/lib/postgresql/data",
|
||||
"name": "lab5-check-d233766d34_postgres_data"
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/postgres_password",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "c658ec3ef1e14ca2f1ca8f1530e6f13b37b99ce8389fafeae84e841e65daa2fc",
|
||||
"service": "publisher",
|
||||
"image": "mtusi-messenger:lab5",
|
||||
"memory_bytes": 402653184,
|
||||
"nano_cpus": 1000000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/run/secrets/database_url",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/broker_url",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "bc944a3a03374052aba43f42767c253755a921090fd15ead92fa2a924507c941",
|
||||
"service": "s3",
|
||||
"image": "chrislusf/seaweedfs:3.85@sha256:49312939c00c01e5ee6afbd7d728b18027821d3764c35a797a72acd4fdf3296a",
|
||||
"memory_bytes": 268435456,
|
||||
"nano_cpus": 500000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/run/secrets/s3_config",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/data",
|
||||
"name": "0496c4617fd5cac2843b4c391fc1eb42e15c90c4d9a35d092e71a6fd8815960f"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "9e7382c34331cb65974fa3feb248105c8055fa25c251151f0f004b65fc388923",
|
||||
"service": "s3-bootstrap",
|
||||
"image": "mtusi-messenger:lab5",
|
||||
"memory_bytes": 134217728,
|
||||
"nano_cpus": 500000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/run/secrets/s3_root_password",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/s3_root_user",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "b9821d3ec48ce18b8255a446cf99b02d6ea9658de406a15cc0ace7ccf7d7350e",
|
||||
"service": "s3-storage",
|
||||
"image": "chrislusf/seaweedfs:3.85@sha256:49312939c00c01e5ee6afbd7d728b18027821d3764c35a797a72acd4fdf3296a",
|
||||
"memory_bytes": 536870912,
|
||||
"nano_cpus": 1000000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/data",
|
||||
"name": "lab5-check-d233766d34_s3_data"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "c8ddc9fede2be42d05e79147dfc7dd87efede3b67f3226afeee1161fcaa49428",
|
||||
"service": "valkey",
|
||||
"image": "valkey/valkey:8.1.3",
|
||||
"memory_bytes": 134217728,
|
||||
"nano_cpus": 500000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/data",
|
||||
"name": "lab5-check-d233766d34_valkey_data"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "9fee2e7832c5c11e81bb56df9d435ad090fdb1fddb85efbf014ea6019559ed75",
|
||||
"service": "worker",
|
||||
"image": "mtusi-messenger:lab5",
|
||||
"memory_bytes": 1610612736,
|
||||
"nano_cpus": 2000000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/run/secrets/broker_url",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/s3_access_key_id",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/s3_secret_access_key",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/database_url",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"passed": true
|
||||
}
|
||||
@@ -0,0 +1,77 @@
|
||||
PASS: topology_and_limits {"api_replicas": 2, "broker": "Valkey Streams with AOF fsync always", "worker_processes": 2}
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab5/mtusi2026lab5/.venv/bin/python scripts/check_contract.py
|
||||
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab5/mtusi2026lab5/.venv/bin/python scripts/validate_openapi.py
|
||||
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab5/mtusi2026lab5/.venv/bin/python tests/smoke.py --base-url "http://127.0.0.1:57277"
|
||||
PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.
|
||||
PASS: public_smoke_and_schema {}
|
||||
PASS: 10 clients sent 100 messages; history contains exactly 100 unique messages.
|
||||
test_chat_access (test_api.ApiTests.test_chat_access) ... ok
|
||||
test_chat_pagination_and_cursor_owner (test_api.ApiTests.test_chat_pagination_and_cursor_owner) ... ok
|
||||
test_cursor_cannot_be_used_for_another_chat_or_changed (test_api.ApiTests.test_cursor_cannot_be_used_for_another_chat_or_changed) ... ok
|
||||
test_default_page_size (test_api.ApiTests.test_default_page_size) ... ok
|
||||
test_duplicate_username (test_api.ApiTests.test_duplicate_username) ... ok
|
||||
test_framework_errors_and_request_ids (test_api.ApiTests.test_framework_errors_and_request_ids) ... ok
|
||||
test_identification_and_authentication_order (test_api.ApiTests.test_identification_and_authentication_order) ... ok
|
||||
test_invalid_chat_members (test_api.ApiTests.test_invalid_chat_members) ... ok
|
||||
test_invalid_chat_title (test_api.ApiTests.test_invalid_chat_title) ... ok
|
||||
test_invalid_limits_and_cursors (test_api.ApiTests.test_invalid_limits_and_cursors) ... ok
|
||||
test_invalid_message_fields (test_api.ApiTests.test_invalid_message_fields) ... ok
|
||||
test_invalid_user_fields (test_api.ApiTests.test_invalid_user_fields) ... ok
|
||||
test_malformed_json_and_encoding (test_api.ApiTests.test_malformed_json_and_encoding) ... ok
|
||||
test_message_pagination_and_changing_limit (test_api.ApiTests.test_message_pagination_and_changing_limit) ... ok
|
||||
test_unicode_and_text_preservation (test_api.ApiTests.test_unicode_and_text_preservation) ... ok
|
||||
test_wrong_content_type (test_api.ApiTests.test_wrong_content_type) ... ok
|
||||
test_access_and_authentication (test_attachments.AttachmentTests.test_access_and_authentication) ... ok
|
||||
test_attachment_only_message_and_other_member (test_attachments.AttachmentTests.test_attachment_only_message_and_other_member) ... ok
|
||||
test_chunked_limit_without_content_length (test_attachments.AttachmentTests.test_chunked_limit_without_content_length) ... ok
|
||||
test_content_length_limit_before_body (test_attachments.AttachmentTests.test_content_length_limit_before_body) ... ok
|
||||
test_corruption_and_mime_spoof (test_attachments.AttachmentTests.test_corruption_and_mime_spoof) ... ok
|
||||
test_filename_rejected_before_storage (test_attachments.AttachmentTests.test_filename_rejected_before_storage) ... ok
|
||||
test_foreign_and_unknown_attachment_ids (test_attachments.AttachmentTests.test_foreign_and_unknown_attachment_ids) ... ok
|
||||
test_idempotency_includes_ordered_attachment_ids (test_attachments.AttachmentTests.test_idempotency_includes_ordered_attachment_ids) ... ok
|
||||
test_invalid_attachment_lists (test_attachments.AttachmentTests.test_invalid_attachment_lists) ... ok
|
||||
test_pixel_limit_before_decode (test_attachments.AttachmentTests.test_pixel_limit_before_decode) ... ok
|
||||
test_png_and_jpeg_originals (test_attachments.AttachmentTests.test_png_and_jpeg_originals) ... ok
|
||||
test_simultaneous_registration_of_one_username (test_concurrency.ConcurrencyTests.test_simultaneous_registration_of_one_username) ... ok
|
||||
test_ten_clients_send_one_hundred_messages (test_concurrency.ConcurrencyTests.test_ten_clients_send_one_hundred_messages) ... ok
|
||||
test_conflict_and_semantic_json_replay (test_idempotency.IdempotencyTests.test_conflict_and_semantic_json_replay) ... ok
|
||||
test_key_format_and_missing_key (test_idempotency.IdempotencyTests.test_key_format_and_missing_key) ... ok
|
||||
test_parallel_replay_across_replicas (test_idempotency.IdempotencyTests.test_parallel_replay_across_replicas) ... ok
|
||||
test_replay_rechecks_session_and_access (test_idempotency.IdempotencyTests.test_replay_rechecks_session_and_access) ... ok
|
||||
test_scope_is_user_and_chat (test_idempotency.IdempotencyTests.test_scope_is_user_and_chat) ... ok
|
||||
test_logout_revokes_only_one_session (test_sessions.SessionTests.test_logout_revokes_only_one_session) ... ok
|
||||
test_password_validation (test_sessions.SessionTests.test_password_validation) ... ok
|
||||
test_sql_like_message_is_plain_text (test_sessions.SessionTests.test_sql_like_message_is_plain_text) ... ok
|
||||
test_unicode_password_with_colon_and_private_response (test_sessions.SessionTests.test_unicode_password_with_colon_and_private_response) ... ok
|
||||
test_wrong_password_and_basic_only_for_login (test_sessions.SessionTests.test_wrong_password_and_basic_only_for_login) ... ok
|
||||
|
||||
----------------------------------------------------------------------
|
||||
Ran 39 tests in 6.981s
|
||||
|
||||
OK
|
||||
PASS: student_api_regressions {}
|
||||
PASS: worker_offline_api_broker_restart {"attachment_id": "63406ab0-cc3c-4fa2-a0aa-e5e3353d5ae3", "job_id": "7f450ff7-cb2d-4189-a668-f1967dbd94e8", "message_id": "bc37bafb-efe1-4481-b304-c725c6e8368d", "variants": {"thumbnail": {"size": [256, 160], "bytes": 5813}, "crop": {"size": [128, 128], "bytes": 2462}, "optimized": {"size": [320, 200], "bytes": 2462}}}
|
||||
PASS: outbox_crash_before_publish {"job_id": "a3973e2b-ff73-4bca-bc41-394f2b875d13", "automatic_recovery": true}
|
||||
PASS: outbox_crash_after_publish {"job_id": "007cb1d1-4b3b-4d12-9b2b-91a4682e8b08", "automatic_recovery": true}
|
||||
PASS: worker_crash_after_objects {"job_id": "21129707-7dd0-4a72-ac33-5691dae3194e", "attempt": 2, "canonical_objects": 4, "duplicate_noop": true}
|
||||
PASS: worker_crash_before_ack {"job_id": "2eeba68e-3fc7-42e0-adf9-13ac01c6c982", "attempt": 1, "canonical_objects": 4, "duplicate_noop": true}
|
||||
PASS: bounded_retry_failed_registry_replay {"job_id": "bf756bf3-c3ca-45d2-acdd-3e67eb882d4d", "max_automatic_attempts": 5, "audit": [{"event": "processing", "attempt": 1, "error_code": null, "at": "2026-11-12T15:25:18.431558+00:00"}, {"event": "retry", "attempt": 1, "error_code": "storage_unavailable", "at": "2026-11-12T15:25:18.432821+00:00"}, {"event": "processing", "attempt": 2, "error_code": null, "at": "2026-11-12T15:25:18.692895+00:00"}, {"event": "retry", "attempt": 2, "error_code": "storage_unavailable", "at": "2026-11-12T15:25:18.69618+00:00"}, {"event": "processing", "attempt": 3, "error_code": null, "at": "2026-11-12T15:25:19.206061+00:00"}, {"event": "retry", "attempt": 3, "error_code": "storage_unavailable", "at": "2026-11-12T15:25:19.208065+00:00"}, {"event": "processing", "attempt": 4, "error_code": null, "at": "2026-11-12T15:25:19.97778+00:00"}, {"event": "retry", "attempt": 4, "error_code": "storage_unavailable", "at": "2026-11-12T15:25:19.978984+00:00"}, {"event": "processing", "attempt": 5, "error_code": null, "at": "2026-11-12T15:25:21.777757+00:00"}, {"event": "failed", "attempt": 5, "error_code": "retry_exhausted", "at": "2026-11-12T15:25:21.78129+00:00"}, {"event": "operator_replay", "attempt": 5, "error_code": null, "at": "2026-11-12T15:25:22.680791+00:00"}, {"event": "processing", "attempt": 6, "error_code": null, "at": "2026-11-12T15:25:22.802027+00:00"}, {"event": "ready", "attempt": 6, "error_code": null, "at": "2026-11-12T15:25:23.151043+00:00"}], "ready_replay_rejected": true}
|
||||
PASS: permanent_error {"job_id": "c2056d32-08bf-4c8a-8980-2f8b6c2511c9", "attempt": 1, "error_code": "invalid_image"}
|
||||
PASS: image_subprocess_timeout {"job_id": "6e8efff5-d82d-45f4-af12-3b05dcdb0076", "timeout_seconds": 5, "task_memory_mb": 640, "recovered": true}
|
||||
PASS: image_subprocess_memory {"job_id": "ff1e7e76-b82d-4ba1-8ca9-79f3d78a4dfe", "timeout_seconds": 5, "task_memory_mb": 640, "recovered": true}
|
||||
PASS: image_semantics {"cases": [{"case": "small_alpha_on_white", "variants": {"thumbnail": {"size": [17, 9], "bytes": 635}, "crop": {"size": [128, 128], "bytes": 883}, "optimized": {"size": [17, 9], "bytes": 56}}}, {"case": "exif_orientation_6_and_floor", "variants": {"thumbnail": {"size": [160, 256], "bytes": 1270}, "crop": {"size": [128, 128], "bytes": 886}, "optimized": {"size": [201, 321], "bytes": 210}}}]}
|
||||
PASS: staging_cleanup {"active_preserved": true, "orphan_removed": true, "ready_removed": true, "expired_failed_removed": true, "expired_replay_rejected": true}
|
||||
PASS: broker_offline_outbox {"attachment_id": "3b3d3600-efcc-4840-ab76-07868501c21d", "recovered": true}
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab5/mtusi2026lab5/.venv/bin/python tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://127.0.0.1:57277"
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
PASS: public_persistence {}
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab5/mtusi2026lab5/.venv/bin/python tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:57277"
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||
{"observed_instances_before": 2, "successful_writes": 110, "failed_attempts": 1, "first_recovery_seconds": 2.797}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
PASS: public_api_failover {}
|
||||
PASS: all lab5 grade-4 criteria completed
|
||||
Removed isolated test stack: lab5-check-d233766d34
|
||||
@@ -0,0 +1,140 @@
|
||||
{
|
||||
"utc": "2026-11-12T15:28:31.543303+00:00",
|
||||
"command": ".venv/bin/python student-tests/verify_restart.py",
|
||||
"passed": true,
|
||||
"before": {
|
||||
"9e5a9d3d2bf0b2e47ee6d85b40b1899881db0d4d91d9ec09e50fb36aca67833c": {
|
||||
"service": "api",
|
||||
"started_at": "2026-11-12T15:28:13.153166256Z",
|
||||
"volumes": []
|
||||
},
|
||||
"83bab824b05accd2aa8829e83e272d72d4105644f8270cf3dc15f3412d569f5b": {
|
||||
"service": "api",
|
||||
"started_at": "2026-11-12T15:28:13.038450506Z",
|
||||
"volumes": []
|
||||
},
|
||||
"6f72735cbaed44f82555dcd550199764ab53a8c61a63c88f25c612fd39b6baba": {
|
||||
"service": "broker",
|
||||
"started_at": "2026-11-12T15:27:46.698266716Z",
|
||||
"volumes": [
|
||||
"mtusi-lab5_broker_data"
|
||||
]
|
||||
},
|
||||
"0f9060c7dc8473fa926c5affc560b3478b90c77771de9810ebba7858a30838d5": {
|
||||
"service": "cleanup",
|
||||
"started_at": "2026-11-12T15:28:13.036348173Z",
|
||||
"volumes": []
|
||||
},
|
||||
"d381adfb7273a5dd2f22d361a6e1ff490e5c2cf44ff147a8253f7f940cd01de4": {
|
||||
"service": "lb",
|
||||
"started_at": "2026-11-12T15:28:18.76757655Z",
|
||||
"volumes": []
|
||||
},
|
||||
"9906c3edca610723e6f09b64867fa71a543f18ef4b474804414e9f818a803115": {
|
||||
"service": "postgres",
|
||||
"started_at": "2026-11-12T15:27:46.700095216Z",
|
||||
"volumes": [
|
||||
"mtusi-lab5_postgres_data"
|
||||
]
|
||||
},
|
||||
"d705dceffd740900bbad04ee520bb96247f286c7c5381734a28bc1c9e30f27e4": {
|
||||
"service": "publisher",
|
||||
"started_at": "2026-11-12T15:27:49.924416218Z",
|
||||
"volumes": []
|
||||
},
|
||||
"cbe9063ff82960d9db58bc5713af90e4be3e72fdd1c08b0a92a93849dc6ad1aa": {
|
||||
"service": "s3",
|
||||
"started_at": "2026-11-12T15:28:09.808290754Z",
|
||||
"volumes": [
|
||||
"fe34bef7724beefe41335cd197f83aae91b16df70766cee6321bff12f765bc66"
|
||||
]
|
||||
},
|
||||
"6d1b6c402e19a068a0f2f4b9c06d5abbec7e41b5d778a883e6d045be409f6d4c": {
|
||||
"service": "s3-storage",
|
||||
"started_at": "2026-11-12T15:27:46.699445508Z",
|
||||
"volumes": [
|
||||
"mtusi-lab5_s3_data"
|
||||
]
|
||||
},
|
||||
"6d8c3e69c4b60acaf42e0259d367a0a3f2c06081a6349c284ddf9f2e9b5ba399": {
|
||||
"service": "valkey",
|
||||
"started_at": "2026-11-12T15:27:46.697619716Z",
|
||||
"volumes": [
|
||||
"mtusi-lab5_valkey_data"
|
||||
]
|
||||
},
|
||||
"28fabe35a9a5b57e0a3a96bb4ae5308fbd2fa825470799180a2471b55400206e": {
|
||||
"service": "worker",
|
||||
"started_at": "2026-11-12T15:28:13.037172673Z",
|
||||
"volumes": []
|
||||
}
|
||||
},
|
||||
"after": {
|
||||
"9e5a9d3d2bf0b2e47ee6d85b40b1899881db0d4d91d9ec09e50fb36aca67833c": {
|
||||
"service": "api",
|
||||
"started_at": "2026-11-12T15:28:33.373854585Z",
|
||||
"volumes": []
|
||||
},
|
||||
"83bab824b05accd2aa8829e83e272d72d4105644f8270cf3dc15f3412d569f5b": {
|
||||
"service": "api",
|
||||
"started_at": "2026-11-12T15:28:33.338416001Z",
|
||||
"volumes": []
|
||||
},
|
||||
"6f72735cbaed44f82555dcd550199764ab53a8c61a63c88f25c612fd39b6baba": {
|
||||
"service": "broker",
|
||||
"started_at": "2026-11-12T15:28:33.257433543Z",
|
||||
"volumes": [
|
||||
"mtusi-lab5_broker_data"
|
||||
]
|
||||
},
|
||||
"0f9060c7dc8473fa926c5affc560b3478b90c77771de9810ebba7858a30838d5": {
|
||||
"service": "cleanup",
|
||||
"started_at": "2026-11-12T15:28:40.990298547Z",
|
||||
"volumes": []
|
||||
},
|
||||
"d381adfb7273a5dd2f22d361a6e1ff490e5c2cf44ff147a8253f7f940cd01de4": {
|
||||
"service": "lb",
|
||||
"started_at": "2026-11-12T15:28:33.069501668Z",
|
||||
"volumes": []
|
||||
},
|
||||
"9906c3edca610723e6f09b64867fa71a543f18ef4b474804414e9f818a803115": {
|
||||
"service": "postgres",
|
||||
"started_at": "2026-11-12T15:28:33.134879876Z",
|
||||
"volumes": [
|
||||
"mtusi-lab5_postgres_data"
|
||||
]
|
||||
},
|
||||
"d705dceffd740900bbad04ee520bb96247f286c7c5381734a28bc1c9e30f27e4": {
|
||||
"service": "publisher",
|
||||
"started_at": "2026-11-12T15:28:40.90857463Z",
|
||||
"volumes": []
|
||||
},
|
||||
"cbe9063ff82960d9db58bc5713af90e4be3e72fdd1c08b0a92a93849dc6ad1aa": {
|
||||
"service": "s3",
|
||||
"started_at": "2026-11-12T15:28:33.167460626Z",
|
||||
"volumes": [
|
||||
"fe34bef7724beefe41335cd197f83aae91b16df70766cee6321bff12f765bc66"
|
||||
]
|
||||
},
|
||||
"6d1b6c402e19a068a0f2f4b9c06d5abbec7e41b5d778a883e6d045be409f6d4c": {
|
||||
"service": "s3-storage",
|
||||
"started_at": "2026-11-12T15:28:40.950084172Z",
|
||||
"volumes": [
|
||||
"mtusi-lab5_s3_data"
|
||||
]
|
||||
},
|
||||
"6d8c3e69c4b60acaf42e0259d367a0a3f2c06081a6349c284ddf9f2e9b5ba399": {
|
||||
"service": "valkey",
|
||||
"started_at": "2026-11-12T15:28:33.205284001Z",
|
||||
"volumes": [
|
||||
"mtusi-lab5_valkey_data"
|
||||
]
|
||||
},
|
||||
"28fabe35a9a5b57e0a3a96bb4ae5308fbd2fa825470799180a2471b55400206e": {
|
||||
"service": "worker",
|
||||
"started_at": "2026-11-12T15:28:33.299354043Z",
|
||||
"volumes": []
|
||||
}
|
||||
},
|
||||
"same_containers_and_volumes": true
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab5/mtusi2026lab5/.venv/bin/python tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:8080"
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
PASS: all persistent services actually restarted; container IDs and volumes preserved
|
||||
@@ -0,0 +1,51 @@
|
||||
.venv/bin/python scripts/check_contract.py
|
||||
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||
.venv/bin/python scripts/validate_openapi.py
|
||||
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
|
||||
.venv/bin/python tests/smoke.py --base-url "http://localhost:8080"
|
||||
PASS: lab6 public API scenario; infrastructure and grading evidence remain separate.
|
||||
test_chat_access (test_api.ApiTests.test_chat_access) ... ok
|
||||
test_chat_pagination_and_cursor_owner (test_api.ApiTests.test_chat_pagination_and_cursor_owner) ... ok
|
||||
test_cursor_cannot_be_used_for_another_chat_or_changed (test_api.ApiTests.test_cursor_cannot_be_used_for_another_chat_or_changed) ... ok
|
||||
test_default_page_size (test_api.ApiTests.test_default_page_size) ... ok
|
||||
test_duplicate_username (test_api.ApiTests.test_duplicate_username) ... ok
|
||||
test_framework_errors_and_request_ids (test_api.ApiTests.test_framework_errors_and_request_ids) ... ok
|
||||
test_identification_and_authentication_order (test_api.ApiTests.test_identification_and_authentication_order) ... ok
|
||||
test_invalid_chat_members (test_api.ApiTests.test_invalid_chat_members) ... ok
|
||||
test_invalid_chat_title (test_api.ApiTests.test_invalid_chat_title) ... ok
|
||||
test_invalid_limits_and_cursors (test_api.ApiTests.test_invalid_limits_and_cursors) ... ok
|
||||
test_invalid_message_fields (test_api.ApiTests.test_invalid_message_fields) ... ok
|
||||
test_invalid_user_fields (test_api.ApiTests.test_invalid_user_fields) ... ok
|
||||
test_malformed_json_and_encoding (test_api.ApiTests.test_malformed_json_and_encoding) ... ok
|
||||
test_message_pagination_and_changing_limit (test_api.ApiTests.test_message_pagination_and_changing_limit) ... ok
|
||||
test_unicode_and_text_preservation (test_api.ApiTests.test_unicode_and_text_preservation) ... ok
|
||||
test_wrong_content_type (test_api.ApiTests.test_wrong_content_type) ... ok
|
||||
test_access_and_authentication (test_attachments.AttachmentTests.test_access_and_authentication) ... ok
|
||||
test_attachment_only_message_and_other_member (test_attachments.AttachmentTests.test_attachment_only_message_and_other_member) ... ok
|
||||
test_chunked_limit_without_content_length (test_attachments.AttachmentTests.test_chunked_limit_without_content_length) ... ok
|
||||
test_content_length_limit_before_body (test_attachments.AttachmentTests.test_content_length_limit_before_body) ... ok
|
||||
test_corruption_and_mime_spoof (test_attachments.AttachmentTests.test_corruption_and_mime_spoof) ... ok
|
||||
test_filename_rejected_before_storage (test_attachments.AttachmentTests.test_filename_rejected_before_storage) ... ok
|
||||
test_foreign_and_unknown_attachment_ids (test_attachments.AttachmentTests.test_foreign_and_unknown_attachment_ids) ... ok
|
||||
test_idempotency_includes_ordered_attachment_ids (test_attachments.AttachmentTests.test_idempotency_includes_ordered_attachment_ids) ... ok
|
||||
test_invalid_attachment_lists (test_attachments.AttachmentTests.test_invalid_attachment_lists) ... ok
|
||||
test_pixel_limit_before_decode (test_attachments.AttachmentTests.test_pixel_limit_before_decode) ... ok
|
||||
test_png_and_jpeg_originals (test_attachments.AttachmentTests.test_png_and_jpeg_originals) ... ok
|
||||
test_simultaneous_registration_of_one_username (test_concurrency.ConcurrencyTests.test_simultaneous_registration_of_one_username) ... ok
|
||||
test_ten_clients_send_one_hundred_messages (test_concurrency.ConcurrencyTests.test_ten_clients_send_one_hundred_messages) ... ok
|
||||
test_conflict_and_semantic_json_replay (test_idempotency.IdempotencyTests.test_conflict_and_semantic_json_replay) ... ok
|
||||
test_key_format_and_missing_key (test_idempotency.IdempotencyTests.test_key_format_and_missing_key) ... ok
|
||||
test_parallel_replay_across_replicas (test_idempotency.IdempotencyTests.test_parallel_replay_across_replicas) ... ok
|
||||
test_replay_rechecks_session_and_access (test_idempotency.IdempotencyTests.test_replay_rechecks_session_and_access) ... ok
|
||||
test_scope_is_user_and_chat (test_idempotency.IdempotencyTests.test_scope_is_user_and_chat) ... ok
|
||||
test_logout_revokes_only_one_session (test_sessions.SessionTests.test_logout_revokes_only_one_session) ... ok
|
||||
test_password_validation (test_sessions.SessionTests.test_password_validation) ... ok
|
||||
test_sql_like_message_is_plain_text (test_sessions.SessionTests.test_sql_like_message_is_plain_text) ... ok
|
||||
test_unicode_password_with_colon_and_private_response (test_sessions.SessionTests.test_unicode_password_with_colon_and_private_response) ... ok
|
||||
test_wrong_password_and_basic_only_for_login (test_sessions.SessionTests.test_wrong_password_and_basic_only_for_login) ... ok
|
||||
|
||||
----------------------------------------------------------------------
|
||||
Ran 39 tests in 7.518s
|
||||
|
||||
OK
|
||||
PASS: 10 clients sent 100 messages; history contains exactly 100 unique messages.
|
||||
@@ -0,0 +1,102 @@
|
||||
{
|
||||
"utc": "2026-11-19T15:09:33.967862+00:00",
|
||||
"checks": {
|
||||
"retry_trace_and_logs": {
|
||||
"passed": true,
|
||||
"trace_id": "93e1177f9568f5bbfc0b6967a1324a84",
|
||||
"request_id": "422b23e9-c8f9-4810-ad09-1cb99b6c9576",
|
||||
"job_id": "fd4d7cce-b0dc-4edd-9938-61230afa7370",
|
||||
"span_count": 59
|
||||
},
|
||||
"metrics_and_cardinality": {
|
||||
"passed": true,
|
||||
"series": 1192
|
||||
},
|
||||
"backend_outage": {
|
||||
"passed": true,
|
||||
"backend": "loki",
|
||||
"successful_writes": 24,
|
||||
"outage_seconds": 51.39,
|
||||
"missing_request_logs": 6,
|
||||
"tested_request_ids": [
|
||||
"a5835a4b-40b6-4383-90a3-82d07b5bdcfb",
|
||||
"ee8b61c0-9acc-4ff4-a0fd-f4a7d84e29a2",
|
||||
"bb492157-8ffd-49a6-84ef-c2b17379bc8e",
|
||||
"6a076038-13e0-415d-b961-254cb7d48194",
|
||||
"59ab7cdd-f319-4828-a0b2-4bb9ecef9464",
|
||||
"0b472396-a1f4-453c-aff2-0a8e2650f3e3",
|
||||
"b0916ed0-b218-423a-ac39-1c6e904300e6",
|
||||
"bcd32c58-953b-4759-ac8b-e87fc66c4fd4",
|
||||
"0a065b55-3fac-46fc-9d92-c67f8ba6e409",
|
||||
"11d46ead-2941-4279-aa86-0abf111bcaba",
|
||||
"f6807acd-4e5f-4bef-9f35-93b2ac581390",
|
||||
"fec97823-5700-4fab-a18a-8a98f11f5c4e",
|
||||
"a03b0d52-34ea-4b0a-86f2-ac408a0a107c",
|
||||
"6d44ee0d-f039-406b-b0f7-4ceb7743d020",
|
||||
"efc0b1d3-e889-4ab4-a169-0369c4cb260b",
|
||||
"0ae460c2-6b26-4f5c-8959-ecb48a61ca92",
|
||||
"0f59b3eb-2187-412b-b569-c2373748020e",
|
||||
"086464d0-5b79-4c9d-aa95-e08af7858c43",
|
||||
"4c42e246-b973-4e79-8a37-be114b255b80",
|
||||
"2065d6ba-c0f8-43a5-999f-55ec5299d855",
|
||||
"8ccb2503-c32f-4b3d-9763-2d9455f13a99",
|
||||
"979ef391-bc49-42af-95b8-194e028dd796",
|
||||
"fe3a4f81-114e-4055-b160-7f47a15acfff",
|
||||
"e6019f30-5b26-4bc1-a630-bcfbbeb6ba96"
|
||||
],
|
||||
"missing_request_ids": [
|
||||
"a5835a4b-40b6-4383-90a3-82d07b5bdcfb",
|
||||
"ee8b61c0-9acc-4ff4-a0fd-f4a7d84e29a2",
|
||||
"bb492157-8ffd-49a6-84ef-c2b17379bc8e",
|
||||
"6a076038-13e0-415d-b961-254cb7d48194",
|
||||
"59ab7cdd-f319-4828-a0b2-4bb9ecef9464",
|
||||
"0b472396-a1f4-453c-aff2-0a8e2650f3e3"
|
||||
],
|
||||
"collector_memory_samples": [
|
||||
"{\"BlockIO\":\"0B / 0B\",\"CPUPerc\":\"0.12%\",\"Container\":\"6e21c63012d160a406e28a342d668de0be55107aa16228ed9d5dc15c34b447be\",\"ID\":\"6e21c63012d1\",\"MemPerc\":\"28.44%\",\"MemUsage\":\"72.8MiB / 256MiB\",\"Name\":\"mtusi-lab6-otel-collector-1\",\"NetIO\":\"7.32MB / 2.49MB\",\"PIDs\":\"15\"}",
|
||||
"{\"BlockIO\":\"0B / 0B\",\"CPUPerc\":\"0.43%\",\"Container\":\"6e21c63012d160a406e28a342d668de0be55107aa16228ed9d5dc15c34b447be\",\"ID\":\"6e21c63012d1\",\"MemPerc\":\"27.52%\",\"MemUsage\":\"70.46MiB / 256MiB\",\"Name\":\"mtusi-lab6-otel-collector-1\",\"NetIO\":\"7.48MB / 2.53MB\",\"PIDs\":\"15\"}",
|
||||
"{\"BlockIO\":\"0B / 0B\",\"CPUPerc\":\"0.42%\",\"Container\":\"6e21c63012d160a406e28a342d668de0be55107aa16228ed9d5dc15c34b447be\",\"ID\":\"6e21c63012d1\",\"MemPerc\":\"27.95%\",\"MemUsage\":\"71.56MiB / 256MiB\",\"Name\":\"mtusi-lab6-otel-collector-1\",\"NetIO\":\"7.64MB / 2.58MB\",\"PIDs\":\"15\"}",
|
||||
"{\"BlockIO\":\"0B / 0B\",\"CPUPerc\":\"0.20%\",\"Container\":\"6e21c63012d160a406e28a342d668de0be55107aa16228ed9d5dc15c34b447be\",\"ID\":\"6e21c63012d1\",\"MemPerc\":\"27.96%\",\"MemUsage\":\"71.57MiB / 256MiB\",\"Name\":\"mtusi-lab6-otel-collector-1\",\"NetIO\":\"7.81MB / 2.62MB\",\"PIDs\":\"15\"}"
|
||||
],
|
||||
"collector_limit_bytes": 268435456,
|
||||
"oom_killed": false,
|
||||
"failed_exports_before": [
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otelcol_exporter_send_failed_log_records_total",
|
||||
"exporter": "otlphttp/loki",
|
||||
"instance": "otel-collector:8888",
|
||||
"job": "collector",
|
||||
"otel_scope_name": "go.opentelemetry.io/collector/exporter/exporterhelper"
|
||||
},
|
||||
"value": [
|
||||
1795100986.923,
|
||||
"6"
|
||||
]
|
||||
}
|
||||
],
|
||||
"failed_exports_after": [
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otelcol_exporter_send_failed_log_records_total",
|
||||
"exporter": "otlphttp/loki",
|
||||
"instance": "otel-collector:8888",
|
||||
"job": "collector",
|
||||
"otel_scope_name": "go.opentelemetry.io/collector/exporter/exporterhelper"
|
||||
},
|
||||
"value": [
|
||||
1795101059.133,
|
||||
"12"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"provisioned_dashboard_queries": {
|
||||
"passed": true,
|
||||
"panels": 19
|
||||
}
|
||||
},
|
||||
"dashboard": "messenger-lab6",
|
||||
"passed": true,
|
||||
"duration_seconds": 85.23
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
{
|
||||
"utc": "2026-11-19T15:09:33.967862+00:00",
|
||||
"checks": {
|
||||
"retry_trace_and_logs": {
|
||||
"passed": true,
|
||||
"trace_id": "93e1177f9568f5bbfc0b6967a1324a84",
|
||||
"request_id": "422b23e9-c8f9-4810-ad09-1cb99b6c9576",
|
||||
"job_id": "fd4d7cce-b0dc-4edd-9938-61230afa7370",
|
||||
"span_count": 59
|
||||
},
|
||||
"metrics_and_cardinality": {
|
||||
"passed": true,
|
||||
"series": 1192
|
||||
},
|
||||
"backend_outage": {
|
||||
"passed": true,
|
||||
"backend": "loki",
|
||||
"successful_writes": 24,
|
||||
"outage_seconds": 51.39,
|
||||
"missing_request_logs": 6,
|
||||
"tested_request_ids": [
|
||||
"a5835a4b-40b6-4383-90a3-82d07b5bdcfb",
|
||||
"ee8b61c0-9acc-4ff4-a0fd-f4a7d84e29a2",
|
||||
"bb492157-8ffd-49a6-84ef-c2b17379bc8e",
|
||||
"6a076038-13e0-415d-b961-254cb7d48194",
|
||||
"59ab7cdd-f319-4828-a0b2-4bb9ecef9464",
|
||||
"0b472396-a1f4-453c-aff2-0a8e2650f3e3",
|
||||
"b0916ed0-b218-423a-ac39-1c6e904300e6",
|
||||
"bcd32c58-953b-4759-ac8b-e87fc66c4fd4",
|
||||
"0a065b55-3fac-46fc-9d92-c67f8ba6e409",
|
||||
"11d46ead-2941-4279-aa86-0abf111bcaba",
|
||||
"f6807acd-4e5f-4bef-9f35-93b2ac581390",
|
||||
"fec97823-5700-4fab-a18a-8a98f11f5c4e",
|
||||
"a03b0d52-34ea-4b0a-86f2-ac408a0a107c",
|
||||
"6d44ee0d-f039-406b-b0f7-4ceb7743d020",
|
||||
"efc0b1d3-e889-4ab4-a169-0369c4cb260b",
|
||||
"0ae460c2-6b26-4f5c-8959-ecb48a61ca92",
|
||||
"0f59b3eb-2187-412b-b569-c2373748020e",
|
||||
"086464d0-5b79-4c9d-aa95-e08af7858c43",
|
||||
"4c42e246-b973-4e79-8a37-be114b255b80",
|
||||
"2065d6ba-c0f8-43a5-999f-55ec5299d855",
|
||||
"8ccb2503-c32f-4b3d-9763-2d9455f13a99",
|
||||
"979ef391-bc49-42af-95b8-194e028dd796",
|
||||
"fe3a4f81-114e-4055-b160-7f47a15acfff",
|
||||
"e6019f30-5b26-4bc1-a630-bcfbbeb6ba96"
|
||||
],
|
||||
"missing_request_ids": [
|
||||
"a5835a4b-40b6-4383-90a3-82d07b5bdcfb",
|
||||
"ee8b61c0-9acc-4ff4-a0fd-f4a7d84e29a2",
|
||||
"bb492157-8ffd-49a6-84ef-c2b17379bc8e",
|
||||
"6a076038-13e0-415d-b961-254cb7d48194",
|
||||
"59ab7cdd-f319-4828-a0b2-4bb9ecef9464",
|
||||
"0b472396-a1f4-453c-aff2-0a8e2650f3e3"
|
||||
],
|
||||
"collector_memory_samples": [
|
||||
"{\"BlockIO\":\"0B / 0B\",\"CPUPerc\":\"0.12%\",\"Container\":\"6e21c63012d160a406e28a342d668de0be55107aa16228ed9d5dc15c34b447be\",\"ID\":\"6e21c63012d1\",\"MemPerc\":\"28.44%\",\"MemUsage\":\"72.8MiB / 256MiB\",\"Name\":\"mtusi-lab6-otel-collector-1\",\"NetIO\":\"7.32MB / 2.49MB\",\"PIDs\":\"15\"}",
|
||||
"{\"BlockIO\":\"0B / 0B\",\"CPUPerc\":\"0.43%\",\"Container\":\"6e21c63012d160a406e28a342d668de0be55107aa16228ed9d5dc15c34b447be\",\"ID\":\"6e21c63012d1\",\"MemPerc\":\"27.52%\",\"MemUsage\":\"70.46MiB / 256MiB\",\"Name\":\"mtusi-lab6-otel-collector-1\",\"NetIO\":\"7.48MB / 2.53MB\",\"PIDs\":\"15\"}",
|
||||
"{\"BlockIO\":\"0B / 0B\",\"CPUPerc\":\"0.42%\",\"Container\":\"6e21c63012d160a406e28a342d668de0be55107aa16228ed9d5dc15c34b447be\",\"ID\":\"6e21c63012d1\",\"MemPerc\":\"27.95%\",\"MemUsage\":\"71.56MiB / 256MiB\",\"Name\":\"mtusi-lab6-otel-collector-1\",\"NetIO\":\"7.64MB / 2.58MB\",\"PIDs\":\"15\"}",
|
||||
"{\"BlockIO\":\"0B / 0B\",\"CPUPerc\":\"0.20%\",\"Container\":\"6e21c63012d160a406e28a342d668de0be55107aa16228ed9d5dc15c34b447be\",\"ID\":\"6e21c63012d1\",\"MemPerc\":\"27.96%\",\"MemUsage\":\"71.57MiB / 256MiB\",\"Name\":\"mtusi-lab6-otel-collector-1\",\"NetIO\":\"7.81MB / 2.62MB\",\"PIDs\":\"15\"}"
|
||||
],
|
||||
"collector_limit_bytes": 268435456,
|
||||
"oom_killed": false,
|
||||
"failed_exports_before": [
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otelcol_exporter_send_failed_log_records_total",
|
||||
"exporter": "otlphttp/loki",
|
||||
"instance": "otel-collector:8888",
|
||||
"job": "collector",
|
||||
"otel_scope_name": "go.opentelemetry.io/collector/exporter/exporterhelper"
|
||||
},
|
||||
"value": [
|
||||
1795100986.923,
|
||||
"6"
|
||||
]
|
||||
}
|
||||
],
|
||||
"failed_exports_after": [
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otelcol_exporter_send_failed_log_records_total",
|
||||
"exporter": "otlphttp/loki",
|
||||
"instance": "otel-collector:8888",
|
||||
"job": "collector",
|
||||
"otel_scope_name": "go.opentelemetry.io/collector/exporter/exporterhelper"
|
||||
},
|
||||
"value": [
|
||||
1795101059.133,
|
||||
"12"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"provisioned_dashboard_queries": {
|
||||
"passed": true,
|
||||
"panels": 19
|
||||
}
|
||||
},
|
||||
"dashboard": "messenger-lab6",
|
||||
"passed": true,
|
||||
"duration_seconds": 85.23
|
||||
}
|
||||
@@ -0,0 +1,306 @@
|
||||
{
|
||||
"status": "success",
|
||||
"data": {
|
||||
"resultType": "streams",
|
||||
"result": [
|
||||
{
|
||||
"stream": {
|
||||
"attachment_id": "0eb04e8c-8e7b-42a9-ad63-8fca3390d11c",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "INFO",
|
||||
"event": "attachment.accepted",
|
||||
"flags": "3",
|
||||
"job_id": "fd4d7cce-b0dc-4edd-9938-61230afa7370",
|
||||
"scope_name": "messenger",
|
||||
"scope_version": "6.0.0",
|
||||
"service_instance_id": "166e3f521d4f:1",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "6.0.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "e04c2c74d09fa066",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1",
|
||||
"trace_id": "93e1177f9568f5bbfc0b6967a1324a84"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1795100974767540043",
|
||||
"{\"timestamp\": \"2026-11-19T15:09:34.767497+00:00\", \"severity\": \"INFO\", \"service.name\": \"messenger-api\", \"event\": \"attachment.accepted\", \"trace_id\": \"93e1177f9568f5bbfc0b6967a1324a84\", \"span_id\": \"e04c2c74d09fa066\", \"job_id\": \"fd4d7cce-b0dc-4edd-9938-61230afa7370\", \"attachment_id\": \"0eb04e8c-8e7b-42a9-ad63-8fca3390d11c\"}"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"attempt": "1",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "ERROR",
|
||||
"error_code": "storage_unavailable",
|
||||
"event": "job.retry",
|
||||
"flags": "3",
|
||||
"job_id": "fd4d7cce-b0dc-4edd-9938-61230afa7370",
|
||||
"scope_name": "messenger",
|
||||
"scope_version": "6.0.0",
|
||||
"service_instance_id": "4a4dd93b6332:8",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "6.0.0",
|
||||
"severity_number": "17",
|
||||
"severity_text": "ERROR",
|
||||
"span_id": "f9ff281d3dbfaedd",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1",
|
||||
"trace_id": "93e1177f9568f5bbfc0b6967a1324a84"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1795100977543203752",
|
||||
"{\"timestamp\": \"2026-11-19T15:09:37.543167+00:00\", \"severity\": \"ERROR\", \"service.name\": \"messenger-worker\", \"event\": \"job.retry\", \"trace_id\": \"93e1177f9568f5bbfc0b6967a1324a84\", \"span_id\": \"f9ff281d3dbfaedd\", \"job_id\": \"fd4d7cce-b0dc-4edd-9938-61230afa7370\", \"attempt\": 1, \"error_code\": \"storage_unavailable\"}"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"attempt": "1",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "ERROR",
|
||||
"event": "job.finished",
|
||||
"flags": "3",
|
||||
"job_id": "fd4d7cce-b0dc-4edd-9938-61230afa7370",
|
||||
"outcome": "retry",
|
||||
"scope_name": "messenger",
|
||||
"scope_version": "6.0.0",
|
||||
"service_instance_id": "4a4dd93b6332:8",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "6.0.0",
|
||||
"severity_number": "17",
|
||||
"severity_text": "ERROR",
|
||||
"span_id": "2bb32cd3759889ae",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1",
|
||||
"trace_id": "93e1177f9568f5bbfc0b6967a1324a84"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1795100977544205419",
|
||||
"{\"timestamp\": \"2026-11-19T15:09:37.544179+00:00\", \"severity\": \"ERROR\", \"service.name\": \"messenger-worker\", \"event\": \"job.finished\", \"trace_id\": \"93e1177f9568f5bbfc0b6967a1324a84\", \"span_id\": \"2bb32cd3759889ae\", \"job_id\": \"fd4d7cce-b0dc-4edd-9938-61230afa7370\", \"attempt\": 1, \"outcome\": \"retry\"}"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"attempt": "2",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "INFO",
|
||||
"event": "job.finished",
|
||||
"flags": "3",
|
||||
"job_id": "fd4d7cce-b0dc-4edd-9938-61230afa7370",
|
||||
"outcome": "ready",
|
||||
"scope_name": "messenger",
|
||||
"scope_version": "6.0.0",
|
||||
"service_instance_id": "4a4dd93b6332:9",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "6.0.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "26b0193c185f8761",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1",
|
||||
"trace_id": "93e1177f9568f5bbfc0b6967a1324a84"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1795100979222424711",
|
||||
"{\"timestamp\": \"2026-11-19T15:09:39.222385+00:00\", \"severity\": \"INFO\", \"service.name\": \"messenger-worker\", \"event\": \"job.finished\", \"trace_id\": \"93e1177f9568f5bbfc0b6967a1324a84\", \"span_id\": \"26b0193c185f8761\", \"job_id\": \"fd4d7cce-b0dc-4edd-9938-61230afa7370\", \"attempt\": 2, \"outcome\": \"ready\"}"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "INFO",
|
||||
"duration_seconds": "0.009846",
|
||||
"event": "http.request",
|
||||
"flags": "3",
|
||||
"method": "POST",
|
||||
"request_id": "422b23e9-c8f9-4810-ad09-1cb99b6c9576",
|
||||
"route": "/api/v1/chats/{chat_id}/attachments",
|
||||
"scope_name": "messenger",
|
||||
"scope_version": "6.0.0",
|
||||
"service_instance_id": "166e3f521d4f:1",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "6.0.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "e04c2c74d09fa066",
|
||||
"status": "202",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1",
|
||||
"trace_id": "93e1177f9568f5bbfc0b6967a1324a84"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1795100974767795501",
|
||||
"{\"timestamp\": \"2026-11-19T15:09:34.767775+00:00\", \"severity\": \"INFO\", \"service.name\": \"messenger-api\", \"event\": \"http.request\", \"trace_id\": \"93e1177f9568f5bbfc0b6967a1324a84\", \"span_id\": \"e04c2c74d09fa066\", \"method\": \"POST\", \"route\": \"/api/v1/chats/{chat_id}/attachments\", \"status\": 202, \"duration_seconds\": 0.009846, \"request_id\": \"422b23e9-c8f9-4810-ad09-1cb99b6c9576\"}"
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"stats": {
|
||||
"summary": {
|
||||
"bytesProcessedPerSecond": 80485548,
|
||||
"linesProcessedPerSecond": 110946,
|
||||
"totalBytesProcessed": 1706250,
|
||||
"totalLinesProcessed": 2352,
|
||||
"execTime": 0.021199,
|
||||
"queueTime": 0.000413,
|
||||
"subqueries": 0,
|
||||
"totalEntriesReturned": 5,
|
||||
"splits": 2,
|
||||
"shards": 2,
|
||||
"totalPostFilterLines": 5,
|
||||
"totalStructuredMetadataBytesProcessed": 853184
|
||||
},
|
||||
"querier": {
|
||||
"store": {
|
||||
"totalChunksRef": 0,
|
||||
"totalChunksDownloaded": 0,
|
||||
"chunksDownloadTime": 0,
|
||||
"queryReferencedStructuredMetadata": false,
|
||||
"chunk": {
|
||||
"headChunkBytes": 0,
|
||||
"headChunkLines": 0,
|
||||
"decompressedBytes": 0,
|
||||
"decompressedLines": 0,
|
||||
"compressedBytes": 0,
|
||||
"totalDuplicates": 0,
|
||||
"postFilterLines": 0,
|
||||
"headChunkStructuredMetadataBytes": 0,
|
||||
"decompressedStructuredMetadataBytes": 0
|
||||
},
|
||||
"chunkRefsFetchTime": 0,
|
||||
"congestionControlLatency": 0,
|
||||
"pipelineWrapperFilteredLines": 0
|
||||
}
|
||||
},
|
||||
"ingester": {
|
||||
"totalReached": 2,
|
||||
"totalChunksMatched": 2,
|
||||
"totalBatches": 3,
|
||||
"totalLinesSent": 5,
|
||||
"store": {
|
||||
"totalChunksRef": 0,
|
||||
"totalChunksDownloaded": 0,
|
||||
"chunksDownloadTime": 0,
|
||||
"queryReferencedStructuredMetadata": false,
|
||||
"chunk": {
|
||||
"headChunkBytes": 114091,
|
||||
"headChunkLines": 228,
|
||||
"decompressedBytes": 1592159,
|
||||
"decompressedLines": 2124,
|
||||
"compressedBytes": 190062,
|
||||
"totalDuplicates": 0,
|
||||
"postFilterLines": 5,
|
||||
"headChunkStructuredMetadataBytes": 34104,
|
||||
"decompressedStructuredMetadataBytes": 819080
|
||||
},
|
||||
"chunkRefsFetchTime": 180667,
|
||||
"congestionControlLatency": 0,
|
||||
"pipelineWrapperFilteredLines": 0
|
||||
}
|
||||
},
|
||||
"cache": {
|
||||
"chunk": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"index": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"result": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"statsResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"volumeResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"seriesResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"labelResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"instantMetricResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
}
|
||||
},
|
||||
"index": {
|
||||
"totalChunks": 0,
|
||||
"postFilterChunks": 0,
|
||||
"shardsDuration": 0,
|
||||
"usedBloomFilters": false
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,100 @@
|
||||
{
|
||||
"passed": true,
|
||||
"panels": [
|
||||
{
|
||||
"panel": "Business API RPS",
|
||||
"source": "prometheus",
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"panel": "API 5xx / all business requests",
|
||||
"source": "prometheus",
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"panel": "API latency: aggregated histogram",
|
||||
"source": "prometheus",
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"panel": "Time from acceptance to ready: completed jobs",
|
||||
"source": "prometheus",
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"panel": "Job attempts by outcome",
|
||||
"source": "prometheus",
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"panel": "Worker processing p95",
|
||||
"source": "prometheus",
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"panel": "Outstanding jobs (DB source of truth)",
|
||||
"source": "prometheus",
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"panel": "Oldest unfinished job age",
|
||||
"source": "prometheus",
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"panel": "DB pool used / waiting",
|
||||
"source": "prometheus",
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"panel": "Worker busy / capacity",
|
||||
"source": "prometheus",
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"panel": "Process CPU: cores consumed",
|
||||
"source": "prometheus",
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"panel": "Process RSS",
|
||||
"source": "prometheus",
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"panel": "API availability SLO: >= 99% / 1h",
|
||||
"source": "prometheus",
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"panel": "Ready SLO: >= 95% within 30s / 1h",
|
||||
"source": "prometheus",
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"panel": "API burn rate (budget / hour)",
|
||||
"source": "prometheus",
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"panel": "Ready burn rate (budget / hour)",
|
||||
"source": "prometheus",
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"panel": "Local alert states",
|
||||
"source": "prometheus",
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"panel": "API / worker structured logs (click TraceID)",
|
||||
"source": "loki",
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"panel": "Recent real worker traces",
|
||||
"source": "tempo",
|
||||
"passed": true
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,191 @@
|
||||
{
|
||||
"status": "success",
|
||||
"data": {
|
||||
"resultType": "streams",
|
||||
"result": [
|
||||
{
|
||||
"stream": {
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "ERROR",
|
||||
"duration_seconds": "2.003678",
|
||||
"event": "http.request",
|
||||
"flags": "3",
|
||||
"method": "POST",
|
||||
"request_id": "3c5e5b14-5825-41b8-a227-93cba420a532",
|
||||
"route": "/api/v1/chats/{chat_id}/messages",
|
||||
"scope_name": "messenger",
|
||||
"scope_version": "6.0.0",
|
||||
"service_instance_id": "2a50b728ce10:1",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "6.0.0",
|
||||
"severity_number": "17",
|
||||
"severity_text": "ERROR",
|
||||
"span_id": "3cec0bb07faa77ff",
|
||||
"status": "503",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1",
|
||||
"trace_id": "f124d93f84dd25c5af978e78e2028646"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1795102249376144341",
|
||||
"{\"timestamp\": \"2026-11-19T15:30:49.376082+00:00\", \"severity\": \"ERROR\", \"service.name\": \"messenger-api\", \"event\": \"http.request\", \"trace_id\": \"f124d93f84dd25c5af978e78e2028646\", \"span_id\": \"3cec0bb07faa77ff\", \"method\": \"POST\", \"route\": \"/api/v1/chats/{chat_id}/messages\", \"status\": 503, \"duration_seconds\": 2.003678, \"request_id\": \"3c5e5b14-5825-41b8-a227-93cba420a532\"}"
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"stats": {
|
||||
"summary": {
|
||||
"bytesProcessedPerSecond": 61790648,
|
||||
"linesProcessedPerSecond": 82173,
|
||||
"totalBytesProcessed": 809854,
|
||||
"totalLinesProcessed": 1077,
|
||||
"execTime": 0.013106,
|
||||
"queueTime": 0.000136,
|
||||
"subqueries": 0,
|
||||
"totalEntriesReturned": 1,
|
||||
"splits": 0,
|
||||
"shards": 1,
|
||||
"totalPostFilterLines": 1,
|
||||
"totalStructuredMetadataBytesProcessed": 400756
|
||||
},
|
||||
"querier": {
|
||||
"store": {
|
||||
"totalChunksRef": 0,
|
||||
"totalChunksDownloaded": 0,
|
||||
"chunksDownloadTime": 0,
|
||||
"queryReferencedStructuredMetadata": false,
|
||||
"chunk": {
|
||||
"headChunkBytes": 0,
|
||||
"headChunkLines": 0,
|
||||
"decompressedBytes": 0,
|
||||
"decompressedLines": 0,
|
||||
"compressedBytes": 0,
|
||||
"totalDuplicates": 0,
|
||||
"postFilterLines": 0,
|
||||
"headChunkStructuredMetadataBytes": 0,
|
||||
"decompressedStructuredMetadataBytes": 0
|
||||
},
|
||||
"chunkRefsFetchTime": 0,
|
||||
"congestionControlLatency": 0,
|
||||
"pipelineWrapperFilteredLines": 0
|
||||
}
|
||||
},
|
||||
"ingester": {
|
||||
"totalReached": 1,
|
||||
"totalChunksMatched": 1,
|
||||
"totalBatches": 2,
|
||||
"totalLinesSent": 1,
|
||||
"store": {
|
||||
"totalChunksRef": 0,
|
||||
"totalChunksDownloaded": 0,
|
||||
"chunksDownloadTime": 0,
|
||||
"queryReferencedStructuredMetadata": false,
|
||||
"chunk": {
|
||||
"headChunkBytes": 23978,
|
||||
"headChunkLines": 47,
|
||||
"decompressedBytes": 785876,
|
||||
"decompressedLines": 1030,
|
||||
"compressedBytes": 95894,
|
||||
"totalDuplicates": 0,
|
||||
"postFilterLines": 1,
|
||||
"headChunkStructuredMetadataBytes": 6976,
|
||||
"decompressedStructuredMetadataBytes": 393780
|
||||
},
|
||||
"chunkRefsFetchTime": 72000,
|
||||
"congestionControlLatency": 0,
|
||||
"pipelineWrapperFilteredLines": 0
|
||||
}
|
||||
},
|
||||
"cache": {
|
||||
"chunk": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"index": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"result": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"statsResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"volumeResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"seriesResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"labelResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"instantMetricResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
}
|
||||
},
|
||||
"index": {
|
||||
"totalChunks": 0,
|
||||
"postFilterChunks": 0,
|
||||
"shardsDuration": 0,
|
||||
"usedBloomFilters": false
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,275 @@
|
||||
{
|
||||
"batches": [
|
||||
{
|
||||
"resource": {
|
||||
"attributes": [
|
||||
{
|
||||
"key": "deployment.environment.name",
|
||||
"value": {
|
||||
"stringValue": "lab"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.instance.id",
|
||||
"value": {
|
||||
"stringValue": "2a50b728ce10:1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.version",
|
||||
"value": {
|
||||
"stringValue": "6.0.0"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.language",
|
||||
"value": {
|
||||
"stringValue": "python"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.name",
|
||||
"value": {
|
||||
"stringValue": "opentelemetry"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.version",
|
||||
"value": {
|
||||
"stringValue": "1.45.1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.name",
|
||||
"value": {
|
||||
"stringValue": "messenger-api"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"scopeSpans": [
|
||||
{
|
||||
"scope": {
|
||||
"name": "messenger",
|
||||
"version": "6.0.0"
|
||||
},
|
||||
"spans": [
|
||||
{
|
||||
"traceId": "8STZP4TdJcWvl4544gKGRg==",
|
||||
"spanId": "6YhkxV7CY4k=",
|
||||
"parentSpanId": "zAWgCBMyg3I=",
|
||||
"name": "postgresql.QUERY",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"startTimeUnixNano": "1795102247372852382",
|
||||
"endTimeUnixNano": "1795102247372984882",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "db.system.name",
|
||||
"value": {
|
||||
"stringValue": "postgresql"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "db.operation.name",
|
||||
"value": {
|
||||
"stringValue": "QUERY"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "8STZP4TdJcWvl4544gKGRg==",
|
||||
"spanId": "9dARsClEhgI=",
|
||||
"parentSpanId": "zAWgCBMyg3I=",
|
||||
"name": "postgresql.SELECT",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"startTimeUnixNano": "1795102247373056215",
|
||||
"endTimeUnixNano": "1795102247373325590",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "db.system.name",
|
||||
"value": {
|
||||
"stringValue": "postgresql"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "db.operation.name",
|
||||
"value": {
|
||||
"stringValue": "SELECT"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "8STZP4TdJcWvl4544gKGRg==",
|
||||
"spanId": "zAWgCBMyg3I=",
|
||||
"parentSpanId": "POwLsH+qd/8=",
|
||||
"name": "postgresql.transaction",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1795102247372781257",
|
||||
"endTimeUnixNano": "1795102247373442673",
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "8STZP4TdJcWvl4544gKGRg==",
|
||||
"spanId": "HOilfsT0R4s=",
|
||||
"parentSpanId": "ZMWryXhMXeM=",
|
||||
"name": "postgresql.QUERY",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"startTimeUnixNano": "1795102247374040798",
|
||||
"endTimeUnixNano": "1795102247374149048",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "db.system.name",
|
||||
"value": {
|
||||
"stringValue": "postgresql"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "db.operation.name",
|
||||
"value": {
|
||||
"stringValue": "QUERY"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "8STZP4TdJcWvl4544gKGRg==",
|
||||
"spanId": "tNH67JGBzzQ=",
|
||||
"parentSpanId": "ZMWryXhMXeM=",
|
||||
"name": "postgresql.SELECT",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"startTimeUnixNano": "1795102247374219298",
|
||||
"endTimeUnixNano": "1795102247374661423",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "db.system.name",
|
||||
"value": {
|
||||
"stringValue": "postgresql"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "db.operation.name",
|
||||
"value": {
|
||||
"stringValue": "SELECT"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "8STZP4TdJcWvl4544gKGRg==",
|
||||
"spanId": "pO76DIPXzps=",
|
||||
"parentSpanId": "ZMWryXhMXeM=",
|
||||
"name": "postgresql.SELECT",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"startTimeUnixNano": "1795102247374726007",
|
||||
"endTimeUnixNano": "1795102247374869840",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "db.system.name",
|
||||
"value": {
|
||||
"stringValue": "postgresql"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "db.operation.name",
|
||||
"value": {
|
||||
"stringValue": "SELECT"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "8STZP4TdJcWvl4544gKGRg==",
|
||||
"spanId": "bxfNsCS1EJs=",
|
||||
"parentSpanId": "ZMWryXhMXeM=",
|
||||
"name": "postgresql.SELECT",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"startTimeUnixNano": "1795102247374927173",
|
||||
"endTimeUnixNano": "1795102249375042508",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "db.system.name",
|
||||
"value": {
|
||||
"stringValue": "postgresql"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "db.operation.name",
|
||||
"value": {
|
||||
"stringValue": "SELECT"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "error.type",
|
||||
"value": {
|
||||
"stringValue": "_WaitTimeout"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {
|
||||
"code": "STATUS_CODE_ERROR"
|
||||
}
|
||||
},
|
||||
{
|
||||
"traceId": "8STZP4TdJcWvl4544gKGRg==",
|
||||
"spanId": "ZMWryXhMXeM=",
|
||||
"parentSpanId": "POwLsH+qd/8=",
|
||||
"name": "postgresql.transaction",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1795102247373987923",
|
||||
"endTimeUnixNano": "1795102249375157091",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "error.type",
|
||||
"value": {
|
||||
"stringValue": "_WaitTimeout"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {
|
||||
"code": "STATUS_CODE_ERROR"
|
||||
}
|
||||
},
|
||||
{
|
||||
"traceId": "8STZP4TdJcWvl4544gKGRg==",
|
||||
"spanId": "POwLsH+qd/8=",
|
||||
"name": "POST /api/v1/chats/{chat_id}/messages",
|
||||
"kind": "SPAN_KIND_SERVER",
|
||||
"startTimeUnixNano": "1795102247372424840",
|
||||
"endTimeUnixNano": "1795102249376209799",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "http.request.method",
|
||||
"value": {
|
||||
"stringValue": "POST"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "http.route",
|
||||
"value": {
|
||||
"stringValue": "/api/v1/chats/{chat_id}/messages"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "http.response.status_code",
|
||||
"value": {
|
||||
"intValue": "503"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {
|
||||
"code": "STATUS_CODE_ERROR"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,795 @@
|
||||
{
|
||||
"utc": "2026-11-19T15:09:44.028971+00:00",
|
||||
"command": "./scripts/check-docker.sh",
|
||||
"project": "lab5-check-c81d43a829",
|
||||
"host": "Darwin arm64",
|
||||
"source_commit": "470d9d6f1e4fe4f594e37c2ba59550ee5ca54fc1",
|
||||
"working_tree": true,
|
||||
"configuration": {
|
||||
"concurrency": 2,
|
||||
"prefetch": 1,
|
||||
"attempts": 5,
|
||||
"timeout_seconds": 5,
|
||||
"lease_seconds": 15,
|
||||
"backoff_base_seconds": 0.15
|
||||
},
|
||||
"checks": {
|
||||
"topology_and_limits": {
|
||||
"passed": true,
|
||||
"api_replicas": 2,
|
||||
"broker": "Valkey Streams with AOF fsync always",
|
||||
"worker_processes": 2
|
||||
},
|
||||
"public_smoke_and_schema": {
|
||||
"passed": true
|
||||
},
|
||||
"student_api_regressions": {
|
||||
"passed": true
|
||||
},
|
||||
"worker_offline_api_broker_restart": {
|
||||
"passed": true,
|
||||
"attachment_id": "ef3471d3-538e-4d3f-8406-edfb2e78d803",
|
||||
"job_id": "29387d95-7e9f-4d86-b645-024cb36049cf",
|
||||
"message_id": "4ae76a83-fddf-4d6a-aa4c-1dffe8910540",
|
||||
"variants": {
|
||||
"thumbnail": {
|
||||
"size": [
|
||||
256,
|
||||
160
|
||||
],
|
||||
"bytes": 5813
|
||||
},
|
||||
"crop": {
|
||||
"size": [
|
||||
128,
|
||||
128
|
||||
],
|
||||
"bytes": 2462
|
||||
},
|
||||
"optimized": {
|
||||
"size": [
|
||||
320,
|
||||
200
|
||||
],
|
||||
"bytes": 2462
|
||||
}
|
||||
}
|
||||
},
|
||||
"outbox_crash_before_publish": {
|
||||
"passed": true,
|
||||
"job_id": "630dde85-21c6-4a82-bcee-5ea7aafa21b8",
|
||||
"automatic_recovery": true
|
||||
},
|
||||
"outbox_crash_after_publish": {
|
||||
"passed": true,
|
||||
"job_id": "5c9d0c57-07b5-41d9-951e-d07ed3ce2126",
|
||||
"automatic_recovery": true
|
||||
},
|
||||
"worker_crash_after_objects": {
|
||||
"passed": true,
|
||||
"job_id": "e7e67187-45e2-49ad-b845-7c80952abaed",
|
||||
"attempt": 2,
|
||||
"canonical_objects": 4,
|
||||
"duplicate_noop": true
|
||||
},
|
||||
"worker_crash_before_ack": {
|
||||
"passed": true,
|
||||
"job_id": "646cabc2-c06f-43a9-98ed-97a0ed430eef",
|
||||
"attempt": 1,
|
||||
"canonical_objects": 4,
|
||||
"duplicate_noop": true
|
||||
},
|
||||
"bounded_retry_failed_registry_replay": {
|
||||
"passed": true,
|
||||
"job_id": "9f0223e7-c7f7-41ce-a6b6-a6b2a6827db7",
|
||||
"max_automatic_attempts": 5,
|
||||
"audit": [
|
||||
{
|
||||
"event": "processing",
|
||||
"attempt": 1,
|
||||
"error_code": null,
|
||||
"at": "2026-11-19T15:11:43.910272+00:00"
|
||||
},
|
||||
{
|
||||
"event": "retry",
|
||||
"attempt": 1,
|
||||
"error_code": "storage_unavailable",
|
||||
"at": "2026-11-19T15:11:43.913931+00:00"
|
||||
},
|
||||
{
|
||||
"event": "processing",
|
||||
"attempt": 2,
|
||||
"error_code": null,
|
||||
"at": "2026-11-19T15:11:44.162093+00:00"
|
||||
},
|
||||
{
|
||||
"event": "retry",
|
||||
"attempt": 2,
|
||||
"error_code": "storage_unavailable",
|
||||
"at": "2026-11-19T15:11:44.163784+00:00"
|
||||
},
|
||||
{
|
||||
"event": "processing",
|
||||
"attempt": 3,
|
||||
"error_code": null,
|
||||
"at": "2026-11-19T15:11:44.672423+00:00"
|
||||
},
|
||||
{
|
||||
"event": "retry",
|
||||
"attempt": 3,
|
||||
"error_code": "storage_unavailable",
|
||||
"at": "2026-11-19T15:11:44.674937+00:00"
|
||||
},
|
||||
{
|
||||
"event": "processing",
|
||||
"attempt": 4,
|
||||
"error_code": null,
|
||||
"at": "2026-11-19T15:11:45.176057+00:00"
|
||||
},
|
||||
{
|
||||
"event": "retry",
|
||||
"attempt": 4,
|
||||
"error_code": "storage_unavailable",
|
||||
"at": "2026-11-19T15:11:45.17712+00:00"
|
||||
},
|
||||
{
|
||||
"event": "processing",
|
||||
"attempt": 5,
|
||||
"error_code": null,
|
||||
"at": "2026-11-19T15:11:45.941529+00:00"
|
||||
},
|
||||
{
|
||||
"event": "failed",
|
||||
"attempt": 5,
|
||||
"error_code": "retry_exhausted",
|
||||
"at": "2026-11-19T15:11:45.944804+00:00"
|
||||
},
|
||||
{
|
||||
"event": "operator_replay",
|
||||
"attempt": 5,
|
||||
"error_code": null,
|
||||
"at": "2026-11-19T15:11:47.029399+00:00"
|
||||
},
|
||||
{
|
||||
"event": "processing",
|
||||
"attempt": 6,
|
||||
"error_code": null,
|
||||
"at": "2026-11-19T15:11:47.203814+00:00"
|
||||
},
|
||||
{
|
||||
"event": "ready",
|
||||
"attempt": 6,
|
||||
"error_code": null,
|
||||
"at": "2026-11-19T15:11:47.601272+00:00"
|
||||
}
|
||||
],
|
||||
"ready_replay_rejected": true
|
||||
},
|
||||
"permanent_error": {
|
||||
"passed": true,
|
||||
"job_id": "28022e30-0cd6-4df1-a41b-56905cf4feda",
|
||||
"attempt": 1,
|
||||
"error_code": "invalid_image"
|
||||
},
|
||||
"image_subprocess_timeout": {
|
||||
"passed": true,
|
||||
"job_id": "c0f7e1b2-59a1-496e-a60a-8ed3fdb19f59",
|
||||
"timeout_seconds": 5,
|
||||
"task_memory_mb": 640,
|
||||
"recovered": true
|
||||
},
|
||||
"image_subprocess_memory": {
|
||||
"passed": true,
|
||||
"job_id": "f1e4caee-86d7-4185-9787-92687c0650a3",
|
||||
"timeout_seconds": 5,
|
||||
"task_memory_mb": 640,
|
||||
"recovered": true
|
||||
},
|
||||
"image_semantics": {
|
||||
"passed": true,
|
||||
"cases": [
|
||||
{
|
||||
"case": "small_alpha_on_white",
|
||||
"variants": {
|
||||
"thumbnail": {
|
||||
"size": [
|
||||
17,
|
||||
9
|
||||
],
|
||||
"bytes": 635
|
||||
},
|
||||
"crop": {
|
||||
"size": [
|
||||
128,
|
||||
128
|
||||
],
|
||||
"bytes": 883
|
||||
},
|
||||
"optimized": {
|
||||
"size": [
|
||||
17,
|
||||
9
|
||||
],
|
||||
"bytes": 56
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"case": "exif_orientation_6_and_floor",
|
||||
"variants": {
|
||||
"thumbnail": {
|
||||
"size": [
|
||||
160,
|
||||
256
|
||||
],
|
||||
"bytes": 1270
|
||||
},
|
||||
"crop": {
|
||||
"size": [
|
||||
128,
|
||||
128
|
||||
],
|
||||
"bytes": 886
|
||||
},
|
||||
"optimized": {
|
||||
"size": [
|
||||
201,
|
||||
321
|
||||
],
|
||||
"bytes": 210
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"staging_cleanup": {
|
||||
"passed": true,
|
||||
"active_preserved": true,
|
||||
"orphan_removed": true,
|
||||
"ready_removed": true,
|
||||
"expired_failed_removed": true,
|
||||
"expired_replay_rejected": true
|
||||
},
|
||||
"broker_offline_outbox": {
|
||||
"passed": true,
|
||||
"attachment_id": "28321e5b-8b43-471a-973d-9b55cbc07b39",
|
||||
"recovered": true
|
||||
},
|
||||
"public_persistence": {
|
||||
"passed": true,
|
||||
"restart_proof": {
|
||||
"passed": true,
|
||||
"before": {
|
||||
"ea2b13aebb92a1ea2741de72078489e6b05ebcc41f1ca74a2604c0ad7d355983": {
|
||||
"service": "api",
|
||||
"started_at": "2026-11-19T15:10:30.841890222Z",
|
||||
"volumes": []
|
||||
},
|
||||
"67eade4e281a4cfed3f55d87ab398097e12ee12155f82cbe19eb240a4e92b41a": {
|
||||
"service": "api",
|
||||
"started_at": "2026-11-19T15:10:30.809775805Z",
|
||||
"volumes": []
|
||||
},
|
||||
"d675c92d195a1bb37db5860c3cf49a2a918eb3e98cfd9c855f8f89931138724b": {
|
||||
"service": "broker",
|
||||
"started_at": "2026-11-19T15:12:24.444290385Z",
|
||||
"volumes": [
|
||||
"lab5-check-c81d43a829_broker_data"
|
||||
]
|
||||
},
|
||||
"f9b2e0b6e448dda6d845d9715d64e1216fc0f3f58c3e05ec22de30d305c2d3e6": {
|
||||
"service": "cleanup",
|
||||
"started_at": "2026-11-19T15:10:11.903050713Z",
|
||||
"volumes": []
|
||||
},
|
||||
"e4abde6d7d326be2a1425d81667ab967dd655ce1eb39c1b0c1bd2f9956f3e91f": {
|
||||
"service": "grafana",
|
||||
"started_at": "2026-11-19T15:09:46.76721934Z",
|
||||
"volumes": [
|
||||
"lab5-check-c81d43a829_grafana_data"
|
||||
]
|
||||
},
|
||||
"1e9b604ee9fe91572a4efd3b629a00819d86ab2b45ed1f04a1df66428cfe1920": {
|
||||
"service": "lb",
|
||||
"started_at": "2026-11-19T15:10:17.66400909Z",
|
||||
"volumes": []
|
||||
},
|
||||
"487c1937cced6d3b86d6a67cae2b76fdfc2b534e6cf15b6cf2242b96b19c8af4": {
|
||||
"service": "loki",
|
||||
"started_at": "2026-11-19T15:09:46.088598506Z",
|
||||
"volumes": [
|
||||
"lab5-check-c81d43a829_loki_data"
|
||||
]
|
||||
},
|
||||
"f31970f75546cfcf6c390fad538eb9a7ebc25d26c481caef09bf679a5af8e8d9": {
|
||||
"service": "otel-collector",
|
||||
"started_at": "2026-11-19T15:09:46.409972798Z",
|
||||
"volumes": []
|
||||
},
|
||||
"82576012c7b7b5a5a352207679f4c327fe50cbd5c56337de33c16ec15b2b0501": {
|
||||
"service": "postgres",
|
||||
"started_at": "2026-11-19T15:09:46.08945334Z",
|
||||
"volumes": [
|
||||
"lab5-check-c81d43a829_postgres_data"
|
||||
]
|
||||
},
|
||||
"6e305e5121dc3113f906c5689d1006091714d72b452acaabd07e3d366f1d13f6": {
|
||||
"service": "prometheus",
|
||||
"started_at": "2026-11-19T15:09:46.555234256Z",
|
||||
"volumes": [
|
||||
"lab5-check-c81d43a829_prometheus_data"
|
||||
]
|
||||
},
|
||||
"fb35c350ab7fcff516b44e4f6933b1e293c896e6cee9bf29d06e71ba99a0f931": {
|
||||
"service": "publisher",
|
||||
"started_at": "2026-11-19T15:12:10.913909129Z",
|
||||
"volumes": []
|
||||
},
|
||||
"8cdcd8bdac73e3bc827ef7bc5a77bad0696c1023cd330eb10206dcc6b29529cf": {
|
||||
"service": "s3",
|
||||
"started_at": "2026-11-19T15:10:08.720162878Z",
|
||||
"volumes": [
|
||||
"d6e81fdf6ec0a4bd30cb0461085c08461b951f8ae9c696222d04b21152ff6560"
|
||||
]
|
||||
},
|
||||
"053f3a4b4e5cd9b28510f08e9bd8d07374c586af075cb62d5c5b49f62af5a6ac": {
|
||||
"service": "s3-storage",
|
||||
"started_at": "2026-11-19T15:09:46.086474548Z",
|
||||
"volumes": [
|
||||
"lab5-check-c81d43a829_s3_data"
|
||||
]
|
||||
},
|
||||
"84473a25dbb67b410a0db8d8b0e029b84b264b3238da1961c7693f13d9e99634": {
|
||||
"service": "tempo",
|
||||
"started_at": "2026-11-19T15:09:46.086451215Z",
|
||||
"volumes": [
|
||||
"lab5-check-c81d43a829_tempo_data"
|
||||
]
|
||||
},
|
||||
"7f3ff970cf57bedf3a8800a6bed5cc3a83f749dab07d9ac42425d14c7eafee80": {
|
||||
"service": "valkey",
|
||||
"started_at": "2026-11-19T15:09:46.08714634Z",
|
||||
"volumes": [
|
||||
"lab5-check-c81d43a829_valkey_data"
|
||||
]
|
||||
},
|
||||
"521131dedb68623718c9d5d5c8dc8b0924dbfe6ec90acd3b21e08189a9a1a1e5": {
|
||||
"service": "worker",
|
||||
"started_at": "2026-11-19T15:12:21.892301301Z",
|
||||
"volumes": []
|
||||
}
|
||||
},
|
||||
"after": {
|
||||
"ea2b13aebb92a1ea2741de72078489e6b05ebcc41f1ca74a2604c0ad7d355983": {
|
||||
"service": "api",
|
||||
"started_at": "2026-11-19T15:12:30.097473346Z",
|
||||
"volumes": []
|
||||
},
|
||||
"67eade4e281a4cfed3f55d87ab398097e12ee12155f82cbe19eb240a4e92b41a": {
|
||||
"service": "api",
|
||||
"started_at": "2026-11-19T15:12:30.048159138Z",
|
||||
"volumes": []
|
||||
},
|
||||
"d675c92d195a1bb37db5860c3cf49a2a918eb3e98cfd9c855f8f89931138724b": {
|
||||
"service": "broker",
|
||||
"started_at": "2026-11-19T15:12:28.168864679Z",
|
||||
"volumes": [
|
||||
"lab5-check-c81d43a829_broker_data"
|
||||
]
|
||||
},
|
||||
"f9b2e0b6e448dda6d845d9715d64e1216fc0f3f58c3e05ec22de30d305c2d3e6": {
|
||||
"service": "cleanup",
|
||||
"started_at": "2026-11-19T15:12:35.75098096Z",
|
||||
"volumes": []
|
||||
},
|
||||
"e4abde6d7d326be2a1425d81667ab967dd655ce1eb39c1b0c1bd2f9956f3e91f": {
|
||||
"service": "grafana",
|
||||
"started_at": "2026-11-19T15:12:28.016722095Z",
|
||||
"volumes": [
|
||||
"lab5-check-c81d43a829_grafana_data"
|
||||
]
|
||||
},
|
||||
"1e9b604ee9fe91572a4efd3b629a00819d86ab2b45ed1f04a1df66428cfe1920": {
|
||||
"service": "lb",
|
||||
"started_at": "2026-11-19T15:12:27.799505637Z",
|
||||
"volumes": []
|
||||
},
|
||||
"487c1937cced6d3b86d6a67cae2b76fdfc2b534e6cf15b6cf2242b96b19c8af4": {
|
||||
"service": "loki",
|
||||
"started_at": "2026-11-19T15:12:29.867573388Z",
|
||||
"volumes": [
|
||||
"lab5-check-c81d43a829_loki_data"
|
||||
]
|
||||
},
|
||||
"f31970f75546cfcf6c390fad538eb9a7ebc25d26c481caef09bf679a5af8e8d9": {
|
||||
"service": "otel-collector",
|
||||
"started_at": "2026-11-19T15:12:29.704735929Z",
|
||||
"volumes": []
|
||||
},
|
||||
"82576012c7b7b5a5a352207679f4c327fe50cbd5c56337de33c16ec15b2b0501": {
|
||||
"service": "postgres",
|
||||
"started_at": "2026-11-19T15:12:28.084774804Z",
|
||||
"volumes": [
|
||||
"lab5-check-c81d43a829_postgres_data"
|
||||
]
|
||||
},
|
||||
"6e305e5121dc3113f906c5689d1006091714d72b452acaabd07e3d366f1d13f6": {
|
||||
"service": "prometheus",
|
||||
"started_at": "2026-11-19T15:12:28.052574179Z",
|
||||
"volumes": [
|
||||
"lab5-check-c81d43a829_prometheus_data"
|
||||
]
|
||||
},
|
||||
"fb35c350ab7fcff516b44e4f6933b1e293c896e6cee9bf29d06e71ba99a0f931": {
|
||||
"service": "publisher",
|
||||
"started_at": "2026-11-19T15:12:35.770825668Z",
|
||||
"volumes": []
|
||||
},
|
||||
"8cdcd8bdac73e3bc827ef7bc5a77bad0696c1023cd330eb10206dcc6b29529cf": {
|
||||
"service": "s3",
|
||||
"started_at": "2026-11-19T15:12:28.115603887Z",
|
||||
"volumes": [
|
||||
"d6e81fdf6ec0a4bd30cb0461085c08461b951f8ae9c696222d04b21152ff6560"
|
||||
]
|
||||
},
|
||||
"053f3a4b4e5cd9b28510f08e9bd8d07374c586af075cb62d5c5b49f62af5a6ac": {
|
||||
"service": "s3-storage",
|
||||
"started_at": "2026-11-19T15:12:35.714312418Z",
|
||||
"volumes": [
|
||||
"lab5-check-c81d43a829_s3_data"
|
||||
]
|
||||
},
|
||||
"84473a25dbb67b410a0db8d8b0e029b84b264b3238da1961c7693f13d9e99634": {
|
||||
"service": "tempo",
|
||||
"started_at": "2026-11-19T15:12:28.142091095Z",
|
||||
"volumes": [
|
||||
"lab5-check-c81d43a829_tempo_data"
|
||||
]
|
||||
},
|
||||
"7f3ff970cf57bedf3a8800a6bed5cc3a83f749dab07d9ac42425d14c7eafee80": {
|
||||
"service": "valkey",
|
||||
"started_at": "2026-11-19T15:12:28.230646637Z",
|
||||
"volumes": [
|
||||
"lab5-check-c81d43a829_valkey_data"
|
||||
]
|
||||
},
|
||||
"521131dedb68623718c9d5d5c8dc8b0924dbfe6ec90acd3b21e08189a9a1a1e5": {
|
||||
"service": "worker",
|
||||
"started_at": "2026-11-19T15:12:27.946355887Z",
|
||||
"volumes": []
|
||||
}
|
||||
},
|
||||
"same_containers_and_volumes": true
|
||||
}
|
||||
},
|
||||
"public_api_failover": {
|
||||
"passed": true
|
||||
}
|
||||
},
|
||||
"docker": "29.0.1 aarch64 CPUs=10 RAM=8217448448",
|
||||
"containers": [
|
||||
{
|
||||
"id": "ea2b13aebb92a1ea2741de72078489e6b05ebcc41f1ca74a2604c0ad7d355983",
|
||||
"service": "api",
|
||||
"image": "mtusi-messenger:lab6",
|
||||
"memory_bytes": 402653184,
|
||||
"nano_cpus": 1000000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/run/secrets/database_url",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/cache_url",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/s3_access_key_id",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/s3_secret_access_key",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "67eade4e281a4cfed3f55d87ab398097e12ee12155f82cbe19eb240a4e92b41a",
|
||||
"service": "api",
|
||||
"image": "mtusi-messenger:lab6",
|
||||
"memory_bytes": 402653184,
|
||||
"nano_cpus": 1000000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/run/secrets/cache_url",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/s3_access_key_id",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/s3_secret_access_key",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/database_url",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "d675c92d195a1bb37db5860c3cf49a2a918eb3e98cfd9c855f8f89931138724b",
|
||||
"service": "broker",
|
||||
"image": "valkey/valkey:8.1.3",
|
||||
"memory_bytes": 268435456,
|
||||
"nano_cpus": 500000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/data",
|
||||
"name": "lab5-check-c81d43a829_broker_data"
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/broker_password",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "f9b2e0b6e448dda6d845d9715d64e1216fc0f3f58c3e05ec22de30d305c2d3e6",
|
||||
"service": "cleanup",
|
||||
"image": "mtusi-messenger:lab6",
|
||||
"memory_bytes": 402653184,
|
||||
"nano_cpus": 1000000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/run/secrets/s3_access_key_id",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/s3_secret_access_key",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/database_url",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "e4abde6d7d326be2a1425d81667ab967dd655ce1eb39c1b0c1bd2f9956f3e91f",
|
||||
"service": "grafana",
|
||||
"image": "grafana/grafana:12.0.2",
|
||||
"memory_bytes": 402653184,
|
||||
"nano_cpus": 1000000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/etc/grafana/provisioning",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/var/lib/grafana/dashboards",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/var/lib/grafana",
|
||||
"name": "lab5-check-c81d43a829_grafana_data"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "1e9b604ee9fe91572a4efd3b629a00819d86ab2b45ed1f04a1df66428cfe1920",
|
||||
"service": "lb",
|
||||
"image": "haproxy:3.2.6-alpine",
|
||||
"memory_bytes": 67108864,
|
||||
"nano_cpus": 500000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/usr/local/etc/haproxy/haproxy.cfg",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "487c1937cced6d3b86d6a67cae2b76fdfc2b534e6cf15b6cf2242b96b19c8af4",
|
||||
"service": "loki",
|
||||
"image": "grafana/loki:3.5.1",
|
||||
"memory_bytes": 402653184,
|
||||
"nano_cpus": 1000000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/etc/loki/config.yaml",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/loki",
|
||||
"name": "lab5-check-c81d43a829_loki_data"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "02572452b7c401e702c6ad13daac234d9e3c069bb791e44770a7f7fc11dca885",
|
||||
"service": "migrate",
|
||||
"image": "mtusi-messenger:lab6",
|
||||
"memory_bytes": 402653184,
|
||||
"nano_cpus": 1000000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/run/secrets/database_url",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "f31970f75546cfcf6c390fad538eb9a7ebc25d26c481caef09bf679a5af8e8d9",
|
||||
"service": "otel-collector",
|
||||
"image": "otel/opentelemetry-collector-contrib:0.123.0",
|
||||
"memory_bytes": 268435456,
|
||||
"nano_cpus": 1000000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/etc/otelcol/config.yaml",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "82576012c7b7b5a5a352207679f4c327fe50cbd5c56337de33c16ec15b2b0501",
|
||||
"service": "postgres",
|
||||
"image": "postgres:17.6-bookworm",
|
||||
"memory_bytes": 536870912,
|
||||
"nano_cpus": 1000000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/run/secrets/postgres_password",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/var/lib/postgresql/data",
|
||||
"name": "lab5-check-c81d43a829_postgres_data"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "6e305e5121dc3113f906c5689d1006091714d72b452acaabd07e3d366f1d13f6",
|
||||
"service": "prometheus",
|
||||
"image": "prom/prometheus:v3.4.1",
|
||||
"memory_bytes": 402653184,
|
||||
"nano_cpus": 1000000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/prometheus",
|
||||
"name": "lab5-check-c81d43a829_prometheus_data"
|
||||
},
|
||||
{
|
||||
"destination": "/etc/prometheus/prometheus.yaml",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/etc/prometheus/alerts.yaml",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "fb35c350ab7fcff516b44e4f6933b1e293c896e6cee9bf29d06e71ba99a0f931",
|
||||
"service": "publisher",
|
||||
"image": "mtusi-messenger:lab6",
|
||||
"memory_bytes": 402653184,
|
||||
"nano_cpus": 1000000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/run/secrets/database_url",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/broker_url",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "8cdcd8bdac73e3bc827ef7bc5a77bad0696c1023cd330eb10206dcc6b29529cf",
|
||||
"service": "s3",
|
||||
"image": "chrislusf/seaweedfs:3.85@sha256:49312939c00c01e5ee6afbd7d728b18027821d3764c35a797a72acd4fdf3296a",
|
||||
"memory_bytes": 268435456,
|
||||
"nano_cpus": 500000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/run/secrets/s3_config",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/data",
|
||||
"name": "d6e81fdf6ec0a4bd30cb0461085c08461b951f8ae9c696222d04b21152ff6560"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "e491085d55967c2cc9a4c907385fa87d8ee2e7d072ae77082a39ffde575cf0e8",
|
||||
"service": "s3-bootstrap",
|
||||
"image": "mtusi-messenger:lab6",
|
||||
"memory_bytes": 134217728,
|
||||
"nano_cpus": 500000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/run/secrets/s3_root_user",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/s3_root_password",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "053f3a4b4e5cd9b28510f08e9bd8d07374c586af075cb62d5c5b49f62af5a6ac",
|
||||
"service": "s3-storage",
|
||||
"image": "chrislusf/seaweedfs:3.85@sha256:49312939c00c01e5ee6afbd7d728b18027821d3764c35a797a72acd4fdf3296a",
|
||||
"memory_bytes": 536870912,
|
||||
"nano_cpus": 1000000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/data",
|
||||
"name": "lab5-check-c81d43a829_s3_data"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "84473a25dbb67b410a0db8d8b0e029b84b264b3238da1961c7693f13d9e99634",
|
||||
"service": "tempo",
|
||||
"image": "grafana/tempo:2.8.1",
|
||||
"memory_bytes": 536870912,
|
||||
"nano_cpus": 1000000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/etc/tempo/config.yaml",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/var/tempo",
|
||||
"name": "lab5-check-c81d43a829_tempo_data"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "7f3ff970cf57bedf3a8800a6bed5cc3a83f749dab07d9ac42425d14c7eafee80",
|
||||
"service": "valkey",
|
||||
"image": "valkey/valkey:8.1.3",
|
||||
"memory_bytes": 134217728,
|
||||
"nano_cpus": 500000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/data",
|
||||
"name": "lab5-check-c81d43a829_valkey_data"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "521131dedb68623718c9d5d5c8dc8b0924dbfe6ec90acd3b21e08189a9a1a1e5",
|
||||
"service": "worker",
|
||||
"image": "mtusi-messenger:lab6",
|
||||
"memory_bytes": 1610612736,
|
||||
"nano_cpus": 2000000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/run/secrets/s3_secret_access_key",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/database_url",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/broker_url",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/s3_access_key_id",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"passed": true
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1 @@
|
||||
SUCCESS
|
||||
@@ -0,0 +1,206 @@
|
||||
{
|
||||
"utc": "2026-11-19T15:11:37.994748+00:00",
|
||||
"command": ".venv/bin/python student-tests/verify_restart.py",
|
||||
"passed": true,
|
||||
"before": {
|
||||
"166e3f521d4f8b2206766859d89df193d0ebc44fb7b8c2d597da65ebefaaf1af": {
|
||||
"service": "api",
|
||||
"started_at": "2026-11-19T15:08:38.453676544Z",
|
||||
"volumes": []
|
||||
},
|
||||
"863415a50886df6498a2c5d137298288e403eb8f628fd93088a93174bb744d8c": {
|
||||
"service": "api",
|
||||
"started_at": "2026-11-19T15:08:38.611590211Z",
|
||||
"volumes": []
|
||||
},
|
||||
"ea51e5f3e0e7921a34c8678a17da799c5f440f0423fc5cc0d0ebc88b6be109b8": {
|
||||
"service": "broker",
|
||||
"started_at": "2026-11-19T15:00:33.36494925Z",
|
||||
"volumes": [
|
||||
"mtusi-lab6_broker_data"
|
||||
]
|
||||
},
|
||||
"d24bddbf342d9bc0f3b43d8462e7d991d82e38c848d4ef762a8b0b1cbb5959ba": {
|
||||
"service": "cleanup",
|
||||
"started_at": "2026-11-19T15:08:38.446185961Z",
|
||||
"volumes": []
|
||||
},
|
||||
"375a3083ca7919b26cdc327adf969072cd09d91f3dc664037b63fb890c91849a": {
|
||||
"service": "grafana",
|
||||
"started_at": "2026-11-19T15:00:33.781731Z",
|
||||
"volumes": [
|
||||
"mtusi-lab6_grafana_data"
|
||||
]
|
||||
},
|
||||
"f56ed57943f2889033e15612d17d633e0b7bb3a9cceb9cb2ab336524117e6ffa": {
|
||||
"service": "lb",
|
||||
"started_at": "2026-11-19T15:00:58.904707679Z",
|
||||
"volumes": []
|
||||
},
|
||||
"e107e6ede3c3b2be8a6995ef242f614dc6f592cf230f10d7862685696991ae71": {
|
||||
"service": "loki",
|
||||
"started_at": "2026-11-19T15:10:28.625123846Z",
|
||||
"volumes": [
|
||||
"mtusi-lab6_loki_data"
|
||||
]
|
||||
},
|
||||
"6e21c63012d160a406e28a342d668de0be55107aa16228ed9d5dc15c34b447be": {
|
||||
"service": "otel-collector",
|
||||
"started_at": "2026-11-19T15:01:52.896095343Z",
|
||||
"volumes": []
|
||||
},
|
||||
"1b3cdbc4689c6b855a435acab8112f86a3c9dc5e0207257c31e7aaa650635a0a": {
|
||||
"service": "postgres",
|
||||
"started_at": "2026-11-19T15:00:33.362963334Z",
|
||||
"volumes": [
|
||||
"mtusi-lab6_postgres_data"
|
||||
]
|
||||
},
|
||||
"d54da34ab176f240c2519bb6fcc46417b5ea3d27de86e2ed05321675f14e1e23": {
|
||||
"service": "prometheus",
|
||||
"started_at": "2026-11-19T15:00:33.65938775Z",
|
||||
"volumes": [
|
||||
"mtusi-lab6_prometheus_data"
|
||||
]
|
||||
},
|
||||
"0cd7cf668f46d355febf0dd912003289a65966692b21a759997e0a8d1c5d83eb": {
|
||||
"service": "publisher",
|
||||
"started_at": "2026-11-19T15:08:38.448713669Z",
|
||||
"volumes": []
|
||||
},
|
||||
"b7b3fe9f4f328f57dc0df0a3af84d8426e431505ffd236da9d6cc143123380c9": {
|
||||
"service": "s3",
|
||||
"started_at": "2026-11-19T15:00:49.997656091Z",
|
||||
"volumes": [
|
||||
"a6b780554a2ebb58edb3e2f7b77fe7ee705ce9543ef0dbe0c8d9c4d80bc31da2"
|
||||
]
|
||||
},
|
||||
"1b69e90a9f7108e2e5d2017365e82fa30495c42f5f93809d42b2fdfcc1f0a9a0": {
|
||||
"service": "s3-storage",
|
||||
"started_at": "2026-11-19T15:00:33.3498635Z",
|
||||
"volumes": [
|
||||
"mtusi-lab6_s3_data"
|
||||
]
|
||||
},
|
||||
"7dc0833e289e6e708dff042076bf4753617ba5e61122a9a67dd61c37384f1c9b": {
|
||||
"service": "tempo",
|
||||
"started_at": "2026-11-19T15:08:38.454968128Z",
|
||||
"volumes": [
|
||||
"mtusi-lab6_tempo_data"
|
||||
]
|
||||
},
|
||||
"f4c736f41e428fb83e0effb49e8a56d9efe23ea6bd979a16fdcf072af0b342b0": {
|
||||
"service": "valkey",
|
||||
"started_at": "2026-11-19T15:00:33.364171167Z",
|
||||
"volumes": [
|
||||
"mtusi-lab6_valkey_data"
|
||||
]
|
||||
},
|
||||
"ef1c4fbd5187a8773b7b448cb5f0b0534e83bddb9b12a51f2ffcd7a326e5e729": {
|
||||
"service": "worker",
|
||||
"started_at": "2026-11-19T15:10:59.920254179Z",
|
||||
"volumes": []
|
||||
}
|
||||
},
|
||||
"after": {
|
||||
"166e3f521d4f8b2206766859d89df193d0ebc44fb7b8c2d597da65ebefaaf1af": {
|
||||
"service": "api",
|
||||
"started_at": "2026-11-19T15:11:41.117466337Z",
|
||||
"volumes": []
|
||||
},
|
||||
"863415a50886df6498a2c5d137298288e403eb8f628fd93088a93174bb744d8c": {
|
||||
"service": "api",
|
||||
"started_at": "2026-11-19T15:11:41.169749671Z",
|
||||
"volumes": []
|
||||
},
|
||||
"ea51e5f3e0e7921a34c8678a17da799c5f440f0423fc5cc0d0ebc88b6be109b8": {
|
||||
"service": "broker",
|
||||
"started_at": "2026-11-19T15:11:40.171899503Z",
|
||||
"volumes": [
|
||||
"mtusi-lab6_broker_data"
|
||||
]
|
||||
},
|
||||
"d24bddbf342d9bc0f3b43d8462e7d991d82e38c848d4ef762a8b0b1cbb5959ba": {
|
||||
"service": "cleanup",
|
||||
"started_at": "2026-11-19T15:11:47.626797257Z",
|
||||
"volumes": []
|
||||
},
|
||||
"375a3083ca7919b26cdc327adf969072cd09d91f3dc664037b63fb890c91849a": {
|
||||
"service": "grafana",
|
||||
"started_at": "2026-11-19T15:11:39.799567837Z",
|
||||
"volumes": [
|
||||
"mtusi-lab6_grafana_data"
|
||||
]
|
||||
},
|
||||
"f56ed57943f2889033e15612d17d633e0b7bb3a9cceb9cb2ab336524117e6ffa": {
|
||||
"service": "lb",
|
||||
"started_at": "2026-11-19T15:11:39.674464462Z",
|
||||
"volumes": []
|
||||
},
|
||||
"e107e6ede3c3b2be8a6995ef242f614dc6f592cf230f10d7862685696991ae71": {
|
||||
"service": "loki",
|
||||
"started_at": "2026-11-19T15:11:40.282189378Z",
|
||||
"volumes": [
|
||||
"mtusi-lab6_loki_data"
|
||||
]
|
||||
},
|
||||
"6e21c63012d160a406e28a342d668de0be55107aa16228ed9d5dc15c34b447be": {
|
||||
"service": "otel-collector",
|
||||
"started_at": "2026-11-19T15:11:39.989740337Z",
|
||||
"volumes": []
|
||||
},
|
||||
"1b3cdbc4689c6b855a435acab8112f86a3c9dc5e0207257c31e7aaa650635a0a": {
|
||||
"service": "postgres",
|
||||
"started_at": "2026-11-19T15:11:40.043447628Z",
|
||||
"volumes": [
|
||||
"mtusi-lab6_postgres_data"
|
||||
]
|
||||
},
|
||||
"d54da34ab176f240c2519bb6fcc46417b5ea3d27de86e2ed05321675f14e1e23": {
|
||||
"service": "prometheus",
|
||||
"started_at": "2026-11-19T15:11:39.859058378Z",
|
||||
"volumes": [
|
||||
"mtusi-lab6_prometheus_data"
|
||||
]
|
||||
},
|
||||
"0cd7cf668f46d355febf0dd912003289a65966692b21a759997e0a8d1c5d83eb": {
|
||||
"service": "publisher",
|
||||
"started_at": "2026-11-19T15:11:47.594100007Z",
|
||||
"volumes": []
|
||||
},
|
||||
"b7b3fe9f4f328f57dc0df0a3af84d8426e431505ffd236da9d6cc143123380c9": {
|
||||
"service": "s3",
|
||||
"started_at": "2026-11-19T15:11:40.13670567Z",
|
||||
"volumes": [
|
||||
"a6b780554a2ebb58edb3e2f7b77fe7ee705ce9543ef0dbe0c8d9c4d80bc31da2"
|
||||
]
|
||||
},
|
||||
"1b69e90a9f7108e2e5d2017365e82fa30495c42f5f93809d42b2fdfcc1f0a9a0": {
|
||||
"service": "s3-storage",
|
||||
"started_at": "2026-11-19T15:11:47.65993734Z",
|
||||
"volumes": [
|
||||
"mtusi-lab6_s3_data"
|
||||
]
|
||||
},
|
||||
"7dc0833e289e6e708dff042076bf4753617ba5e61122a9a67dd61c37384f1c9b": {
|
||||
"service": "tempo",
|
||||
"started_at": "2026-11-19T15:11:40.243825753Z",
|
||||
"volumes": [
|
||||
"mtusi-lab6_tempo_data"
|
||||
]
|
||||
},
|
||||
"f4c736f41e428fb83e0effb49e8a56d9efe23ea6bd979a16fdcf072af0b342b0": {
|
||||
"service": "valkey",
|
||||
"started_at": "2026-11-19T15:11:40.087174462Z",
|
||||
"volumes": [
|
||||
"mtusi-lab6_valkey_data"
|
||||
]
|
||||
},
|
||||
"ef1c4fbd5187a8773b7b448cb5f0b0534e83bddb9b12a51f2ffcd7a326e5e729": {
|
||||
"service": "worker",
|
||||
"started_at": "2026-11-19T15:11:40.21509467Z",
|
||||
"volumes": []
|
||||
}
|
||||
},
|
||||
"same_containers_and_volumes": true
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab6/mtusi2026lab6/.venv/bin/python tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:8080"
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
PASS: all persistent services actually restarted; container IDs and volumes preserved
|
||||
@@ -0,0 +1,322 @@
|
||||
{
|
||||
"passed": true,
|
||||
"services": [
|
||||
{
|
||||
"service": "api",
|
||||
"container_id": "33daefd247cfa9bdb99c0e9d10913c5305de5db293431162467a9fb0f55f3e4c",
|
||||
"image": "mtusi-messenger:lab6",
|
||||
"image_id": "sha256:a7932cb2a948db89797bf90f560ecf0046cac576c7fff97412aa55d8c4446a62",
|
||||
"state": "running",
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 402653184,
|
||||
"nano_cpus": 1000000000,
|
||||
"oom_killed": false,
|
||||
"published_ports": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"service": "api",
|
||||
"container_id": "2a50b728ce1094a536622ed51528f1e359b5bbd51685bb3b368dfb9327a2f9d8",
|
||||
"image": "mtusi-messenger:lab6",
|
||||
"image_id": "sha256:a7932cb2a948db89797bf90f560ecf0046cac576c7fff97412aa55d8c4446a62",
|
||||
"state": "running",
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 402653184,
|
||||
"nano_cpus": 1000000000,
|
||||
"oom_killed": false,
|
||||
"published_ports": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"service": "broker",
|
||||
"container_id": "ea51e5f3e0e7921a34c8678a17da799c5f440f0423fc5cc0d0ebc88b6be109b8",
|
||||
"image": "valkey/valkey:8.1.3",
|
||||
"image_id": "sha256:fea8b3e67b15729d4bb70589eb03367bab9ad1ee89c876f54327fc7c6e618571",
|
||||
"state": "running",
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 268435456,
|
||||
"nano_cpus": 500000000,
|
||||
"oom_killed": false,
|
||||
"published_ports": {},
|
||||
"volumes": [
|
||||
"mtusi-lab6_broker_data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "cleanup",
|
||||
"container_id": "e3328a92cd7b264a34d3ba2863ebd509ac105e68d507ce39062181cbe34e91bc",
|
||||
"image": "mtusi-messenger:lab6",
|
||||
"image_id": "sha256:a7932cb2a948db89797bf90f560ecf0046cac576c7fff97412aa55d8c4446a62",
|
||||
"state": "running",
|
||||
"health": null,
|
||||
"memory_limit_bytes": 402653184,
|
||||
"nano_cpus": 1000000000,
|
||||
"oom_killed": false,
|
||||
"published_ports": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"service": "grafana",
|
||||
"container_id": "9d373574ca2ee05f2e55ffc54a58266bfe13ac1c12d46c5275355a716fa8bd70",
|
||||
"image": "grafana/grafana:12.0.2",
|
||||
"image_id": "sha256:b5b59bfc7561634c2d7b136c4543d702ebcc94a3da477f21ff26f89ffd4214fa",
|
||||
"state": "running",
|
||||
"health": null,
|
||||
"memory_limit_bytes": 402653184,
|
||||
"nano_cpus": 1000000000,
|
||||
"oom_killed": false,
|
||||
"published_ports": {
|
||||
"3000/tcp": [
|
||||
{
|
||||
"HostIp": "127.0.0.1",
|
||||
"HostPort": "3000"
|
||||
}
|
||||
]
|
||||
},
|
||||
"volumes": [
|
||||
"mtusi-lab6_grafana_data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "lb",
|
||||
"container_id": "f56ed57943f2889033e15612d17d633e0b7bb3a9cceb9cb2ab336524117e6ffa",
|
||||
"image": "haproxy:3.2.6-alpine",
|
||||
"image_id": "sha256:3a819392f5a6af2a9203d1c63ff0feba74f0c940623a558511be06d7d85e361c",
|
||||
"state": "running",
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 67108864,
|
||||
"nano_cpus": 500000000,
|
||||
"oom_killed": false,
|
||||
"published_ports": {
|
||||
"8080/tcp": [
|
||||
{
|
||||
"HostIp": "127.0.0.1",
|
||||
"HostPort": "8080"
|
||||
}
|
||||
]
|
||||
},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"service": "loki",
|
||||
"container_id": "e107e6ede3c3b2be8a6995ef242f614dc6f592cf230f10d7862685696991ae71",
|
||||
"image": "grafana/loki:3.5.1",
|
||||
"image_id": "sha256:a74594532eec4cc313401beedc4dd2708c43674c032084b1aeb87c14a5be1745",
|
||||
"state": "running",
|
||||
"health": null,
|
||||
"memory_limit_bytes": 402653184,
|
||||
"nano_cpus": 1000000000,
|
||||
"oom_killed": false,
|
||||
"published_ports": {},
|
||||
"volumes": [
|
||||
"mtusi-lab6_loki_data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "migrate",
|
||||
"container_id": "a01249e7317b529181758de5244cd14133cc7b39ba5077ee3d844d240ca98cb2",
|
||||
"image": "mtusi-messenger:lab6",
|
||||
"image_id": "sha256:0189284e9808367735919f731c1169eb64ba34743d8f5dcaebf73a9f6788902c",
|
||||
"state": "exited",
|
||||
"health": null,
|
||||
"memory_limit_bytes": 402653184,
|
||||
"nano_cpus": 1000000000,
|
||||
"oom_killed": false,
|
||||
"published_ports": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"service": "otel-collector",
|
||||
"container_id": "6e21c63012d160a406e28a342d668de0be55107aa16228ed9d5dc15c34b447be",
|
||||
"image": "otel/opentelemetry-collector-contrib:0.123.0",
|
||||
"image_id": "sha256:e39311df1f3d941923c00da79ac7ba6269124a870ee87e3c3ad24d60f8aee4d2",
|
||||
"state": "running",
|
||||
"health": null,
|
||||
"memory_limit_bytes": 268435456,
|
||||
"nano_cpus": 1000000000,
|
||||
"oom_killed": false,
|
||||
"published_ports": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"service": "postgres",
|
||||
"container_id": "1b3cdbc4689c6b855a435acab8112f86a3c9dc5e0207257c31e7aaa650635a0a",
|
||||
"image": "postgres:17.6-bookworm",
|
||||
"image_id": "sha256:f3bd19c606e442c3d7bdfa8002e03fe260a1023351e0ea4598032022b68dd6e3",
|
||||
"state": "running",
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 536870912,
|
||||
"nano_cpus": 1000000000,
|
||||
"oom_killed": false,
|
||||
"published_ports": {},
|
||||
"volumes": [
|
||||
"mtusi-lab6_postgres_data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "prometheus",
|
||||
"container_id": "d54da34ab176f240c2519bb6fcc46417b5ea3d27de86e2ed05321675f14e1e23",
|
||||
"image": "prom/prometheus:v3.4.1",
|
||||
"image_id": "sha256:9abc6cf6aea7710d163dbb28d8eeb7dc5baef01e38fa4cd146a406dd9f07f70d",
|
||||
"state": "running",
|
||||
"health": null,
|
||||
"memory_limit_bytes": 402653184,
|
||||
"nano_cpus": 1000000000,
|
||||
"oom_killed": false,
|
||||
"published_ports": {},
|
||||
"volumes": [
|
||||
"mtusi-lab6_prometheus_data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "publisher",
|
||||
"container_id": "02738dc58bbbadc9dc35910593e6a75bcb1794821b38f892d22f05c0f9df7772",
|
||||
"image": "mtusi-messenger:lab6",
|
||||
"image_id": "sha256:a7932cb2a948db89797bf90f560ecf0046cac576c7fff97412aa55d8c4446a62",
|
||||
"state": "running",
|
||||
"health": null,
|
||||
"memory_limit_bytes": 402653184,
|
||||
"nano_cpus": 1000000000,
|
||||
"oom_killed": false,
|
||||
"published_ports": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"service": "s3",
|
||||
"container_id": "b7b3fe9f4f328f57dc0df0a3af84d8426e431505ffd236da9d6cc143123380c9",
|
||||
"image": "chrislusf/seaweedfs:3.85@sha256:49312939c00c01e5ee6afbd7d728b18027821d3764c35a797a72acd4fdf3296a",
|
||||
"image_id": "sha256:49312939c00c01e5ee6afbd7d728b18027821d3764c35a797a72acd4fdf3296a",
|
||||
"state": "running",
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 268435456,
|
||||
"nano_cpus": 500000000,
|
||||
"oom_killed": false,
|
||||
"published_ports": {},
|
||||
"volumes": [
|
||||
"a6b780554a2ebb58edb3e2f7b77fe7ee705ce9543ef0dbe0c8d9c4d80bc31da2"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "s3-bootstrap",
|
||||
"container_id": "4c9c5e1d677c45da83a1651c20809f58457588843791e01d5b6e22ab486c9467",
|
||||
"image": "mtusi-messenger:lab6",
|
||||
"image_id": "sha256:0189284e9808367735919f731c1169eb64ba34743d8f5dcaebf73a9f6788902c",
|
||||
"state": "exited",
|
||||
"health": null,
|
||||
"memory_limit_bytes": 134217728,
|
||||
"nano_cpus": 500000000,
|
||||
"oom_killed": false,
|
||||
"published_ports": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"service": "s3-storage",
|
||||
"container_id": "1b69e90a9f7108e2e5d2017365e82fa30495c42f5f93809d42b2fdfcc1f0a9a0",
|
||||
"image": "chrislusf/seaweedfs:3.85@sha256:49312939c00c01e5ee6afbd7d728b18027821d3764c35a797a72acd4fdf3296a",
|
||||
"image_id": "sha256:49312939c00c01e5ee6afbd7d728b18027821d3764c35a797a72acd4fdf3296a",
|
||||
"state": "running",
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 536870912,
|
||||
"nano_cpus": 1000000000,
|
||||
"oom_killed": false,
|
||||
"published_ports": {},
|
||||
"volumes": [
|
||||
"mtusi-lab6_s3_data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "tempo",
|
||||
"container_id": "7dc0833e289e6e708dff042076bf4753617ba5e61122a9a67dd61c37384f1c9b",
|
||||
"image": "grafana/tempo:2.8.1",
|
||||
"image_id": "sha256:bc9245fe3da4e63dc4c6862d9c2dad9bcd8be13d0ba4f7705fa6acda4c904d0e",
|
||||
"state": "running",
|
||||
"health": null,
|
||||
"memory_limit_bytes": 536870912,
|
||||
"nano_cpus": 1000000000,
|
||||
"oom_killed": false,
|
||||
"published_ports": {},
|
||||
"volumes": [
|
||||
"mtusi-lab6_tempo_data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "valkey",
|
||||
"container_id": "f4c736f41e428fb83e0effb49e8a56d9efe23ea6bd979a16fdcf072af0b342b0",
|
||||
"image": "valkey/valkey:8.1.3",
|
||||
"image_id": "sha256:fea8b3e67b15729d4bb70589eb03367bab9ad1ee89c876f54327fc7c6e618571",
|
||||
"state": "running",
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 134217728,
|
||||
"nano_cpus": 500000000,
|
||||
"oom_killed": false,
|
||||
"published_ports": {},
|
||||
"volumes": [
|
||||
"mtusi-lab6_valkey_data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "worker",
|
||||
"container_id": "168b98028f54233914e74d5a5e4e0f75b68968230fce783e9835e768e1450813",
|
||||
"image": "mtusi-messenger:lab6",
|
||||
"image_id": "sha256:a7932cb2a948db89797bf90f560ecf0046cac576c7fff97412aa55d8c4446a62",
|
||||
"state": "running",
|
||||
"health": null,
|
||||
"memory_limit_bytes": 1610612736,
|
||||
"nano_cpus": 2000000000,
|
||||
"oom_killed": false,
|
||||
"published_ports": {},
|
||||
"volumes": []
|
||||
}
|
||||
],
|
||||
"grafana_version": "12.0.2",
|
||||
"datasources": [
|
||||
{
|
||||
"uid": "loki",
|
||||
"name": "Loki",
|
||||
"type": "loki",
|
||||
"url": "http://loki:3100",
|
||||
"jsonData": {
|
||||
"derivedFields": [
|
||||
{
|
||||
"datasourceUid": "tempo",
|
||||
"matcherRegex": "\"trace_id\": \"([a-f0-9]{32})\"",
|
||||
"name": "TraceID",
|
||||
"url": "${__value.raw}"
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"uid": "prometheus",
|
||||
"name": "Prometheus",
|
||||
"type": "prometheus",
|
||||
"url": "http://prometheus:9090",
|
||||
"jsonData": {
|
||||
"timeInterval": "5s"
|
||||
}
|
||||
},
|
||||
{
|
||||
"uid": "tempo",
|
||||
"name": "Tempo",
|
||||
"type": "tempo",
|
||||
"url": "http://tempo:3200",
|
||||
"jsonData": {
|
||||
"nodeGraph": {
|
||||
"enabled": true
|
||||
},
|
||||
"tracesToLogs": {
|
||||
"datasourceUid": "loki",
|
||||
"filterByTraceID": true,
|
||||
"spanEndTimeShift": "1m",
|
||||
"spanStartTimeShift": "-1m",
|
||||
"tags": [
|
||||
{
|
||||
"key": "service.name",
|
||||
"value": "service_name"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,666 @@
|
||||
{
|
||||
"passed": true,
|
||||
"metrics": [
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_log_queue_capacity",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "02738dc58bbb:1",
|
||||
"exported_job": "messenger-publisher",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_log_processor/0",
|
||||
"otel_component_type": "batching_log_processor",
|
||||
"service_instance_id": "02738dc58bbb:1",
|
||||
"service_name": "messenger-publisher",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"256"
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_log_queue_capacity",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "168b98028f54:8",
|
||||
"exported_job": "messenger-worker",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_log_processor/0",
|
||||
"otel_component_type": "batching_log_processor",
|
||||
"service_instance_id": "168b98028f54:8",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"256"
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_log_queue_capacity",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "168b98028f54:9",
|
||||
"exported_job": "messenger-worker",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_log_processor/0",
|
||||
"otel_component_type": "batching_log_processor",
|
||||
"service_instance_id": "168b98028f54:9",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"256"
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_log_queue_capacity",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "2a50b728ce10:1",
|
||||
"exported_job": "messenger-api",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_log_processor/0",
|
||||
"otel_component_type": "batching_log_processor",
|
||||
"service_instance_id": "2a50b728ce10:1",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"256"
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_log_queue_capacity",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "33daefd247cf:1",
|
||||
"exported_job": "messenger-api",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_log_processor/0",
|
||||
"otel_component_type": "batching_log_processor",
|
||||
"service_instance_id": "33daefd247cf:1",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"256"
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_log_queue_size",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "02738dc58bbb:1",
|
||||
"exported_job": "messenger-publisher",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_log_processor/0",
|
||||
"otel_component_type": "batching_log_processor",
|
||||
"service_instance_id": "02738dc58bbb:1",
|
||||
"service_name": "messenger-publisher",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"0"
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_log_queue_size",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "168b98028f54:8",
|
||||
"exported_job": "messenger-worker",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_log_processor/0",
|
||||
"otel_component_type": "batching_log_processor",
|
||||
"service_instance_id": "168b98028f54:8",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"0"
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_log_queue_size",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "168b98028f54:9",
|
||||
"exported_job": "messenger-worker",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_log_processor/0",
|
||||
"otel_component_type": "batching_log_processor",
|
||||
"service_instance_id": "168b98028f54:9",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"0"
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_log_queue_size",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "2a50b728ce10:1",
|
||||
"exported_job": "messenger-api",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_log_processor/0",
|
||||
"otel_component_type": "batching_log_processor",
|
||||
"service_instance_id": "2a50b728ce10:1",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"0"
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_log_queue_size",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "33daefd247cf:1",
|
||||
"exported_job": "messenger-api",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_log_processor/0",
|
||||
"otel_component_type": "batching_log_processor",
|
||||
"service_instance_id": "33daefd247cf:1",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"0"
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_span_queue_capacity",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "02738dc58bbb:1",
|
||||
"exported_job": "messenger-publisher",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_span_processor/0",
|
||||
"otel_component_type": "batching_span_processor",
|
||||
"service_instance_id": "02738dc58bbb:1",
|
||||
"service_name": "messenger-publisher",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"256"
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_span_queue_capacity",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "168b98028f54:8",
|
||||
"exported_job": "messenger-worker",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_span_processor/0",
|
||||
"otel_component_type": "batching_span_processor",
|
||||
"service_instance_id": "168b98028f54:8",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"256"
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_span_queue_capacity",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "168b98028f54:9",
|
||||
"exported_job": "messenger-worker",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_span_processor/0",
|
||||
"otel_component_type": "batching_span_processor",
|
||||
"service_instance_id": "168b98028f54:9",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"256"
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_span_queue_capacity",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "2a50b728ce10:1",
|
||||
"exported_job": "messenger-api",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_span_processor/0",
|
||||
"otel_component_type": "batching_span_processor",
|
||||
"service_instance_id": "2a50b728ce10:1",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"256"
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_span_queue_capacity",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "33daefd247cf:1",
|
||||
"exported_job": "messenger-api",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_span_processor/0",
|
||||
"otel_component_type": "batching_span_processor",
|
||||
"service_instance_id": "33daefd247cf:1",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"256"
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_span_queue_size",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "02738dc58bbb:1",
|
||||
"exported_job": "messenger-publisher",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_span_processor/0",
|
||||
"otel_component_type": "batching_span_processor",
|
||||
"service_instance_id": "02738dc58bbb:1",
|
||||
"service_name": "messenger-publisher",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"0"
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_span_queue_size",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "168b98028f54:8",
|
||||
"exported_job": "messenger-worker",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_span_processor/0",
|
||||
"otel_component_type": "batching_span_processor",
|
||||
"service_instance_id": "168b98028f54:8",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"0"
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_span_queue_size",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "168b98028f54:9",
|
||||
"exported_job": "messenger-worker",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_span_processor/0",
|
||||
"otel_component_type": "batching_span_processor",
|
||||
"service_instance_id": "168b98028f54:9",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"0"
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_span_queue_size",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "2a50b728ce10:1",
|
||||
"exported_job": "messenger-api",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_span_processor/0",
|
||||
"otel_component_type": "batching_span_processor",
|
||||
"service_instance_id": "2a50b728ce10:1",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"0"
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_span_queue_size",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "33daefd247cf:1",
|
||||
"exported_job": "messenger-api",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_span_processor/0",
|
||||
"otel_component_type": "batching_span_processor",
|
||||
"service_instance_id": "33daefd247cf:1",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"0"
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_log_processed_total",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "02738dc58bbb:1",
|
||||
"exported_job": "messenger-publisher",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_log_processor/0",
|
||||
"otel_component_type": "batching_log_processor",
|
||||
"service_instance_id": "02738dc58bbb:1",
|
||||
"service_name": "messenger-publisher",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"318"
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_log_processed_total",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "168b98028f54:9",
|
||||
"exported_job": "messenger-worker",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_log_processor/0",
|
||||
"otel_component_type": "batching_log_processor",
|
||||
"service_instance_id": "168b98028f54:9",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"66"
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_log_processed_total",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "2a50b728ce10:1",
|
||||
"exported_job": "messenger-api",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_log_processor/0",
|
||||
"otel_component_type": "batching_log_processor",
|
||||
"service_instance_id": "2a50b728ce10:1",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"1296"
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_log_processed_total",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "33daefd247cf:1",
|
||||
"exported_job": "messenger-api",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_log_processor/0",
|
||||
"otel_component_type": "batching_log_processor",
|
||||
"service_instance_id": "33daefd247cf:1",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"1289"
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_span_processed_total",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "02738dc58bbb:1",
|
||||
"exported_job": "messenger-publisher",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_span_processor/0",
|
||||
"otel_component_type": "batching_span_processor",
|
||||
"service_instance_id": "02738dc58bbb:1",
|
||||
"service_name": "messenger-publisher",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"318"
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_span_processed_total",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "168b98028f54:9",
|
||||
"exported_job": "messenger-worker",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_span_processor/0",
|
||||
"otel_component_type": "batching_span_processor",
|
||||
"service_instance_id": "168b98028f54:9",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"1320"
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_span_processed_total",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "2a50b728ce10:1",
|
||||
"exported_job": "messenger-api",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_span_processor/0",
|
||||
"otel_component_type": "batching_span_processor",
|
||||
"service_instance_id": "2a50b728ce10:1",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"13025"
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_span_processed_total",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "33daefd247cf:1",
|
||||
"exported_job": "messenger-api",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_span_processor/0",
|
||||
"otel_component_type": "batching_span_processor",
|
||||
"service_instance_id": "33daefd247cf:1",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"13012"
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_log_processed_total",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "168b98028f54:8",
|
||||
"exported_job": "messenger-worker",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_log_processor/0",
|
||||
"otel_component_type": "batching_log_processor",
|
||||
"service_instance_id": "168b98028f54:8",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"54"
|
||||
]
|
||||
},
|
||||
{
|
||||
"metric": {
|
||||
"__name__": "otel_sdk_processor_span_processed_total",
|
||||
"deployment_environment_name": "lab",
|
||||
"exported_instance": "168b98028f54:8",
|
||||
"exported_job": "messenger-worker",
|
||||
"instance": "otel-collector:9464",
|
||||
"job": "messenger",
|
||||
"otel_component_name": "batching_span_processor/0",
|
||||
"otel_component_type": "batching_span_processor",
|
||||
"service_instance_id": "168b98028f54:8",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "6.0.0",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1"
|
||||
},
|
||||
"value": [
|
||||
1795102653.838,
|
||||
"1080"
|
||||
]
|
||||
}
|
||||
],
|
||||
"per_process_queue_capacity": 256
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,246 @@
|
||||
PHASE: baseline 120 seconds
|
||||
PHASE: short_db_spike 4 seconds
|
||||
PHASE: after_short_spike 60 seconds
|
||||
PHASE: sustained_db_lock 90 seconds
|
||||
PHASE: after_db_recovery 100 seconds
|
||||
PHASE: worker_stopped 120 seconds
|
||||
PHASE: after_worker_recovery 120 seconds
|
||||
{
|
||||
"utc": "2026-11-19T15:26:14.987289+00:00",
|
||||
"hypothesis": "Multiwindow burn alerts ignore a 4s spike, detect sustained DB/worker degradation and resolve after recovery.",
|
||||
"profile": {
|
||||
"message_arrival_per_second": 2,
|
||||
"image_arrival_per_second": 0.5,
|
||||
"max_inflight": 16,
|
||||
"sampling": 1.0,
|
||||
"retries": 0,
|
||||
"slo_window_seconds": 3600,
|
||||
"api_slo": 0.99,
|
||||
"ready_slo": 0.95,
|
||||
"ready_deadline_seconds": 30
|
||||
},
|
||||
"docker": "29.0.1 aarch64 CPUs=10 RAM=8217448448",
|
||||
"db_diagnosis": {
|
||||
"trace_id": "f124d93f84dd25c5af978e78e2028646",
|
||||
"request_id": "3c5e5b14-5825-41b8-a227-93cba420a532",
|
||||
"slow_error_spans": [
|
||||
{
|
||||
"name": "postgresql.SELECT",
|
||||
"status": {
|
||||
"code": "STATUS_CODE_ERROR"
|
||||
},
|
||||
"duration_ms": 2000.115335,
|
||||
"attributes": [
|
||||
{
|
||||
"key": "db.system.name",
|
||||
"value": {
|
||||
"stringValue": "postgresql"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "db.operation.name",
|
||||
"value": {
|
||||
"stringValue": "SELECT"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "error.type",
|
||||
"value": {
|
||||
"stringValue": "_WaitTimeout"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "postgresql.transaction",
|
||||
"status": {
|
||||
"code": "STATUS_CODE_ERROR"
|
||||
},
|
||||
"duration_ms": 2001.169168,
|
||||
"attributes": [
|
||||
{
|
||||
"key": "error.type",
|
||||
"value": {
|
||||
"stringValue": "_WaitTimeout"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "POST /api/v1/chats/{chat_id}/messages",
|
||||
"status": {
|
||||
"code": "STATUS_CODE_ERROR"
|
||||
},
|
||||
"duration_ms": 2003.784959,
|
||||
"attributes": [
|
||||
{
|
||||
"key": "http.request.method",
|
||||
"value": {
|
||||
"stringValue": "POST"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "http.route",
|
||||
"value": {
|
||||
"stringValue": "/api/v1/chats/{chat_id}/messages"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "http.response.status_code",
|
||||
"value": {
|
||||
"intValue": "503"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"short_spike_observed_error_ratio_peak": 0.06373490138007629,
|
||||
"passed": true,
|
||||
"false_positives": {
|
||||
"baseline": 0,
|
||||
"burn_alerts_after_short_spike": 0,
|
||||
"single_window_alerts_after_short_spike": [
|
||||
"APIHighErrorRate"
|
||||
]
|
||||
},
|
||||
"start": 1795101974.987257,
|
||||
"end": 1795102594.377363,
|
||||
"duration_seconds": 619.3901059627533,
|
||||
"phases": [
|
||||
{
|
||||
"name": "baseline",
|
||||
"start": 1795101975.364327,
|
||||
"planned_seconds": 120,
|
||||
"active_at": 1795101975.364415,
|
||||
"end": 1795102095.3712242
|
||||
},
|
||||
{
|
||||
"name": "short_db_spike",
|
||||
"start": 1795102095.371294,
|
||||
"planned_seconds": 4,
|
||||
"active_at": 1795102095.527463,
|
||||
"end": 1795102099.381341
|
||||
},
|
||||
{
|
||||
"name": "after_short_spike",
|
||||
"start": 1795102099.381383,
|
||||
"planned_seconds": 60,
|
||||
"active_at": 1795102099.381435,
|
||||
"end": 1795102159.386289
|
||||
},
|
||||
{
|
||||
"name": "sustained_db_lock",
|
||||
"start": 1795102159.386304,
|
||||
"planned_seconds": 90,
|
||||
"active_at": 1795102159.552147,
|
||||
"end": 1795102249.3866298
|
||||
},
|
||||
{
|
||||
"name": "after_db_recovery",
|
||||
"start": 1795102252.4302201,
|
||||
"planned_seconds": 100,
|
||||
"active_at": 1795102252.430262,
|
||||
"end": 1795102352.43217
|
||||
},
|
||||
{
|
||||
"name": "worker_stopped",
|
||||
"start": 1795102352.4322171,
|
||||
"planned_seconds": 120,
|
||||
"active_at": 1795102352.766314,
|
||||
"end": 1795102472.4357738
|
||||
},
|
||||
{
|
||||
"name": "after_worker_recovery",
|
||||
"start": 1795102472.435835,
|
||||
"planned_seconds": 120,
|
||||
"active_at": 1795102474.322918,
|
||||
"end": 1795102592.4362948
|
||||
}
|
||||
],
|
||||
"skipped_arrivals": 0,
|
||||
"phase_summary": {
|
||||
"baseline": {
|
||||
"requests": 301,
|
||||
"5xx": 0,
|
||||
"successful_rps": 2.5081891707489476,
|
||||
"p50_ms": 16.607,
|
||||
"p95_ms": 31.266,
|
||||
"p99_ms": 35.193
|
||||
},
|
||||
"short_db_spike": {
|
||||
"requests": 10,
|
||||
"5xx": 4,
|
||||
"successful_rps": 1.4962418299487805,
|
||||
"p50_ms": 1670.714,
|
||||
"p95_ms": 2017.188,
|
||||
"p99_ms": 2017.188
|
||||
},
|
||||
"after_short_spike": {
|
||||
"requests": 150,
|
||||
"5xx": 0,
|
||||
"successful_rps": 2.499795602583431,
|
||||
"p50_ms": 16.475,
|
||||
"p95_ms": 30.739,
|
||||
"p99_ms": 32.763
|
||||
},
|
||||
"sustained_db_lock": {
|
||||
"requests": 225,
|
||||
"5xx": 176,
|
||||
"successful_rps": 0.5444424728476727,
|
||||
"p50_ms": 2011.805,
|
||||
"p95_ms": 2022.545,
|
||||
"p99_ms": 2025.758
|
||||
},
|
||||
"after_db_recovery": {
|
||||
"requests": 250,
|
||||
"5xx": 0,
|
||||
"successful_rps": 2.4999512562719017,
|
||||
"p50_ms": 16.283,
|
||||
"p95_ms": 30.965,
|
||||
"p99_ms": 33.443
|
||||
},
|
||||
"worker_stopped": {
|
||||
"requests": 300,
|
||||
"5xx": 0,
|
||||
"successful_rps": 2.4999259036886072,
|
||||
"p50_ms": 16.221,
|
||||
"p95_ms": 31.277,
|
||||
"p99_ms": 35.499
|
||||
},
|
||||
"after_worker_recovery": {
|
||||
"requests": 300,
|
||||
"5xx": 0,
|
||||
"successful_rps": 2.4999904185900736,
|
||||
"p50_ms": 13.134,
|
||||
"p95_ms": 30.749,
|
||||
"p99_ms": 35.272
|
||||
}
|
||||
},
|
||||
"alert_timings": {
|
||||
"APIAvailabilitySlowBurn": {
|
||||
"detection_seconds": 60.28,
|
||||
"resolution_after_restore_seconds": 65.27
|
||||
},
|
||||
"APIHighErrorRate": {
|
||||
"detection_seconds": 25.11,
|
||||
"resolution_after_restore_seconds": 35.15
|
||||
},
|
||||
"APIAvailabilityFastBurn": {
|
||||
"detection_seconds": 40.18,
|
||||
"resolution_after_restore_seconds": 35.15
|
||||
},
|
||||
"AttachmentQueueLag": {
|
||||
"detection_seconds": 45.21,
|
||||
"resolution_after_restore_seconds": 30.12
|
||||
},
|
||||
"AttachmentReadyFastBurn": {
|
||||
"detection_seconds": 95.5,
|
||||
"resolution_after_restore_seconds": 45.18
|
||||
},
|
||||
"AttachmentReadySlowBurn": {
|
||||
"detection_seconds": 131.6,
|
||||
"resolution_after_restore_seconds": 75.32
|
||||
}
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,130 @@
|
||||
{
|
||||
"base_commit": "470d9d6f1e4fe4f594e37c2ba59550ee5ca54fc1",
|
||||
"working_tree": true,
|
||||
"files": {
|
||||
".dockerignore": "a092ac618bd90f555f48c85fc58cad3ac9f51f4de4379b242f51e14e3d468b38",
|
||||
".env.example": "690df84f2c2f7200bd05a6ae325277f0771df9c57e581003746e2c20e8119c05",
|
||||
".github/pull_request_template.lab5.md": "e3e9afa285870e2133b3dbc30fec881bae7418605789bdb1eaf497a2eb62914a",
|
||||
".github/pull_request_template.md": "fd1a90691c2197f05c4aa0dd43aea9eda4be1e6abb0e1bb1430991228eda8e64",
|
||||
".github/workflows/application.yml": "e9b1309023f630f7773c7bb7c525e68662245c1657e5b773a9393ec50d0d88a5",
|
||||
".github/workflows/contracts.yml": "a45d7932ef14130aff8020bce79f043cccaf9692d697088a754d309a03f7d3e8",
|
||||
".gitignore": "c26ab45b3a778a5376f6db4f624a51c5fdbf1a404f9a26df8613449277d95438",
|
||||
"CONTRIBUTING.md": "2c01dd698cc9653dece49a94c9fd3f58d36e84764aaa643b46e26d7e599dc12e",
|
||||
"COURSE.md": "c34d53cd082d194aa4b58f504c4e0348d25b6afec3d6a9c0963dd01c9c81e971",
|
||||
"Dockerfile": "e2a683428301e5bf584fb0c067b447c22e2a088ff13ce147b7f7b8b2261cd8ea",
|
||||
"Dockerfile.example": "fd7ac9491c0c3c663b083c41a762fb38bc2d212fa4a2ed3e6a0702af72108709",
|
||||
"MAINTAINERS.md": "f2e7e9d59e06f260ccf3acc9baf79c4a3b66e1bc370ff6e21cbefb338210630a",
|
||||
"Makefile": "c05bd07058ae0949dd4905afdae7674bc7e3e63863a6cdf8b72661140d445826",
|
||||
"README.md": "583fa7625d728ead8a6ee2cd2de9f117ed3f4a1d9696c0bb9fb602fe549a31db",
|
||||
"REPORT.lab5.md": "ce77ed0e6cc43a6283a9a2964a2508375496b274996222ae1ab5045b508f1d5f",
|
||||
"REPORT.md": "4107c5e486c13d8e48eb0f2d37bbdabe267d857af56a17fe9ac879b5b652a6ea",
|
||||
"compose.example.yaml": "9bfd9eacae286dd9bffe29d6fe349534552560f966c3c3e6e6540e92d7357afb",
|
||||
"compose.yaml": "aed7a12a1ceb68bc729a102b7d76ab9354d426a136245921c37556f20a06a79c",
|
||||
"contracts/README.md": "72df27a559f88ffa3fd8c3c79d3c32f6db2c5986a0602677c51994c434a3fc4f",
|
||||
"contracts/attachment-states.md": "cf0519e137410e4b34ab632aadb8ad8a381d4b73033e27b354c58eeacbdf86d6",
|
||||
"contracts/examples.json": "381cc18c03789e935b391d007fb5843964be9cfd9f7530cb6b1996ed4a1c4c37",
|
||||
"contracts/infrastructure.json": "45264a3db653bb79de8d8fcb8d70733924f3cbfcba0fa1253cecbc6828bce217",
|
||||
"contracts/jobs.schema.json": "9ccf9b11466c0c3a26a941520ac5ff543241fc4935e3a6eebb01ef303779b10d",
|
||||
"contracts/observability.md": "882c120c0e153a06c9cc824a802e317c58eb6590bc3d3a576f2e0a96f8597b62",
|
||||
"contracts/openapi.json": "2c4074e8fac6bed6337fbbc0aaed0ae2321a76ef30ea68afad80a1f39699535a",
|
||||
"docs/runbook.md": "5b9b9529ecce1f0c510e9463783db3d5d0dd14f709d2a2d571c0094714067b49",
|
||||
"docs/schema.md": "205d9a1524343b602f87b14f64ebc207372c747c1e38a3222d1c733c002093ec",
|
||||
"examples/requests.http": "8c086fd540092b3e31a945a416632bcda6c4bea39253f86b076f791b362cf5e4",
|
||||
"infra/README.md": "0e804fa51debba4c38d3cc84abf4704897665e0335b272060363f03aadd6354f",
|
||||
"infra/grafana/dashboards/messenger.json": "ed102001c4002a3e63617216571979727c05b10866a6eb58c4ac795110402510",
|
||||
"infra/grafana/provisioning/dashboards/provider.yaml": "d7aa7568978a55168509eb23f2522d081627cf3c1e141a356b2c6d8e3ff86f62",
|
||||
"infra/grafana/provisioning/datasources/sources.yaml": "b89e3fa70a4432fac952a2bb7a55a0dbb6cae92787090a8d1b30f3468db56d82",
|
||||
"infra/haproxy.cfg": "c80c30717494c8185eda0aa0437734e3b011bff3d213ca2fa2b2e0af83c88600",
|
||||
"infra/otel/README.md": "0d6c93589f7b96b7045c1b82a261362697d25f14c9e1a4946b8ea8a977c1e842",
|
||||
"infra/otel/alerts.yaml": "e7a1223c64b7040cbd76b754e19ed17a6d321aaf6da1996dafcd608b3ca669af",
|
||||
"infra/otel/collector.yaml": "92c759d393d9669815131acca5c8df7ed1c71dc7c7818a7b445cefcb8c040379",
|
||||
"infra/otel/loki.yaml": "a1b1e6af51a0f34a7b8356d6eab523b855731b292e129cbe6a480fe0551c2eeb",
|
||||
"infra/otel/prometheus.yaml": "df8340a67cb44830d1d1a6c19b87acb22d00b8b85c2993ab37a28599b43829ff",
|
||||
"infra/otel/tempo.yaml": "b7d437347fe6b344de2e6499426931330c6d41ce1ba2f39492e03069e2908a43",
|
||||
"lab.json": "807dbb02b180ddb13dc898c117e09495e15a462e83f4b55133ba174f089594ce",
|
||||
"migrations/001_initial.sql": "a319949c962c54c5ba537a284ff38f77f2ba04c5bbc748873889d49c2bf48a46",
|
||||
"migrations/002_message_idempotency.sql": "8e484326b7861b8acf4b847d7126639bf6216e635012e1e969006e86caf902ed",
|
||||
"migrations/003_attachments.sql": "9deda173c984675092869022715c32648aa7b91cf8f8641bb5daf68c6f871aba",
|
||||
"migrations/004_attachment_jobs.sql": "d88a95701d0b7359a731fa81aff22b5b3e1948fb29aacdf1e7e27b9136d27427",
|
||||
"migrations/005_observability.sql": "fb23df863aedb6686c185d59a2499134c09c2e6ad9859c2b7e1648027aa815ea",
|
||||
"requirements-dev.txt": "4cefdb67007caf026d1613be7c519c01b4c8343db7c4f4b618677e01cf113525",
|
||||
"requirements.txt": "861b5ad246e5cfa706a7cdcf7d3a4a9e9feca3261d89b387e1bb553244339ed6",
|
||||
"research/README.lab2.md": "59606b7a6325ac75749963f45b6b367c93d5a3a97093cb8850587ee9ae92b5da",
|
||||
"research/README.md": "cd14eeb3cfdb81a70269f4503f4e91383c99e482757bb462f9120f684c6dcca7",
|
||||
"research/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
|
||||
"research/benchmark.py": "d66405d705f6941735efb7f297df2539ec73a31f3efa989723e4ed02f72af721",
|
||||
"research/observations.py": "238b8da50a1c48656daf2fe424e60d0de46ae5a4743c17ac17dda064b555a7f6",
|
||||
"research/run.py": "843a33bb642ebe0648ab8ccf551adfb2e05d9fe1127d8cbd83bf762f872b243b",
|
||||
"research/slo.py": "7bf6848d94e483f65598681d39b908635d205893c89874fcbe14c229de8ca233",
|
||||
"scripts/build.sh": "8f67493f0563b0dce0e0f327655c4838a21c04d9d887a03820798f4f56518c2e",
|
||||
"scripts/check-docker.sh": "43e6d221f386af27eec2b49a18ed6c48c284331c7a06dcb5ee03e189e0bca3ba",
|
||||
"scripts/check-lab6.sh": "eb8b12eb0b770a5503ffeb349cd3b79b2777e6c7b5bb6a41186d14ef3b1f2d7a",
|
||||
"scripts/check.sh": "c0de27e3d079b503f6cf2a1606bc6d156b5ac35c61455582bab3d5ee809a17a6",
|
||||
"scripts/check_contract.py": "7c4402c2a6ab80d3b4676e1866d6ecc87b6daea14c080bcb9be9a7e8b959d671",
|
||||
"scripts/demo.sh": "8363935669d34bf56f1e170261d590c75dce9470308da90f6e700034a9bd01f3",
|
||||
"scripts/init_local.py": "279f03cc5a4bc1735396594ec238c3946bc97a7343baf93f94e6abbe12f43e1f",
|
||||
"scripts/migrate.sh": "8700e123e7f154b2b80926c83ffe384cd39572bb3e063d807dfa3c144c135913",
|
||||
"scripts/replicas.py": "fa2700c2a486c5919cfa1095068307ed445a6268d8dfcf5b75ce15a1629509eb",
|
||||
"scripts/research-lab2.sh": "081ea2b07c338b03e08eaba8316c1de4c9b49e0d83bdded7b4de956dc77352dc",
|
||||
"scripts/research.sh": "84f34fcb0cf8338d8ff8749524db68aab0e2c7b328e2f4b45ad2cf980d179a66",
|
||||
"scripts/restart.example.sh": "0328246ba75defbced992104e986fde973f63e3830d83251eced5953263e944b",
|
||||
"scripts/restart.sh": "6fd3eb1e0e65df0f600d88cccce52068d60ea1f58a8a125b87622ec62b82a5de",
|
||||
"scripts/rolling-update.sh": "dc09335e3e8989b42bbac75a9aec9e722ab234541361830bd924f544b45f489d",
|
||||
"scripts/run-local.sh": "61d5021aaca500ae8ac4249d6d63a839edbd811df006a56ea27a916f9fcbeb71",
|
||||
"scripts/start-worker.sh": "b5e8df907b645ab2d81799933f9e94cf160f4922ebdba21b9d1e52ee59c8d93f",
|
||||
"scripts/start.sh": "5314c8d051541b330510d6f59d6aefebd5d3ee020464e14dd9f46c09af2f6c61",
|
||||
"scripts/stop-one.example.sh": "1bcf34394520b89d95b7605ec56675be799eea920e83cfbba306d5dc0a6d9811",
|
||||
"scripts/stop-one.sh": "fd6e58a604aaed3833c220f4e828ce52bd2c61a905a232a0c50da74b6a3b2edb",
|
||||
"scripts/stop-worker.sh": "32ef73cfc0ef91e5acd437934a46e4567d83ab926247bf624be5aeb4ce3d3993",
|
||||
"scripts/stop.sh": "db7ef5f1723566859b211f02f753c65657e6820f8135a4bc07ce6c544284df0d",
|
||||
"scripts/validate_openapi.py": "4e981acbfc10dac58817bf922d5896e996798551fedc1b2385130e1cddb13d54",
|
||||
"scripts/wait-observability.py": "af89dbdf9c0e4d9ae11bb7b80143fd8af08b254578159e695e660c55345693c7",
|
||||
"scripts/wait-ready.sh": "c24e7a2297bbcbeb69451d17875a669b61a0b6b86e9a03fb33dabb2daea0dce7",
|
||||
"src/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
|
||||
"src/__main__.py": "91819b838eef82165a9ef67aaf45cbb7a54d131f170076e1dde672d9a77e121b",
|
||||
"src/app.py": "796aee198f9aa934bac388aa05e1cc2b067643db5f1bd2a843cc0618e1b9b09d",
|
||||
"src/attachments.py": "9d4c4d3f76e489dfe3b96d89f892f7b9e9222e4ffec667d419df7dcc5850e7d5",
|
||||
"src/auth.py": "75ad98d17940efb2e4f6dfc584b371a4edc6c397af4fb46f3666ee9e2d043866",
|
||||
"src/bootstrap_s3.py": "ad8330225e4697873d2961f874cbdfbb69b66301c6d0ccf28d6999cc5b9c8191",
|
||||
"src/broker.py": "13c5c025338ccd90eb74e48d8892aba4547adc33b71807fad021a99eb37d795e",
|
||||
"src/cleanup.py": "839162ed4bf01db7734025b3b14e21a30449e2b7e5b6c3d60b39c9bd723c98b5",
|
||||
"src/config.py": "ad9d1f94e796b82a7a7b9b93a98d94ca588decf02a65e588b34cab362ef0a6a0",
|
||||
"src/db.py": "803446c82ebbfa11b33ef41c100b4b06204f7ba3fffcec8092e0e91f6b992882",
|
||||
"src/errors.py": "f020bf3e5c9619c6ef47e1c8910a19e3331140eb9dd2bba74981e89e855e1149",
|
||||
"src/faults.py": "b1ca87cf1c75a050c8572c9a3b05b569d2e459c7c5d4ce23efefbeffd2774e6b",
|
||||
"src/healthcheck.py": "bb8c06c1836f601f77755de3aca4b1092294784d3c33d60bd27b41778cd130be",
|
||||
"src/image_pipeline.py": "f1800c26913612739f919b6880e5b673e986c198cb4ea9509e105240f40f1fa8",
|
||||
"src/job_admin.py": "51781bdeacf644c36118a9972eab125893d6bf65a3eaa1143d8eb3f3d24631e1",
|
||||
"src/jobs.py": "9bbed284d1f61355d65e84271191242989b10f97bb803792907444908ea15165",
|
||||
"src/migrate.py": "66b66b9d244925957ee8559dc734e872172da403c688c2bf20ae8312d7c3b260",
|
||||
"src/models.py": "6f9adcc886e05c7c28c540393ca4a3d22083693f30e33a51a074306f511acc15",
|
||||
"src/object_store.py": "177a9cd69e0a4f494a1cdafc92544cf4a0f150842a0b7d5e826b72c13b49532a",
|
||||
"src/pagination.py": "ca07be71f21ba56873d77c9422530a7a5050679b57c055b88fcf547efc112558",
|
||||
"src/passwords.py": "9cb66e8e841faee89868af041a793b0b511df49b85b8d5b1a27ca296165bf1de",
|
||||
"src/publisher.py": "2f99db91efdf3f7e06383954fe54c30745af402efd07052bd372ed8e1ab78211",
|
||||
"src/storage.py": "58c49912188944d739d8ebc9a0d834958b605b56f9e6296d59e9b99c2735961f",
|
||||
"src/telemetry.py": "3151acbe5eed23c92cc2bf2a644bc0035a12be34fdec99737042487c1ef87af5",
|
||||
"src/worker.py": "1215386583876119eb817934461b56edc5416369282af017b33e4a6a9d79da0a",
|
||||
"student-tests/attachment_checks.py": "219648a7ea58d81ed2c709c1961d382558654c230db369f320d4e39f6b5ba4fb",
|
||||
"student-tests/docker_lifecycle.py": "06a49931d151419eea5f78668a737bc17a5d31f0559d78e5429be882fd10046c",
|
||||
"student-tests/helpers.py": "3723976953cb1212e6773e8fc9771148232e4d95ff665f049e5f47d23a97d7ba",
|
||||
"student-tests/lab5_pipeline.py": "c23d50767759df0ea6642bff7a56b2f3cd0eb1010dd77b42cd941e15ca3650d3",
|
||||
"student-tests/lab6_observability.py": "1a16f9e7315f980089dfb82c45e009ad3979c9754dd305b2628fe16c36e68b6c",
|
||||
"student-tests/postgres_pause.py": "e81dee40fe1356940b7b8a65e2c9cf5cc1510adc6bf831dd235041044f654548",
|
||||
"student-tests/promtool-slo.yaml": "f265001af6026fadc0677132a510f8dccf8ff1b5e1590ea156efec571040779e",
|
||||
"student-tests/test_api.py": "5a60014b01ebc40a14c037ded532995ebb680c0aa5923a910460a8da181fc32f",
|
||||
"student-tests/test_attachments.py": "8660c45842e9987b814f7075f3ae23c446c156cb7b755dd2734fed0b5b11af56",
|
||||
"student-tests/test_concurrency.py": "ae48809fb1acef260abb04ba3598475a9456908d3193c90445458837547a9d84",
|
||||
"student-tests/test_idempotency.py": "1f34c297029b182a0f66b99f007bc1dfebf7e6e7ca6fb5d695d0a8be37135d32",
|
||||
"student-tests/test_sessions.py": "4d8fe299727266bf1ca1d7fd25b8ee3dd1780b10cce23e0af48bd47da95a852e",
|
||||
"student-tests/verify_restart.py": "5c7d5197ede3d92351e07cc6768a7acc3ce204120e21f744ca48a98594def33f",
|
||||
"tests/README.md": "e0bfa7e196797f6a5bf6e44d8ad2ffeb82651927ff5d8484eb1c2dda1ec7dec5",
|
||||
"tests/client.py": "8d50976fc7c7bbb81e88a152f613c61ab25eaf53dba33c6a9dfab19f868d546d",
|
||||
"tests/fixtures/README.md": "c001783760b4f87a7a8441a7ba55f5d080ff48e1397d3581d72a88d7691f96c2",
|
||||
"tests/fixtures/sample.png": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
|
||||
"tests/image_probe.py": "a9a4f89181c8e0fda6ce42e1a1af32fb42d8e3ae519e314207127c9b9b448b13",
|
||||
"tests/load.py": "6d3511ea719c7a7e841083e73d435c86dcbfd9f4e5c2a07d62eb9614b01d9933",
|
||||
"tests/resilience.py": "1f96663a174b3978d24eff66f2741f5f8c9b788d3242261d37a972720c52788b",
|
||||
"tests/smoke.py": "52ac38eb343ff7d43e7543588e1da58ca06c9fce74b2b0d0bf0e4982bc23c1db"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"passed": true,
|
||||
"trace_id": "93e1177f9568f5bbfc0b6967a1324a84",
|
||||
"stored_span_count": 59,
|
||||
"after_full_restart": true,
|
||||
"prometheus_series": 4,
|
||||
"datasources": [
|
||||
"loki",
|
||||
"prometheus",
|
||||
"tempo"
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,248 @@
|
||||
{
|
||||
"status": "success",
|
||||
"data": {
|
||||
"resultType": "streams",
|
||||
"result": [
|
||||
{
|
||||
"stream": {
|
||||
"attachment_id": "d0c6d064-5a57-47eb-a5b1-c8d2f7a9870f",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "INFO",
|
||||
"event": "attachment.accepted",
|
||||
"flags": "3",
|
||||
"job_id": "03320707-a810-4a0e-a72d-b10ef7734c32",
|
||||
"scope_name": "messenger",
|
||||
"scope_version": "6.0.0",
|
||||
"service_instance_id": "33daefd247cf:1",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "6.0.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "65e8265a35af1c1d",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1",
|
||||
"trace_id": "2b1592ca18aac08bbd14b3f1825f0816"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1795102383403137083",
|
||||
"{\"timestamp\": \"2026-11-19T15:33:03.403092+00:00\", \"severity\": \"INFO\", \"service.name\": \"messenger-api\", \"event\": \"attachment.accepted\", \"trace_id\": \"2b1592ca18aac08bbd14b3f1825f0816\", \"span_id\": \"65e8265a35af1c1d\", \"job_id\": \"03320707-a810-4a0e-a72d-b10ef7734c32\", \"attachment_id\": \"d0c6d064-5a57-47eb-a5b1-c8d2f7a9870f\"}"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"attempt": "1",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "INFO",
|
||||
"event": "job.finished",
|
||||
"flags": "3",
|
||||
"job_id": "03320707-a810-4a0e-a72d-b10ef7734c32",
|
||||
"outcome": "ready",
|
||||
"scope_name": "messenger",
|
||||
"scope_version": "6.0.0",
|
||||
"service_instance_id": "168b98028f54:8",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "6.0.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "ba2d3f8e53a54fd2",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1",
|
||||
"trace_id": "2b1592ca18aac08bbd14b3f1825f0816"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1795102477984404627",
|
||||
"{\"timestamp\": \"2026-11-19T15:34:37.984358+00:00\", \"severity\": \"INFO\", \"service.name\": \"messenger-worker\", \"event\": \"job.finished\", \"trace_id\": \"2b1592ca18aac08bbd14b3f1825f0816\", \"span_id\": \"ba2d3f8e53a54fd2\", \"job_id\": \"03320707-a810-4a0e-a72d-b10ef7734c32\", \"attempt\": 1, \"outcome\": \"ready\"}"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "INFO",
|
||||
"duration_seconds": "0.024863",
|
||||
"event": "http.request",
|
||||
"flags": "3",
|
||||
"method": "POST",
|
||||
"request_id": "bad9f74b-1237-475e-b7a9-13e04b9457fb",
|
||||
"route": "/api/v1/chats/{chat_id}/attachments",
|
||||
"scope_name": "messenger",
|
||||
"scope_version": "6.0.0",
|
||||
"service_instance_id": "33daefd247cf:1",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "6.0.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "65e8265a35af1c1d",
|
||||
"status": "202",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1",
|
||||
"trace_id": "2b1592ca18aac08bbd14b3f1825f0816"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1795102383403584208",
|
||||
"{\"timestamp\": \"2026-11-19T15:33:03.403545+00:00\", \"severity\": \"INFO\", \"service.name\": \"messenger-api\", \"event\": \"http.request\", \"trace_id\": \"2b1592ca18aac08bbd14b3f1825f0816\", \"span_id\": \"65e8265a35af1c1d\", \"method\": \"POST\", \"route\": \"/api/v1/chats/{chat_id}/attachments\", \"status\": 202, \"duration_seconds\": 0.024863, \"request_id\": \"bad9f74b-1237-475e-b7a9-13e04b9457fb\"}"
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"stats": {
|
||||
"summary": {
|
||||
"bytesProcessedPerSecond": 99017520,
|
||||
"linesProcessedPerSecond": 135509,
|
||||
"totalBytesProcessed": 2015283,
|
||||
"totalLinesProcessed": 2758,
|
||||
"execTime": 0.020353,
|
||||
"queueTime": 0.000121,
|
||||
"subqueries": 0,
|
||||
"totalEntriesReturned": 3,
|
||||
"splits": 0,
|
||||
"shards": 1,
|
||||
"totalPostFilterLines": 3,
|
||||
"totalStructuredMetadataBytesProcessed": 1011858
|
||||
},
|
||||
"querier": {
|
||||
"store": {
|
||||
"totalChunksRef": 0,
|
||||
"totalChunksDownloaded": 0,
|
||||
"chunksDownloadTime": 0,
|
||||
"queryReferencedStructuredMetadata": false,
|
||||
"chunk": {
|
||||
"headChunkBytes": 0,
|
||||
"headChunkLines": 0,
|
||||
"decompressedBytes": 0,
|
||||
"decompressedLines": 0,
|
||||
"compressedBytes": 0,
|
||||
"totalDuplicates": 0,
|
||||
"postFilterLines": 0,
|
||||
"headChunkStructuredMetadataBytes": 0,
|
||||
"decompressedStructuredMetadataBytes": 0
|
||||
},
|
||||
"chunkRefsFetchTime": 0,
|
||||
"congestionControlLatency": 0,
|
||||
"pipelineWrapperFilteredLines": 0
|
||||
}
|
||||
},
|
||||
"ingester": {
|
||||
"totalReached": 1,
|
||||
"totalChunksMatched": 2,
|
||||
"totalBatches": 2,
|
||||
"totalLinesSent": 3,
|
||||
"store": {
|
||||
"totalChunksRef": 0,
|
||||
"totalChunksDownloaded": 0,
|
||||
"chunksDownloadTime": 0,
|
||||
"queryReferencedStructuredMetadata": false,
|
||||
"chunk": {
|
||||
"headChunkBytes": 39416,
|
||||
"headChunkLines": 85,
|
||||
"decompressedBytes": 1975867,
|
||||
"decompressedLines": 2673,
|
||||
"compressedBytes": 241323,
|
||||
"totalDuplicates": 0,
|
||||
"postFilterLines": 3,
|
||||
"headChunkStructuredMetadataBytes": 12008,
|
||||
"decompressedStructuredMetadataBytes": 999850
|
||||
},
|
||||
"chunkRefsFetchTime": 293083,
|
||||
"congestionControlLatency": 0,
|
||||
"pipelineWrapperFilteredLines": 0
|
||||
}
|
||||
},
|
||||
"cache": {
|
||||
"chunk": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"index": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"result": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"statsResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"volumeResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"seriesResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"labelResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"instantMetricResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
}
|
||||
},
|
||||
"index": {
|
||||
"totalChunks": 0,
|
||||
"postFilterChunks": 0,
|
||||
"shardsDuration": 0,
|
||||
"usedBloomFilters": false
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,76 @@
|
||||
{
|
||||
"utc": "2026-11-06T07:27:40.514878+00:00",
|
||||
"project": "lab4-pg-pause-30f1d978ce",
|
||||
"command": ".venv/bin/python student-tests/postgres_pause.py",
|
||||
"passed": true,
|
||||
"sql_without_server_timeout": {
|
||||
"elapsed_seconds": 2.001,
|
||||
"connection_closed": true
|
||||
},
|
||||
"direct_ready": {
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.012
|
||||
},
|
||||
"direct_live": {
|
||||
"status": 200,
|
||||
"elapsed_seconds": 0.006
|
||||
},
|
||||
"repeated_ready": [
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.11
|
||||
},
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.011
|
||||
},
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.014
|
||||
},
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.097
|
||||
},
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.194
|
||||
},
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.013
|
||||
},
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.299
|
||||
},
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.203
|
||||
},
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.013
|
||||
},
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.094
|
||||
}
|
||||
],
|
||||
"ready_after_12_seconds": {
|
||||
"status": 503,
|
||||
"elapsed_seconds": 0.009
|
||||
},
|
||||
"live_after_repeated_requests": {
|
||||
"status": 200,
|
||||
"elapsed_seconds": 0.006
|
||||
},
|
||||
"paused_seconds": 12.565,
|
||||
"recovered_ready": {
|
||||
"status": 200,
|
||||
"elapsed_seconds": 0.009
|
||||
},
|
||||
"recovery_seconds": 2.874,
|
||||
"api_not_restarted": true,
|
||||
"existing_session_and_message_preserved": true
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,32 @@
|
||||
Research project: lab2-research-85021a09d9
|
||||
Measuring two indexes on the same 100000 messages, three repeats...
|
||||
{
|
||||
"chat_only": {
|
||||
"insert_100000_median_ms": 1386.7587500000127,
|
||||
"index_bytes": 671744,
|
||||
"wal_bytes_median": 38822560,
|
||||
"queries": 180,
|
||||
"errors": 0,
|
||||
"concurrency": 1,
|
||||
"p50_ms": 1.6852079999978287,
|
||||
"p95_ms": 7.566457999985232,
|
||||
"p99_ms": 46.40233399999261,
|
||||
"successful_sql_rps": 282.01364182715855
|
||||
},
|
||||
"history": {
|
||||
"insert_100000_median_ms": 1404.4910420000178,
|
||||
"index_bytes": 7315456,
|
||||
"wal_bytes_median": 43108512,
|
||||
"queries": 180,
|
||||
"errors": 0,
|
||||
"concurrency": 1,
|
||||
"p50_ms": 0.22610399997802233,
|
||||
"p95_ms": 0.29704099995342403,
|
||||
"p99_ms": 0.3384169999662845,
|
||||
"successful_sql_rps": 4322.810483886352
|
||||
}
|
||||
}
|
||||
PASS: backup 0.54s; restore 0.68s; 100000 messages, all hashes and 12 samples match
|
||||
PASS: restored into a different container and fresh volume; one post-backup commit is absent as expected
|
||||
Results: evidence/research.json, evidence/query-plans.json; dump stays outside Git in backups/
|
||||
Removed only temporary research project: lab2-research-85021a09d9
|
||||
@@ -0,0 +1,618 @@
|
||||
{
|
||||
"uid": "messenger-lab6",
|
||||
"title": "Messenger — Lab 6: signals and SLO",
|
||||
"schemaVersion": 41,
|
||||
"version": 1,
|
||||
"refresh": "5s",
|
||||
"time": {
|
||||
"from": "now-30m",
|
||||
"to": "now"
|
||||
},
|
||||
"tags": [
|
||||
"lab6",
|
||||
"slo"
|
||||
],
|
||||
"panels": [
|
||||
{
|
||||
"id": 1,
|
||||
"title": "Business API RPS",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 0,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "reqps"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"description": "",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum by (method, route) (rate(messenger_http_requests_total[1m]))",
|
||||
"legendFormat": "{{method}} {{route}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"title": "API 5xx / all business requests",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 0,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"description": "",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "lab6:api_bad_ratio:1m",
|
||||
"legendFormat": "API 5xx / all business requests"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"title": "API latency: aggregated histogram",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 8,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"description": "Seconds to response headers; excludes health. Sum buckets across replicas before percentile.",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "histogram_quantile(0.5, sum by (le) (rate(messenger_http_duration_seconds_bucket[1m])))",
|
||||
"legendFormat": "p50"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "histogram_quantile(0.95, sum by (le) (rate(messenger_http_duration_seconds_bucket[1m])))",
|
||||
"legendFormat": "p95"
|
||||
},
|
||||
{
|
||||
"refId": "C",
|
||||
"expr": "histogram_quantile(0.99, sum by (le) (rate(messenger_http_duration_seconds_bucket[1m])))",
|
||||
"legendFormat": "p99"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"title": "Time from acceptance to ready: completed jobs",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 8,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"description": "Success histogram only. SLO below also includes jobs that never complete.",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "histogram_quantile(0.5, sum by (le) (rate(messenger_attachment_ready_duration_seconds_bucket[2m])))",
|
||||
"legendFormat": "p50"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "histogram_quantile(0.95, sum by (le) (rate(messenger_attachment_ready_duration_seconds_bucket[2m])))",
|
||||
"legendFormat": "p95"
|
||||
},
|
||||
{
|
||||
"refId": "C",
|
||||
"expr": "histogram_quantile(0.99, sum by (le) (rate(messenger_attachment_ready_duration_seconds_bucket[2m])))",
|
||||
"legendFormat": "p99"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"title": "Job attempts by outcome",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 16,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"description": "",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum by (outcome) (rate(messenger_job_attempts_total[1m]))",
|
||||
"legendFormat": "{{outcome}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"title": "Worker processing p95",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 16,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"description": "",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "histogram_quantile(0.95, sum by (le, outcome) (rate(messenger_job_duration_seconds_bucket[1m])))",
|
||||
"legendFormat": "{{outcome}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"title": "Outstanding jobs (DB source of truth)",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 24,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"description": "Queued + processing durable jobs, including retry/outbox; not Redis XLEN.",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "max(messenger_queue_depth)",
|
||||
"legendFormat": "Outstanding jobs (DB source of truth)"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"title": "Oldest unfinished job age",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 24,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"description": "",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "max(messenger_queue_oldest_age_seconds)",
|
||||
"legendFormat": "Oldest unfinished job age"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 9,
|
||||
"title": "DB pool used / waiting",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 32,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"description": "",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum by (service_name, state) (messenger_db_pool)",
|
||||
"legendFormat": "{{service_name}} {{state}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 10,
|
||||
"title": "Worker busy / capacity",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 32,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"description": "",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum(messenger_worker_busy) or vector(0)",
|
||||
"legendFormat": "busy"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "sum(messenger_worker_capacity) or vector(0)",
|
||||
"legendFormat": "capacity"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 11,
|
||||
"title": "Process CPU: cores consumed",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 40,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"description": "",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum by (service_name) (rate(messenger_process_cpu_time_seconds_total[1m]))",
|
||||
"legendFormat": "{{service_name}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 12,
|
||||
"title": "Process RSS",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 40,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "bytes"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"description": "",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum by (service_name) (messenger_process_memory_bytes)",
|
||||
"legendFormat": "{{service_name}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 13,
|
||||
"title": "API availability SLO: >= 99% / 1h",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 48,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"description": "5xx bad; all instrumented business requests denominator; 4xx not server failures.",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "1-lab6:api_bad_ratio:1h",
|
||||
"legendFormat": "API availability SLO: >= 99% / 1h"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 14,
|
||||
"title": "Ready SLO: >= 95% within 30s / 1h",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 48,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"description": "Acceptance cohorts mature after 30s. Failed, stuck and late jobs are BAD. No ready-only bias.",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "1-lab6:ready_bad_ratio:1h",
|
||||
"legendFormat": "Ready SLO: >= 95% within 30s / 1h"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 15,
|
||||
"title": "API burn rate (budget / hour)",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 56,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"description": "Fast: 2m AND 30s > 6x, for 15s. Slow: 6m AND 1m > 3x, for 30s.",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "lab6:api_bad_ratio:30s/0.01",
|
||||
"legendFormat": "30s"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "lab6:api_bad_ratio:2m/0.01",
|
||||
"legendFormat": "2m"
|
||||
},
|
||||
{
|
||||
"refId": "C",
|
||||
"expr": "lab6:api_bad_ratio:1m/0.01",
|
||||
"legendFormat": "1m"
|
||||
},
|
||||
{
|
||||
"refId": "D",
|
||||
"expr": "lab6:api_bad_ratio:6m/0.01",
|
||||
"legendFormat": "6m"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 16,
|
||||
"title": "Ready burn rate (budget / hour)",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 56,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"description": "Fast: 2m AND 30s > 6x, for 15s. Slow: 6m AND 1m > 3x, for 30s.",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "lab6:ready_bad_ratio:30s/0.05",
|
||||
"legendFormat": "30s"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "lab6:ready_bad_ratio:2m/0.05",
|
||||
"legendFormat": "2m"
|
||||
},
|
||||
{
|
||||
"refId": "C",
|
||||
"expr": "lab6:ready_bad_ratio:1m/0.05",
|
||||
"legendFormat": "1m"
|
||||
},
|
||||
{
|
||||
"refId": "D",
|
||||
"expr": "lab6:ready_bad_ratio:6m/0.05",
|
||||
"legendFormat": "6m"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 17,
|
||||
"title": "Local alert states",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 64,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"description": "",
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "ALERTS{alertstate=~\"pending|firing\"}",
|
||||
"legendFormat": "{{alertname}} {{alertstate}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 18,
|
||||
"title": "API / worker structured logs (click TraceID)",
|
||||
"type": "logs",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 64,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "{service_name=~\"messenger-api|messenger-worker\"} | json"
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"wrapLogMessage": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 19,
|
||||
"title": "Recent real worker traces",
|
||||
"type": "table",
|
||||
"datasource": {
|
||||
"type": "tempo",
|
||||
"uid": "tempo"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 72,
|
||||
"w": 24,
|
||||
"h": 9
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"queryType": "traceql",
|
||||
"query": "{ resource.service.name = \"messenger-worker\" }",
|
||||
"limit": 20,
|
||||
"tableType": "traces",
|
||||
"filters": []
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"annotations": {
|
||||
"list": []
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
apiVersion: 1
|
||||
providers:
|
||||
- name: messenger
|
||||
folder: Lab 6
|
||||
type: file
|
||||
disableDeletion: false
|
||||
updateIntervalSeconds: 10
|
||||
options:
|
||||
path: /var/lib/grafana/dashboards
|
||||
@@ -0,0 +1,35 @@
|
||||
apiVersion: 1
|
||||
datasources:
|
||||
- name: Prometheus
|
||||
uid: prometheus
|
||||
type: prometheus
|
||||
access: proxy
|
||||
url: http://prometheus:9090
|
||||
isDefault: true
|
||||
jsonData:
|
||||
timeInterval: 5s
|
||||
- name: Loki
|
||||
uid: loki
|
||||
type: loki
|
||||
access: proxy
|
||||
url: http://loki:3100
|
||||
jsonData:
|
||||
derivedFields:
|
||||
- name: TraceID
|
||||
matcherRegex: '"trace_id": "([a-f0-9]{32})"'
|
||||
datasourceUid: tempo
|
||||
url: '$${__value.raw}'
|
||||
- name: Tempo
|
||||
uid: tempo
|
||||
type: tempo
|
||||
access: proxy
|
||||
url: http://tempo:3200
|
||||
jsonData:
|
||||
tracesToLogs:
|
||||
datasourceUid: loki
|
||||
spanStartTimeShift: '-1m'
|
||||
spanEndTimeShift: '1m'
|
||||
filterByTraceID: true
|
||||
tags: [{key: service.name, value: service_name}]
|
||||
nodeGraph:
|
||||
enabled: true
|
||||
@@ -0,0 +1,40 @@
|
||||
global
|
||||
log stdout format raw local0
|
||||
maxconn 256
|
||||
|
||||
defaults
|
||||
mode http
|
||||
log global
|
||||
option dontlognull
|
||||
option http-server-close
|
||||
timeout connect 1s
|
||||
timeout client 40s
|
||||
timeout server 60s
|
||||
timeout http-request 5s
|
||||
timeout check 1s
|
||||
retries 1
|
||||
|
||||
resolvers docker
|
||||
nameserver dns 127.0.0.11:53
|
||||
accepted_payload_size 8192
|
||||
resolve_retries 3
|
||||
timeout resolve 1s
|
||||
timeout retry 1s
|
||||
hold valid 1s
|
||||
hold obsolete 1s
|
||||
hold nx 1s
|
||||
hold timeout 1s
|
||||
hold refused 1s
|
||||
hold other 1s
|
||||
|
||||
frontend public
|
||||
bind :8080
|
||||
default_backend api
|
||||
|
||||
backend api
|
||||
balance roundrobin
|
||||
option httpchk
|
||||
http-check send meth GET uri /health/ready ver HTTP/1.1 hdr Host localhost
|
||||
http-check expect status 200
|
||||
default-server inter 1s fastinter 1s downinter 1s fall 2 rise 2
|
||||
server-template replica 10 api:8080 check resolvers docker resolve-prefer ipv4 init-addr none
|
||||
@@ -0,0 +1,113 @@
|
||||
groups:
|
||||
- name: messenger-slo
|
||||
interval: 5s
|
||||
rules:
|
||||
- record: lab6:api_bad_ratio:30s
|
||||
expr: (sum(rate(messenger_http_requests_total{status_class="5xx"}[30s])) or vector(0))
|
||||
/ sum(rate(messenger_http_requests_total[30s]))
|
||||
- record: lab6:ready_bad_ratio:30s
|
||||
expr: sum(rate(messenger_ready_deadlines_total{outcome="bad"}[30s])) / sum(rate(messenger_ready_deadlines_total[30s]))
|
||||
- record: lab6:api_bad_ratio:1m
|
||||
expr: (sum(rate(messenger_http_requests_total{status_class="5xx"}[1m])) or vector(0))
|
||||
/ sum(rate(messenger_http_requests_total[1m]))
|
||||
- record: lab6:ready_bad_ratio:1m
|
||||
expr: sum(rate(messenger_ready_deadlines_total{outcome="bad"}[1m])) / sum(rate(messenger_ready_deadlines_total[1m]))
|
||||
- record: lab6:api_bad_ratio:2m
|
||||
expr: (sum(rate(messenger_http_requests_total{status_class="5xx"}[2m])) or vector(0))
|
||||
/ sum(rate(messenger_http_requests_total[2m]))
|
||||
- record: lab6:ready_bad_ratio:2m
|
||||
expr: sum(rate(messenger_ready_deadlines_total{outcome="bad"}[2m])) / sum(rate(messenger_ready_deadlines_total[2m]))
|
||||
- record: lab6:api_bad_ratio:6m
|
||||
expr: (sum(rate(messenger_http_requests_total{status_class="5xx"}[6m])) or vector(0))
|
||||
/ sum(rate(messenger_http_requests_total[6m]))
|
||||
- record: lab6:ready_bad_ratio:6m
|
||||
expr: sum(rate(messenger_ready_deadlines_total{outcome="bad"}[6m])) / sum(rate(messenger_ready_deadlines_total[6m]))
|
||||
- record: lab6:api_bad_ratio:1h
|
||||
expr: (sum(rate(messenger_http_requests_total{status_class="5xx"}[1h])) or vector(0))
|
||||
/ sum(rate(messenger_http_requests_total[1h]))
|
||||
- record: lab6:ready_bad_ratio:1h
|
||||
expr: sum(rate(messenger_ready_deadlines_total{outcome="bad"}[1h])) / sum(rate(messenger_ready_deadlines_total[1h]))
|
||||
- alert: APIAvailabilityFastBurn
|
||||
expr: (lab6:api_bad_ratio:2m > 6 * 0.01) and (lab6:api_bad_ratio:30s > 6 * 0.01)
|
||||
and (sum(increase(messenger_http_requests_total[2m])) >= 30) and (sum(increase(messenger_http_requests_total[30s]))
|
||||
>= 2)
|
||||
for: 15s
|
||||
labels:
|
||||
severity: critical
|
||||
slo: api
|
||||
annotations:
|
||||
summary: 'api: 6x budget burn in both 2m and 30s'
|
||||
description: Budget 0.01; rolling SLO window 1h. Requires real traffic, not
|
||||
health requests.
|
||||
runbook_url: https://gitea.privcloud.tech/Savelev_Ivan/mtusi2026lab6/src/branch/main/docs/runbook.md#api
|
||||
runbook: docs/runbook.md#api
|
||||
- alert: APIAvailabilitySlowBurn
|
||||
expr: (lab6:api_bad_ratio:6m > 3 * 0.01) and (lab6:api_bad_ratio:1m > 3 * 0.01)
|
||||
and (sum(increase(messenger_http_requests_total[6m])) >= 30) and (sum(increase(messenger_http_requests_total[1m]))
|
||||
>= 2)
|
||||
for: 30s
|
||||
labels:
|
||||
severity: warning
|
||||
slo: api
|
||||
annotations:
|
||||
summary: 'api: 3x budget burn in both 6m and 1m'
|
||||
description: Budget 0.01; rolling SLO window 1h. Requires real traffic, not
|
||||
health requests.
|
||||
runbook_url: https://gitea.privcloud.tech/Savelev_Ivan/mtusi2026lab6/src/branch/main/docs/runbook.md#api
|
||||
runbook: docs/runbook.md#api
|
||||
- alert: AttachmentReadyFastBurn
|
||||
expr: (lab6:ready_bad_ratio:2m > 6 * 0.05) and (lab6:ready_bad_ratio:30s > 6 *
|
||||
0.05) and (sum(increase(messenger_ready_deadlines_total[2m])) >= 5) and (sum(increase(messenger_ready_deadlines_total[30s]))
|
||||
>= 2)
|
||||
for: 15s
|
||||
labels:
|
||||
severity: critical
|
||||
slo: ready
|
||||
annotations:
|
||||
summary: 'ready: 6x budget burn in both 2m and 30s'
|
||||
description: Budget 0.05; rolling SLO window 1h. Requires real traffic, not
|
||||
health requests.
|
||||
runbook_url: https://gitea.privcloud.tech/Savelev_Ivan/mtusi2026lab6/src/branch/main/docs/runbook.md#ready
|
||||
runbook: docs/runbook.md#ready
|
||||
- alert: AttachmentReadySlowBurn
|
||||
expr: (lab6:ready_bad_ratio:6m > 3 * 0.05) and (lab6:ready_bad_ratio:1m > 3 *
|
||||
0.05) and (sum(increase(messenger_ready_deadlines_total[6m])) >= 5) and (sum(increase(messenger_ready_deadlines_total[1m]))
|
||||
>= 2)
|
||||
for: 30s
|
||||
labels:
|
||||
severity: warning
|
||||
slo: ready
|
||||
annotations:
|
||||
summary: 'ready: 3x budget burn in both 6m and 1m'
|
||||
description: Budget 0.05; rolling SLO window 1h. Requires real traffic, not
|
||||
health requests.
|
||||
runbook_url: https://gitea.privcloud.tech/Savelev_Ivan/mtusi2026lab6/src/branch/main/docs/runbook.md#ready
|
||||
runbook: docs/runbook.md#ready
|
||||
- alert: APIHighErrorRate
|
||||
expr: lab6:api_bad_ratio:30s > 0.05
|
||||
for: 5s
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: API 5xx > 5% for 30s (single-window comparator)
|
||||
runbook: docs/runbook.md#api
|
||||
runbook_url: https://gitea.privcloud.tech/Savelev_Ivan/mtusi2026lab6/src/branch/main/docs/runbook.md#api
|
||||
- alert: AttachmentQueueLag
|
||||
expr: (max(messenger_queue_oldest_age_seconds) > 20) and (max(messenger_queue_depth)
|
||||
> 0)
|
||||
for: 10s
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: Oldest unfinished job exceeds 20s; queue not empty
|
||||
runbook: docs/runbook.md#ready
|
||||
runbook_url: https://gitea.privcloud.tech/Savelev_Ivan/mtusi2026lab6/src/branch/main/docs/runbook.md#ready
|
||||
- alert: TelemetryCollectorUnavailable
|
||||
expr: up{job="collector"} == 0
|
||||
for: 15s
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: Collector scrape unavailable; SLI may be incomplete
|
||||
runbook: docs/runbook.md#telemetry
|
||||
runbook_url: https://gitea.privcloud.tech/Savelev_Ivan/mtusi2026lab6/src/branch/main/docs/runbook.md#telemetry
|
||||
@@ -0,0 +1,74 @@
|
||||
extensions:
|
||||
health_check:
|
||||
endpoint: 0.0.0.0:13133
|
||||
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
http:
|
||||
endpoint: 0.0.0.0:4318
|
||||
|
||||
processors:
|
||||
memory_limiter:
|
||||
check_interval: 1s
|
||||
limit_mib: 192
|
||||
spike_limit_mib: 32
|
||||
batch:
|
||||
timeout: 1s
|
||||
send_batch_size: 64
|
||||
send_batch_max_size: 128
|
||||
|
||||
exporters:
|
||||
prometheus:
|
||||
endpoint: 0.0.0.0:9464
|
||||
metric_expiration: 30s
|
||||
resource_to_telemetry_conversion:
|
||||
enabled: true
|
||||
otlphttp/loki:
|
||||
endpoint: http://loki:3100/otlp
|
||||
timeout: 2s
|
||||
sending_queue:
|
||||
enabled: true
|
||||
num_consumers: 2
|
||||
queue_size: 64
|
||||
retry_on_failure:
|
||||
initial_interval: 1s
|
||||
max_interval: 5s
|
||||
max_elapsed_time: 15s
|
||||
otlp/tempo:
|
||||
endpoint: tempo:4317
|
||||
tls:
|
||||
insecure: true
|
||||
timeout: 2s
|
||||
sending_queue:
|
||||
enabled: true
|
||||
num_consumers: 2
|
||||
queue_size: 64
|
||||
retry_on_failure:
|
||||
initial_interval: 1s
|
||||
max_interval: 5s
|
||||
max_elapsed_time: 15s
|
||||
|
||||
service:
|
||||
extensions: [health_check]
|
||||
telemetry:
|
||||
metrics:
|
||||
readers:
|
||||
- pull:
|
||||
exporter:
|
||||
prometheus:
|
||||
host: 0.0.0.0
|
||||
port: 8888
|
||||
pipelines:
|
||||
traces:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, batch]
|
||||
exporters: [otlp/tempo]
|
||||
metrics:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, batch]
|
||||
exporters: [prometheus]
|
||||
logs:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, batch]
|
||||
exporters: [otlphttp/loki]
|
||||
@@ -0,0 +1,37 @@
|
||||
auth_enabled: false
|
||||
server:
|
||||
http_listen_port: 3100
|
||||
common:
|
||||
path_prefix: /loki
|
||||
replication_factor: 1
|
||||
ring:
|
||||
kvstore:
|
||||
store: inmemory
|
||||
storage:
|
||||
filesystem:
|
||||
chunks_directory: /loki/chunks
|
||||
rules_directory: /loki/rules
|
||||
schema_config:
|
||||
configs:
|
||||
- from: 2024-01-01
|
||||
store: tsdb
|
||||
object_store: filesystem
|
||||
schema: v13
|
||||
index:
|
||||
prefix: index_
|
||||
period: 24h
|
||||
limits_config:
|
||||
allow_structured_metadata: true
|
||||
retention_period: 168h
|
||||
otlp_config:
|
||||
resource_attributes:
|
||||
ignore_defaults: true
|
||||
attributes_config:
|
||||
- action: index_label
|
||||
attributes: [service.name, deployment.environment.name]
|
||||
- action: structured_metadata
|
||||
attributes: [service.instance.id, service.version]
|
||||
compactor:
|
||||
working_directory: /loki/compactor
|
||||
retention_enabled: true
|
||||
delete_request_store: filesystem
|
||||
@@ -0,0 +1,15 @@
|
||||
global:
|
||||
scrape_interval: 5s
|
||||
evaluation_interval: 5s
|
||||
rule_files:
|
||||
- /etc/prometheus/alerts.yaml
|
||||
scrape_configs:
|
||||
- job_name: messenger
|
||||
static_configs:
|
||||
- targets: [otel-collector:9464]
|
||||
- job_name: collector
|
||||
static_configs:
|
||||
- targets: [otel-collector:8888]
|
||||
- job_name: prometheus
|
||||
static_configs:
|
||||
- targets: [localhost:9090]
|
||||
@@ -0,0 +1,20 @@
|
||||
server:
|
||||
http_listen_port: 3200
|
||||
distributor:
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
grpc:
|
||||
endpoint: 0.0.0.0:4317
|
||||
ingester:
|
||||
max_block_duration: 1m
|
||||
compactor:
|
||||
compaction:
|
||||
block_retention: 168h
|
||||
storage:
|
||||
trace:
|
||||
backend: local
|
||||
wal:
|
||||
path: /var/tempo/wal
|
||||
local:
|
||||
path: /var/tempo/blocks
|
||||
@@ -0,0 +1,30 @@
|
||||
CREATE TABLE users (
|
||||
id uuid PRIMARY KEY,
|
||||
username varchar(32) NOT NULL UNIQUE CHECK (username ~ '^[a-z0-9_]{3,32}$'),
|
||||
display_name varchar(100) NOT NULL CHECK (char_length(display_name) >= 1),
|
||||
password_hash text NOT NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
|
||||
CREATE TABLE chats (
|
||||
id uuid PRIMARY KEY,
|
||||
title varchar(100) NOT NULL CHECK (char_length(title) >= 1),
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
|
||||
CREATE TABLE chat_members (
|
||||
chat_id uuid NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||
user_id uuid NOT NULL REFERENCES users(id),
|
||||
PRIMARY KEY (chat_id, user_id)
|
||||
);
|
||||
CREATE INDEX chat_members_user_idx ON chat_members (user_id, chat_id);
|
||||
|
||||
CREATE TABLE messages (
|
||||
id uuid PRIMARY KEY,
|
||||
chat_id uuid NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||
sender_id uuid NOT NULL,
|
||||
text text NOT NULL CHECK (char_length(text) BETWEEN 1 AND 4000 AND text !~ '^\s*$'),
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
FOREIGN KEY (chat_id, sender_id) REFERENCES chat_members(chat_id, user_id)
|
||||
);
|
||||
CREATE INDEX messages_history_idx ON messages (chat_id, created_at, id);
|
||||
@@ -0,0 +1,10 @@
|
||||
-- Добавочная миграция: существующие данные и таблицы lab2 сохраняются.
|
||||
CREATE TABLE message_idempotency (
|
||||
user_id uuid NOT NULL REFERENCES users(id),
|
||||
chat_id uuid NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||
key varchar(128) NOT NULL CHECK (key ~ '^[A-Za-z0-9._:-]{1,128}$'),
|
||||
request_text text NOT NULL,
|
||||
message_id uuid NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
|
||||
expires_at timestamptz NOT NULL,
|
||||
PRIMARY KEY (user_id, chat_id, key)
|
||||
);
|
||||
@@ -0,0 +1,34 @@
|
||||
-- Только добавления к lab3; старые сообщения и ключи сохраняются.
|
||||
CREATE TABLE attachments (
|
||||
id uuid PRIMARY KEY,
|
||||
chat_id uuid NOT NULL REFERENCES chats(id),
|
||||
owner_id uuid NOT NULL,
|
||||
object_key text NOT NULL UNIQUE,
|
||||
filename varchar(255) NOT NULL,
|
||||
content_type text NOT NULL CHECK (content_type IN ('image/png', 'image/jpeg')),
|
||||
size_bytes integer NOT NULL CHECK (size_bytes BETWEEN 1 AND 10485760),
|
||||
sha256 char(64) NOT NULL CHECK (sha256 ~ '^[a-f0-9]{64}$'),
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
FOREIGN KEY (chat_id, owner_id) REFERENCES chat_members(chat_id, user_id)
|
||||
);
|
||||
CREATE INDEX attachments_chat_idx ON attachments(chat_id);
|
||||
|
||||
ALTER TABLE messages DROP CONSTRAINT messages_text_check;
|
||||
ALTER TABLE messages ADD CONSTRAINT messages_text_check CHECK (char_length(text) <= 4000);
|
||||
CREATE TABLE message_attachments (
|
||||
message_id uuid NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
|
||||
attachment_id uuid NOT NULL REFERENCES attachments(id),
|
||||
position integer NOT NULL CHECK (position BETWEEN 0 AND 9),
|
||||
PRIMARY KEY (message_id, attachment_id),
|
||||
UNIQUE (message_id, position)
|
||||
);
|
||||
ALTER TABLE message_idempotency
|
||||
ADD COLUMN request_attachment_ids uuid[] NOT NULL DEFAULT '{}';
|
||||
|
||||
-- Фиксируется ДО S3 PUT. После COMMIT ready-метаданных удаляется атомарно с ними.
|
||||
CREATE TABLE attachment_uploads (
|
||||
id uuid PRIMARY KEY,
|
||||
object_key text NOT NULL UNIQUE,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
CREATE INDEX attachment_uploads_created_idx ON attachment_uploads(created_at);
|
||||
@@ -0,0 +1,42 @@
|
||||
-- Lab5: существующие ready-вложения lab4 остаются доступными.
|
||||
ALTER TABLE attachments
|
||||
ADD COLUMN status text NOT NULL DEFAULT 'ready' CHECK (status IN ('queued','processing','ready','failed')),
|
||||
ADD COLUMN variants jsonb NOT NULL DEFAULT '[]',
|
||||
ADD COLUMN error_code text,
|
||||
ADD COLUMN updated_at timestamptz NOT NULL DEFAULT clock_timestamp();
|
||||
UPDATE attachments SET updated_at=created_at;
|
||||
|
||||
CREATE TABLE attachment_jobs (
|
||||
id uuid PRIMARY KEY,
|
||||
attachment_id uuid NOT NULL UNIQUE REFERENCES attachments(id),
|
||||
pipeline_version integer NOT NULL DEFAULT 1 CHECK (pipeline_version=1),
|
||||
staging_key text NOT NULL UNIQUE,
|
||||
attempt integer NOT NULL DEFAULT 1 CHECK (attempt>=1),
|
||||
max_attempts integer NOT NULL CHECK (max_attempts>=1),
|
||||
replay_count integer NOT NULL DEFAULT 0,
|
||||
lease_token uuid,
|
||||
lease_until timestamptz,
|
||||
next_attempt_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
last_error text,
|
||||
staging_deleted_at timestamptz,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
CREATE INDEX attachment_jobs_lease_idx ON attachment_jobs(lease_until) WHERE lease_until IS NOT NULL;
|
||||
|
||||
CREATE TABLE attachment_outbox (
|
||||
job_id uuid NOT NULL REFERENCES attachment_jobs(id),
|
||||
attempt integer NOT NULL,
|
||||
available_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
published_at timestamptz,
|
||||
PRIMARY KEY(job_id, attempt)
|
||||
);
|
||||
CREATE INDEX attachment_outbox_due_idx ON attachment_outbox(available_at) WHERE published_at IS NULL;
|
||||
|
||||
CREATE TABLE attachment_job_events (
|
||||
id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
job_id uuid NOT NULL REFERENCES attachment_jobs(id),
|
||||
attempt integer NOT NULL,
|
||||
event text NOT NULL,
|
||||
error_code text,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
@@ -0,0 +1,9 @@
|
||||
-- Context belongs to the durable job, not the lifetime of the HTTP process.
|
||||
ALTER TABLE attachment_jobs
|
||||
ADD COLUMN traceparent text,
|
||||
ADD COLUMN tracestate text,
|
||||
ADD COLUMN ready_at timestamptz;
|
||||
-- Existing completed jobs predate instrumentation; preserve their actual update time.
|
||||
UPDATE attachment_jobs j SET ready_at=a.updated_at
|
||||
FROM attachments a WHERE a.id=j.attachment_id AND a.status='ready';
|
||||
CREATE INDEX attachment_jobs_created_idx ON attachment_jobs(created_at);
|
||||
@@ -0,0 +1,50 @@
|
||||
# Основные библиотеки
|
||||
fastapi==0.142.4
|
||||
uvicorn==0.54.0
|
||||
pydantic==2.13.5
|
||||
|
||||
# Их зависимости
|
||||
annotated-doc==0.0.5
|
||||
annotated-types==0.8.0
|
||||
anyio==4.15.1
|
||||
click==8.5.0
|
||||
h11==0.16.0
|
||||
idna==3.20
|
||||
opentelemetry-api==1.45.1
|
||||
pydantic-core==2.46.5
|
||||
starlette==1.7.0
|
||||
typing-extensions==4.16.0
|
||||
typing-inspection==0.4.4
|
||||
|
||||
# PostgreSQL, сессии и Argon2id
|
||||
psycopg[binary,pool]==3.3.6
|
||||
redis==8.1.0
|
||||
argon2-cffi==25.1.0
|
||||
|
||||
# Транзитивные зависимости новых библиотек
|
||||
psycopg-binary==3.3.6
|
||||
psycopg-pool==3.3.3
|
||||
argon2-cffi-bindings==26.1.0
|
||||
cffi==2.1.1
|
||||
pycparser==3.11
|
||||
|
||||
# S3 и проверка настоящих PNG/JPEG (без изменения оригинала)
|
||||
minio==7.2.20
|
||||
Pillow==13.0.0
|
||||
certifi==2026.10.23
|
||||
pycryptodome==4.3.0
|
||||
urllib3==2.8.0
|
||||
|
||||
# Explicit OpenTelemetry SDK (all signals), no automatic header/SQL/body capture.
|
||||
opentelemetry-sdk==1.45.1
|
||||
opentelemetry-exporter-otlp-proto-http==1.45.1
|
||||
opentelemetry-semantic-conventions==0.66b1
|
||||
opentelemetry-exporter-otlp-proto-common==1.45.1
|
||||
opentelemetry-proto==1.45.1
|
||||
opentelemetry-exporter-http-transport==0.66b1
|
||||
opentelemetry-exporter-otlp-common==0.66b1
|
||||
googleapis-common-protos==1.76.0
|
||||
protobuf==7.36.2
|
||||
requests==2.34.2
|
||||
charset-normalizer==3.5.2
|
||||
psutil==7.2.2
|
||||
@@ -0,0 +1,12 @@
|
||||
# Трек 2 на оценку 5
|
||||
|
||||
Запуск из корня: `./scripts/research-lab2.sh`. Нужны Docker Compose, Python 3 и интернет для первой сборки. Скрипт собирает API-образ и использует отдельный Compose project с новыми volumes. Текущий стенд не участвует.
|
||||
|
||||
- run.py управляет контейнерами, pg_dump, pg_restore, файлами evidence и cleanup.
|
||||
- benchmark.py выполняет SQL и сверку внутри API-образа. Перед заполнением требует пустую БД. Исследовательский флаг устанавливает только драйвер.
|
||||
|
||||
Датасет: 100000 сообщений в 4 чатах (70000/20000/9000/1000). Сравниваются индекс chat_id и индекс (chat_id,created_at,id), по 3 повтора с одинаковыми ограничениями и записями. На каждый чат — страницы около 10/50/90%, один прогрев и пять измерений, 51 строка. Измеряется INSERT SELECT с COMMIT, WAL и размер индекса. EXPLAIN и основные цифры сохраняются в evidence.
|
||||
|
||||
После dump создаётся ещё одно сообщение. Backup восстанавливается в другой контейнер и свежий volume. Сравниваются все счётчики, 12 полных записей и SHA-256 всех сообщений. Последующая запись ожидаемо отсутствует в восстановленном снимке.
|
||||
|
||||
Файлы research.json, query-plans.json и research.txt обновляются автоматически. Dump остаётся в игнорируемом backups/. Временные исследовательские volumes удаляются после проверки. Методика, фактические результаты и ограничения лабораторной 2 описаны в [REPORT.lab2.md](../REPORT.lab2.md).
|
||||
@@ -0,0 +1,37 @@
|
||||
# Лабораторная 6, оценка 5, трек 2: SLO и обнаружение инцидентов
|
||||
|
||||
После `./scripts/start.sh`: `./scripts/research.sh` (около 11 минут).
|
||||
Полный набор проверок вместе с сохранённым минимумом lab5:
|
||||
`./scripts/check-lab6.sh` (около 17–20 минут при готовых образах).
|
||||
Эксперимент намеренно блокирует messages и останавливает worker своего локального
|
||||
синтетического стенда. Отказы ограничены по времени, worker восстанавливается
|
||||
в `finally`. Нужны Docker Compose, Python 3.11 и `.venv` с корневыми requirements.
|
||||
|
||||
Гипотеза: alert по двум окнам не сработает на 4s всплеск, но обнаружит длительную
|
||||
деградацию БД и worker; после восстановления станет inactive. Сравнение — простой
|
||||
5%/30s alert. Параметры заранее заданы в `infra/otel/alerts.yaml` и `docs/runbook.md`;
|
||||
эксперимент их не меняет.
|
||||
|
||||
Профиль: 2 POST сообщения/сек и 0.5 PNG upload/сек, fixed arrival, максимум
|
||||
16 in-flight, без retries. Jobs реально обрабатываются через S3 и broker.
|
||||
Baseline 120s → DB spike 4s → recovery 60s → DB lock 90s → recovery 100s →
|
||||
worker stopped 120s → recovery 120s. Состояния alerts и backlog читаются каждые 5s
|
||||
из Prometheus. SQL/Docker создают и снимают воздействие; причина определяется
|
||||
по сохранённым metrics/logs/traces, а не по выводу команды внедрения отказа.
|
||||
|
||||
Результаты: `evidence/lab6-slo-research.json` (каждый запрос без body/credentials,
|
||||
фазы, detection/resolution, p50/p95/p99, RPS), `lab6-slo-series.json`
|
||||
(CPU/RSS, pool, saturation и остальные временные ряды),
|
||||
`lab6-db-incident-trace.json`, `lab6-db-incident-logs.json`,
|
||||
`lab6-worker-incident-trace.json`, `lab6-worker-incident-logs.json`.
|
||||
`student-tests/promtool-slo.yaml` дополнительно проверяет правила на повторяемых
|
||||
синтетических временных рядах: всплеск, длительный отказ, recovery, stalled jobs,
|
||||
нет трафика. Это дополнение к live эксперименту, не замена ему.
|
||||
|
||||
Ограничения: один Docker-host, небольшие PNG, один live прогон каждого отказа;
|
||||
шаг измерения alerts 5s, задержки SDK export/scrape. Часовой SLO задан, но
|
||||
эксперимент короче часа: результат проверяет детектор, а не долгосрочную надёжность.
|
||||
False-positive count нельзя переносить на другие нагрузки/окна.
|
||||
|
||||
Историческое исследование lab2: `run.py`, `benchmark.py`, `scripts/research-lab2.sh`,
|
||||
[README.lab2.md](README.lab2.md).
|
||||
@@ -0,0 +1,168 @@
|
||||
"""SQL-эксперимент запускается только в отдельном research Compose project."""
|
||||
from datetime import datetime, timedelta, timezone
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import secrets
|
||||
import statistics
|
||||
import sys
|
||||
import time
|
||||
from uuid import UUID
|
||||
|
||||
import psycopg
|
||||
from psycopg.conninfo import make_conninfo
|
||||
from psycopg.rows import dict_row
|
||||
|
||||
from src import db, migrate, passwords
|
||||
from src.config import secret
|
||||
|
||||
|
||||
SIZES = [70000, 20000, 9000, 1000]
|
||||
START = datetime(2026, 1, 1, tzinfo=timezone.utc)
|
||||
QUERY = """SELECT * FROM messages WHERE chat_id=%s AND (created_at, id)>(%s, %s)
|
||||
ORDER BY created_at, id LIMIT %s"""
|
||||
|
||||
|
||||
def identifier(value):
|
||||
return UUID(hashlib.md5(value.encode(), usedforsecurity=False).hexdigest())
|
||||
|
||||
|
||||
def connect(restored=False):
|
||||
url = secret("DATABASE_URL")
|
||||
if restored:
|
||||
url = make_conninfo(url, host="postgres-restore")
|
||||
return psycopg.connect(url, autocommit=True, row_factory=dict_row,
|
||||
options="-c timezone=UTC -c statement_timeout=120000")
|
||||
|
||||
|
||||
def snapshot(conn):
|
||||
counts = {}
|
||||
for table in ("users", "chats", "chat_members", "messages", "schema_migrations"):
|
||||
# Имена таблиц — фиксированный список выше, не входные данные клиента.
|
||||
counts[table] = conn.execute("SELECT count(*) AS n FROM " + table).fetchone()["n"]
|
||||
digest = hashlib.sha256()
|
||||
for row in conn.execute("SELECT * FROM messages ORDER BY id"):
|
||||
digest.update((json.dumps(row, default=str, sort_keys=True, ensure_ascii=False) + "\n").encode())
|
||||
samples = []
|
||||
offset = 0
|
||||
for size in SIZES:
|
||||
for number in (offset + 1, offset + size // 2, offset + size):
|
||||
row = conn.execute("SELECT * FROM messages WHERE id=%s", (identifier(f"message:{number}"),)).fetchone()
|
||||
samples.append(json.loads(json.dumps(row, default=str, sort_keys=True, ensure_ascii=False)))
|
||||
offset += size
|
||||
return {"counts": counts, "messages_sha256": digest.hexdigest(), "samples": samples}
|
||||
|
||||
|
||||
def benchmark():
|
||||
db.pool.open(wait=True, timeout=15)
|
||||
try:
|
||||
migrate.apply()
|
||||
finally:
|
||||
db.pool.close()
|
||||
results, plans = [], []
|
||||
with connect() as conn:
|
||||
assert conn.execute("SELECT count(*) AS n FROM users").fetchone()["n"] == 0, "Research requires an empty database"
|
||||
encoded = passwords.hash_password(secrets.token_urlsafe(24))
|
||||
for n in range(1, 11):
|
||||
conn.execute("INSERT INTO users(id,username,display_name,password_hash) VALUES(%s,%s,%s,%s)",
|
||||
(identifier(f"user:{n}"), f"research_{n}", "Synthetic research user", encoded))
|
||||
for n in range(4):
|
||||
chat = identifier(f"chat:{n}")
|
||||
conn.execute("INSERT INTO chats(id,title) VALUES(%s,%s)", (chat, f"Research chat {n}"))
|
||||
for user in (1, 2):
|
||||
conn.execute("INSERT INTO chat_members(chat_id,user_id) VALUES(%s,%s)", (chat, identifier(f"user:{user}")))
|
||||
variants = [("chat_only", "chat_id"), ("history", "chat_id, created_at, id")]
|
||||
for repeat in range(3):
|
||||
order = variants if repeat % 2 == 0 else variants[::-1]
|
||||
for variant, columns in order:
|
||||
conn.execute("TRUNCATE messages")
|
||||
conn.execute("DROP INDEX IF EXISTS messages_history_idx")
|
||||
conn.execute("DROP INDEX IF EXISTS messages_chat_idx")
|
||||
index = "messages_history_idx" if variant == "history" else "messages_chat_idx"
|
||||
conn.execute("CREATE INDEX " + index + " ON messages (" + columns + ")")
|
||||
lsn = conn.execute("SELECT pg_current_wal_insert_lsn() AS lsn").fetchone()["lsn"]
|
||||
started = time.perf_counter()
|
||||
conn.execute("""INSERT INTO messages(id,chat_id,sender_id,text,created_at)
|
||||
SELECT md5('message:'||n)::uuid,
|
||||
md5('chat:'||(CASE WHEN n<=70000 THEN 0 WHEN n<=90000 THEN 1 WHEN n<=99000 THEN 2 ELSE 3 END))::uuid,
|
||||
md5('user:'||(1+n%2))::uuid,
|
||||
'Synthetic message '||n||' '||repeat('x',80),
|
||||
timestamptz '2026-01-01 00:00:00+00' + n*interval '1 microsecond'
|
||||
FROM generate_series(1,100000) AS n""")
|
||||
insert_ms = (time.perf_counter() - started) * 1000
|
||||
wal = conn.execute("SELECT pg_wal_lsn_diff(pg_current_wal_insert_lsn(), %s) AS bytes", (lsn,)).fetchone()["bytes"]
|
||||
conn.execute("VACUUM (ANALYZE) messages")
|
||||
index_bytes = conn.execute("SELECT pg_relation_size(%s::regclass) AS bytes", (index,)).fetchone()["bytes"]
|
||||
result = {"variant": variant, "repeat": repeat + 1, "insert_100000_ms": insert_ms,
|
||||
"wal_bytes": int(wal), "index_bytes": index_bytes, "queries": []}
|
||||
offset = 0
|
||||
for chat, size in enumerate(SIZES):
|
||||
for fraction in (0.1, 0.5, 0.9):
|
||||
number = offset + int(size * fraction)
|
||||
params = (identifier(f"chat:{chat}"), START + timedelta(microseconds=number), identifier(f"message:{number}"), 51)
|
||||
conn.execute(QUERY, params).fetchall() # один прогрев именно этой страницы
|
||||
times = []
|
||||
for _ in range(5):
|
||||
started = time.perf_counter()
|
||||
rows = conn.execute(QUERY, params).fetchall()
|
||||
times.append((time.perf_counter() - started) * 1000)
|
||||
assert len(rows) == 51
|
||||
plan = conn.execute("EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) " + QUERY, params).fetchone()["QUERY PLAN"][0]
|
||||
result["queries"].append({"chat_size": size, "position": fraction, "latency_ms": times,
|
||||
"explain_execution_ms": plan["Execution Time"]})
|
||||
if repeat == 0:
|
||||
plans.append({"variant": variant, "chat_size": size, "position": fraction, "plan": plan})
|
||||
offset += size
|
||||
results.append(result)
|
||||
# Последняя итерация заканчивается production-индексом history.
|
||||
assert conn.execute("SELECT to_regclass('messages_history_idx') AS name").fetchone()["name"]
|
||||
baseline = snapshot(conn)
|
||||
assert baseline["counts"]["messages"] == 100000
|
||||
summary = {}
|
||||
for variant, _ in variants:
|
||||
trials = [row for row in results if row["variant"] == variant]
|
||||
latencies = sorted(t for row in trials for q in row["queries"] for t in q["latency_ms"])
|
||||
summary[variant] = {
|
||||
"insert_100000_median_ms": statistics.median(row["insert_100000_ms"] for row in trials),
|
||||
"index_bytes": trials[-1]["index_bytes"],
|
||||
"wal_bytes_median": statistics.median(row["wal_bytes"] for row in trials),
|
||||
"queries": len(latencies), "errors": 0, "concurrency": 1,
|
||||
"p50_ms": statistics.median(latencies),
|
||||
"p95_ms": latencies[int(0.95 * (len(latencies) - 1))],
|
||||
"p99_ms": latencies[int(0.99 * (len(latencies) - 1))],
|
||||
"successful_sql_rps": len(latencies) / (sum(latencies) / 1000),
|
||||
}
|
||||
return {"postgres": conn.execute("SELECT version() AS version").fetchone()["version"],
|
||||
"messages": 100000, "chat_sizes": SIZES, "repeats": 3, "warmup_per_case": 1,
|
||||
"measurements_per_case": 5, "page_size": 51, "query": QUERY,
|
||||
"summary": summary, "trials": results, "plans": plans, "snapshot": baseline}
|
||||
|
||||
|
||||
def after_backup():
|
||||
with connect() as conn:
|
||||
message_id = identifier("after-backup")
|
||||
conn.execute("INSERT INTO messages(id,chat_id,sender_id,text) VALUES(%s,%s,%s,%s)",
|
||||
(message_id, identifier("chat:0"), identifier("user:1"), "Committed AFTER backup"))
|
||||
return {"id": str(message_id), "source_messages": conn.execute("SELECT count(*) AS n FROM messages").fetchone()["n"]}
|
||||
|
||||
|
||||
def verify():
|
||||
expected = json.load(sys.stdin)
|
||||
with connect(restored=True) as restored, connect() as source:
|
||||
actual = snapshot(restored)
|
||||
assert actual == expected, "Restored data differs from the snapshot"
|
||||
marker = (identifier("after-backup"),)
|
||||
assert restored.execute("SELECT 1 FROM messages WHERE id=%s", marker).fetchone() is None
|
||||
assert source.execute("SELECT 1 FROM messages WHERE id=%s", marker).fetchone() is not None
|
||||
return {"counts": actual["counts"], "messages_sha256": actual["messages_sha256"],
|
||||
"samples_verified": len(actual["samples"]), "all_messages_match": True,
|
||||
"after_backup_message_missing_in_restore": True,
|
||||
"source_messages": source.execute("SELECT count(*) AS n FROM messages").fetchone()["n"]}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if os.environ.get("LAB2_RESEARCH") != "1":
|
||||
raise SystemExit("Use ./scripts/research.sh: an isolated research database is required")
|
||||
operation = sys.argv[1] if len(sys.argv) > 1 else "benchmark"
|
||||
result = {"benchmark": benchmark, "after-backup": after_backup, "verify": verify}[operation]()
|
||||
print(json.dumps(result, ensure_ascii=False, default=str))
|
||||
@@ -0,0 +1,111 @@
|
||||
"""Read stored observations through Grafana's provisioned datasource proxy."""
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import time
|
||||
from urllib.parse import urlencode
|
||||
from urllib.request import urlopen
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
GRAFANA = os.environ.get('GRAFANA_URL')
|
||||
if not GRAFANA:
|
||||
# Also supports an ephemeral host port (GRAFANA_PORT=0) in isolated checks.
|
||||
address = subprocess.check_output(['docker','compose','port','grafana','3000'], cwd=ROOT, text=True).strip()
|
||||
GRAFANA = 'http://' + address
|
||||
GRAFANA = GRAFANA.rstrip('/')
|
||||
|
||||
|
||||
def get(path, **params):
|
||||
url = GRAFANA + path + ('?' + urlencode(params) if params else '')
|
||||
with urlopen(url, timeout=15) as response:
|
||||
return json.load(response)
|
||||
|
||||
|
||||
def proxy(source, path, **params):
|
||||
return get('/api/datasources/proxy/uid/' + source + path, **params)
|
||||
|
||||
|
||||
def query(expr):
|
||||
result = proxy('prometheus', '/api/v1/query', query=expr)
|
||||
assert result['status'] == 'success', result
|
||||
return result['data']['result']
|
||||
|
||||
|
||||
def scalar(expr, default=0):
|
||||
result = query(expr)
|
||||
return float(result[0]['value'][1]) if result else default
|
||||
|
||||
|
||||
def logs(expr, start=None, end=None, limit=1000):
|
||||
return proxy('loki', '/loki/api/v1/query_range', query=expr,
|
||||
start=str(int((start or time.time()-600)*1e9)),
|
||||
end=str(int((end or time.time())*1e9)), limit=limit, direction='forward')
|
||||
|
||||
|
||||
def trace(trace_id):
|
||||
return proxy('tempo', '/api/traces/' + trace_id)
|
||||
|
||||
|
||||
def compose(*args, check=True, timeout=180):
|
||||
return subprocess.run(['docker', 'compose', *args], cwd=ROOT, text=True,
|
||||
capture_output=True, check=check, timeout=timeout)
|
||||
|
||||
|
||||
def sql(statement):
|
||||
return compose('exec', '-T', 'postgres', 'psql', '-U', 'messenger', '-d', 'messenger', '-At', '-c', statement).stdout.strip()
|
||||
|
||||
|
||||
def wait(predicate, timeout=60):
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
result = predicate()
|
||||
if result:
|
||||
return result
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
time.sleep(1)
|
||||
raise AssertionError('Timed out waiting for observed condition')
|
||||
|
||||
|
||||
def save(name, value):
|
||||
path = ROOT / 'evidence' / name
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(value, ensure_ascii=False, indent=2) + '\n')
|
||||
|
||||
|
||||
def spans(value):
|
||||
result = []
|
||||
def walk(item, service=''):
|
||||
if isinstance(item, dict):
|
||||
if 'resource' in item:
|
||||
for attr in item['resource'].get('attributes', []):
|
||||
if attr['key'] == 'service.name':
|
||||
service = attr['value']['stringValue']
|
||||
if 'spanId' in item and 'name' in item:
|
||||
result.append({'service': service, **item})
|
||||
for child in item.values():
|
||||
walk(child, service)
|
||||
elif isinstance(item, list):
|
||||
for child in item:
|
||||
walk(child, service)
|
||||
walk(value)
|
||||
return result
|
||||
|
||||
|
||||
def check_dashboard():
|
||||
dashboard = get('/api/dashboards/uid/messenger-lab6')['dashboard']
|
||||
checked = []
|
||||
for panel in dashboard['panels']:
|
||||
source = panel.get('datasource', {}).get('uid')
|
||||
for target in panel.get('targets', []):
|
||||
if source == 'prometheus':
|
||||
query(target['expr'])
|
||||
elif source == 'loki':
|
||||
assert logs(target['expr'])['status'] == 'success'
|
||||
elif source == 'tempo':
|
||||
# Grafana v12 routes traceql frontend searches to this proxy endpoint.
|
||||
assert proxy('tempo', '/api/search', q=target['query'], limit=20).get('traces')
|
||||
checked.append({'panel': panel['title'], 'source': source, 'passed': True})
|
||||
return {'passed': True, 'panels': checked}
|
||||
@@ -0,0 +1,99 @@
|
||||
"""Создаёт изолированный стенд, измеряет SQL, делает dump и restore; пишет реальные результаты."""
|
||||
from datetime import datetime, timezone
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import platform
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from uuid import uuid4
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
project = "lab2-research-" + uuid4().hex[:10]
|
||||
env = dict(os.environ, COMPOSE_PROJECT_NAME=project)
|
||||
command = ["docker", "compose", "--profile", "research"]
|
||||
|
||||
|
||||
def compose(*args, stdin=None, stdout=subprocess.PIPE):
|
||||
return subprocess.run([*command, *args], cwd=ROOT, env=env, input=stdin,
|
||||
stdout=stdout, stderr=subprocess.PIPE, check=True, timeout=600)
|
||||
|
||||
|
||||
def run_module(operation, stdin=None):
|
||||
result = compose("run", "--rm", "--no-deps", "-T", "-e", "LAB2_RESEARCH=1", "api", "python", "-m", "research.benchmark", operation, stdin=stdin)
|
||||
return json.loads(result.stdout)
|
||||
|
||||
|
||||
def wait_database(service):
|
||||
for _ in range(60):
|
||||
cid = compose("ps", "-q", service).stdout.decode().strip()
|
||||
state = subprocess.run(["docker", "inspect", "--format", "{{.State.Health.Status}}", cid], capture_output=True, text=True, check=True)
|
||||
if state.stdout.strip() == "healthy":
|
||||
return
|
||||
time.sleep(1)
|
||||
raise RuntimeError(service + " did not become healthy")
|
||||
|
||||
|
||||
def main():
|
||||
evidence = ROOT / "evidence"
|
||||
evidence.mkdir(exist_ok=True)
|
||||
backups = ROOT / "backups"
|
||||
backups.mkdir(exist_ok=True)
|
||||
dump = backups / (project + ".dump")
|
||||
metadata = {"utc": datetime.now(timezone.utc).isoformat(), "project": project,
|
||||
"command": "./scripts/research.sh", "host": platform.system() + " " + platform.machine(),
|
||||
"commit": subprocess.run(["git", "rev-parse", "--short", "HEAD"], cwd=ROOT, capture_output=True, text=True).stdout.strip() + " + working tree",
|
||||
"postgres_limits": {"cpus": 1, "memory_mib": 512}, "api_runner_limits": {"cpus": 1, "memory_mib": 256}}
|
||||
started = time.perf_counter()
|
||||
try:
|
||||
print("Research project:", project, flush=True)
|
||||
compose("up", "-d", "postgres", "postgres-restore")
|
||||
wait_database("postgres")
|
||||
wait_database("postgres-restore")
|
||||
print("Measuring two indexes on the same 100000 messages, three repeats...", flush=True)
|
||||
result = run_module("benchmark")
|
||||
plans = result.pop("plans")
|
||||
(evidence / "query-plans.json").write_text(json.dumps(plans, ensure_ascii=False, indent=2))
|
||||
source_id = compose("ps", "-q", "postgres").stdout.decode().strip()
|
||||
restore_id = compose("ps", "-q", "postgres-restore").stdout.decode().strip()
|
||||
assert source_id != restore_id
|
||||
for label, cid in (("source", source_id), ("restored", restore_id)):
|
||||
inspected = json.loads(subprocess.run(["docker", "inspect", cid], capture_output=True, check=True).stdout)[0]
|
||||
metadata[label] = {"container": cid, "image": inspected["Image"],
|
||||
"volumes": [m["Name"] for m in inspected["Mounts"] if m["Type"] == "volume"]}
|
||||
assert metadata["source"]["volumes"] != metadata["restored"]["volumes"]
|
||||
metadata["docker"] = subprocess.run(["docker", "info", "--format", "{{.ServerVersion}} {{.Architecture}} CPUs={{.NCPU}} RAM={{.MemTotal}}"], capture_output=True, text=True, check=True).stdout.strip()
|
||||
metadata["stats_after_queries"] = subprocess.run(["docker", "stats", "--no-stream", "--format", "{{json .}}", source_id], capture_output=True, text=True, check=True).stdout.strip()
|
||||
backup_started = time.perf_counter()
|
||||
with dump.open("wb") as output:
|
||||
compose("exec", "-T", "postgres", "pg_dump", "-U", "messenger", "-d", "messenger", "--format=custom", stdout=output)
|
||||
backup_seconds = time.perf_counter() - backup_started
|
||||
marker = run_module("after-backup")
|
||||
restore_started = time.perf_counter()
|
||||
# Dump подаётся через stdin, без вывода данных или credentials в журнал.
|
||||
with dump.open("rb") as input_file:
|
||||
subprocess.run([*command, "exec", "-T", "postgres-restore", "pg_restore", "-U", "messenger", "-d", "messenger",
|
||||
"--exit-on-error", "--single-transaction", "--no-owner"], cwd=ROOT, env=env, stdin=input_file,
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, check=True, timeout=300)
|
||||
restore_seconds = time.perf_counter() - restore_started
|
||||
verification_started = time.perf_counter()
|
||||
verification = run_module("verify", stdin=json.dumps(result["snapshot"]).encode())
|
||||
result.update(metadata)
|
||||
result["backup"] = {"file": "backups/" + dump.name, "bytes": dump.stat().st_size, "seconds": backup_seconds}
|
||||
result["restore"] = {"seconds": restore_seconds, "verification_seconds": time.perf_counter() - verification_started,
|
||||
"verification": verification, "post_backup_insert": marker}
|
||||
result["total_seconds"] = time.perf_counter() - started
|
||||
(evidence / "research.json").write_text(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
print(json.dumps(result["summary"], ensure_ascii=False, indent=2), flush=True)
|
||||
print(f"PASS: backup {backup_seconds:.2f}s; restore {restore_seconds:.2f}s; 100000 messages, all hashes and 12 samples match", flush=True)
|
||||
print("PASS: restored into a different container and fresh volume; one post-backup commit is absent as expected", flush=True)
|
||||
print("Results: evidence/research.json, evidence/query-plans.json; dump stays outside Git in backups/", flush=True)
|
||||
finally:
|
||||
compose("down", "--volumes", "--timeout", "10")
|
||||
print("Removed only temporary research project:", project, flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+207
@@ -0,0 +1,207 @@
|
||||
"""Track 2: real-time SLO alert experiment against the current synthetic lab6 stack."""
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from datetime import datetime, timezone
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import statistics
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from uuid import uuid4
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path[:0] = [str(ROOT / 'tests'), str(ROOT)]
|
||||
from client import Client
|
||||
from research.observations import compose, logs, proxy, query, save, spans, sql, trace, wait
|
||||
|
||||
|
||||
def worker_evidence(requests, phases, started):
|
||||
stopped = next(p for p in phases if p['name'] == 'worker_stopped')
|
||||
restored = next(p for p in phases if p['name'] == 'after_worker_recovery')
|
||||
upload = next(r for r in requests if r['kind'] == 'upload' and r['status'] == 202
|
||||
and stopped['active_at'] + 30 < r['utc'] < stopped['end'] - 30)
|
||||
stored = trace(upload['trace_id'])
|
||||
chain = spans(stored)
|
||||
consumers = [s for s in chain if s['name'] == 'attachments process']
|
||||
assert consumers, 'Accepted upload must eventually have a real consumer span'
|
||||
consumed_at = min(int(s['startTimeUnixNano']) / 1e9 for s in consumers)
|
||||
assert consumed_at >= restored['active_at'] - 2, 'Job was consumed while worker was stopped'
|
||||
assert {'s3.get_object', 's3.put_object', 'postgresql.UPDATE'} <= {s['name'] for s in chain}
|
||||
save('lab6-worker-incident-trace.json', stored)
|
||||
save('lab6-worker-incident-logs.json', logs(
|
||||
'{service_name=~"messenger-api|messenger-worker"} |= "' + upload['trace_id'] + '"', start=started))
|
||||
return {'trace_id': upload['trace_id'], 'request_id': upload['request_id'],
|
||||
'accept_to_consume_seconds': round(consumed_at - upload['utc'], 2),
|
||||
'consumer_after_restore': True}
|
||||
|
||||
|
||||
def main():
|
||||
client = Client()
|
||||
alice, _, _, chat = client.bootstrap()
|
||||
png = (ROOT / 'tests/fixtures/sample.png').read_bytes()
|
||||
started = time.time()
|
||||
stop = threading.Event()
|
||||
lock = threading.Lock()
|
||||
requests, samples, phases = [], [], []
|
||||
skipped = 0
|
||||
phase_name = 'baseline'
|
||||
pending = set()
|
||||
|
||||
def send(kind):
|
||||
now = time.time()
|
||||
name_at_start = phase_name
|
||||
try:
|
||||
if kind == 'message':
|
||||
r = client.call('POST', '/api/v1/chats/' + chat['id'] + '/messages', [201,500,503],
|
||||
actor=alice,body={'text':'synthetic SLO experiment'},headers={'Idempotency-Key':uuid4().hex},schema='Message')
|
||||
else:
|
||||
r = client.call('POST', '/api/v1/chats/' + chat['id'] + '/attachments?filename=synthetic.png', [202,500,503],
|
||||
actor=alice,raw=png,headers={'Content-Type':'image/png'},schema='Attachment')
|
||||
row = {'utc':now,'phase':name_at_start,'kind':kind,'status':r['status'],
|
||||
'duration_seconds':r['elapsed'],'trace_id':r['headers'].get('X-Trace-Id',''),
|
||||
'request_id':r['headers']['X-Request-Id']}
|
||||
except Exception as error:
|
||||
row = {'utc':now,'phase':name_at_start,'kind':kind,'status':0,'error_type':type(error).__name__,
|
||||
'duration_seconds':time.time()-now}
|
||||
with lock:
|
||||
requests.append(row)
|
||||
|
||||
def load():
|
||||
nonlocal skipped
|
||||
tick = 0
|
||||
next_tick = time.monotonic()
|
||||
with ThreadPoolExecutor(max_workers=16) as pool:
|
||||
while not stop.is_set():
|
||||
pending.difference_update({f for f in pending if f.done()})
|
||||
kinds = ['message'] + (['upload'] if tick % 4 == 0 else [])
|
||||
for kind in kinds:
|
||||
if len(pending) >= 16:
|
||||
skipped += 1
|
||||
else:
|
||||
pending.add(pool.submit(send, kind))
|
||||
tick += 1
|
||||
next_tick += .5
|
||||
stop.wait(max(0,next_tick-time.monotonic()))
|
||||
|
||||
def phase(name, seconds, action=None):
|
||||
nonlocal phase_name
|
||||
phase_name = name
|
||||
row = {'name':name,'start':time.time(),'planned_seconds':seconds}
|
||||
print('PHASE:', name, seconds, 'seconds', flush=True)
|
||||
if action:
|
||||
action()
|
||||
row['active_at'] = time.time()
|
||||
deadline = row['start'] + seconds
|
||||
while time.time() < deadline:
|
||||
alerts = proxy('prometheus','/api/v1/alerts')['data']['alerts']
|
||||
samples.append({'utc':time.time(),'phase':name,'alerts':[
|
||||
{'name':a['labels']['alertname'],'state':a['state'],'activeAt':a['activeAt']} for a in alerts],
|
||||
'queue_depth':query('max(messenger_queue_depth)'),
|
||||
'queue_age':query('max(messenger_queue_oldest_age_seconds)'),
|
||||
'api_bad_30s':query('lab6:api_bad_ratio:30s'),
|
||||
'api_bad_2m':query('lab6:api_bad_ratio:2m')})
|
||||
time.sleep(min(5,max(0,deadline-time.time())))
|
||||
row['end'] = time.time()
|
||||
phases.append(row)
|
||||
|
||||
def db_lock(seconds):
|
||||
# Named session and bounded pg_sleep: the fault self-releases even if the driver exits.
|
||||
p = subprocess.Popen(['docker','compose','exec','-T','-e','PGAPPNAME=lab6-incident','postgres',
|
||||
'psql','-v','ON_ERROR_STOP=1','-U','messenger','-d','messenger','-c',
|
||||
f"BEGIN; LOCK TABLE messages IN ACCESS EXCLUSIVE MODE; SELECT pg_sleep({seconds}); COMMIT;"],
|
||||
cwd=ROOT, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
wait(lambda: sql("SELECT count(*) FROM pg_stat_activity WHERE application_name='lab6-incident' AND wait_event='PgSleep'") == '1',timeout=15)
|
||||
return p
|
||||
|
||||
thread = threading.Thread(target=load)
|
||||
result = {'utc':datetime.now(timezone.utc).isoformat(),'hypothesis':
|
||||
'Multiwindow burn alerts ignore a 4s spike, detect sustained DB/worker degradation and resolve after recovery.',
|
||||
'profile':{'message_arrival_per_second':2,'image_arrival_per_second':.5,'max_inflight':16,
|
||||
'sampling':1.0,'retries':0,'slo_window_seconds':3600,'api_slo':.99,'ready_slo':.95,'ready_deadline_seconds':30},
|
||||
'docker':subprocess.check_output(['docker','info','--format','{{.ServerVersion}} {{.Architecture}} CPUs={{.NCPU}} RAM={{.MemTotal}}'],text=True).strip()}
|
||||
thread.start()
|
||||
try:
|
||||
phase('baseline',120)
|
||||
phase('short_db_spike',4,lambda:db_lock(4))
|
||||
phase('after_short_spike',60)
|
||||
phase('sustained_db_lock',90,lambda:db_lock(90))
|
||||
# Read only saved signals to retain the diagnosis: error trace + correlated log.
|
||||
failing = next(r for r in reversed(requests) if r['status'] >= 500 and r.get('trace_id'))
|
||||
time.sleep(3)
|
||||
saved_trace = trace(failing['trace_id'])
|
||||
save('lab6-db-incident-trace.json',saved_trace)
|
||||
save('lab6-db-incident-logs.json',logs('{service_name="messenger-api"} |= "'+failing['trace_id']+'"',start=started))
|
||||
result['db_diagnosis'] = {'trace_id':failing['trace_id'],'request_id':failing['request_id'],
|
||||
'slow_error_spans':[{'name':s['name'],'status':s.get('status'),
|
||||
'duration_ms':(int(s['endTimeUnixNano'])-int(s['startTimeUnixNano']))/1e6,
|
||||
'attributes':s.get('attributes',[])} for s in spans(saved_trace) if s.get('status',{}).get('code') in [2,'STATUS_CODE_ERROR']]}
|
||||
phase('after_db_recovery',100)
|
||||
phase('worker_stopped',120,lambda:compose('stop','--timeout','5','worker'))
|
||||
phase('after_worker_recovery',120,lambda:compose('start','worker'))
|
||||
result['worker_diagnosis'] = worker_evidence(requests, phases, started)
|
||||
assert all(r['status'] in {201,202,500,503} for r in requests), 'Transport/client errors occurred'
|
||||
baseline_firing = {a['name'] for s in samples if s['phase']=='baseline' for a in s['alerts'] if a['state']=='firing'}
|
||||
assert not baseline_firing, baseline_firing
|
||||
spike_firing = {a['name'] for s in samples if s['phase'] in {'short_db_spike','after_short_spike'} for a in s['alerts'] if a['state']=='firing'}
|
||||
assert not any('Burn' in a for a in spike_firing), spike_firing
|
||||
spike_peak = max([float(v['value'][1]) for s in samples
|
||||
if s['phase'] in {'short_db_spike','after_short_spike'} for v in s['api_bad_30s']] or [0])
|
||||
assert spike_peak > 0, 'The real short spike must be visible in the stored error ratio'
|
||||
result['short_spike_observed_error_ratio_peak'] = spike_peak
|
||||
firing = {a['name'] for s in samples for a in s['alerts'] if a['state']=='firing'}
|
||||
expected = {'APIAvailabilityFastBurn','APIAvailabilitySlowBurn','AttachmentReadyFastBurn',
|
||||
'AttachmentReadySlowBurn','AttachmentQueueLag','APIHighErrorRate'}
|
||||
assert expected <= firing, expected-firing
|
||||
assert not [a for a in samples[-1]['alerts'] if a['state']=='firing'], samples[-1]
|
||||
result['passed']=True
|
||||
result['false_positives']={'baseline':len(baseline_firing),'burn_alerts_after_short_spike':0,
|
||||
'single_window_alerts_after_short_spike':sorted(spike_firing)}
|
||||
finally:
|
||||
stop.set(); thread.join(timeout=20)
|
||||
# End the bounded lock early on interruption; restore the worker in all cases.
|
||||
sql("SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE application_name='lab6-incident'")
|
||||
compose('start','worker')
|
||||
ended=time.time()
|
||||
result.update({'start':started,'end':ended,'duration_seconds':ended-started,'phases':phases,
|
||||
'samples':samples,'requests':requests,'skipped_arrivals':skipped})
|
||||
summary={}
|
||||
for p in phases:
|
||||
rows=[r for r in requests if p['start']<=r['utc']<p['end']]
|
||||
latencies=sorted(r['duration_seconds']*1000 for r in rows)
|
||||
summary[p['name']]={'requests':len(rows),'5xx':sum(r['status']>=500 for r in rows),
|
||||
'successful_rps':sum(200<=r['status']<300 for r in rows)/(p['end']-p['start']),
|
||||
**{f'p{q}_ms':round(latencies[min(len(latencies)-1,int(len(latencies)*q/100))],3)
|
||||
for q in [50,95,99] if latencies}}
|
||||
result['phase_summary']=summary
|
||||
# Actual detection and resolution times, measured from observed Prometheus states.
|
||||
timings={}
|
||||
for prefix,incident,recovery in [('API','sustained_db_lock','after_db_recovery'),('Attachment','worker_stopped','after_worker_recovery')]:
|
||||
ip=next((p for p in phases if p['name']==incident),None)
|
||||
rp=next((p for p in phases if p['name']==recovery),None)
|
||||
if not ip or not rp: continue
|
||||
names={a['name'] for s in samples for a in s['alerts'] if a['name'].startswith(prefix) and a['state']=='firing'}
|
||||
for name in names:
|
||||
fired=[s['utc'] for s in samples if s['utc']>=ip['start'] and any(a['name']==name and a['state']=='firing' for a in s['alerts'])]
|
||||
if not fired: continue
|
||||
resolved=next((s['utc'] for s in samples if s['utc']>=max(rp['active_at'],fired[0]) and not any(a['name']==name and a['state']=='firing' for a in s['alerts'])),None)
|
||||
timings[name]={'detection_seconds':round(fired[0]-ip['active_at'],2),
|
||||
'resolution_after_restore_seconds':round(resolved-rp['active_at'],2) if resolved else None}
|
||||
result['alert_timings']=timings
|
||||
expressions={'http_rps':'sum(rate(messenger_http_requests_total[1m]))',
|
||||
'http_p95':'histogram_quantile(0.95,sum by(le)(rate(messenger_http_duration_seconds_bucket[1m])))',
|
||||
'queue_depth':'max(messenger_queue_depth)','queue_age':'max(messenger_queue_oldest_age_seconds)',
|
||||
'db_pool':'sum by(state)(messenger_db_pool)','worker_busy':'sum(messenger_worker_busy)',
|
||||
'worker_capacity':'sum(messenger_worker_capacity)',
|
||||
'cpu':'sum by(service_name)(rate(messenger_process_cpu_time_seconds_total[1m]))',
|
||||
'rss':'sum by(service_name)(messenger_process_memory_bytes)',
|
||||
'api_bad':'lab6:api_bad_ratio:2m','ready_bad':'lab6:ready_bad_ratio:2m','alerts':'ALERTS{alertstate="firing"}'}
|
||||
series={name:proxy('prometheus','/api/v1/query_range',query=expr,start=started,end=ended,step=5) for name,expr in expressions.items()}
|
||||
save('lab6-slo-series.json',series)
|
||||
save('lab6-slo-research.json',result)
|
||||
print(json.dumps({k:v for k,v in result.items() if k not in {'requests','samples'}},ensure_ascii=False,indent=2),flush=True)
|
||||
|
||||
|
||||
if __name__=='__main__':
|
||||
main()
|
||||
Executable
+6
@@ -0,0 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$project_dir"
|
||||
python3 scripts/init_local.py
|
||||
docker compose build "$@" api
|
||||
Executable
+11
@@ -0,0 +1,11 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$project_dir"
|
||||
if [[ ! -x .venv/bin/python ]]; then
|
||||
echo "Создай .venv и установи requirements.txt по REPORT.md" >&2
|
||||
exit 1
|
||||
fi
|
||||
./scripts/build.sh
|
||||
mkdir -p evidence
|
||||
.venv/bin/python -u student-tests/lab5_pipeline.py 2>&1 | sed -E 's/[[:blank:]]+$//' | tee evidence/lab5-checks.txt
|
||||
Executable
+29
@@ -0,0 +1,29 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$project_dir"
|
||||
mkdir -p evidence
|
||||
./scripts/build.sh
|
||||
# Inherited grade-4 lab5 criteria, on a separate temporary project and fresh volumes.
|
||||
LAB5_EVIDENCE_FILE=evidence/lab6-lab5-regression.json \
|
||||
.venv/bin/python -u student-tests/lab5_pipeline.py 2>&1 | tee evidence/lab6-lab5-regression.txt
|
||||
./scripts/start.sh
|
||||
make check validate test PYTHON=.venv/bin/python 2>&1 | tee evidence/lab6-api-checks.txt
|
||||
.venv/bin/python -m unittest discover -s student-tests -v 2>&1 | tee -a evidence/lab6-api-checks.txt
|
||||
docker run --rm --entrypoint /bin/promtool -v "$project_dir:/work:ro" -w /work \
|
||||
prom/prometheus:v3.4.1 test rules student-tests/promtool-slo.yaml | tee evidence/lab6-promtool.txt
|
||||
# Only this short check enables the already inherited one-shot worker failpoint.
|
||||
trap 'ENABLE_FAULT_INJECTION=0 docker compose up -d --no-deps worker >/dev/null 2>&1' EXIT
|
||||
ENABLE_FAULT_INJECTION=1 docker compose up -d --no-deps worker
|
||||
.venv/bin/python -u student-tests/lab6_observability.py 2>&1 | tee evidence/lab6-checks.txt
|
||||
ENABLE_FAULT_INJECTION=0 docker compose up -d --no-deps worker
|
||||
.venv/bin/python -u student-tests/verify_restart.py 2>&1 | tee evidence/lab6-restart.txt
|
||||
.venv/bin/python - <<'PY'
|
||||
import json
|
||||
from research.observations import ROOT, spans, trace, wait
|
||||
evidence=json.loads((ROOT/'evidence/lab6-checks.json').read_text())
|
||||
trace_id=evidence['checks']['retry_trace_and_logs']['trace_id']
|
||||
wait(lambda: len(spans(trace(trace_id))) >= 10)
|
||||
print('PASS: trace remains queryable after restarting all telemetry stores with their volumes')
|
||||
PY
|
||||
.venv/bin/python -u research/slo.py 2>&1 | tee evidence/lab6-slo-research.txt
|
||||
Executable
+17
@@ -0,0 +1,17 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$project_dir"
|
||||
|
||||
if [[ ! -x .venv/bin/python ]]; then
|
||||
echo "Не хватает .venv" >&2
|
||||
echo "python3 -m venv .venv" >&2
|
||||
echo ".venv/bin/python -m pip install -r requirements.txt" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
export BASE_URL="${BASE_URL:-http://localhost:8080}"
|
||||
echo "Проверяем сервер: $BASE_URL"
|
||||
make check test PYTHON="$project_dir/.venv/bin/python" BASE_URL="$BASE_URL"
|
||||
.venv/bin/python -m unittest discover -s student-tests -v
|
||||
Executable
+28
@@ -0,0 +1,28 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
base_url="${BASE_URL:-http://localhost:8080}"
|
||||
suffix="$(date +%s)_$RANDOM"
|
||||
# Только синтетические пароли; токены остаются в памяти shell и не выводятся.
|
||||
password="Demo-password-$suffix"
|
||||
register() {
|
||||
local username="$1" display_name="$2"
|
||||
jq -nc --arg u "$username" --arg d "$display_name" --arg p "$password" \
|
||||
'{username:$u, display_name:$d, password:$p}' |
|
||||
curl -fsS -H 'Content-Type: application/json' --data-binary @- "$base_url/api/v1/users" | jq -er '.id'
|
||||
}
|
||||
login() {
|
||||
curl -fsS -X POST -u "$1:$password" "$base_url/api/v1/auth/sessions" | jq -er '.token'
|
||||
}
|
||||
alice_id="$(register "alice_$suffix" Алиса)"
|
||||
bob_id="$(register "bob_$suffix" Боб)"
|
||||
alice_token="$(login "alice_$suffix")"
|
||||
bob_token="$(login "bob_$suffix")"
|
||||
chat_id="$(jq -nc --arg a "$alice_id" --arg b "$bob_id" '{title:"Учебный чат",member_ids:[$a,$b]}' |
|
||||
curl -fsS -H "Authorization: Bearer $alice_token" -H 'Content-Type: application/json' \
|
||||
--data-binary @- "$base_url/api/v1/chats" | jq -er '.id')"
|
||||
for token in "$alice_token" "$bob_token"; do
|
||||
curl -fsS -H "Authorization: Bearer $token" -H 'Content-Type: application/json' \
|
||||
-H "Idempotency-Key: demo-$RANDOM-$RANDOM" \
|
||||
-d '{"text":"Привет из lab5! 👋"}' "$base_url/api/v1/chats/$chat_id/messages" | jq .
|
||||
done
|
||||
curl -fsS -H "Authorization: Bearer $bob_token" "$base_url/api/v1/chats/$chat_id/messages" | jq .
|
||||
@@ -0,0 +1,62 @@
|
||||
"""Создаёт локальное окружение и secrets; существующие значения сохраняет."""
|
||||
from pathlib import Path
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import secrets
|
||||
import shutil
|
||||
from urllib.parse import quote
|
||||
|
||||
|
||||
root = Path(__file__).resolve().parents[1]
|
||||
if not (root / ".env").exists():
|
||||
shutil.copyfile(root / ".env.example", root / ".env")
|
||||
directory = root / "secrets"
|
||||
directory.mkdir(exist_ok=True, mode=0o700)
|
||||
directory.chmod(0o700)
|
||||
password_file = directory / "postgres_password"
|
||||
if not password_file.exists():
|
||||
password_file.write_text(secrets.token_hex(24))
|
||||
password = password_file.read_text().strip()
|
||||
values = {
|
||||
"database_url": f"postgresql://messenger:{quote(password, safe='')}@postgres:5432/messenger",
|
||||
"cache_url": "redis://valkey:6379/0",
|
||||
"broker_password": secrets.token_hex(24),
|
||||
"s3_root_user": "root-" + secrets.token_hex(8),
|
||||
"s3_root_password": secrets.token_urlsafe(32),
|
||||
"s3_access_key_id": "api-" + secrets.token_hex(8),
|
||||
"s3_secret_access_key": secrets.token_urlsafe(32),
|
||||
}
|
||||
for name, value in values.items():
|
||||
path = directory / name
|
||||
if not path.exists():
|
||||
path.write_text(value)
|
||||
broker_url = directory / "broker_url"
|
||||
if not broker_url.exists():
|
||||
broker_url.write_text("redis://:" + (directory / "broker_password").read_text().strip() + "@broker:6379/0")
|
||||
broker_url.chmod(0o444)
|
||||
dotenv = dict(line.split("=", 1) for line in (root / ".env").read_text().splitlines()
|
||||
if line.strip() and not line.lstrip().startswith("#") and "=" in line)
|
||||
bucket = os.environ.get("S3_BUCKET", dotenv.get("S3_BUCKET", "messenger-attachments"))
|
||||
if re.fullmatch(r"[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]", bucket) is None:
|
||||
raise ValueError("Недопустимое имя S3 bucket")
|
||||
def local_secret(name):
|
||||
return (directory / name).read_text().strip()
|
||||
config = {"accounts": [{"id": "messenger-api", "displayName": "messenger-api"}], "identities": [
|
||||
{"name": "bootstrap", "credentials": [{"accessKey": local_secret("s3_root_user"),
|
||||
"secretKey": local_secret("s3_root_password")}], "actions": ["Admin"]},
|
||||
{"name": "messenger-api", "account": {"id": "messenger-api", "displayName": "messenger-api"},
|
||||
"credentials": [{"accessKey": local_secret("s3_access_key_id"), "secretKey": local_secret("s3_secret_access_key")}],
|
||||
"actions": ["Read:" + bucket, "List:" + bucket, "Write:" + bucket + "/attachments/*"]},
|
||||
]}
|
||||
config_file = directory / "s3_config"
|
||||
encoded_config = json.dumps(config)
|
||||
if not config_file.exists() or config_file.read_text() != encoded_config:
|
||||
temporary = directory / "s3_config.tmp"
|
||||
temporary.write_text(encoded_config)
|
||||
temporary.chmod(0o444)
|
||||
temporary.replace(config_file)
|
||||
# Каталог 0700 защищает файлы на хосте; 0444 позволяет USER 10001 читать bind-mounted secret.
|
||||
for name in ("postgres_password", "s3_config", *values):
|
||||
(directory / name).chmod(0o444)
|
||||
print("Local .env and secrets are ready; values are not printed")
|
||||
Executable
+6
@@ -0,0 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$project_dir"
|
||||
./scripts/build.sh
|
||||
docker compose run --rm migrate
|
||||
@@ -0,0 +1,90 @@
|
||||
"""Управление репликами только текущего Compose project через Docker CLI."""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import time
|
||||
from urllib.request import urlopen
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def command(*args):
|
||||
return subprocess.check_output(args, cwd=ROOT, text=True, timeout=120).strip()
|
||||
|
||||
|
||||
def replicas():
|
||||
ids = command("docker", "compose", "ps", "-q", "api").split()
|
||||
return json.loads(command("docker", "inspect", *ids)) if ids else []
|
||||
|
||||
|
||||
def base_url():
|
||||
config = json.loads(command("docker", "compose", "config", "--format", "json"))
|
||||
port = config["services"]["lb"]["ports"][0]["published"]
|
||||
return os.environ.get("BASE_URL", "http://127.0.0.1:" + str(port)).rstrip("/")
|
||||
|
||||
|
||||
def wait_ready():
|
||||
url = base_url() + "/health/ready"
|
||||
deadline = time.monotonic() + 90
|
||||
seen = set()
|
||||
while time.monotonic() < deadline:
|
||||
current = replicas()
|
||||
expected = {info["Config"]["Hostname"] for info in current}
|
||||
if expected and all(info["State"].get("Health", {}).get("Status") == "healthy" for info in current):
|
||||
try:
|
||||
with urlopen(url, timeout=2) as response:
|
||||
if response.status == 200:
|
||||
seen.add(response.headers.get("X-Instance-Id"))
|
||||
if expected <= seen:
|
||||
print("API ready through LB:", ", ".join(sorted(expected)), flush=True)
|
||||
return
|
||||
except OSError:
|
||||
pass
|
||||
time.sleep(0.2)
|
||||
raise RuntimeError("Не все работающие API-реплики готовы через LB за 90 секунд")
|
||||
|
||||
|
||||
def stop_one():
|
||||
target = os.environ.get("TARGET_INSTANCE_ID")
|
||||
if not target:
|
||||
raise RuntimeError("Задайте TARGET_INSTANCE_ID из X-Instance-Id")
|
||||
current = replicas()
|
||||
matches = [info for info in current if info["Config"]["Hostname"] == target]
|
||||
if len(current) < 2 or len(matches) != 1:
|
||||
raise RuntimeError("Нужны минимум две работающие реплики и ровно одна выбранная")
|
||||
selected = matches[0]
|
||||
command("docker", "stop", "--time", "8", selected["Id"])
|
||||
print("Stopped API:", target, "container=" + selected["Id"], flush=True)
|
||||
|
||||
|
||||
def rolling():
|
||||
wait_ready()
|
||||
originals = replicas()
|
||||
count = len(originals)
|
||||
if count < 2:
|
||||
raise RuntimeError("Для поочерёдной замены нужны минимум две API-реплики")
|
||||
for old in originals:
|
||||
active = replicas()
|
||||
healthy = [info for info in active if info["Id"] != old["Id"]
|
||||
and info["State"].get("Health", {}).get("Status") == "healthy"]
|
||||
if not healthy:
|
||||
raise RuntimeError("Нет готовой реплики, которая продолжит обслуживать запросы")
|
||||
print("Replacing API:", old["Config"]["Hostname"], flush=True)
|
||||
command("docker", "stop", "--time", "8", old["Id"])
|
||||
command("docker", "rm", old["Id"])
|
||||
# --no-recreate сохраняет остальные процессы; Compose создаёт только недостающий.
|
||||
command("docker", "compose", "up", "-d", "--no-deps", "--no-recreate", "--scale", f"api={count}", "api")
|
||||
wait_ready()
|
||||
assert len(replicas()) == count
|
||||
final = replicas()
|
||||
assert not ({info["Id"] for info in originals} & {info["Id"] for info in final})
|
||||
print("PASS: all API containers replaced sequentially; stores and LB kept running", flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("action", choices=["wait", "stop", "rolling"])
|
||||
action = parser.parse_args().action
|
||||
{"wait": wait_ready, "stop": stop_one, "rolling": rolling}[action]()
|
||||
Executable
+7
@@ -0,0 +1,7 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$project_dir"
|
||||
./scripts/build.sh
|
||||
mkdir -p evidence
|
||||
python3 -u research/run.py 2>&1 | tee evidence/research.txt
|
||||
Executable
+7
@@ -0,0 +1,7 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$project_dir"
|
||||
python3 scripts/wait-observability.py
|
||||
mkdir -p evidence
|
||||
.venv/bin/python -u research/slo.py 2>&1 | tee evidence/lab6-slo-research.txt
|
||||
Executable
+7
@@ -0,0 +1,7 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$project_dir"
|
||||
docker compose restart --timeout 8 postgres valkey s3-storage s3 broker api lb cleanup publisher worker otel-collector prometheus loki tempo grafana
|
||||
./scripts/wait-ready.sh
|
||||
python3 scripts/wait-observability.py
|
||||
Executable
+7
@@ -0,0 +1,7 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$project_dir"
|
||||
# Сначала собираем образ и применяем только обратно совместимые миграции.
|
||||
./scripts/migrate.sh
|
||||
exec python3 scripts/replicas.py rolling
|
||||
Executable
+5
@@ -0,0 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
# Локальный стенд: HAProxy, две API-реплики, PostgreSQL и Valkey.
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
exec "$project_dir/scripts/start.sh"
|
||||
Executable
+5
@@ -0,0 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$project_dir"
|
||||
docker compose start worker
|
||||
Executable
+8
@@ -0,0 +1,8 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$project_dir"
|
||||
python3 scripts/init_local.py
|
||||
docker compose up -d --build lb cleanup publisher worker otel-collector prometheus grafana
|
||||
./scripts/wait-ready.sh
|
||||
python3 scripts/wait-observability.py
|
||||
Executable
+5
@@ -0,0 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$project_dir"
|
||||
exec python3 scripts/replicas.py stop
|
||||
Executable
+5
@@ -0,0 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$project_dir"
|
||||
docker compose stop --timeout 8 worker
|
||||
Executable
+6
@@ -0,0 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$project_dir"
|
||||
docker compose down --timeout 10
|
||||
echo "Контейнеры удалены; именованные volumes сохранены"
|
||||
@@ -0,0 +1,26 @@
|
||||
"""Readiness of provisioned telemetry, queried through the local Grafana proxy."""
|
||||
from pathlib import Path
|
||||
import sys
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
from research.observations import get, proxy, query, wait
|
||||
|
||||
|
||||
def ready():
|
||||
assert get('/api/health')['database'] == 'ok'
|
||||
assert get('/api/dashboards/uid/messenger-lab6')['dashboard']['uid'] == 'messenger-lab6'
|
||||
assert len(get('/api/datasources')) == 3
|
||||
assert len(query('up == 1')) == 3
|
||||
proxy('loki', '/loki/api/v1/labels')
|
||||
proxy('tempo', '/api/search', limit=1)
|
||||
return True
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
# Assertion failures are readiness failures, not successful probes.
|
||||
def probe():
|
||||
try:
|
||||
return ready()
|
||||
except (AssertionError, OSError, ValueError):
|
||||
return False
|
||||
wait(probe, timeout=90)
|
||||
print('PASS: Collector scrape, stores, Grafana datasources and dashboard are ready')
|
||||
Executable
+5
@@ -0,0 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$project_dir"
|
||||
exec python3 scripts/replicas.py wait
|
||||
@@ -0,0 +1,16 @@
|
||||
"""Общий запуск"""
|
||||
|
||||
import os
|
||||
|
||||
import uvicorn
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
uvicorn.run(
|
||||
"src.app:app",
|
||||
host=os.environ.get("APP_HOST", "0.0.0.0"),
|
||||
port=int(os.environ.get("APP_PORT", "8080")),
|
||||
workers=1,
|
||||
access_log=False,
|
||||
timeout_graceful_shutdown=5,
|
||||
)
|
||||
+222
@@ -0,0 +1,222 @@
|
||||
"""HTTP-адреса, определение пользователя и обработка запросов"""
|
||||
|
||||
from contextlib import asynccontextmanager
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
from time import perf_counter
|
||||
from uuid import UUID, uuid4
|
||||
from urllib.parse import quote
|
||||
|
||||
from fastapi import FastAPI, Request, Response
|
||||
from fastapi.responses import StreamingResponse
|
||||
from starlette.concurrency import run_in_threadpool
|
||||
from fastapi.exceptions import RequestValidationError
|
||||
from starlette.exceptions import HTTPException
|
||||
|
||||
from src import attachments, auth, db, migrate, object_store, passwords, storage, telemetry
|
||||
from src.errors import ApiError, error_response
|
||||
from src.models import CreateChat, CreateMessage, CreateUser
|
||||
from src.config import INSTANCE_ID
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app):
|
||||
telemetry.setup('messenger-api', db.pool)
|
||||
# Expose zeros before the first error. Otherwise rate() cannot see a short
|
||||
# burst that finishes before the first scrape of a new status-class series.
|
||||
for route in app.routes:
|
||||
if getattr(route, 'path', '').startswith('/api/v1/'):
|
||||
for method in route.methods:
|
||||
for status_class in ('2xx', '3xx', '4xx', '5xx'):
|
||||
telemetry.http_requests.add(0, {'method': method, 'route': route.path,
|
||||
'status_class': status_class})
|
||||
await run_in_threadpool(db.pool.open, wait=True, timeout=15)
|
||||
try:
|
||||
await run_in_threadpool(migrate.check)
|
||||
yield
|
||||
finally:
|
||||
await run_in_threadpool(db.pool.close)
|
||||
await run_in_threadpool(telemetry.shutdown)
|
||||
|
||||
|
||||
app = FastAPI(title="Учебный мессенджер", redirect_slashes=False, lifespan=lifespan,
|
||||
telemetry={'auto_configure': False, 'tracing': False, 'metrics': False,
|
||||
'logs': False, 'operation_spans': False})
|
||||
logger = logging.getLogger("uvicorn.error")
|
||||
|
||||
|
||||
@app.middleware("http")
|
||||
async def handle_request(request: Request, call_next):
|
||||
path = request.url.path
|
||||
route = next((route.path for route in app.routes
|
||||
if hasattr(route, 'path_regex') and route.path_regex.fullmatch(path)), 'unmatched')
|
||||
method = request.method if request.method in {'GET', 'POST', 'DELETE', 'PUT', 'PATCH', 'HEAD', 'OPTIONS'} else 'OTHER'
|
||||
attrs = {'method': method, 'route': route}
|
||||
started = perf_counter()
|
||||
if path.startswith('/health/') or not path.startswith('/api/v1/'):
|
||||
return await process_request(request, call_next)
|
||||
with telemetry.span(method + ' ' + route, kind=telemetry.SpanKind.SERVER, parent=request.headers,
|
||||
attributes={'http.request.method': method, 'http.route': route}) as current:
|
||||
response = await process_request(request, call_next)
|
||||
current.set_attribute('http.response.status_code', response.status_code)
|
||||
if response.status_code >= 500:
|
||||
current.set_status(telemetry.Status(telemetry.StatusCode.ERROR))
|
||||
context = current.get_span_context()
|
||||
if context.is_valid:
|
||||
response.headers['X-Trace-Id'] = format(context.trace_id, '032x')
|
||||
elapsed = perf_counter() - started
|
||||
attrs['status_class'] = str(response.status_code // 100) + 'xx'
|
||||
telemetry.http_requests.add(1, attrs)
|
||||
telemetry.http_duration.record(elapsed, attrs)
|
||||
telemetry.event('http.request', 'ERROR' if response.status_code >= 500 else 'INFO',
|
||||
method=method, route=route, status=response.status_code, duration_seconds=round(elapsed, 6),
|
||||
request_id=request.state.request_id)
|
||||
return response
|
||||
|
||||
|
||||
async def process_request(request: Request, call_next):
|
||||
request.state.request_id = str(uuid4())
|
||||
started = perf_counter()
|
||||
path = request.url.path
|
||||
try:
|
||||
is_registration = request.method == "POST" and path == "/api/v1/users"
|
||||
is_login = request.method == "POST" and path == "/api/v1/auth/sessions"
|
||||
is_upload = request.method == "POST" and re.fullmatch(r"/api/v1/chats/[^/]+/attachments", path)
|
||||
if is_login:
|
||||
request.state.user = await run_in_threadpool(auth.basic_user, request.headers.get("Authorization"))
|
||||
if await request.body():
|
||||
raise ApiError(400, "invalid_request", "Вход через Basic не принимает тело запроса")
|
||||
elif path.startswith("/api/v1/") and not is_registration:
|
||||
request.state.user = await run_in_threadpool(auth.current_user, request.headers.get("Authorization"))
|
||||
if request.method == "POST" and path.startswith("/api/v1/") and not is_login and not is_upload:
|
||||
content_type = request.headers.get("Content-Type", "").split(";", 1)[0].strip().lower()
|
||||
if content_type != "application/json":
|
||||
raise ApiError(415, "unsupported_media_type", "Ожидается application/json")
|
||||
try:
|
||||
json.loads((await request.body()).decode("utf-8"))
|
||||
except (ValueError, RecursionError):
|
||||
raise ApiError(400, "invalid_request", "Ожидается корректный JSON в UTF-8")
|
||||
response = await call_next(request)
|
||||
except ApiError as error:
|
||||
response = error_response(request, error)
|
||||
except Exception as error:
|
||||
# Не печатаем тело запроса или текст исключения: там могут быть сообщения.
|
||||
telemetry.event('http.exception', 'ERROR', error_type=type(error).__name__, request_id=request.state.request_id)
|
||||
response = error_response(request, ApiError(500, "internal_error", "Внутренняя ошибка сервера"))
|
||||
response.headers["X-Request-Id"] = request.state.request_id
|
||||
response.headers["X-Instance-Id"] = INSTANCE_ID
|
||||
return response
|
||||
|
||||
|
||||
@app.exception_handler(ApiError)
|
||||
def handle_api_error(request: Request, error: ApiError):
|
||||
return error_response(request, error)
|
||||
|
||||
|
||||
@app.exception_handler(RequestValidationError)
|
||||
def handle_validation_error(request: Request, error: RequestValidationError):
|
||||
# Контракт требует 400 и наш формат ошибки, а не стандартный ответ FastAPI.
|
||||
return error_response(request, ApiError(400, "invalid_request", "Неверные поля или параметры запроса"))
|
||||
|
||||
|
||||
@app.exception_handler(HTTPException)
|
||||
def handle_http_error(request: Request, error: HTTPException):
|
||||
code = "not_found" if error.status_code == 404 else "invalid_request"
|
||||
message = "Адрес не найден" if error.status_code == 404 else "Недопустимый HTTP-запрос"
|
||||
response = error_response(request, ApiError(error.status_code, code, message))
|
||||
if error.headers:
|
||||
response.headers.update(error.headers)
|
||||
return response
|
||||
|
||||
|
||||
@app.get("/health/live")
|
||||
def health_live():
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
@app.get("/health/ready")
|
||||
def health_ready():
|
||||
db.ready()
|
||||
auth.ready()
|
||||
object_store.ready()
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
@app.post("/api/v1/users", status_code=201)
|
||||
def create_user(data: CreateUser):
|
||||
return storage.create_user(data.username, data.display_name, passwords.hash_password(data.password))
|
||||
|
||||
|
||||
@app.get("/api/v1/users/me")
|
||||
def current_user(request: Request):
|
||||
return request.state.user
|
||||
|
||||
|
||||
@app.post("/api/v1/chats", status_code=201)
|
||||
def create_chat(request: Request, data: CreateChat):
|
||||
return storage.create_chat(request.state.user["id"], data.title, data.member_ids)
|
||||
|
||||
|
||||
@app.get("/api/v1/chats")
|
||||
def list_chats(request: Request, limit: str = "50", cursor: str | None = None):
|
||||
user_id = request.state.user["id"]
|
||||
return storage.list_chats(user_id, limit, cursor)
|
||||
|
||||
|
||||
@app.get("/api/v1/chats/{chat_id}")
|
||||
def get_chat(request: Request, chat_id: UUID):
|
||||
return storage.get_chat(request.state.user["id"], str(chat_id))
|
||||
|
||||
|
||||
@app.post("/api/v1/chats/{chat_id}/messages", status_code=201)
|
||||
def create_message(request: Request, chat_id: UUID, data: CreateMessage):
|
||||
keys = request.headers.getlist("Idempotency-Key")
|
||||
if len(keys) != 1 or re.fullmatch(r"[A-Za-z0-9._:-]{1,128}", keys[0]) is None:
|
||||
raise ApiError(400, "invalid_request", "Нужен Idempotency-Key: 1–128 символов ASCII [A-Za-z0-9._:-]")
|
||||
return storage.create_message(request.state.user["id"], str(chat_id), data.text, keys[0], data.attachment_ids)
|
||||
|
||||
|
||||
@app.post("/api/v1/chats/{chat_id}/attachments", status_code=202)
|
||||
async def upload_attachment(request: Request, chat_id: UUID, filename: str):
|
||||
return await attachments.upload(request, str(chat_id), filename)
|
||||
|
||||
|
||||
@app.get("/api/v1/attachments/{attachment_id}")
|
||||
def get_attachment(request: Request, attachment_id: UUID):
|
||||
return attachments.public_attachment(attachments.find(request.state.user["id"], str(attachment_id)))
|
||||
|
||||
|
||||
@app.get("/api/v1/attachments/{attachment_id}/content")
|
||||
def download_attachment(request: Request, attachment_id: UUID, variant: str = "original"):
|
||||
row = attachments.find(request.state.user["id"], str(attachment_id))
|
||||
if row["status"] != "ready":
|
||||
raise ApiError(409, "conflict", "Вложение не готово")
|
||||
info = row
|
||||
key = row["object_key"]
|
||||
if variant != "original":
|
||||
info = next((item for item in row["variants"] if item["name"] == variant), None)
|
||||
if info is None:
|
||||
raise ApiError(404, "not_found", "Вариант вложения не найден")
|
||||
key = row["object_key"].rsplit("/", 1)[0] + "/" + variant
|
||||
response = object_store.get(key)
|
||||
return StreamingResponse(attachments.chunks(response), media_type=info["content_type"], headers={
|
||||
"Content-Length": str(info["size_bytes"]), "X-Content-Type-Options": "nosniff",
|
||||
"Content-Disposition": "attachment; filename*=UTF-8''" + quote(row["filename"], safe=""),
|
||||
})
|
||||
|
||||
|
||||
@app.get("/api/v1/chats/{chat_id}/messages")
|
||||
def list_messages(request: Request, chat_id: UUID, limit: str = "50", cursor: str | None = None):
|
||||
return storage.list_messages(request.state.user["id"], str(chat_id), limit, cursor)
|
||||
|
||||
|
||||
@app.post("/api/v1/auth/sessions", status_code=201)
|
||||
def create_session(request: Request):
|
||||
return auth.create_session(request.state.user)
|
||||
|
||||
|
||||
@app.delete("/api/v1/auth/sessions/current", status_code=204)
|
||||
def delete_session(request: Request):
|
||||
auth.logout(request.headers.get("Authorization"))
|
||||
return Response(status_code=204)
|
||||
@@ -0,0 +1,154 @@
|
||||
"""Приём и проверка -> durable S3 staging + PostgreSQL outbox -> 202."""
|
||||
import asyncio
|
||||
import hashlib
|
||||
import os
|
||||
from tempfile import TemporaryFile
|
||||
import unicodedata
|
||||
from uuid import UUID, uuid4
|
||||
import warnings
|
||||
|
||||
from PIL import Image, UnidentifiedImageError
|
||||
from starlette.concurrency import run_in_threadpool
|
||||
from starlette.requests import ClientDisconnect
|
||||
|
||||
from src import db, jobs, object_store, storage, telemetry
|
||||
from src.errors import ApiError
|
||||
|
||||
|
||||
MAX_BYTES = int(os.environ.get("MAX_UPLOAD_BYTES", "10485760"))
|
||||
MAX_PIXELS = int(os.environ.get("MAX_IMAGE_PIXELS", "40000000"))
|
||||
if not 1 <= MAX_BYTES <= 10485760 or not 1 <= MAX_PIXELS <= 40000000:
|
||||
raise ValueError("Лимиты загрузки должны быть положительными и не превышать контракт")
|
||||
Image.MAX_IMAGE_PIXELS = MAX_PIXELS
|
||||
warnings.filterwarnings("error", category=Image.DecompressionBombWarning)
|
||||
# Максимум одно полное декодирование на реплику, независимо от числа клиентов.
|
||||
upload_slot = asyncio.Semaphore(1)
|
||||
|
||||
|
||||
def public_attachment(row):
|
||||
return {"id": str(row["id"]), "chat_id": str(row["chat_id"]), "owner_id": str(row["owner_id"]),
|
||||
"filename": row["filename"], "content_type": row["content_type"],
|
||||
"size_bytes": row["size_bytes"], "sha256": row["sha256"], "status": row["status"],
|
||||
"variants": row["variants"], "error_code": row["error_code"], "created_at": storage.utc(row["created_at"]),
|
||||
"updated_at": storage.utc(row["updated_at"])}
|
||||
|
||||
|
||||
def validate_filename(filename):
|
||||
if not 1 <= len(filename) <= 255 or not filename.strip() or any(
|
||||
char in "/\\" or unicodedata.category(char) == "Cc" for char in filename
|
||||
):
|
||||
raise ApiError(400, "invalid_request", "Недопустимое имя файла")
|
||||
try:
|
||||
filename.encode("utf-8")
|
||||
except UnicodeError:
|
||||
raise ApiError(400, "invalid_request", "Недопустимое имя файла") from None
|
||||
|
||||
|
||||
def validate_image(file, declared_type):
|
||||
try:
|
||||
file.seek(0)
|
||||
with Image.open(file, formats=["PNG", "JPEG"]) as image:
|
||||
true_type = {"PNG": "image/png", "JPEG": "image/jpeg"}[image.format]
|
||||
if image.width * image.height > MAX_PIXELS:
|
||||
raise ApiError(413, "payload_too_large", "Слишком много пикселей")
|
||||
if true_type != declared_type:
|
||||
raise ApiError(415, "unsupported_media_type", "MIME не совпадает с содержимым")
|
||||
image.verify()
|
||||
# verify проверяет структуру/CRC, load дополнительно обнаруживает обрыв pixel data.
|
||||
file.seek(0)
|
||||
with Image.open(file, formats=["PNG", "JPEG"]) as image:
|
||||
image.load()
|
||||
return true_type
|
||||
except (Image.DecompressionBombError, Image.DecompressionBombWarning):
|
||||
raise ApiError(413, "payload_too_large", "Слишком много пикселей") from None
|
||||
except (UnidentifiedImageError, OSError, ValueError, SyntaxError):
|
||||
raise ApiError(415, "unsupported_media_type", "Повреждённый PNG/JPEG или другой формат") from None
|
||||
|
||||
|
||||
def lock_upload(conn, key, blocking=True):
|
||||
function = "pg_advisory_xact_lock" if blocking else "pg_try_advisory_xact_lock"
|
||||
return conn.execute(f"SELECT {function}(hashtextextended(%s, 4)) AS locked", (key,)).fetchone()["locked"]
|
||||
|
||||
|
||||
def save(user_id, chat_id, filename, content_type, file, size, sha256):
|
||||
attachment_id, job_id = uuid4(), uuid4()
|
||||
staging_key = "attachments/staging/" + attachment_id.hex
|
||||
final_key = "attachments/" + attachment_id.hex + "/v1/original"
|
||||
# Журнал фиксируется до PUT: crash до записи job оставит управляемый orphan.
|
||||
with db.connection() as conn:
|
||||
storage.require_member(conn, user_id, chat_id)
|
||||
conn.execute("INSERT INTO attachment_uploads(id, object_key) VALUES (%s, %s)", (attachment_id, staging_key))
|
||||
with db.connection() as conn:
|
||||
conn.execute("SET LOCAL idle_in_transaction_session_timeout='30s'")
|
||||
lock_upload(conn, staging_key)
|
||||
if not conn.execute("SELECT 1 FROM attachment_uploads WHERE id=%s", (attachment_id,)).fetchone():
|
||||
raise ApiError(503, "unavailable", "Загрузка больше не активна")
|
||||
object_store.put(staging_key, file, size, content_type)
|
||||
row = conn.execute("""INSERT INTO attachments(id,chat_id,owner_id,object_key,filename,
|
||||
content_type,size_bytes,sha256,status) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,'queued') RETURNING *""",
|
||||
(attachment_id, UUID(chat_id), UUID(user_id), final_key, filename, content_type, size, sha256)).fetchone()
|
||||
context = telemetry.carrier()
|
||||
conn.execute("""INSERT INTO attachment_jobs(id,attachment_id,staging_key,max_attempts,traceparent,tracestate)
|
||||
VALUES (%s,%s,%s,%s,%s,%s)""", (job_id, attachment_id, staging_key, jobs.MAX_ATTEMPTS,
|
||||
context.get('traceparent'), context.get('tracestate')))
|
||||
conn.execute("INSERT INTO attachment_outbox(job_id,attempt) VALUES (%s,1)", (job_id,))
|
||||
conn.execute("DELETE FROM attachment_uploads WHERE id=%s", (attachment_id,))
|
||||
result = public_attachment(row)
|
||||
# Снимок queued даже если worker успеет закончить до отправки ответа.
|
||||
telemetry.event('attachment.accepted', job_id=str(job_id), attachment_id=str(attachment_id))
|
||||
return result
|
||||
|
||||
|
||||
async def upload(request, chat_id, filename):
|
||||
user_id = request.state.user["id"]
|
||||
await run_in_threadpool(storage.get_chat, user_id, chat_id)
|
||||
validate_filename(filename)
|
||||
content_type = request.headers.get("Content-Type", "").split(";", 1)[0].strip().lower()
|
||||
if content_type not in {"image/png", "image/jpeg"}:
|
||||
raise ApiError(415, "unsupported_media_type", "Ожидаются сырые bytes PNG/JPEG")
|
||||
length = request.headers.get("Content-Length")
|
||||
if length is not None:
|
||||
if not length.isascii() or not length.isdecimal():
|
||||
raise ApiError(400, "invalid_request", "Неверный Content-Length")
|
||||
if len(length) > 10 or int(length) > MAX_BYTES:
|
||||
raise ApiError(413, "payload_too_large", "Файл больше допустимого размера")
|
||||
try:
|
||||
await asyncio.wait_for(upload_slot.acquire(), timeout=20)
|
||||
except TimeoutError:
|
||||
raise ApiError(503, "unavailable", "Загрузки заняты; повторите позднее") from None
|
||||
try:
|
||||
with TemporaryFile() as file:
|
||||
size = 0
|
||||
digest = hashlib.sha256()
|
||||
try:
|
||||
async with asyncio.timeout(30):
|
||||
async for chunk in request.stream():
|
||||
size += len(chunk)
|
||||
if size > MAX_BYTES:
|
||||
raise ApiError(413, "payload_too_large", "Файл больше допустимого размера")
|
||||
digest.update(chunk)
|
||||
await run_in_threadpool(file.write, chunk)
|
||||
except (TimeoutError, ClientDisconnect):
|
||||
raise ApiError(400, "invalid_request", "Загрузка прервана или истёк таймаут") from None
|
||||
true_type = await run_in_threadpool(validate_image, file, content_type)
|
||||
return await run_in_threadpool(save, user_id, chat_id, filename, true_type, file, size, digest.hexdigest())
|
||||
finally:
|
||||
upload_slot.release()
|
||||
|
||||
|
||||
def find(user_id, attachment_id):
|
||||
with db.connection() as conn:
|
||||
row = conn.execute("""SELECT a.* FROM attachments a JOIN chat_members m ON m.chat_id=a.chat_id
|
||||
WHERE a.id=%s AND m.user_id=%s""", (UUID(attachment_id), UUID(user_id))).fetchone()
|
||||
if row is None:
|
||||
raise ApiError(404, "not_found", "Вложение не найдено")
|
||||
return row
|
||||
|
||||
|
||||
def chunks(response):
|
||||
try:
|
||||
while chunk := response.read(65536):
|
||||
yield chunk
|
||||
finally:
|
||||
response.close()
|
||||
response.release_conn()
|
||||
+95
@@ -0,0 +1,95 @@
|
||||
"""Basic только для входа; Bearer-сессии с TTL и AOF находятся в Valkey."""
|
||||
import base64
|
||||
import binascii
|
||||
from contextlib import contextmanager
|
||||
from datetime import datetime, timedelta, timezone
|
||||
import hashlib
|
||||
import json
|
||||
import secrets
|
||||
|
||||
import redis
|
||||
|
||||
from src import passwords, storage
|
||||
from src.config import SESSION_TTL, secret
|
||||
from src.errors import ApiError
|
||||
|
||||
|
||||
sessions = redis.Redis.from_url(
|
||||
secret("CACHE_URL"), decode_responses=True,
|
||||
socket_connect_timeout=0.5, socket_timeout=0.5, max_connections=10,
|
||||
retry=redis.retry.Retry(redis.backoff.NoBackoff(), 0),
|
||||
)
|
||||
|
||||
|
||||
@contextmanager
|
||||
def available():
|
||||
try:
|
||||
yield
|
||||
except redis.RedisError:
|
||||
raise ApiError(503, "unavailable", "Valkey временно недоступен") from None
|
||||
|
||||
|
||||
def unauthorized(basic=False):
|
||||
challenge = 'Basic realm="messenger", charset="UTF-8"' if basic else 'Bearer realm="messenger"'
|
||||
return ApiError(401, "unauthorized", "Неверная или отсутствующая идентификация", {"WWW-Authenticate": challenge})
|
||||
|
||||
|
||||
def credential(header, scheme):
|
||||
parts = (header or "").split()
|
||||
if len(parts) != 2 or parts[0].lower() != scheme.lower():
|
||||
raise unauthorized(scheme == "Basic")
|
||||
return parts[1]
|
||||
|
||||
|
||||
def basic_user(header):
|
||||
value = credential(header, "Basic")
|
||||
try:
|
||||
decoded = base64.b64decode(value, validate=True).decode("utf-8")
|
||||
username, password = decoded.split(":", 1)
|
||||
except (ValueError, UnicodeError, binascii.Error):
|
||||
raise unauthorized(True) from None
|
||||
user = storage.find_login(username)
|
||||
if user is None or not passwords.matches(user["password_hash"], password):
|
||||
raise unauthorized(True)
|
||||
return storage.public_user(user)
|
||||
|
||||
|
||||
def session_key(token):
|
||||
return "session:" + hashlib.sha256(token.encode()).hexdigest()
|
||||
|
||||
|
||||
def create_session(user):
|
||||
token = secrets.token_urlsafe(32)
|
||||
expires = datetime.now(timezone.utc) + timedelta(seconds=SESSION_TTL)
|
||||
value = {"user_id": user["id"], "expires_at": storage.utc(expires)}
|
||||
with available():
|
||||
sessions.set(session_key(token), json.dumps(value), ex=SESSION_TTL)
|
||||
return {"token": token, "token_type": "Bearer", "expires_at": value["expires_at"], "user": user}
|
||||
|
||||
|
||||
def current_user(header):
|
||||
token = credential(header, "Bearer")
|
||||
with available():
|
||||
value = sessions.get(session_key(token))
|
||||
if value is None:
|
||||
raise unauthorized()
|
||||
data = json.loads(value)
|
||||
if datetime.fromisoformat(data["expires_at"].replace("Z", "+00:00")) <= datetime.now(timezone.utc):
|
||||
raise unauthorized()
|
||||
user = storage.find_user(data["user_id"])
|
||||
if user is None:
|
||||
raise unauthorized()
|
||||
return user
|
||||
|
||||
|
||||
def logout(header):
|
||||
token = credential(header, "Bearer")
|
||||
with available():
|
||||
removed = sessions.delete(session_key(token))
|
||||
if not removed:
|
||||
raise unauthorized()
|
||||
|
||||
|
||||
def ready():
|
||||
with available():
|
||||
sessions.ping()
|
||||
@@ -0,0 +1,38 @@
|
||||
"""Идемпотентное создание приватного bucket отдельной административной учёткой."""
|
||||
import os
|
||||
import time
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from minio import Minio
|
||||
from minio.error import MinioException, S3Error
|
||||
import urllib3
|
||||
|
||||
from src.config import secret
|
||||
|
||||
|
||||
def bootstrap():
|
||||
endpoint = urlsplit(os.environ.get("S3_ENDPOINT", "http://s3:9000"))
|
||||
client = Minio(endpoint.netloc, access_key=secret("S3_ROOT_USER"), secret_key=secret("S3_ROOT_PASSWORD"),
|
||||
secure=endpoint.scheme == "https", region=os.environ.get("S3_REGION", "us-east-1"),
|
||||
http_client=urllib3.PoolManager(timeout=urllib3.Timeout(connect=1, read=3), retries=False))
|
||||
bucket = os.environ.get("S3_BUCKET", "messenger-attachments")
|
||||
deadline = time.monotonic() + 60
|
||||
while True:
|
||||
try:
|
||||
if not client.bucket_exists(bucket):
|
||||
client.make_bucket(bucket)
|
||||
print("Private bucket is ready; static IAM permits only authenticated accounts", flush=True)
|
||||
return
|
||||
except S3Error as error:
|
||||
if error.code in {"BucketAlreadyOwnedByYou", "BucketAlreadyExists"}:
|
||||
return
|
||||
if time.monotonic() >= deadline:
|
||||
raise RuntimeError("S3 bootstrap failed") from None
|
||||
except (MinioException, urllib3.exceptions.HTTPError, OSError):
|
||||
if time.monotonic() >= deadline:
|
||||
raise RuntimeError("S3 bootstrap timed out") from None
|
||||
time.sleep(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
bootstrap()
|
||||
@@ -0,0 +1,46 @@
|
||||
"""Один durable Stream, consumer group; ACK только после эффекта/устойчивого retry."""
|
||||
import json
|
||||
|
||||
import redis
|
||||
|
||||
from src.config import secret
|
||||
from src import telemetry
|
||||
|
||||
STREAM = 'attachments:v1'
|
||||
GROUP = 'image-workers'
|
||||
|
||||
|
||||
def connect():
|
||||
return redis.Redis.from_url(secret('BROKER_URL'), decode_responses=True,
|
||||
socket_connect_timeout=2, socket_timeout=3,
|
||||
retry_on_timeout=False)
|
||||
|
||||
|
||||
def bootstrap(client):
|
||||
try:
|
||||
client.xgroup_create(STREAM, GROUP, id='0', mkstream=True)
|
||||
except redis.ResponseError as error:
|
||||
if 'BUSYGROUP' not in str(error):
|
||||
raise
|
||||
|
||||
|
||||
def publish(client, job):
|
||||
payload = {'schema_version': 1, 'job_id': str(job['id']), 'type': 'attachment.process.v1',
|
||||
'attachment_id': str(job['attachment_id']), 'pipeline_version': job['pipeline_version'],
|
||||
'attempt': job['attempt'], 'created_at': job['created_at'].isoformat().replace('+00:00', 'Z')}
|
||||
with telemetry.span('attachments publish', kind=telemetry.SpanKind.PRODUCER,
|
||||
parent={key: job[key] for key in ('traceparent', 'tracestate') if job.get(key)},
|
||||
attributes={'messaging.system': 'redis', 'messaging.destination.name': STREAM,
|
||||
'messaging.operation.type': 'send', 'job.attempt': job['attempt']}):
|
||||
payload.update(telemetry.carrier())
|
||||
result = client.xadd(STREAM, {'job': json.dumps(payload)})
|
||||
telemetry.event('job.published', job_id=str(job['id']), attempt=job['attempt'])
|
||||
return result
|
||||
|
||||
|
||||
def acknowledge(client, message_id):
|
||||
# Единственная группа: подтверждённые записи можно удалить. Pending не trim-им.
|
||||
with client.pipeline(transaction=True) as pipe:
|
||||
pipe.xack(STREAM, GROUP, message_id)
|
||||
pipe.xdel(STREAM, message_id)
|
||||
pipe.execute()
|
||||
@@ -0,0 +1,87 @@
|
||||
"""Очистка orphan PUT и staging только ready/просроченных failed; текущую работу не трогает."""
|
||||
import argparse
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
|
||||
from src import db, object_store
|
||||
from src.attachments import lock_upload
|
||||
|
||||
|
||||
GRACE_SECONDS = int(os.environ.get("UPLOAD_CLEANUP_GRACE_SECONDS", "3600"))
|
||||
if GRACE_SECONDS < 60:
|
||||
raise ValueError("Безопасный grace period должен быть не меньше 60 секунд")
|
||||
|
||||
|
||||
def cleanup_orphans():
|
||||
result = {"removed": 0, "kept_ready": 0, "skipped_active": 0}
|
||||
with db.connection() as conn:
|
||||
candidates = conn.execute("""SELECT id, object_key FROM attachment_uploads
|
||||
WHERE created_at < clock_timestamp() - %s * interval '1 second'
|
||||
ORDER BY created_at LIMIT 100""", (GRACE_SECONDS,)).fetchall()
|
||||
for row in candidates:
|
||||
with db.connection() as conn:
|
||||
if not lock_upload(conn, row["object_key"], blocking=False):
|
||||
result["skipped_active"] += 1
|
||||
continue
|
||||
pending = conn.execute("""SELECT 1 FROM attachment_uploads WHERE id=%s
|
||||
AND created_at < clock_timestamp() - %s * interval '1 second' FOR UPDATE""",
|
||||
(row["id"], GRACE_SECONDS)).fetchone()
|
||||
if not pending:
|
||||
continue
|
||||
ready = conn.execute("SELECT 1 FROM attachments WHERE object_key=%s", (row["object_key"],)).fetchone()
|
||||
if ready:
|
||||
result["kept_ready"] += 1
|
||||
else:
|
||||
object_store.remove(row["object_key"])
|
||||
result["removed"] += 1
|
||||
conn.execute("DELETE FROM attachment_uploads WHERE id=%s", (row["id"],))
|
||||
return result
|
||||
|
||||
|
||||
RETENTION = int(os.environ.get("STAGING_RETENTION_SECONDS", "86400"))
|
||||
if RETENTION < 60:
|
||||
raise ValueError("STAGING_RETENTION_SECONDS должен быть >=60")
|
||||
|
||||
|
||||
def cleanup_staging():
|
||||
removed = 0
|
||||
with db.connection() as conn:
|
||||
rows = conn.execute("""SELECT j.*,a.status,a.object_key FROM attachment_jobs j
|
||||
JOIN attachments a ON a.id=j.attachment_id WHERE j.staging_deleted_at IS NULL
|
||||
AND (a.status='ready' OR (a.status='failed' AND a.updated_at<clock_timestamp()-%s*interval '1 second'))
|
||||
LIMIT 20 FOR UPDATE OF j,a SKIP LOCKED""", (RETENTION,)).fetchall()
|
||||
for job in rows:
|
||||
object_store.remove(job['staging_key'])
|
||||
if job['status'] == 'failed':
|
||||
prefix = job['object_key'].rsplit('/', 1)[0] + '/'
|
||||
for variant in ('original', 'thumbnail', 'crop', 'optimized'):
|
||||
object_store.remove(prefix + variant)
|
||||
conn.execute('UPDATE attachment_jobs SET staging_deleted_at=clock_timestamp() WHERE id=%s', (job['id'],))
|
||||
removed += 1
|
||||
return removed
|
||||
|
||||
|
||||
def cleanup_once():
|
||||
return {**cleanup_orphans(), "staging_removed": cleanup_staging()}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--once", action="store_true")
|
||||
args = parser.parse_args()
|
||||
db.pool.open(wait=True, timeout=15)
|
||||
try:
|
||||
while True:
|
||||
try:
|
||||
print(json.dumps(cleanup_once()), flush=True)
|
||||
except Exception as error:
|
||||
if args.once:
|
||||
raise
|
||||
logging.error("Cleanup postponed: %s", type(error).__name__)
|
||||
if args.once:
|
||||
break
|
||||
time.sleep(30)
|
||||
finally:
|
||||
db.pool.close()
|
||||
@@ -0,0 +1,20 @@
|
||||
"""Настройки и чтение локальных Docker secrets."""
|
||||
import os
|
||||
import socket
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def secret(name):
|
||||
return Path(os.environ[name + "_FILE"]).read_text().strip()
|
||||
|
||||
|
||||
SESSION_TTL = int(os.environ.get("SESSION_TTL_SECONDS", "3600"))
|
||||
if SESSION_TTL < 1:
|
||||
raise ValueError("SESSION_TTL_SECONDS должен быть положительным")
|
||||
if os.environ.get("SESSION_STORE", "valkey") != "valkey":
|
||||
raise ValueError("В этой реализации SESSION_STORE=valkey")
|
||||
|
||||
INSTANCE_ID = socket.gethostname()
|
||||
IDEMPOTENCY_TTL = int(os.environ.get("IDEMPOTENCY_TTL_SECONDS", "86400"))
|
||||
if IDEMPOTENCY_TTL < 86400:
|
||||
raise ValueError("IDEMPOTENCY_TTL_SECONDS должен быть не меньше 86400")
|
||||
@@ -0,0 +1,63 @@
|
||||
"""Ограниченный пул; каждый with connection() — отдельная транзакция."""
|
||||
from contextlib import contextmanager
|
||||
|
||||
import psycopg
|
||||
from psycopg.rows import dict_row
|
||||
from psycopg_pool import ConnectionPool, PoolTimeout, TooManyRequests
|
||||
|
||||
from src.config import secret
|
||||
from src.errors import ApiError
|
||||
from src import telemetry
|
||||
|
||||
|
||||
IO_TIMEOUT = 2.0
|
||||
|
||||
|
||||
class TimedConnection(psycopg.Connection):
|
||||
def execute(self, query, params=None, **kwargs):
|
||||
# Only the SQL verb is allowed: never statement text or parameters.
|
||||
verb = query.lstrip().split(None, 1)[0].upper() if isinstance(query, str) and query.strip() else 'QUERY'
|
||||
if verb not in {'SELECT', 'INSERT', 'UPDATE', 'DELETE', 'SET'}:
|
||||
verb = 'QUERY'
|
||||
with telemetry.span('postgresql.' + verb, kind=telemetry.SpanKind.CLIENT,
|
||||
attributes={'db.system.name': 'postgresql', 'db.operation.name': verb}, require_parent=True):
|
||||
return super().execute(query, params, **kwargs)
|
||||
|
||||
def wait(self, gen, interval=0.1, timeout=None):
|
||||
# statement_timeout работает на сервере и не помогает при docker pause.
|
||||
# Ограничиваем само ожидание I/O: проверку пула, SQL, COMMIT и ROLLBACK.
|
||||
timeout = IO_TIMEOUT if timeout is None else min(timeout, IO_TIMEOUT)
|
||||
try:
|
||||
return super().wait(gen, interval=interval, timeout=timeout)
|
||||
except psycopg.OperationalError:
|
||||
# Не ждём отмены/ROLLBACK от зависшего сервера. Пул заменит соединение.
|
||||
self.close()
|
||||
raise
|
||||
|
||||
|
||||
pool = ConnectionPool(
|
||||
conninfo=secret("DATABASE_URL"), open=False,
|
||||
connection_class=TimedConnection,
|
||||
min_size=1, max_size=5, max_waiting=20, timeout=2,
|
||||
check=ConnectionPool.check_connection,
|
||||
kwargs={
|
||||
"row_factory": dict_row,
|
||||
"connect_timeout": 2,
|
||||
"options": "-c timezone=UTC -c statement_timeout=3000 -c lock_timeout=2000 -c idle_in_transaction_session_timeout=5000",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@contextmanager
|
||||
def connection():
|
||||
try:
|
||||
with telemetry.span('postgresql.transaction', require_parent=True):
|
||||
with pool.connection() as conn:
|
||||
yield conn
|
||||
except (psycopg.OperationalError, PoolTimeout, TooManyRequests):
|
||||
raise ApiError(503, "unavailable", "PostgreSQL временно недоступен") from None
|
||||
|
||||
|
||||
def ready():
|
||||
with connection() as conn:
|
||||
conn.execute("SELECT 1")
|
||||
@@ -0,0 +1,27 @@
|
||||
"""общий формат ошибок"""
|
||||
|
||||
from fastapi import Request
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
|
||||
class ApiError(Exception):
|
||||
def __init__(self, status: int, code: str, message: str, headers=None):
|
||||
super().__init__(message)
|
||||
self.status = status
|
||||
self.code = code
|
||||
self.message = message
|
||||
self.headers = headers or {}
|
||||
|
||||
|
||||
def error_response(request: Request, error: ApiError):
|
||||
return JSONResponse(
|
||||
status_code=error.status,
|
||||
content={
|
||||
"error": {
|
||||
"code": error.code,
|
||||
"message": error.message,
|
||||
"request_id": request.state.request_id,
|
||||
}
|
||||
},
|
||||
headers={"X-Request-Id": request.state.request_id, **error.headers},
|
||||
)
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user