1 Commits
127 changed files with 53496 additions and 61 deletions
+11 -20
View File
@@ -1,20 +1,11 @@
.git
.github
.env
.env.*
!.env.example
secrets
certs
volumes
data
backups
*.key
*.pem
*.p12
*.pfx
*.jks
.venv
__pycache__
node_modules
evidence
.DS_Store
*
!Dockerfile
!requirements.txt
!src/
!src/**
!migrations/
!migrations/**
!research/
!research/**
**/__pycache__/
**/*.py[cod]
+10 -2
View File
@@ -3,11 +3,12 @@
APP_HOST=0.0.0.0
APP_PORT=8080
APP_ENV=lab
HOST_PORT=8080
SESSION_TTL_SECONDS=3600
# Choose credentials locally. File paths contain no secret values.
DATABASE_URL_FILE=/run/secrets/database_url
# Optional on 3, required use on 4 in lab2; Redis OR Valkey:
SESSION_STORE=postgres
# Shared sessions carried over from lab4 use Valkey.
SESSION_STORE=valkey
CACHE_URL_FILE=/run/secrets/cache_url
API_REPLICAS=2
IDEMPOTENCY_TTL_SECONDS=86400
@@ -18,13 +19,20 @@ S3_ACCESS_KEY_ID_FILE=/run/secrets/s3_access_key_id
S3_SECRET_ACCESS_KEY_FILE=/run/secrets/s3_secret_access_key
MAX_UPLOAD_BYTES=10485760
MAX_IMAGE_PIXELS=40000000
UPLOAD_CLEANUP_GRACE_SECONDS=3600
BROKER_URL_FILE=/run/secrets/broker_url
WORKER_CONCURRENCY=2
JOB_MAX_ATTEMPTS=5
JOB_TIMEOUT_SECONDS=60
STAGING_RETENTION_SECONDS=86400
JOB_LEASE_SECONDS=90
JOB_BACKOFF_SECONDS=2
WORKER_TASK_MEMORY_MB=640
OTEL_SERVICE_NAME=messenger-api
# Worker must set a distinct service.name.
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=lab
GRAFANA_PORT=3000
OTEL_SDK_DISABLED=false
OTEL_PYTHON_SDK_INTERNAL_METRICS_ENABLED=true
+58
View File
@@ -0,0 +1,58 @@
## Лабораторная 5
Автор / группа: Савельев Иван / БВТ2304
Целевая оценка (3/4/5): **4** — выполнены требования уровней 3 и 4.
Трек 5, если заявлен: не заявлен.
### Что реализовано и что перенесено
Из lab4 перенесены API мессенджера, авторизация и сессии, ACL, идемпотентность сообщений, миграции, приватный S3, две API-реплики за HAProxy, Docker-конфигурация и собственные тесты.
- Загрузка возвращает `202 queued` после устойчивого сохранения оригинала в S3 staging и транзакции metadata + job + outbox. Статус доступен через GET; content до готовности возвращает 409.
- Добавлены отдельные Valkey Streams broker с AOF и volume, outbox publisher и worker. Доставка at least once, ACK после устойчивого результата; дубли не создают второй набор объектов.
- Worker сохраняет неизменённый оригинал, JPEG thumbnail, центрированный crop 128×128 и optimized WebP по контракту, с учётом EXIF orientation и прозрачности.
- Реализованы пять автоматических попыток, backoff с jitter, lease/recovery, безопасные коды ошибок, реестр failed и явный replay с audit. Ready-задания повторно не запускаются.
- Добавлены ограничения bytes/pixels/времени/памяти, ограниченная конкуренция и очистка staging после фиксации результата либо истечения retention failed.
### Запуск из чистого clone
Нужны Docker Engine с Compose v2, Python 3.11, Bash и make. Команды из корня репозитория:
```sh
python3 -m venv .venv
.venv/bin/python -m pip install -r requirements.txt -r requirements-dev.txt
make check validate PYTHON=.venv/bin/python
./scripts/start.sh
./scripts/check.sh
# Полная проверка на отдельном временном Docker Compose project:
./scripts/check-docker.sh
# Остановка обычного стенда с сохранением volumes:
./scripts/stop.sh
```
`start.sh` создаёт локальные `.env` и secrets, собирает приложение, выполняет миграции и bootstrap, ждёт готовности обеих API-реплик. API доступен на `http://localhost:8080`.
### Результаты проверок
Проверено 12.11.2026: `make check validate test`, 39 собственных API-тестов и все 17 групп полного сценария прошли. Воспроизведены остановка worker и рестарты API/broker, оба разрыва outbox, crash worker после PUT и перед ACK, дубли, исчерпание retry/replay, постоянная ошибка, timeout/MemoryError, очистка staging, недоступность broker, persistence и failover API. Отдельно подтверждён фактический restart всех 11 постоянных контейнеров с сохранением их ID и volumes.
- [Отчёт, архитектура, команды и ограничения](../REPORT.md).
- [Полный вывод проверок](../evidence/lab5-checks.txt) и [JSON результатов](../evidence/lab5-checks.json).
- [Доказательство рестартов](../evidence/lab5-restart.json) и [вывод persistence](../evidence/lab5-restart.txt).
### Ограничения / известные ошибки
Стенд учебный, на одном Docker-host; потеря host/disk/volumes и ручное удаление объектов не покрываются. Поддерживается pipeline_version=1; исследовательские треки на 5 не реализованы. При длительном простое worker outbox и staging требуют свободного диска; общей квоты принятой работы нет. Failed staging хранится 24 часа, после удаления replay запрещён. Превышение ресурсов обработки заканчивается failed после ограниченного числа попыток. Старые ready-вложения lab4 остаются доступны без автоматического создания вариантов.
### Источники и использование ИИ
Основа — ТЗ, OpenAPI, `attachment-states.md` и `jobs.schema.json` этого репозитория; официальная документация Redis Streams/XAUTOCLAIM, Valkey persistence, Pillow ImageOps и Python resource. Ссылки приведены в [REPORT.md](../REPORT.md#первичные-источники).
- [x] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
- [x] Контракт и публичные тесты не ослаблены.
- [x] make check и make test выполнены; приложен реальный вывод.
- [x] Сценарии отказов/безопасности своего уровня воспроизведены.
- [x] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
- [x] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
+52 -15
View File
@@ -1,18 +1,55 @@
## Лабораторная 6
## Лабораторная 6 — наблюдаемость, оценка 5, трек 2
Автор / группа: TODO
Целевая оценка (3/4/5): TODO
Трек 5, если заявлен: TODO
Автор / группа: Савельев Иван / БВТ2304.
Что реализовано и что перенесено: TODO
Команды запуска из чистого clone: TODO
Результаты проверок и ссылка на REPORT.md: TODO
Ограничения / известные ошибки: TODO
Источники и использование ИИ: TODO
### Результат
- [ ] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
- [ ] Контракт и публичные тесты не ослаблены.
- [ ] make check и make test выполнены; приложен реальный вывод.
- [ ] Сценарии отказов/безопасности своего уровня воспроизведены.
- [ ] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
- [ ] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
Сохранён работающий pipeline lab5. Добавлены явный OTel SDK для API/worker/publisher,
Collector, Prometheus, Loki, Tempo и Grafana с provisioning трёх источников и
импортируемым dashboard на 19 панелей. W3C context хранится в durable job/outbox,
проходит через broker и сохраняет цепочку при retry до S3/DB ready.
Реализованы SLO доступности API 99%/1h и готовности изображения 95% за 30s/1h.
SLI изображений учитывает failed, late и не завершившиеся jobs. Fast/slow burn
alerts используют пары 2m/30s и 6m/1m, guards и for; есть high error rate и queue lag.
Исследование сравнивает короткий всплеск с длительными отказами DB/worker,
измеряет detection/resolution и ложные срабатывания. Разбор причин использует
сохранённые метрики, логи и реальные traces; запросы собраны в runbook.
### Запуск и проверки
```sh
python3 -m venv .venv
.venv/bin/python -m pip install -r requirements.txt -r requirements-dev.txt
./scripts/start.sh
./scripts/check-lab6.sh
```
- API: http://localhost:8080.
- Dashboard: http://localhost:3000/d/messenger-lab6, локальный Viewer.
- [Отчёт, результаты, версии и ограничения](../REPORT.md).
- [Runbook](../docs/runbook.md), [методика](../research/README.md).
- API/39 собственных tests: `evidence/lab6-api-checks.txt`.
- 17 групп сохранённых критериев lab5: `evidence/lab6-lab5-regression.json`.
- Trace с retry и логи: `evidence/lab6-trace-retry.json`, `evidence/lab6-correlated-logs.json`.
- Loki outage/ограниченная RAM/измеренные потери: `evidence/lab6-checks.json`.
- Реальные restart/volumes: `evidence/lab6-restart.json`, `evidence/lab6-telemetry-persistence.json`.
- Трек 2: `evidence/lab6-slo-research.json`, `evidence/lab6-slo-series.json`.
- Проверки правил: `evidence/lab6-promtool.txt`.
### Ограничения
Один host; потеря диска/volumes не покрыта. Telemetry buffer ограничен и допускает
измеряемые потери при недоступном backend. SLO периода час задан, но один короткий
live эксперимент проверяет детектор, а не долгосрочную надёжность. Отказы LB/сети
до API требуют отдельного внешнего SLI. Внешняя доставка alerts не требуется.
Источники — ТЗ и официальная документация OTel, W3C, Grafana/Prometheus и Google SRE
Workbook; ссылки в REPORT.md. Реализация и проверки подготовлены с помощью Codex.
- [x] Минимум предыдущей работы сохранён.
- [x] Обязательные контракты и публичные tests не изменены.
- [x] Три сигнала поступают в реальные persistent stores; dashboard provisioned из Git.
- [x] Секреты, тела сообщений/изображений и SQL/params не экспортируются.
- [x] Локальные .env/secrets и runtime artifacts исключены из Git.
- [x] Условия alerts, окна, for и runbook сохранены в Git.
+24
View File
@@ -0,0 +1,24 @@
name: Application
on: [push, pull_request]
permissions:
contents: read
jobs:
docker:
runs-on: ubuntu-latest
timeout-minutes: 35
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.11'
- run: python -m venv .venv
- run: .venv/bin/python -m pip install -r requirements.txt -r requirements-dev.txt
- run: ./scripts/check-lab6.sh
- uses: actions/upload-artifact@v4
if: always()
with:
name: lab6-evidence
path: |
evidence/lab6-*.txt
evidence/lab6-*.json
+28
View File
@@ -0,0 +1,28 @@
FROM python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
APP_HOST=0.0.0.0 \
APP_PORT=8080
WORKDIR /app
COPY requirements.txt .
RUN python -m pip install --no-cache-dir -r requirements.txt \
&& python -m pip check
RUN groupadd --gid 10001 app \
&& useradd --uid 10001 --gid app --no-create-home app
COPY src/ ./src/
COPY migrations/ ./migrations/
COPY research/ ./research/
USER 10001:10001
EXPOSE 8080
STOPSIGNAL SIGTERM
HEALTHCHECK --interval=2s --timeout=3s --start-period=5s --retries=3 \
CMD ["python", "-m", "src.healthcheck"]
CMD ["python", "-m", "src"]
+184
View File
@@ -0,0 +1,184 @@
# Отчёт — лабораторная 5
- Автор, группа: Савельев Иван, БВТ2304.
- Целевая оценка: **4** — все требования уровней 3 и 4. Исследовательский трек оценки 5 не заявлен.
- Стенд: Docker Compose; HAProxy, две реплики FastAPI, PostgreSQL, Valkey для сессий, SeaweedFS S3; отдельные Valkey Streams broker, outbox publisher и worker.
- Commit/PR: после публикации. Evidence содержит source_commit и working_tree=true: проверялось рабочее дерево поверх указанного commit.
- Использование ИИ: перенос базы, код, проверки и отчёт подготовлены с помощью Codex. Результаты проверок получены реальным запуском Docker, а не предполагаются по коду.
## Запуск из чистого clone
Нужны Docker Engine с Compose v2, Python 3.11, Bash и make. Все команды выполняются из корня репозитория. Внешние заранее установленные БД/брокер/S3 не нужны.
```sh
python3 -m venv .venv
.venv/bin/python -m pip install -r requirements.txt -r requirements-dev.txt
make check validate PYTHON=.venv/bin/python
./scripts/start.sh
./scripts/check.sh
```
`start.sh` создаёт локальные `.env` и `secrets/`, собирает образ, выполняет миграции и bootstrap приватного bucket, поднимает сервисы и ждёт обе API-реплики через LB. Существующие настройки и пароли сохраняются. `.env`, secrets, виртуальное окружение и runtime data исключены из Git; Docker context содержит только исходники, зависимости и миграции.
API: http://localhost:8080. Если порт занят, задайте `HOST_PORT=8081` в `.env`; HTTP-проверки запускайте с `BASE_URL=http://localhost:8081`. Compose project `mtusi-lab5`, образы `mtusi-messenger:lab5` и volumes отделены от предыдущих лабораторных. Локальные credentials генерируются заново, файлы из lab4 не копируются.
```sh
# Полный автоматический сценарий на отдельном временном project и свободном порту:
./scripts/check-docker.sh
# Обычный стенд, остановка без потери данных:
./scripts/stop.sh
./scripts/start.sh
# Перезапуск всех постоянных процессов и хранилищ с сохранением volumes:
./scripts/restart.sh
# Только для намеренного удаления учебных данных ТЕКУЩЕГО Compose project:
docker compose down --volumes
```
Полный сценарий создаёт синтетические данные в отдельном `lab5-check-<random>` и удаляет только его контейнеры/volumes. Пишет `evidence/lab5-checks.txt` и `evidence/lab5-checks.json`. Для диагностики сбоя можно явно задать `KEEP_TEST_STACK=1`; это оставит только неуспешный временный стенд. Основной стенд не затрагивается.
## Что перенесено и изменено
Из lab4 сохранены регистрация/Argon2id, Basic-вход и Bearer-сессии в Valkey, ACL чатов и вложений, сообщения и keyset-пагинация, X-Request-Id/X-Instance-Id, общая идемпотентность сообщений, две API-реплики за LB, приватный S3, лимиты загрузки, миграции и эксплуатационные скрипты. Предыдущие отчёты — `REPORT.lab2.md`, `REPORT.lab3.md`, `REPORT.lab4.md`; старые evidence относятся к соответствующим работам.
Объявленный переход API: upload возвращает **202 со снимком queued**, а клиент опрашивает GET metadata. Queued/processing уже можно прикрепить к сообщению в том же чате. Для failed новое прикрепление даёт 409. Content любого не-ready вложения даёт 409, отсутствующий вариант ready — 404. Оригинал скачивается побайтно без перекодирования и redirect; API не раскрывает storage keys/credentials. Контракт, README задания и публичные tests не изменялись.
Миграции 001–003 сохранены. Миграция 004 добавляет состояния/metadata вариантов и таблицы jobs, outbox, audit событий. Старые ready-вложения сохраняют свои ключи и читаются после обновления; варианты у старых записей не создаются автоматически. Схема описана в `docs/schema.md`.
## Архитектура и границы устойчивости
```text
Клиент → HAProxy → API ×2 → PostgreSQL: metadata + job + outbox
→ приватный S3 staging
PostgreSQL outbox → publisher → Valkey Streams (AOF, volume)
↓ XREADGROUP, prefetch=1
worker (2 процесса)
↓
отдельный resize subprocess
↓
S3 original + thumbnail + crop + optimized
↓
PostgreSQL COMMIT ready → broker ACK
cleanup → orphan PUT journal + staging только ready/просроченных failed
```
PostgreSQL — источник истины для намерения обработки и состояния. Valkey Streams обеспечивает at-least-once доставку. Это отдельный service `broker`, не Pub/Sub и не тот же экземпляр, что сессии. Stream `attachments:v1`, группа `image-workers`, начало чтения `0`; topology создаётся идемпотентным XGROUP CREATE MKSTREAM. Сообщения не trim-ятся до ACK. После XACK запись удаляется XDEL в одной broker transaction; используется одна группа.
1. API принимает все bytes с ограничением размера, проверяет имя, MIME и настоящий PNG/JPEG. Resize/генерация вариантов в API не выполняются.
2. Перед PUT фиксируется журнал `attachment_uploads`. Полный проверенный оригинал записывается в S3 `attachments/staging/<UUID>`. Это тот же приватный S3, но отдельный staging prefix; на этом этапе конечного original/вариантов ещё нет.
3. Одна транзакция PostgreSQL вставляет queued metadata, job и outbox и удаляет запись журнала. **202 отправляется после PUT и успешного COMMIT.** Локальный TemporaryFile нужен только во время HTTP-приёма, устойчивым staging не является.
4. Отдельный publisher читает due outbox с блокировкой SKIP LOCKED и делает XADD. Broker настроен AOF `appendfsync always`, volume, `noeviction`. Успешный XADD подтверждает приём брокером с этой настройкой persistence; это не consumer ACK. После ответа dispatcher отмечает published_at.
5. Worker захватывает job транзакционно: status=processing, случайный lease_token и deadline. Данные для скачивания берёт из БД, а не из сообщения/filename. Проверяет размер и SHA-256 staging. Записывает original и все три варианта, затем в одной транзакции ставит ready и metadata/checksums. Только после COMMIT подтверждает доставку.
В сообщении только поля `jobs.schema.json`: schema_version, job_id, type, attachment_id, pipeline_version=1, attempt, created_at. Bytes, URL, credentials, filename и storage key отсутствуют. job_id постоянен при redelivery и managed retry; attempt начинается с 1 и меняется при новой управляемой попытке. Native stream ID — отдельный идентификатор доставки.
## Идемпотентность, retry и replay
Ключ эффекта `(attachment_id, pipeline_version, variant)` реализован детерминированными объектами:
```text
attachments/<UUID>/v1/original
attachments/<UUID>/v1/thumbnail
attachments/<UUID>/v1/crop
attachments/<UUID>/v1/optimized
```
Число объектов конечного результата — четыре. Дубль queued/processing не захватывает уже арендованную job. Устаревшая попытка не меняет актуальное состояние. Перед PUT и COMMIT worker блокирует строки job/attachment и сверяет lease_token и deadline. При повторе после незавершённого PUT те же детерминированные объекты могут записаться ещё раз, второй набор не создаётся. Дубль ready проверяет полную metadata и размеры объектов через S3 HEAD, затем ACK, не меняя status/updated_at и не выполняя resize. В этой работе поддерживается только pipeline_version=1; трек versioned pipeline на 5 не заявлен.
Временные ошибки S3, resource limit и timeout дают queued с новым due outbox. Backoff: `uniform(0.5,1.5) × min(60, JOB_BACKOFF_SECONDS × 2^(attempt-1))`; интервал сохраняется в PostgreSQL. Default base=2 s, jitter присутствует. Автоматический бюджет — `JOB_MAX_ATTEMPTS=5`. При исчерпании — failed/error_code=retry_exhausted; последняя безопасная причина сохраняется в job. Нарушение checksum/размера staging или невалидное изображение — постоянный invalid_image, сразу failed без повторов. Тексты исключений и stack traces не передаются в API/registry.
Publisher также восстанавливает истёкший processing lease: записывает worker_lost и создаёт следующую попытку либо failed при исчерпании бюджета. Это не бесконечный retry. Pending-доставки умершего consumer забираются XAUTOCLAIM после lease timeout. Если recovery уже создал новую попытку, старое сообщение становится obsolete и получает ACK; намерение следующей попытки уже устойчиво сохранено.
Реестр failed и явный replay:
```sh
docker compose exec -T worker python -m src.job_admin failed
docker compose exec -T worker python -m src.job_admin replay <job_id>
```
Replay разрешён только для failed с ещё сохранённым staging. Он сохраняет job_id/ключ эффекта, записывает operator_replay в audit, увеличивает глобальный attempt и выделяет новый ограниченный бюджет из пяти попыток. Ready/queued/processing повторять этой командой нельзя; повтор ready завершается ошибкой без изменения БД/объектов. Глобальный attempt монотонен, поэтому старые сообщения не запускают replay как новую работу. После удаления failed staging replay запрещён: исходник надо загрузить заново. HTTP endpoint управления заданиями не добавлен.
## Правила изображений и ограничения ресурсов
- Оригинал остаётся исходными bytes, MIME, size_bytes и SHA-256 относятся к нему.
- EXIF orientation применяется к производным изображениям. Прозрачность компонуется на белый; производные RGB, без EXIF/ICC/comment metadata.
- Thumbnail: JPEG quality=85, вписывание в 256×256 без увеличения, целочисленное округление вниз с минимумом 1 px. Fixture 320×200 → 256×160.
- Crop: JPEG quality=85, центрированный квадрат 128×128, LANCZOS; маленький исходник может увеличиваться.
- Optimized: WebP quality=80, method=4; вписывание в 1280×1280 без увеличения. Уменьшение байтов каждого маленького файла не обещается.
- Загрузка ≤10 MiB и ≤40 млн pixels. Limit проверяется до полной распаковки; Content-Length и фактический поток проверяются независимо. Corrupt/MIME spoof — 415, bytes/pixels overflow — 413. API допускает одну проверяемую загрузку на реплику, таймаут тела 30 s и ожидания слота 20 s.
- Worker concurrency=2, каждый consumer получает максимум одну доставку за раз. Для заданного Compose memory budget разрешено concurrency 1 или 2.
- Каждый resize выполняется отдельным subprocess: wall timeout 60 s, CPU limit 60/61 s, address-space limit 640 MiB (`RLIMIT_AS`). При timeout subprocess убивается; MemoryError/resource termination не убивает основной consumer и проходит bounded retry. Worker container ограничен 1536 MiB, 2 CPU и 96 PID. Lease по умолчанию 90 s и должен превышать resize timeout минимум на 5 s.
- S3 операции имеют connect timeout 1 s/read timeout 3 s, SDK retries отключены. PostgreSQL I/O, pool wait и блокировки ограничены. CPU/resize выполняется вне DB-транзакции; транзакция публикации результата включает только fencing, четыре ограниченных PUT и COMMIT.
`WORKER_TASK_MEMORY_MB` ограничивает адресное пространство процесса, а Compose mem_limit — весь контейнер. Это разные ограничения; поддержка 40 млн pixels не обещает, что любой формат/ориентация уложится в память: превышение ресурсов заканчивается наблюдаемым failed после ограниченного бюджета, а не бесконечной обработкой.
## Staging retention и очистка
`cleanup` запускается раз в 30 s. Журнал orphan PUT старше `UPLOAD_CLEANUP_GRACE_SECONDS=3600` очищается под advisory lock: свежая/активная загрузка не удаляется. Crash после staging, но до metadata/job не дал 202; его журнал позволяет удалить orphan без обхода всего bucket.
Staging queued/processing не удаляется независимо от возраста. После безопасного COMMIT ready staging удаляется при ближайшем проходе; конечные четыре объекта сохраняются. Failed staging и частичные конечные объекты хранятся `STAGING_RETENTION_SECONDS=86400` после последнего перехода в failed, затем удаляются; metadata, failed registry и audit остаются. Операции DELETE идемпотентны, отметка staging_deleted_at фиксируется после удаления. Cleaner и replay блокируют те же строки, поэтому не удаляют staging текущей replay-работы.
## Сервисы, настройки и bootstrap
| Роль / service | Версия | Readiness / persistence / credentials |
| --- | --- | --- |
| API / api ×2 | Python 3.11.15; FastAPI 0.142.4 | non-root UID 10001, health live/ready; 384 MiB/1 CPU на реплику; Docker secrets |
| БД / postgres | PostgreSQL 17.6-bookworm | pg_isready, postgres_data; отдельный migrate с checksum/advisory lock |
| Сессии / valkey | Valkey 8.1.3 | PING, AOF always, valkey_data, внутренний порт |
| Брокер / broker | Valkey 8.1.3 | авторизованный PING, AOF always, broker_data, отдельный пароль через secret |
| Outbox / publisher | тот же собранный Python image | restart unless-stopped; DB+broker secrets; topology bootstrap идемпотентен |
| Worker / worker | тот же Python image; Pillow 13.0.0 | restart supervisor, 2 consumer-процесса, leases; DB+broker+S3 secrets |
| S3 / s3 + s3-storage | SeaweedFS 3.85 с digest | gateway probe + filer HTTP; s3_data; storage только в s3-internal |
| S3 bootstrap / s3-bootstrap | MinIO SDK 7.2.20 | root secrets только для создания приватного bucket; повтор безопасен |
| LB / lb | HAProxy 3.2.6-alpine | readiness backend; единственный порт 127.0.0.1:8080 |
| Cleanup / cleanup | тот же Python image | restart unless-stopped, DB+S3 secrets; без broker credentials |
Python base image закреплён digest в Dockerfile; S3 image — digest в Compose. Все Python-зависимости закреплены в requirements.txt. Административные интерфейсы S3 и внутренние API/БД/broker ports наружу не опубликованы. S3 application credentials разрешают работу только своего bucket/prefix attachments/*; root используется только bootstrap. API/cleaner не получают broker password. `.env.example` содержит пути secret-файлов, а не их значения.
Соответствие настроек: APP_HOST/APP_PORT — адрес API внутри контейнера; HOST_PORT — порт LB; APP_ENV — учебная метка. SESSION_STORE=valkey и SESSION_TTL_SECONDS=3600 — общее session storage; CACHE_URL_FILE/DATABASE_URL_FILE — Docker secrets. API_REPLICAS=2; IDEMPOTENCY_TTL_SECONDS≥86400. S3_ENDPOINT/REGION/BUCKET и S3_*_FILE — приватный bucket. MAX_UPLOAD_BYTES/MAX_IMAGE_PIXELS и UPLOAD_CLEANUP_GRACE_SECONDS — приём и orphan cleanup. BROKER_URL_FILE — авторизованный Redis-compatible URL отдельного broker. WORKER_CONCURRENCY, WORKER_TASK_MEMORY_MB, JOB_TIMEOUT_SECONDS, JOB_LEASE_SECONDS, JOB_MAX_ATTEMPTS, JOB_BACKOFF_SECONDS — ограничения/повторы. STAGING_RETENTION_SECONDS — срок хранения failed исходников.
/health/ready API зависит от PostgreSQL, сессий и S3: они нужны синхронному пути. Broker/worker/publisher не являются зависимостями readiness API: при их остановке original+outbox надёжно принимаются, текстовые сообщения работают, resize синхронно не запускается. У фоновых процессов нет HTTP health endpoint; состояние jobs/pending/failed наблюдается через БД, broker и операторскую CLI.
## Доказательства критериев
Проверено 12.11.2026: все **17 групп проверок** и **39 API regression tests** прошли. Docker 29.0.1, aarch64, 10 CPU, 8 217 448 448 bytes RAM Docker VM.
Полный воспроизводимый запуск: `./scripts/check-docker.sh`. Исходник сценария — `student-tests/lab5_pipeline.py`. Результаты и реальные job/attachment IDs — `evidence/lab5-checks.json`, консольный вывод — `evidence/lab5-checks.txt`. Дополнительное доказательство реального restart обычного стенда с прежними container IDs и volumes — `evidence/lab5-restart.json`; воспроизводится `.venv/bin/python student-tests/verify_restart.py`.
| Критерий README | Команда / ключ проверки evidence | Наблюдаемый результат |
| --- | --- | --- |
| Сохранён lab4, переход 201→202, smoke | make check validate test; public_smoke_and_schema, student_api_regressions | Публичный lab5 сценарий и собственные API regression tests |
| Отдельные broker/worker, persistence, лимиты | topology_and_limits | Реальные контейнеры/volumes, две API, broker AOF always, worker concurrency=2/prefetch=1 |
| Worker остановлен → 202 → сообщение → рестарты → ready | worker_offline_api_broker_restart | Тот же attachment/job после рестарта API и broker; 409 до ready; original/три варианта доступны |
| Original/thumbnail/crop/WebP по контракту | image_semantics | Bytes/checksums/MIME и реальные размеры; small/alpha/EXIF/floor cases |
| Outbox, crash COMMIT→publish | outbox_crash_before_publish | Устойчивое намерение доставляется после автоматического restart publisher |
| Неопределённый XADD, дубли | outbox_crash_after_publish | Повтор publish не создаёт второго результата |
| Crash после side effect до COMMIT/ACK | worker_crash_after_objects, worker_crash_before_ack | Lease/reclaim восстанавливают работу; четыре canonical объекта; ready дубль — no-op |
| Retry/backoff/jitter, failed registry, replay | bounded_retry_failed_registry_replay | 5 автоматических неудач → failed; audit содержит время/попытки; явный replay → ready; ready replay отклонён |
| Постоянная ошибка | permanent_error | invalid_image с первой попытки, безопасный код |
| Ограничение времени и памяти обработки | image_subprocess_timeout, image_subprocess_memory | Реальное убийство зависшего subprocess и MemoryError при выделении 1 GiB под лимитом 640 MiB; следующий retry завершает тот же job |
| Безопасная очистка staging | staging_cleanup | Active staging сохранён; ready/expired failed удалены; replay без исходника запрещён |
| Broker недоступен | broker_offline_outbox | API принимает 202 и текст; outbox доставляет после восстановления |
| Сохранность после всех рестартов | make test-persistence; public_persistence | Сессия, история и original переживают рестарты с volumes |
| Отказ API | make test-failover; public_api_failover | Другая реплика продолжает обслуживать подтверждённые сообщения без дублей |
Failpoints включаются только явным `ENABLE_FAULT_INJECTION=1` во временном тестовом project; по умолчанию отключены. Файлы управления доступны только через docker exec внутри контейнера, HTTP управления ими нет. `before_publish` завершает publisher после DB COMMIT намерения до XADD; `after_publish` — после XADD до фиксации outbox. `after_objects` завершает consumer после PUT четырёх объектов до COMMIT ready; `before_ack` — после COMMIT ready до XACK. Маркеры срабатывания проверяются тестом, поэтому простой быстрый успех без внедрённого сбоя не засчитывается. Отдельный timeout failpoint усыпляет именно ограниченный resize subprocess, не подменяет результат mock-объектом.
## Эксперименты и ограничения
Это проверка критериев на 4; исследование максимального throughput, сравнение concurrency/prefetch и versioned pipeline не выполнялись. Полный сценарий использует concurrency=2/prefetch=1, max_attempts=5; для быстрых fault checks timeout=5 s, lease=15 s, base backoff=0.15 s. Production-подобные default настройки остаются 60/90/2 s в .env.example. Дата, Docker/hardware и лимиты конкретного запуска записаны в evidence. Измерения latency/RSS этой проверки не трактуются как нагрузочный benchmark.
Обработка at-least-once, не exactly-once: XADD и PUT могут повториться, идемпотентен конечный эффект и его публикация. Сбой host/disk с потерей volumes, потеря единственной PostgreSQL/S3 или ручное удаление объектов не покрываются. Backup/replication этих хранилищ в lab5 не добавлялись. Очередь ограничена maxmemory/noeviction broker; outbox и staging во время длительного простоя требуют диска, общей квоты принятой работы эта оценка не требует. Повторы самого HTTP upload могут создать отдельные вложения по контракту.
## Первичные источники
- [Задание и объявленные переходы](README.md), [состояния](contracts/attachment-states.md), [схема job](contracts/jobs.schema.json).
- [Redis Streams: consumer groups, pending, acknowledgements](https://redis.io/docs/latest/develop/data-types/streams/).
- [XAUTOCLAIM: восстановление pending](https://redis.io/docs/latest/commands/xautoclaim/).
- [Valkey persistence: AOF и fsync](https://valkey.io/topics/persistence/).
- [Pillow ImageOps: EXIF transpose и центрированный fit](https://pillow.readthedocs.io/en/stable/reference/ImageOps.html).
- [Python resource limits](https://docs.python.org/3/library/resource.html).
+208 -24
View File
@@ -1,37 +1,221 @@
# Отчёт — лабораторная 6
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
- Автор, группа: Савельев Иван, БВТ2304.
- Целевая оценка: **5**, трек 2 — **SLO и обнаружение инцидентов**. Реализованы уровни 3 и 4.
- Основа: готовая lab5, включая асинхронный pipeline и все её проверки на 4. Исторический отчёт — [REPORT.lab5.md](REPORT.lab5.md).
- Commit/PR: рабочее дерево поверх исходного skeleton; публикация выполняется отдельно. Проверенные файлы перечислены с SHA-256 в `evidence/lab6-source-manifest.json`.
- Помощь ИИ: Codex использован для переноса, реализации, проверок и подготовки отчёта. Evidence получен реальными запусками Docker и запросами к хранилищам.
- Проверено 19.11.2026 на macOS arm64, Docker Desktop 29.0.1: 10 CPU, 8217448448 bytes RAM (≈7.65 GiB) доступны Docker VM. Это учебный стенд на одном host.
## Запуск из чистого clone
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
Нужны Docker Engine/Desktop с Compose v2, Python 3.11, Bash и make. Интернет нужен для первой установки зависимостей и загрузки закреплённых образов. Все команды из корня:
## Архитектура и решения
```sh
python3 -m venv .venv
.venv/bin/python -m pip install -r requirements.txt -r requirements-dev.txt
make check validate PYTHON=.venv/bin/python
./scripts/start.sh
./scripts/check.sh
```
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab5, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
`start.sh` создаёт локальные `.env` и `secrets/`, собирает приложение, применяет миграции и bootstrap приватного S3 bucket, поднимает API/worker и наблюдаемость. Затем ожидает две API-реплики через LB, scrape Collector, все хранилища и provisioning Grafana. Ручной импорт dashboard не нужен.
- API: http://localhost:8080; health: `/health/live`, `/health/ready`.
- [Grafana dashboard](http://localhost:3000/d/messenger-lab6): локальный anonymous Viewer; пароли приложения в Grafana не вводятся.
- Порты меняются через `HOST_PORT` и `GRAFANA_PORT` в `.env`; для API-тестов передать соответствующий `BASE_URL`. Драйвер наблюдаемости определяет опубликованный порт Grafana через Compose; можно задать `GRAFANA_URL`.
- Compose project `mtusi-lab6`, приложение `mtusi-messenger:lab6`; volumes отделены от lab5. Локальные `.env`/secrets не входят в Git и Docker build context. В новом clone credentials генерируются заново.
```sh
# Весь минимум lab5 + API/telemetry/persistence + исследование lab6:
./scripts/check-lab6.sh
# Только исследование трека 2 на уже запущенном синтетическом стенде (~11 минут):
./scripts/research.sh
# Остановка с сохранением данных / повторный запуск:
./scripts/stop.sh
./scripts/start.sh
# Перезапуск процессов и хранилищ без удаления volumes:
./scripts/restart.sh
# Только намеренное удаление данных СВОЕГО учебного Compose project:
docker compose down --volumes
```
Полный script сначала проверяет lab5 в отдельном временном project на свободных портах и новых volumes, затем удаляет только этот временный стенд. Исследование lab6 использует текущий локальный синтетический стенд и намеренно создаёт ограниченные отказы. Стандартная работа приложения не требует `ENABLE_FAULT_INJECTION`; проверка retry временно включает его только для worker и затем выключает.
## Архитектура и версии
```text
HTTP → HAProxy → API ×2 → PostgreSQL / Valkey sessions / private S3
└ durable metadata + job + outbox (W3C context)
→ publisher → Valkey Streams → worker ×2
→ S3 GET/PUT → DB ready → ACK
API / publisher / worker → OTLP HTTP → Collector
├ Prometheus exporter → Prometheus ← scrape
├ OTLP HTTP → Loki
└ OTLP gRPC → Tempo
↑
Grafana: metrics / logs / traces
```
API, ACL, сессии, идемпотентность, ограничения изображений, at-least-once/retry/replay и очистка staging перенесены из lab5. Обязательные контракты и публичные tests сохранены. Миграции 001–004 не изменены; 005 добавляет сохранённый trace context и `ready_at`. Новые поля не меняют бизнес-API. Источник истины — PostgreSQL; ACK по-прежнему выполняется после устойчивого результата/retry.
| Роль / service | Закреплённая версия | Persistence / readiness / credentials |
| --- | --- | --- |
| API / worker / publisher / cleanup / migrate | Python 3.11.15, FastAPI 0.142.4; image `mtusi-messenger:lab6` | Исходники и lock версий в requirements; API `/health/ready`, worker читает broker; отдельный migrate проверяет checksum. Docker secrets, non-root UID 10001 |
| PostgreSQL / postgres | 17.6-bookworm | `postgres_data`, pg_isready; password/DSN из secrets |
| Сессии / valkey | 8.1.3 | `valkey_data`, AOF fsync always; ping; только внутренняя сеть |
| Очередь / broker | Valkey 8.1.3 | `broker_data`, AOF fsync always; authenticated ping; password/URL из secrets |
| S3 / s3, s3-storage | SeaweedFS 3.85 с digest | `s3_data`, внутренний TCP/HTTP readiness; private bucket и service account создаёт bootstrap; secrets |
| LB / lb | HAProxy 3.2.6-alpine | `/health/ready`; только `127.0.0.1:8080`; API-порты напрямую не опубликованы |
| Collector / otel-collector | contrib 0.123.0 | Config `infra/otel/collector.yaml`, health 13133 и self metrics 8888 внутри сети; долговременной БД не является |
| Метрики / prometheus | 3.4.1 | `prometheus_data`, retention 7d, scrape/evaluation 5s; rules в Git |
| Логи / loki | 3.5.1 | `loki_data`, TSDB v13/filesystem, retention 7d, native OTLP со structured metadata |
| Трассы / tempo | 2.8.1 | `tempo_data`, WAL + local blocks, retention 7d; OTLP receiver только внутри сети |
| Grafana / grafana | 12.0.2 | `grafana_data`, `/api/health`; 3 datasources + dashboard provisioned из Git; localhost:3000, Viewer |
Grafana запрашивает хранилища, сама не хранит три сигнала. Collector принимает/буферизует/экспортирует, долгую историю хранят backends. У всех контейнеров заданы CPU/RAM limits; точные значения, volumes и опубликованные порты — `lab6-runtime.json`. Collector: 1 CPU/256 MiB; Prometheus/Loki/Grafana: по 1 CPU/384 MiB; Tempo: 1 CPU/512 MiB с GOMEMLIMIT 384 MiB. Приложение сохранило ограничения lab5.
SDK/API/exporters OTel **1.45.1**, semantic conventions **0.66b1**; полный список зависимостей закреплён в `requirements.txt`. Используются явные span/metric имена, а не автосбор SQL/HTTP headers. Встроенная автоматическая telemetry FastAPI отключена: один SDK и один набор инструментов без дублирования сигналов. Counter series для известных маршрутов и status classes инициализированы нулём до первого scrape: короткий первый всплеск ошибок не теряется из rate из-за отсутствия исходного нулевого sample. Фиктивные запросы или histogram observations для этого не создаются.
`.env.example` соответствует настройкам: `DATABASE_URL_FILE`, `CACHE_URL_FILE`, S3 и `BROKER_URL_FILE` читают Docker secrets; worker concurrency/retry/lease/time/memory — перенесённые настройки pipeline; `OTEL_EXPORTER_OTLP_ENDPOINT` — общий Collector, protocol HTTP/protobuf. Resource `service.name` устанавливается отдельно для API, worker и publisher; `service.version=6.0.0`, instance — hostname:pid, environment — lab. `OTEL_PYTHON_SDK_INTERNAL_METRICS_ENABLED=true` включает контроль очередей SDK. Перезапуск сохраняет volumes всех signal stores.
## Сигналы и запросы
[Dashboard JSON](infra/grafana/dashboards/messenger.json) содержит 19 панелей; datasources/provider рядом в `infra/grafana/provisioning/`. `lab6-dashboard-checks.json` подтверждает выполнение запросов ко всем трём источникам.
| Реальное имя в Prometheus | Тип / единицы / labels |
| --- | --- |
| `messenger_http_requests_total` | counter запросов; method, route template, status_class; health исключён |
| `messenger_http_duration_seconds_*` | histogram, секунды до response headers; те же labels |
| `messenger_job_attempts_total` | counter попыток; job_type, outcome (ready/retry/failed/error/duplicate/busy/obsolete/invalid) |
| `messenger_job_duration_seconds_*` | histogram длительности consumer attempt, секунды; job_type/outcome |
| `messenger_attachment_ready_duration_seconds_*` | histogram acceptance → ready успешных jobs, секунды; job_type |
| `messenger_queue_depth` / `messenger_queue_oldest_age_seconds` | gauges: незавершённые durable jobs / возраст самой старой, секунды; queue=attachments |
| `messenger_db_pool` | gauge соединений; state=used/waiting |
| `messenger_worker_busy` / `messenger_worker_capacity` | gauges занятых/доступных worker процессов |
| `messenger_process_cpu_time_seconds_total` / `messenger_process_memory_bytes` | process CPU seconds / RSS bytes; группировка по service_name |
| `messenger_ready_deadlines_total` | durable cohort counter; outcome=good/bad, включая failed/stuck/late jobs |
`_bucket`, `_sum`, `_count` — части одного histogram. Resource labels добавляются Collector; технический instance ID нужен для суммирования процессов и различения перезапусков. User/chat/message/attachment/request IDs, raw URL, filename и текст labels не являются. Лимит экспирации неактивных metric series Collector — 30s.
```promql
# RPS (без health), ошибки 5xx / все business requests:
sum(rate(messenger_http_requests_total[1m]))
lab6:api_bad_ratio:1m
# p95 по всем API-репликам; для p50/p99 меняется quantile:
histogram_quantile(0.95,
sum by (le) (rate(messenger_http_duration_seconds_bucket[1m])))
max(messenger_queue_depth)
max(messenger_queue_oldest_age_seconds)
sum by (state, service_name) (messenger_db_pool)
sum(messenger_worker_busy) / sum(messenger_worker_capacity)
```
Buckets в секундах: .005, .01, .025, .05, .1, .25, .5, 1, 2, 5, 10, 20, **30**, 60, 120, 300, 600, +Inf. Percentiles считаются из суммарных buckets, не средних latency/p95 реплик. Backlog не подменяет processing duration. `s3.get_object` измеряет получение ответа S3; чтение stream входит в общий consumer span.
Лог — структурный JSON с timestamp, severity, service.name, event, trace_id/span_id; HTTP содержит также `request_id`, совпадающий с X-Request-Id. В ответе есть X-Trace-Id. Только job/attachment IDs разрешены в явных диагностических событиях для синтетических заданий; в labels их нет. Пароли, Authorization/session tokens, bodies, изображения, filenames/storage keys/signed URLs и SQL/params не собираются. Exception events с текстом/stack trace отключены; сохраняется только класс ошибки. Loki индексирует service/environment, instance — structured metadata.
W3C traceparent/tracestate сохраняются в job той же транзакцией, что metadata/outbox. Producer восстанавливает parent, инжектирует свой context в broker payload. Consumer извлекает его; retry сохраняет актуальный context для следующей публикации. После перезапуска процессов связь сохраняется. [Сохранённый trace с retry](evidence/lab6-trace-retry.json) и [логи](evidence/lab6-correlated-logs.json) показывают реальную цепочку API → producer → consumer → retry → producer → consumer → S3/DB. Sampling 100% для учебного стенда.
## SLI/SLO и alerts
- **API availability ≥99% за скользящий час**. Bad — instrumented business responses 5xx; denominator — все business requests, включая 4xx; health исключён. Неверный пароль не является серверной ошибкой. Нужны внешние probes для отказов LB/сети до API; в этот SLI они не входят.
- **Ready ≥95% за ≤30s, окно час**. Acceptance timestamp — `job.created_at` в durable metadata transaction; запись становится видна после COMMIT. По `ready_at` оценивается каждый cohort после истечения deadline. Позднее завершение, failed и незавершившееся задание считаются bad. Часовое окно относится к созревшим cohorts, acceptance сдвинута на 30s. В отличие от histogram только successful jobs, SLI не скрывает остановку worker.
Обоснование: 99% availability задаёт заметный бюджет серверных ошибок для учебного single-host стенда с единственной БД; это проверяемая цель, а не обещание production SLA. Для асинхронного изображения 30s оставляют запас на S3, ожидание публикации и краткий retry; минуты без обработки уже нарушают пользовательское ожидание. Цель 95% допускает редкие тяжёлые изображения/повторы, сохраняя чувствительность к остановке worker. При нашем постоянном профиле час содержит около 9000 HTTP requests и 1800 jobs: budgets — 90 ошибок и 90 просроченных jobs. Четыре ошибки короткого всплеска расходуют около 4.4% часового API budget, ниже порогов эскалации 20%/30%. Час выбран для воспроизводимого учебного исследования; production потребует отдельного обоснования периода и целей.
| Alert policy | Long / short window | Burn rate / порог API / ready | for / guards |
| --- | --- | --- | --- |
| Fast | 2m AND 30s | 6× / 6% / 30% bad | 15s; long ≥30 API / ≥5 jobs, short ≥2 события |
| Slow | 6m AND 1m | 3× / 3% / 15% bad | 30s; те же guards |
| APIHighErrorRate (сравнение) | 30s | >5% 5xx | 5s |
| AttachmentQueueLag | текущие age/depth | oldest >20s и depth >0 | 10s |
Fast и slow одновременно проверяют оба окна. Это учебный **1h SLO**: fast соответствует расходу 20% бюджета за 2m, slow — 30% за 6m (`burn × long_window / 1h`). Production окна из примера 30-дневного SLO не копируются механически. Guards снижают шум при малом числе событий. При нулевом denominator ratio — NaN, а не искусственные 0% ошибок/100% availability; отдельно проверяются scrape и трафик. Есть CollectorUnavailable warning.
Rules/условия/описания/for/runbook находятся в [alerts.yaml](infra/otel/alerts.yaml). Доставка наружу не нужна; firing/inactive (resolved) наблюдаются через Prometheus API и панель ALERTS. [Runbook](docs/runbook.md) содержит запросы и порядок диагностики другим студентом.
## Эксперимент трека 2 и разбор инцидентов
Гипотеза, заранее заданная методика, нагрузка и ограничения — [research/README.md](research/README.md). Один real-time прогон: 2 сообщения/s + 0.5 PNG/s, 16 max in-flight, без retries; baseline 120s, spike БД 4s, recovery 60s, lock messages 90s, recovery 100s, worker stopped 120s, recovery 120s. SQL-lock ограничен pg_sleep; worker восстанавливается в finally. Каждые 5s читаются сохранённые alert states/метрики. Исследование выполняется отдельно от полного lab5 regression, чтобы они не конкурировали за ресурсы.
Финальный прогон **19.11.2026, 18:26:14–18:36:34 МСК**, 619.4s. 1544 ответов, пропущенных arrivals: 0. p50/p95/p99 ниже рассчитаны по всем ответам, включая 5xx; successful RPS — только 2xx.
[Открыть этот интервал в Grafana](http://localhost:3000/d/messenger-lab6?from=1795101974987&to=1795102594377).
| Фаза | Запросы / 5xx | Successful RPS | p50 / p95 / p99, ms |
| --- | --- | --- | --- |
| Baseline | 301 / 0 | 2.51 | 16.6 / 31.3 / 35.2 |
| Короткий DB spike, 4s | 10 / 4 | 1.50 | 1670.7 / 2017.2 / 2017.2 |
| После короткого всплеска | 150 / 0 | 2.50 | 16.5 / 30.7 / 32.8 |
| Длительное ожидание DB, 90s | 225 / 176 | 0.54 | 2011.8 / 2022.5 / 2025.8 |
| После восстановления DB | 250 / 0 | 2.50 | 16.3 / 31.0 / 33.4 |
| Worker stopped, 120s | 300 / 0 | 2.50 | 16.2 / 31.3 / 35.5 |
| После восстановления worker | 300 / 0 | 2.50 | 13.1 / 30.7 / 35.3 |
| Alert | Detection после воздействия, s | Resolved после восстановления, s |
| --- | --- | --- |
| APIAvailabilityFastBurn | 40.2 | 35.1 |
| APIAvailabilitySlowBurn | 60.3 | 65.3 |
| APIHighErrorRate | 25.1 | 35.1 |
| AttachmentQueueLag | 45.2 | 30.1 |
| AttachmentReadyFastBurn | 95.5 | 45.2 |
| AttachmentReadySlowBurn | 131.6 | 75.3 |
Baseline: **0** firing alerts. Короткий реальный всплеск виден в сохранённой 30s error ratio (peak **6.37%**); burn alerts на него сработали **0** раз. Single-window comparator: **APIHighErrorRate**.
В исследовании false positive означает эскалацию короткого всплеска, заранее принятого допустимым для выбранной SLO policy; предупреждение APIHighErrorRate при этом корректно соответствует собственному порогу.
Все заявленные alerts наблюдались firing и затем inactive/resolved. Сохранённый worker trace подтверждает consumer спустя **94.3s** после upload, уже после восстановления worker.
| Процесс | Max 1m CPU rate, cores | Max RSS, MiB |
| --- | --- | --- |
| messenger-api | 0.149 | 603.75 |
| messenger-publisher | 0.012 | 147.05 |
| messenger-worker | 0.013 | 319.40 |
Это maxima точек с шагом 5s; CPU усреднён по минуте, RSS суммирован по процессам сервиса. Два consumer процесса worker показаны суммарно. Исходные samples, timestamps и все запросы без credentials сохранены в `lab6-slo-research.json`, ряды — в `lab6-slo-series.json`.
Причина деградации API локализуется по сохранённым наблюдениям: 5xx и HTTP latency растут, внутри server trace `postgresql.SELECT` занимает около 2s и заканчивается `_WaitTimeout`, тогда как acceptance и worker продолжаются. Это I/O ожидание PostgreSQL, а не искусственный sleep HTTP handler. Конкретные spans/durations — `lab6-db-incident-trace.json`, коррелированные request logs — `lab6-db-incident-logs.json`. Методика внедряет SQL lock, но по одному сохранённому timeout нельзя доказать именно блокировку вместо другой причины медленного ответа БД; этот предел диагностики явно учитывается.
При остановке worker acceptance/producer продолжаются, successful processing прекращается, capacity исчезает после истечения series, backlog/oldest age растут. В trace принятого upload до восстановления отсутствует consumer; после восстановления очередь дренируется, появляются S3/DB ready spans, но просроченные cohorts сохраняют bad исход. Это проверено по timestamps consumer в `lab6-worker-incident-trace.json` и связанным `lab6-worker-incident-logs.json`. Подробные ряды, CPU/RSS, pool и saturation — `lab6-slo-series.json`.
## Отказ telemetry backend и границы буферизации
Collector memory limiter: 192 MiB со spike 32 MiB, check 1s; hard container limit 256 MiB. Batch максимум 128 записей; отдельные exporter queues по 64 batch, 2 consumers, timeout 2s, retry максимум 15s. SDK trace/log queues по 256 записей, batch 64, фоновый export; metric export каждые 5s. Internal SDK queue capacity/size и Collector exporter metrics доступны в Prometheus. Бесконечной буферизации нет: при недоступном backend telemetry теряется, бизнес-запрос не ждёт экспорт.
Контролируемая остановка Loki проверена `student-tests/lab6_observability.py`: отказ около **51.39s**, **24/24 сообщения записаны**, история читается. После восстановления не найдены **6 из 24** соответствующих request logs; это фактически измеренная потеря по известным request IDs. RAM Collector в четырёх samples — **70.46–72.80 MiB** при limit 256 MiB, OOM=false. Отдельный счётчик failed exports вырос на 6; вообще его нельзя считать числом потерянных записей без проверки из-за retries. Raw результаты — `lab6-checks.json`. Grafana/Prometheus/Tempo сохраняли доступность; Loki после запуска снова принимает OTLP. Потери не маскируются debug exporter или stdout.
## Доказательства критериев
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
## Эксперименты и отказы
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
| Критерий | Команда/сценарий | Наблюдение / evidence |
| --- | --- | --- |
| Весь minimum lab5 и сохранённые переходы | `.venv/bin/python -u student-tests/lab5_pipeline.py` (тот же stage в check-lab6) | 17 групп PASS, включая outbox/worker crash, retry/replay, processing limits, cleanup, broker outage, persistence/failover; `lab6-lab5-regression.json/.txt` |
| Контракт, публичный smoke, свои API-тесты с telemetry | make check validate test; unittest discover | PASS и 39 tests OK; `lab6-api-checks.txt` |
| Collector + три persistent stores + Grafana | start.sh; readiness/provisioning | 2 API, CPU/RAM limits, localhost порты, volumes; `lab6-runtime.json` |
| Все обязательные metrics и dashboard queries | lab6_observability; check_dashboard | Реальные series и 19 панелей трёх источников; `lab6-checks.json`, `lab6-dashboard-checks.json` |
| API/worker logs + traces, W3C, retry → S3/DB | Deterministic transient worker failpoint | 59 связанных spans; `lab6-trace-retry.json`, `lab6-correlated-logs.json` |
| Отсутствие секретов/body/ID labels | Canary body/filename/password/token + metric label check | PASS; `lab6-checks.json` |
| Telemetry persistence | Restart всех постоянных контейнеров и запрос старого trace | IDs/volumes сохранены; `lab6-restart.json/.txt`, `lab6-telemetry-persistence.json` |
| Ограниченный отказ Loki | Stop/start Loki, 24 записи под отказом, memory samples | API работает, потери logs измерены, OOM=false; `lab6-checks.json/.txt` |
| SLO/rules и пограничные сценарии | promtool test rules | Spike/sustained/recovery/stalled/no traffic PASS; `lab6-promtool.txt` |
| Два инцидента + трек 2 | research.sh, runbook | Реальные firing/resolved, detection/recovery/false positives, stored observations; `lab6-slo-research.json/.txt`, `lab6-slo-series.json` |
## Ограничения и следующие шаги
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
Один host, единственные PostgreSQL/S3/telemetry stores: потеря host/disk/volumes не покрыта. Persistent volumes и проверенный restart не означают HA или backup. Срок хранения telemetry 7d; heap/queues ограничены, при длительном отказе часть наблюдений теряется. Нет внешней доставки alerts и внешнего black-box SLI LB/сети: это не требуется выбранным минимумом. Histogram HTTP измеряет время до headers; полный stream download отдельно не профилируется. Queue depth берётся из durable state БД, а не только broker.
Эксперимент короче часа и выполнен по одному live прогону каждого отказа на небольшой синтетической нагрузке. Он проверяет работу детектора и воспроизводимость диагностики; не доказывает часовую/месячную надёжность, предел RPS или универсальную частоту ложных срабатываний. Точность detection/resolution ограничена шагом 5s плюс SDK export/scrape. Часовые ratio при неполной истории — только доступная часть окна. CPU/RSS относятся к долгоживущим instrumented процессам; resize subprocess ограничен общим Docker limit и отдельные resource series не экспортирует. SQL cohort scan достаточен для учебного объёма; на больших объёмах понадобится incremental aggregation с сохранением состояния.
## Первичные источники
- [OpenTelemetry Collector configuration](https://opentelemetry.io/docs/collector/configuration/) и [internal telemetry](https://opentelemetry.io/docs/collector/internal-telemetry/).
- [OTel Python SDK](https://opentelemetry.io/docs/languages/python/) и [W3C Trace Context](https://www.w3.org/TR/trace-context/).
- [Native OTLP в Loki](https://grafana.com/docs/loki/latest/send-data/otel/).
- [Prometheus histograms](https://prometheus.io/docs/practices/histograms/) и [configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/).
- [Google SRE Workbook: multiwindow, multi-burn-rate alerts](https://sre.google/workbook/alerting-on-slos/) — принцип двух окон, адаптированный к явно выбранному учебному периоду.
- [Grafana provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/); compatibility TraceQL query проверена по source Grafana v12.0.2 и реальному Tempo search API.
+289
View File
@@ -0,0 +1,289 @@
name: mtusi-lab6
x-application: &application
image: mtusi-messenger:lab6
env_file: .env
environment: &pipeline-environment
OTEL_PYTHON_SDK_INTERNAL_METRICS_ENABLED: "true"
JOB_MAX_ATTEMPTS: ${JOB_MAX_ATTEMPTS:-5}
JOB_TIMEOUT_SECONDS: ${JOB_TIMEOUT_SECONDS:-60}
JOB_LEASE_SECONDS: ${JOB_LEASE_SECONDS:-90}
JOB_BACKOFF_SECONDS: ${JOB_BACKOFF_SECONDS:-2}
WORKER_CONCURRENCY: ${WORKER_CONCURRENCY:-2}
WORKER_TASK_MEMORY_MB: ${WORKER_TASK_MEMORY_MB:-640}
ENABLE_FAULT_INJECTION: ${ENABLE_FAULT_INJECTION:-0}
cpus: 1
mem_limit: 384m
x-postgres: &postgres
image: postgres:17.6-bookworm
environment:
POSTGRES_USER: messenger
POSTGRES_DB: messenger
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
secrets: [postgres_password]
cpus: 1
mem_limit: 512m
healthcheck:
test: ["CMD-SHELL", "pg_isready -U messenger -d messenger"]
interval: 2s
timeout: 3s
retries: 30
services:
postgres:
<<: *postgres
volumes: [postgres_data:/var/lib/postgresql/data]
valkey:
image: valkey/valkey:8.1.3
command: ["valkey-server", "--appendonly", "yes", "--appendfsync", "always", "--save", "", "--maxmemory", "64mb", "--maxmemory-policy", "noeviction"]
volumes: [valkey_data:/data]
cpus: 0.5
mem_limit: 128m
healthcheck:
test: ["CMD", "valkey-cli", "ping"]
interval: 2s
timeout: 3s
retries: 30
broker:
image: valkey/valkey:8.1.3
command: ["sh", "-c", 'exec valkey-server --appendonly yes --appendfsync always --save "" --maxmemory 128mb --maxmemory-policy noeviction --requirepass "$$(cat /run/secrets/broker_password)"']
secrets: [broker_password]
volumes: [broker_data:/data]
cpus: 0.5
mem_limit: 256m
healthcheck:
test: ["CMD-SHELL", 'REDISCLI_AUTH="$$(cat /run/secrets/broker_password)" valkey-cli ping | grep -q PONG']
interval: 2s
timeout: 3s
retries: 30
publisher:
<<: *application
command: ["python", "-m", "src.publisher"]
secrets: [database_url, broker_url]
restart: unless-stopped
healthcheck:
disable: true
depends_on:
migrate: {condition: service_completed_successfully}
broker: {condition: service_healthy}
worker:
<<: *application
command: ["python", "-m", "src.worker"]
secrets: [database_url, broker_url, s3_access_key_id, s3_secret_access_key]
restart: unless-stopped
mem_limit: 1536m
cpus: 2
pids_limit: 96
healthcheck:
disable: true
depends_on:
migrate: {condition: service_completed_successfully}
broker: {condition: service_healthy}
s3-bootstrap: {condition: service_completed_successfully}
stop_grace_period: 8s
api:
<<: *application
build: .
deploy:
replicas: ${API_REPLICAS:-2}
environment:
<<: *pipeline-environment
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
secrets: [database_url, cache_url, s3_access_key_id, s3_secret_access_key]
depends_on:
postgres: {condition: service_healthy}
valkey: {condition: service_healthy}
migrate: {condition: service_completed_successfully}
s3: {condition: service_healthy}
s3-bootstrap: {condition: service_completed_successfully}
stop_grace_period: 8s
migrate:
<<: *application
command: ["python", "-m", "src.migrate"]
secrets: [database_url]
healthcheck:
disable: true
depends_on:
postgres: {condition: service_healthy}
s3:
image: chrislusf/seaweedfs:3.85@sha256:49312939c00c01e5ee6afbd7d728b18027821d3764c35a797a72acd4fdf3296a
command: ["s3", "-filer=s3-storage:8888", "-ip.bind=0.0.0.0", "-port=9000", "-config=/run/secrets/s3_config"]
secrets: [s3_config]
networks: [default, s3-internal]
depends_on:
s3-storage: {condition: service_healthy}
cpus: 0.5
mem_limit: 256m
healthcheck:
test: ["CMD", "nc", "-z", "127.0.0.1", "9000"]
interval: 2s
timeout: 2s
retries: 30
s3-storage:
image: chrislusf/seaweedfs:3.85@sha256:49312939c00c01e5ee6afbd7d728b18027821d3764c35a797a72acd4fdf3296a
command: ["server", "-dir=/data", "-ip=s3-storage", "-ip.bind=0.0.0.0", "-filer", "-master.volumeSizeLimitMB=64", "-volume.max=64", "-volume.preStopSeconds=0"]
volumes: [s3_data:/data]
networks: [s3-internal]
cpus: 1
mem_limit: 512m
healthcheck:
test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:8888/"]
interval: 2s
timeout: 2s
retries: 30
s3-bootstrap:
<<: *application
command: ["python", "-m", "src.bootstrap_s3"]
environment:
S3_ROOT_USER_FILE: /run/secrets/s3_root_user
S3_ROOT_PASSWORD_FILE: /run/secrets/s3_root_password
secrets: [s3_root_user, s3_root_password]
cpus: 0.5
mem_limit: 128m
healthcheck:
disable: true
depends_on:
s3: {condition: service_healthy}
cleanup:
<<: *application
command: ["python", "-m", "src.cleanup"]
secrets: [database_url, s3_access_key_id, s3_secret_access_key]
restart: unless-stopped
healthcheck:
disable: true
depends_on:
migrate: {condition: service_completed_successfully}
s3-bootstrap: {condition: service_completed_successfully}
lb:
image: haproxy:3.2.6-alpine
ports: ["127.0.0.1:${HOST_PORT:-8080}:8080"]
volumes: [./infra/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro]
cpus: 0.5
mem_limit: 64m
depends_on:
api: {condition: service_healthy}
stop_grace_period: 10s
healthcheck:
test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:8080/health/ready"]
interval: 2s
timeout: 2s
retries: 3
otel-collector:
image: otel/opentelemetry-collector-contrib:0.123.0
command: ["--config=/etc/otelcol/config.yaml"]
volumes: [./infra/otel/collector.yaml:/etc/otelcol/config.yaml:ro]
depends_on: [loki, tempo]
cpus: 1
mem_limit: 256m
restart: unless-stopped
prometheus:
image: prom/prometheus:v3.4.1
command: ["--config.file=/etc/prometheus/prometheus.yaml", "--storage.tsdb.path=/prometheus", "--storage.tsdb.retention.time=7d"]
volumes:
- ./infra/otel/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro
- ./infra/otel/alerts.yaml:/etc/prometheus/alerts.yaml:ro
- prometheus_data:/prometheus
depends_on: [otel-collector]
cpus: 1
mem_limit: 384m
restart: unless-stopped
loki:
image: grafana/loki:3.5.1
command: ["-config.file=/etc/loki/config.yaml"]
volumes:
- ./infra/otel/loki.yaml:/etc/loki/config.yaml:ro
- loki_data:/loki
cpus: 1
mem_limit: 384m
restart: unless-stopped
tempo:
image: grafana/tempo:2.8.1
environment:
GOMEMLIMIT: 384MiB
command: ["-config.file=/etc/tempo/config.yaml"]
volumes:
- ./infra/otel/tempo.yaml:/etc/tempo/config.yaml:ro
- tempo_data:/var/tempo
cpus: 1
mem_limit: 512m
restart: unless-stopped
grafana:
image: grafana/grafana:12.0.2
environment:
GF_AUTH_ANONYMOUS_ENABLED: "true"
GF_AUTH_ANONYMOUS_ORG_ROLE: Viewer
GF_AUTH_BASIC_ENABLED: "false"
GF_AUTH_DISABLE_LOGIN_FORM: "true"
GF_ANALYTICS_REPORTING_ENABLED: "false"
GF_ANALYTICS_CHECK_FOR_UPDATES: "false"
GF_PLUGINS_PREINSTALL_DISABLED: "true"
GF_PLUGINS_CHECK_FOR_UPDATES: "false"
GF_NEWS_NEWS_FEED_ENABLED: "false"
ports: ["127.0.0.1:${GRAFANA_PORT:-3000}:3000"]
volumes:
- ./infra/grafana/provisioning:/etc/grafana/provisioning:ro
- ./infra/grafana/dashboards:/var/lib/grafana/dashboards:ro
- grafana_data:/var/lib/grafana
depends_on: [prometheus, loki, tempo]
cpus: 1
mem_limit: 384m
restart: unless-stopped
postgres-restore:
<<: *postgres
profiles: [research]
volumes: [restore_data:/var/lib/postgresql/data]
volumes:
prometheus_data:
loki_data:
tempo_data:
grafana_data:
postgres_data:
valkey_data:
restore_data:
s3_data:
broker_data:
networks:
s3-internal:
internal: true
secrets:
broker_password:
file: ./secrets/broker_password
broker_url:
file: ./secrets/broker_url
postgres_password:
file: ./secrets/postgres_password
database_url:
file: ./secrets/database_url
cache_url:
file: ./secrets/cache_url
s3_root_user:
file: ./secrets/s3_root_user
s3_root_password:
file: ./secrets/s3_root_password
s3_access_key_id:
file: ./secrets/s3_access_key_id
s3_secret_access_key:
file: ./secrets/s3_secret_access_key
s3_config:
file: ./secrets/s3_config
+91
View File
@@ -0,0 +1,91 @@
# Runbook: лабораторная 6
Открыть [dashboard](http://localhost:3000/d/messenger-lab6). Выбрать время инцидента,
а не только последние пять минут. Все шаги диагностики ниже используют сохранённые
Prometheus/Loki/Tempo данные. Команды управления выполняются **после** диагноза.
Внешняя доставка alerts не настроена: состояние проверяется в Prometheus через
Grafana datasource proxy (`research/observations.py`).
## api
1. Проверить `API 5xx / all business requests`, RPS и p95/p99. Health и ошибочный
пароль не являются серверными 5xx. Посмотреть обе пары burn rate: 2m/30s и 6m/1m.
2. В Explore → Loki выполнить
`{service_name="messenger-api"} | json | status >= 500`.
По `request_id` найти ответ пользователя; скопировать `trace_id` или нажать TraceID.
3. В Tempo открыть trace. Развернуть server span и `postgresql.transaction`:
медленный `postgresql.SELECT/INSERT` с `error.type=_WaitTimeout` и длительностью
около 2s указывает на срабатывание I/O timeout ожидания PostgreSQL. Если сервер
успел вернуть `LockNotAvailable`, дополнительно подтверждена блокировка.
По одному `_WaitTimeout` нельзя отличить SQL lock от других причин медленной БД.
Это отличается от медленного handler:
при нём server span длинный, а DB spans короткие. `db.pool{state="used"}`
показывает занятые соединения, `waiting` — ожидание свободного соединения.
Наш эксперимент блокирует только messages, поэтому health остаётся успешным.
4. После определения причины дождаться окончания ограниченного failpoint.
Для досрочного восстановления своего учебного эксперимента:
```sh
docker compose exec -T postgres psql -U messenger -d messenger -c \
"SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE application_name='lab6-incident'"
```
5. Проверить успешные записи, короткие DB spans и спад error ratio.
Alert должен перейти firing → inactive (resolved); long-window ratio может
оставаться высоким, но short window уже должен очищаться. Состояния измеряются
раз в 5s; не объявлять recovery по одному успешному запросу.
## ready
1. Проверить `Outstanding jobs` и `Oldest unfinished job age`. Это очередь
незавершённых **durable jobs в БД**, включая outbox/retry/processing;
длина Redis Stream сама по себе не отражает всю принятую работу.
2. Сопоставить worker capacity/busy, process CPU/RSS и rate успешных jobs.
Если capacity исчезла, successful jobs остановились, а acceptance продолжается
и backlog/age растут — worker не обрабатывает очередь. После 30s устаревшие
gauge series удаляются Collector; краткое сохранение старого значения допустимо.
3. В Loki:
`{service_name="messenger-api"} | json | event="attachment.accepted"`.
Открыть trace: виден API → `attachments publish`, но в интервале простоя
нет нового `attachments process`. После восстановления в том же trace появятся
consumer → S3 GET/PUT → PostgreSQL UPDATE. Retry — отдельные producer/consumer
spans с возрастающим `job.attempt` и логом `job.retry`.
4. Восстановить worker: `docker compose start worker`.
5. Проверить уменьшение backlog до 0, age до 0, возврат successful jobs и capacity.
Ready SLO остаётся нарушенным для опоздавших заданий: позднее ready не превращает
уже нарушенный 30s deadline в успешный. Дождаться resolved в обеих парах окон.
## telemetry
1. Проверить `up{job="collector"}`, `otelcol_exporter_queue_size` и
`otelcol_exporter_send_failed_log_records_total`. Ошибки экспортёра сами по себе
не равны числу окончательно потерянных записей: возможны retries.
2. При живом Collector и ошибках только `otlphttp/loki` проверить доступность
datasource Loki. Если API принимает сообщения и метрики/traces приходят,
это отдельный отказ хранилища логов, а не отказ бизнес-API.
3. Восстановить `docker compose start loki`. Collector ограничен 256 MiB,
memory limiter 192 MiB, очередь каждого экспортёра — 64 batch, максимальный
batch 128 records, retry не дольше 15s. SDK очереди traces/logs — по 256 записей;
сеть работает в фоновых потоках. При исчерпании лимитов telemetry теряется.
4. Проверить возобновление Loki ingestion и опустошение exporter queue.
В `lab6-checks.json` число потерянных request logs считается по известным
request IDs после восстановления, отдельно от счётчиков неудачных экспортов.
## SLO и интерпретация
- API: ≥99% business responses за скользящий час без 5xx. 4xx входят в
denominator, но не numerator. Отказы LB/сети до API требуют внешнего probe
и этим SLI не покрываются.
- Изображения: ≥95% accepted jobs готовы за ≤30s; cohort оценивается после
истечения deadline. Failed и не завершившиеся задания входят в bad.
Скользящее окно — час **созревших** cohorts (acceptance сдвинута на 30s).
- Fast burn: 6× бюджета одновременно за 2m и 30s, `for: 15s`.
Slow burn: 3× за 6m и 1m, `for: 30s`. Это учебные окна для 1h SLO,
а не механически перенесённые production окна для 30 дней.
- Guards: ≥30 API requests / ≥5 mature jobs в long window и ≥2 события в short.
Нет данных — нет утверждения об успешном SLO; проверять scrape и traffic.
- `APIHighErrorRate` — single-window comparator (5%/30s, for 5s),
`AttachmentQueueLag` — отдельный эксплуатационный warning (>20s, for 10s).
Сравнивать их с burn alerts, а не считать любой warning доказательством
исчерпания бюджета. Данные одного короткого эксперимента не доказывают
соблюдение SLO в течение полного часа или отсутствие ложных срабатываний вообще.
+40
View File
@@ -0,0 +1,40 @@
# Схема и транзакции
Лабораторная 5, оценка 4: клиент → HAProxy → FastAPI × 2 → PostgreSQL + Valkey sessions + приватный S3; outbox publisher → Valkey Streams broker → worker.
| Таблица PostgreSQL | Назначение и ограничения |
| --- | --- |
| users | UUID PK; username UNIQUE; password_hash Argon2id; display_name; created_at |
| chats | UUID PK; title; created_at |
| chat_members | PK (chat_id, user_id); FK на chats и users |
| messages | UUID PK; FK на чат и членство отправителя; text до 4000 code points; created_at |
| message_attachments | FK на messages и attachments; до 10 позиций; UNIQUE(message_id, position); PK(message_id, attachment_id) |
| attachments | UUID PK; FK chat_id и (chat_id, owner_id); UNIQUE object_key; original metadata; status, variants, error_code, created_at/updated_at |
| attachment_jobs | UUID job_id PK; UNIQUE attachment_id/staging_key; version, attempt/budget, lease, retention |
| attachment_outbox | PK (job_id, attempt); available_at, published_at; индекс неопубликованных |
| attachment_job_events | Audit переходов, safe error_code, attempt, время |
| attachment_uploads | UUID PK, UNIQUE object_key, created_at; долговечный журнал незавершённых PUT |
| message_idempotency | PK (user_id, chat_id, key); request_text, request_attachment_ids, message_id FK, expires_at |
| schema_migrations | Имя и SHA-256 применённой миграции; applied_at |
В PostgreSQL нет bytes оригинала. `object_key=attachments/<UUID>/v1/original` назначает сервер; filename не участвует в ключе. До ответа 202 в таблице attachments есть status=queued, variants=[], error_code=null. Миграция 004 добавляет status, variants, error_code, updated_at; уже существующие ready-вложения lab4 не меняют ключи и остаются доступны.
`attachment_jobs` хранит один job_id на attachment_id, staging_key, pipeline_version=1, текущую попытку, бюджет, lease_token/lease_until и отметку удаления staging. `attachment_outbox(job_id,attempt)` создаётся в той же транзакции, что metadata/намерение. Dispatcher отмечает published_at после XADD; потеря ответа вызывает безопасный дубль. Отложенный retry получает отдельную строку outbox с available_at. `attachment_job_events` — audit переходов и явных replay без credentials и текста исключений.
Данные при 202: приватный S3 `attachments/staging/<UUID>` и COMMIT metadata+job+outbox. Готовность: worker сохраняет неизменённый original и три варианта по детерминированным ключам, затем атомарно фиксирует status=ready и JSON metadata. До ready content отвечает 409. Worker проверяет lease_token под блокировкой строк перед записью результатов. Cleaner не удаляет staging queued/processing; после ready удаляет staging, а failed хранит до STAGING_RETENTION_SECONDS.
Индексы: UNIQUE users(username); chat_members(user_id, chat_id); messages(chat_id, created_at, id); attachments(chat_id); attachment_uploads(created_at). Первичные/уникальные ключи индексируют поиск ID и object_key.
Создание чата и его участников атомарно. Неизвестный участник откатывает весь чат. Сообщение проверяет членство и все attachment_ids в том же чате, затем одной транзакцией записывает messages, message_attachments и результат идемпотентности. Составной FK дополнительно запрещает отправителя вне чата. Пустой/пробельный text разрешён только с вложением: это проверяет модель запроса; FK таблицы message_attachments запрещают несуществующие ссылки. Удаления вложений в HTTP API нет.
Общая advisory transaction lock сериализует область пользователь + POST + путь чата + Idempotency-Key. Сравниваются text и упорядоченный нормализованный список UUID; отсутствие attachment_ids эквивалентно []. Совпадение возвращает исходный 201 и Message, отличие — 409. TTL не меньше 24 часов. Чтение истории использует keyset (created_at,id), без кэша; на следующем запросе видны COMMIT любой реплики. При конкурентных вставках snapshot между страницами нет; запись с более ранними часами может оказаться до уже пройденного cursor.
Upload сначала полностью принимает ограниченный поток во временный файл и проверяет PNG/JPEG. Затем отдельной транзакцией фиксирует attachment_uploads. В следующей транзакции берётся advisory lock object_key, выполняется S3 PUT, вставляются queued metadata, job и outbox, удаляется журнал. 202 отправляется после COMMIT. S3 и PostgreSQL не образуют распределённую транзакцию.
Если PUT прошёл, а COMMIT нет, остаётся журнал. Если исход COMMIT неизвестен, немедленного удаления нет: metadata/job могли сохраниться, а удаление журнала атомарно с ними. Cleaner раз в 30 секунд берёт до 100 записей старше grace period (по умолчанию 3600 s, минимум 60 s), пробует ту же advisory lock и повторно проверяет запись. Активная загрузка пропускается; при наличии ready-метаданных удаляется только запись журнала; иначе удаляется S3 object и затем журнал. DELETE объекта идемпотентен. Сбой S3/БД откладывает обработку; следующий проход повторяет её. Это компенсация журналируемых загрузок уровня 4, без полного обхода bucket.
Миграции 001 и 002 сохранены. Миграция 003 добавляет вложения, связи, журнал и список вложений в ключе повтора; старые сообщения имеют пустой список. DDL выполняет один migrate-service под общей advisory lock; API проверяют версии и checksum.
Сервис valkey хранит только серверные сессии; отдельный сервис broker хранит Stream. Сессии: session:SHA256(token) → user_id, expires_at, TTL. Включены AOF appendfsync always и постоянный volume. Сырой токен не записывается в evidence. PostgreSQL, Valkey и S3 bucket проверяются /health/ready; /health/live не обращается к зависимостям.
Основа схемы перенесена из lab4; команды, измерения и ограничения прежней версии — в [REPORT.lab4.md](../REPORT.lab4.md). Исходные отчёты и результаты lab2/lab3 сохранены отдельно. Текущая лабораторная описывается в [REPORT.md](../REPORT.md).
+66
View File
@@ -0,0 +1,66 @@
UTC: 2026-10-22T11:17:22.562634+00:00
Host: Darwin arm64 Python 3.11.9
Docker: 29.0.1 aarch64 CPUs=10 RAM=8217448448
Commit: 9961362 + working tree
Command: ./scripts/check-docker.sh; project: lab2-check-ae280d00db
PASS: start.sh built and started the stack, readiness reached
api image=sha256:0b3dc5056cced642eea5ade11d235dce9f69f0d9e6d63956ce9479f2deaa10a0 started=2026-10-22T11:17:27.057932632Z memory=268435456 nano_cpus=1000000000
postgres image=sha256:f3bd19c606e442c3d7bdfa8002e03fe260a1023351e0ea4598032022b68dd6e3 started=2026-10-22T11:17:24.411773881Z memory=536870912 nano_cpus=1000000000
valkey image=sha256:fea8b3e67b15729d4bb70589eb03367bab9ad1ee89c876f54327fc7c6e618571 started=2026-10-22T11:17:24.406509756Z memory=134217728 nano_cpus=500000000
PASS: non-root UID 10001; Python PID 1; healthcheck; named volumes; no API data volume
Проверяем сервер: http://127.0.0.1:61900
/Users/vanek/Desktop/VNPlabs/VNPlab2/mtusi2026lab2/.venv/bin/python scripts/check_contract.py
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
/Users/vanek/Desktop/VNPlabs/VNPlab2/mtusi2026lab2/.venv/bin/python tests/smoke.py --base-url "http://127.0.0.1:61900"
PASS: lab2 public API scenario; infrastructure and grading evidence remain separate.
test_chat_access (test_api.ApiTests.test_chat_access) ... ok
test_chat_pagination_and_cursor_owner (test_api.ApiTests.test_chat_pagination_and_cursor_owner) ... ok
test_cursor_cannot_be_used_for_another_chat_or_changed (test_api.ApiTests.test_cursor_cannot_be_used_for_another_chat_or_changed) ... ok
test_default_page_size (test_api.ApiTests.test_default_page_size) ... ok
test_duplicate_username (test_api.ApiTests.test_duplicate_username) ... ok
test_framework_errors_and_request_ids (test_api.ApiTests.test_framework_errors_and_request_ids) ... ok
test_identification_and_authentication_order (test_api.ApiTests.test_identification_and_authentication_order) ... ok
test_invalid_chat_members (test_api.ApiTests.test_invalid_chat_members) ... ok
test_invalid_chat_title (test_api.ApiTests.test_invalid_chat_title) ... ok
test_invalid_limits_and_cursors (test_api.ApiTests.test_invalid_limits_and_cursors) ... ok
test_invalid_message_fields (test_api.ApiTests.test_invalid_message_fields) ... ok
test_invalid_user_fields (test_api.ApiTests.test_invalid_user_fields) ... ok
test_malformed_json_and_encoding (test_api.ApiTests.test_malformed_json_and_encoding) ... ok
test_message_pagination_and_changing_limit (test_api.ApiTests.test_message_pagination_and_changing_limit) ... ok
test_unicode_and_text_preservation (test_api.ApiTests.test_unicode_and_text_preservation) ... ok
test_wrong_content_type (test_api.ApiTests.test_wrong_content_type) ... ok
test_simultaneous_registration_of_one_username (test_concurrency.ConcurrencyTests.test_simultaneous_registration_of_one_username) ... ok
test_ten_clients_send_one_hundred_messages (test_concurrency.ConcurrencyTests.test_ten_clients_send_one_hundred_messages) ... ok
test_logout_revokes_only_one_session (test_sessions.SessionTests.test_logout_revokes_only_one_session) ... ok
test_password_validation (test_sessions.SessionTests.test_password_validation) ... ok
test_sql_like_message_is_plain_text (test_sessions.SessionTests.test_sql_like_message_is_plain_text) ... ok
test_unicode_password_with_colon_and_private_response (test_sessions.SessionTests.test_unicode_password_with_colon_and_private_response) ... ok
test_wrong_password_and_basic_only_for_login (test_sessions.SessionTests.test_wrong_password_and_basic_only_for_login) ... ok
----------------------------------------------------------------------
Ran 23 tests in 5.743s
OK
PASS: 10 clients sent 100 messages; history contains exactly 100 unique messages.
PASS: demo.sh registered users, logged in and read two messages
PASS: Argon2id in PostgreSQL; logs do not contain message text, passwords or session tokens
PASS: failed membership insert rolled back chat; migrations run twice without duplicate data
/Users/vanek/Desktop/VNPlabs/VNPlab2/mtusi2026lab2/.venv/bin/python tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://127.0.0.1:61900"
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
Restart api 2026-10-22T11:17:27.057932632Z -> 2026-10-22T11:17:45.460071626Z
Restart postgres 2026-10-22T11:17:24.411773881Z -> 2026-10-22T11:17:44.637353001Z
Restart valkey 2026-10-22T11:17:24.406509756Z -> 2026-10-22T11:17:44.670095293Z
PASS: API, PostgreSQL and Valkey really restarted; previous token still works
PASS: valkey unavailable -> 503 in 0.11s; anonymous -> 401; readiness 503, liveness 200
API ready
PASS: valkey recovered without a new login
PASS: postgres unavailable -> 503 in 2.01s; anonymous -> 401; readiness 503, liveness 200
API ready
PASS: postgres recovered without a new login
API ready
PASS: SESSION_TTL_SECONDS=2; valid token expires without logout
PASS: SIGTERM with unfinished request: 5.54s; new requests refused after 0.16s; exit=0; OOM=false
PASS: all lab2 Docker checks completed
PASS: stop.sh removed containers and preserved named volumes
Removed only temporary Compose project and its test volumes: lab2-check-ae280d00db
File diff suppressed because it is too large Load Diff
+403
View File
@@ -0,0 +1,403 @@
Command: ./scripts/check-docker.sh; project: lab3-check-a31f544685
Docker: 29.0.1 aarch64 CPUs=10 RAM=8217448448
Local .env and secrets are ready; values are not printed
#1 [internal] load local bake definitions
#1 reading from stdin 1.05kB done
#1 DONE 0.0s
#2 [migrate internal] load build definition from Dockerfile
#2 transferring dockerfile: 733B done
#2 DONE 0.0s
#3 [api internal] load metadata for docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
#3 DONE 0.0s
#4 [api internal] load .dockerignore
#4 transferring context: 167B done
#4 DONE 0.0s
#5 [migrate internal] load build context
#5 transferring context: 809B done
#5 DONE 0.0s
#6 [migrate 1/8] FROM docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
#6 resolve docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3 done
#6 DONE 0.0s
#7 [migrate 2/8] WORKDIR /app
#7 CACHED
#8 [migrate 3/8] COPY requirements.txt .
#8 CACHED
#9 [migrate 4/8] RUN python -m pip install --no-cache-dir -r requirements.txt && python -m pip check
#9 CACHED
#10 [migrate 6/8] COPY src/ ./src/
#10 CACHED
#11 [migrate 7/8] COPY migrations/ ./migrations/
#11 CACHED
#12 [migrate 5/8] RUN groupadd --gid 10001 app && useradd --uid 10001 --gid app --no-create-home app
#12 CACHED
#13 [migrate 8/8] COPY research/ ./research/
#13 CACHED
#14 [api] exporting to image
#14 exporting layers done
#14 exporting manifest sha256:06163e45d3676c7cf4f7a1c7bfe108ed3b98559ebb485c7a09d1e87e6f418d17 done
#14 exporting config sha256:3fed7460e9510eef7b2928f8d600d06d5d65329f74d820beac13809476b11398 done
#14 exporting attestation manifest sha256:352319cde788733f2f237a2a78204630c542e7ca906cff14c902046e8c1d2e5b 0.0s done
#14 exporting manifest list sha256:848e0c76f1678f35dd1b483e7c54552466999551ba7e77c0062bf346671e2902 done
#14 naming to docker.io/library/mtusi-messenger:lab3 done
#14 unpacking to docker.io/library/mtusi-messenger:lab3 done
#14 DONE 0.1s
#15 [migrate] exporting to image
#15 exporting layers done
#15 exporting manifest sha256:a59568d34e2c5e8b4d23308342601fdaf8c07e78ec3065878679a067c082e952 done
#15 exporting config sha256:b67b805ba1b7dd90f0a3e8444455488fe8c7f4118f14d671501dd63e85b59c97 done
#15 exporting attestation manifest sha256:b1853c07ddac46c4b398b475052f334aa2753c2fe971e0e4fb2517b0e3e13656 0.0s done
#15 exporting manifest list sha256:eff1abdd662eb60e2741e6ce6b8f364467d2817dafeef9754518247124a780eb done
#15 naming to docker.io/library/mtusi-messenger:lab3 done
#15 unpacking to docker.io/library/mtusi-messenger:lab3 done
#15 DONE 0.1s
#16 [api] resolving provenance for metadata file
#16 DONE 0.0s
#17 [migrate] resolving provenance for metadata file
#17 DONE 0.0s
mtusi-messenger:lab3 Built
mtusi-messenger:lab3 Built
Network lab3-check-a31f544685_default Creating
Network lab3-check-a31f544685_default Created
Volume lab3-check-a31f544685_postgres_data Creating
Volume lab3-check-a31f544685_postgres_data Created
Volume lab3-check-a31f544685_valkey_data Creating
Volume lab3-check-a31f544685_valkey_data Created
Container lab3-check-a31f544685-valkey-1 Creating
Container lab3-check-a31f544685-postgres-1 Creating
Container lab3-check-a31f544685-valkey-1 Created
Container lab3-check-a31f544685-postgres-1 Created
Container lab3-check-a31f544685-migrate-1 Creating
Container lab3-check-a31f544685-migrate-1 Created
Container lab3-check-a31f544685-api-2 Creating
Container lab3-check-a31f544685-api-1 Creating
Container lab3-check-a31f544685-api-1 Created
Container lab3-check-a31f544685-api-2 Created
Container lab3-check-a31f544685-lb-1 Creating
Container lab3-check-a31f544685-lb-1 Created
Container lab3-check-a31f544685-valkey-1 Starting
Container lab3-check-a31f544685-postgres-1 Starting
Container lab3-check-a31f544685-valkey-1 Started
Container lab3-check-a31f544685-postgres-1 Started
Container lab3-check-a31f544685-postgres-1 Waiting
Container lab3-check-a31f544685-postgres-1 Healthy
Container lab3-check-a31f544685-migrate-1 Starting
Container lab3-check-a31f544685-migrate-1 Started
Container lab3-check-a31f544685-migrate-1 Waiting
Container lab3-check-a31f544685-postgres-1 Waiting
Container lab3-check-a31f544685-valkey-1 Waiting
Container lab3-check-a31f544685-valkey-1 Healthy
Container lab3-check-a31f544685-postgres-1 Healthy
Container lab3-check-a31f544685-migrate-1 Exited
Container lab3-check-a31f544685-api-2 Starting
Container lab3-check-a31f544685-api-2 Started
Container lab3-check-a31f544685-api-1 Starting
Container lab3-check-a31f544685-api-1 Started
Container lab3-check-a31f544685-api-2 Waiting
Container lab3-check-a31f544685-api-1 Waiting
Container lab3-check-a31f544685-api-2 Healthy
Container lab3-check-a31f544685-api-1 Healthy
Container lab3-check-a31f544685-lb-1 Starting
Container lab3-check-a31f544685-lb-1 Started
API ready through LB: 7f6a80b69e32, f973eaec4d52
PASS: 2 real API containers, CPU/RAM limits, API non-root and no public ports; only LB published
Проверяем сервер: http://127.0.0.1:58474
/Users/vanek/Desktop/VNPlabs/VNPlab3/mtusi2026lab3/.venv/bin/python scripts/check_contract.py
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
/Users/vanek/Desktop/VNPlabs/VNPlab3/mtusi2026lab3/.venv/bin/python tests/smoke.py --base-url "http://127.0.0.1:58474"
PASS: lab3 public API scenario; infrastructure and grading evidence remain separate.
test_chat_access (test_api.ApiTests.test_chat_access) ... ok
test_chat_pagination_and_cursor_owner (test_api.ApiTests.test_chat_pagination_and_cursor_owner) ... ok
test_cursor_cannot_be_used_for_another_chat_or_changed (test_api.ApiTests.test_cursor_cannot_be_used_for_another_chat_or_changed) ... ok
test_default_page_size (test_api.ApiTests.test_default_page_size) ... ok
test_duplicate_username (test_api.ApiTests.test_duplicate_username) ... ok
test_framework_errors_and_request_ids (test_api.ApiTests.test_framework_errors_and_request_ids) ... ok
test_identification_and_authentication_order (test_api.ApiTests.test_identification_and_authentication_order) ... ok
test_invalid_chat_members (test_api.ApiTests.test_invalid_chat_members) ... ok
test_invalid_chat_title (test_api.ApiTests.test_invalid_chat_title) ... ok
test_invalid_limits_and_cursors (test_api.ApiTests.test_invalid_limits_and_cursors) ... ok
test_invalid_message_fields (test_api.ApiTests.test_invalid_message_fields) ... ok
test_invalid_user_fields (test_api.ApiTests.test_invalid_user_fields) ... ok
test_malformed_json_and_encoding (test_api.ApiTests.test_malformed_json_and_encoding) ... ok
test_message_pagination_and_changing_limit (test_api.ApiTests.test_message_pagination_and_changing_limit) ... ok
test_unicode_and_text_preservation (test_api.ApiTests.test_unicode_and_text_preservation) ... ok
test_wrong_content_type (test_api.ApiTests.test_wrong_content_type) ... ok
test_simultaneous_registration_of_one_username (test_concurrency.ConcurrencyTests.test_simultaneous_registration_of_one_username) ... ok
test_ten_clients_send_one_hundred_messages (test_concurrency.ConcurrencyTests.test_ten_clients_send_one_hundred_messages) ... ok
test_conflict_and_semantic_json_replay (test_idempotency.IdempotencyTests.test_conflict_and_semantic_json_replay) ... ok
test_key_format_and_missing_key (test_idempotency.IdempotencyTests.test_key_format_and_missing_key) ... ok
test_parallel_replay_across_replicas (test_idempotency.IdempotencyTests.test_parallel_replay_across_replicas) ... ok
test_replay_rechecks_session_and_access (test_idempotency.IdempotencyTests.test_replay_rechecks_session_and_access) ... ok
test_scope_is_user_and_chat (test_idempotency.IdempotencyTests.test_scope_is_user_and_chat) ... ok
test_logout_revokes_only_one_session (test_sessions.SessionTests.test_logout_revokes_only_one_session) ... ok
test_password_validation (test_sessions.SessionTests.test_password_validation) ... ok
test_sql_like_message_is_plain_text (test_sessions.SessionTests.test_sql_like_message_is_plain_text) ... ok
test_unicode_password_with_colon_and_private_response (test_sessions.SessionTests.test_unicode_password_with_colon_and_private_response) ... ok
test_wrong_password_and_basic_only_for_login (test_sessions.SessionTests.test_wrong_password_and_basic_only_for_login) ... ok
----------------------------------------------------------------------
Ran 28 tests in 6.783s
OK
PASS: 10 clients sent 100 messages; history contains exactly 100 unique messages.
Container lab3-check-a31f544685-postgres-1 Running
Container lab3-check-a31f544685-postgres-1 Running
PASS: migrations only in one-shot service; repeated execution preserves data
/Users/vanek/Desktop/VNPlabs/VNPlab3/mtusi2026lab3/.venv/bin/python tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://127.0.0.1:58474"
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
PASS: both API, LB, PostgreSQL, Valkey actually restarted; session and confirmed data preserved
PASS: SIGTERM with unfinished request: {"instance": "7f6a80b69e32", "unfinished_request": true, "shutdown_seconds": 5.393, "new_connections_refused_seconds": 0.167, "exit_code": 0, "oom_killed": false}
Container lab3-check-a31f544685-postgres-1 Waiting
Container lab3-check-a31f544685-postgres-1 Healthy
Container lab3-check-a31f544685-migrate-1 Starting
Container lab3-check-a31f544685-migrate-1 Started
Container lab3-check-a31f544685-valkey-1 Waiting
Container lab3-check-a31f544685-migrate-1 Waiting
Container lab3-check-a31f544685-postgres-1 Waiting
Container lab3-check-a31f544685-valkey-1 Healthy
Container lab3-check-a31f544685-postgres-1 Healthy
Container lab3-check-a31f544685-migrate-1 Exited
Container lab3-check-a31f544685-api-1 Starting
Container lab3-check-a31f544685-api-1 Started
API ready through LB: 7f6a80b69e32, f973eaec4d52
Container lab3-check-a31f544685-valkey-1 Stopping
Container lab3-check-a31f544685-valkey-1 Stopped
PASS: valkey outage: API live=200, ready=503, LB ready=503
Container lab3-check-a31f544685-valkey-1 Starting
Container lab3-check-a31f544685-valkey-1 Started
API ready through LB: 7f6a80b69e32, f973eaec4d52
Container lab3-check-a31f544685-postgres-1 Stopping
Container lab3-check-a31f544685-postgres-1 Stopped
PASS: postgres outage: API live=200, ready=503, LB ready=503
Container lab3-check-a31f544685-postgres-1 Starting
Container lab3-check-a31f544685-postgres-1 Started
API ready through LB: 7f6a80b69e32, f973eaec4d52
/Users/vanek/Desktop/VNPlabs/VNPlab3/mtusi2026lab3/.venv/bin/python tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:58474"
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 113, "failed_attempts": 1, "first_recovery_seconds": 2.452}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
Container lab3-check-a31f544685-postgres-1 Waiting
Container lab3-check-a31f544685-postgres-1 Healthy
Container lab3-check-a31f544685-migrate-1 Starting
Container lab3-check-a31f544685-migrate-1 Started
Container lab3-check-a31f544685-postgres-1 Waiting
Container lab3-check-a31f544685-valkey-1 Waiting
Container lab3-check-a31f544685-migrate-1 Waiting
Container lab3-check-a31f544685-postgres-1 Healthy
Container lab3-check-a31f544685-valkey-1 Healthy
Container lab3-check-a31f544685-migrate-1 Exited
Container lab3-check-a31f544685-api-1 Starting
Container lab3-check-a31f544685-api-1 Started
API ready through LB: 7f6a80b69e32, f973eaec4d52
Stopped API: 7f6a80b69e32 container=7f6a80b69e3297a4de72e756a6c9d867a108930c1ca0e57922b3821c26fd03e6
{"label": "stop 7f6a80b69e32", "duration_seconds": 16.009, "successful_requests": 1576, "successful_writes": 788, "errors": 4, "successful_rps": 98.44, "first_success_after_action_seconds": {"POST": 0.014, "GET": 0.007}, "maximum_success_gap_seconds": {"POST": 1.957, "GET": 2.004}, "action_start_seconds": 1.003, "action_end_seconds": 1.666, "p50_ms": 7.813, "p95_ms": 16.393, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["7f6a80b69e32", "f973eaec4d52"]}
Container lab3-check-a31f544685-postgres-1 Waiting
Container lab3-check-a31f544685-postgres-1 Healthy
Container lab3-check-a31f544685-migrate-1 Starting
Container lab3-check-a31f544685-migrate-1 Started
Container lab3-check-a31f544685-postgres-1 Waiting
Container lab3-check-a31f544685-valkey-1 Waiting
Container lab3-check-a31f544685-migrate-1 Waiting
Container lab3-check-a31f544685-valkey-1 Healthy
Container lab3-check-a31f544685-postgres-1 Healthy
Container lab3-check-a31f544685-migrate-1 Exited
Container lab3-check-a31f544685-api-1 Starting
Container lab3-check-a31f544685-api-1 Started
API ready through LB: 7f6a80b69e32, f973eaec4d52
Stopped API: f973eaec4d52 container=f973eaec4d5290eb73cd96949877ccb8df1213523fce0ec05c6b40c1136bb31b
{"label": "stop f973eaec4d52", "duration_seconds": 16.015, "successful_requests": 1574, "successful_writes": 786, "errors": 4, "successful_rps": 98.28, "first_success_after_action_seconds": {"POST": 0.002, "GET": 0.008}, "maximum_success_gap_seconds": {"POST": 2.006, "GET": 2.006}, "action_start_seconds": 1.005, "action_end_seconds": 1.713, "p50_ms": 7.98, "p95_ms": 17.217, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["7f6a80b69e32", "f973eaec4d52"]}
Container lab3-check-a31f544685-postgres-1 Waiting
Container lab3-check-a31f544685-postgres-1 Healthy
Container lab3-check-a31f544685-migrate-1 Starting
Container lab3-check-a31f544685-migrate-1 Started
Container lab3-check-a31f544685-postgres-1 Waiting
Container lab3-check-a31f544685-valkey-1 Waiting
Container lab3-check-a31f544685-migrate-1 Waiting
Container lab3-check-a31f544685-valkey-1 Healthy
Container lab3-check-a31f544685-postgres-1 Healthy
Container lab3-check-a31f544685-migrate-1 Exited
Container lab3-check-a31f544685-api-2 Starting
Container lab3-check-a31f544685-api-2 Started
API ready through LB: 7f6a80b69e32, f973eaec4d52
Local .env and secrets are ready; values are not printed
#1 [internal] load local bake definitions
#1 reading from stdin 1.03kB done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 733B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
#3 DONE 0.0s
#4 [internal] load .dockerignore
#4 transferring context: 167B done
#4 DONE 0.0s
#5 [internal] load build context
#5 transferring context: 809B done
#5 DONE 0.0s
#6 [1/8] FROM docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
#6 resolve docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3 0.0s done
#6 DONE 0.0s
#7 [4/8] RUN python -m pip install --no-cache-dir -r requirements.txt && python -m pip check
#7 CACHED
#8 [3/8] COPY requirements.txt .
#8 CACHED
#9 [5/8] RUN groupadd --gid 10001 app && useradd --uid 10001 --gid app --no-create-home app
#9 CACHED
#10 [6/8] COPY src/ ./src/
#10 CACHED
#11 [2/8] WORKDIR /app
#11 CACHED
#12 [7/8] COPY migrations/ ./migrations/
#12 CACHED
#13 [8/8] COPY research/ ./research/
#13 CACHED
#14 exporting to image
#14 exporting layers done
#14 exporting manifest sha256:06163e45d3676c7cf4f7a1c7bfe108ed3b98559ebb485c7a09d1e87e6f418d17 done
#14 exporting config sha256:3fed7460e9510eef7b2928f8d600d06d5d65329f74d820beac13809476b11398 done
#14 exporting attestation manifest sha256:3a88ce0975ddcbab53c3ecb185da08ef114e818e9db952983e997cd47add67bf done
#14 exporting manifest list sha256:433a8a3293202a92f092574052b474e8373c0d58649cbcf1c7ebc1af17933d3c done
#14 naming to docker.io/library/mtusi-messenger:lab3 done
#14 unpacking to docker.io/library/mtusi-messenger:lab3 done
#14 DONE 0.1s
#15 resolving provenance for metadata file
#15 DONE 0.0s
mtusi-messenger:lab3 Built
Local .env and secrets are ready; values are not printed
Container lab3-check-a31f544685-postgres-1 Running
#1 [internal] load local bake definitions
#1 reading from stdin 576B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 733B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
#3 DONE 0.0s
#4 [internal] load .dockerignore
#4 transferring context: 167B done
#4 DONE 0.0s
#5 [internal] load build context
#5 transferring context: 809B done
#5 DONE 0.0s
#6 [1/8] FROM docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
#6 resolve docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3 done
#6 DONE 0.0s
#7 [3/8] COPY requirements.txt .
#7 CACHED
#8 [4/8] RUN python -m pip install --no-cache-dir -r requirements.txt && python -m pip check
#8 CACHED
#9 [5/8] RUN groupadd --gid 10001 app && useradd --uid 10001 --gid app --no-create-home app
#9 CACHED
#10 [6/8] COPY src/ ./src/
#10 CACHED
#11 [7/8] COPY migrations/ ./migrations/
#11 CACHED
#12 [2/8] WORKDIR /app
#12 CACHED
#13 [8/8] COPY research/ ./research/
#13 CACHED
#14 exporting to image
#14 exporting layers done
#14 exporting manifest sha256:a59568d34e2c5e8b4d23308342601fdaf8c07e78ec3065878679a067c082e952 done
#14 exporting config sha256:b67b805ba1b7dd90f0a3e8444455488fe8c7f4118f14d671501dd63e85b59c97 done
#14 exporting attestation manifest sha256:25ff7934ca5161da9b6c345af61b6a10b452580e4d349b35939d32feb66664ea done
#14 exporting manifest list sha256:3e102cd9a2c1e81e95aed4a0d4de1ac94352370595fced350257813f7a07bef8 done
#14 naming to docker.io/library/mtusi-messenger:lab3 done
#14 unpacking to docker.io/library/mtusi-messenger:lab3 done
#14 DONE 0.0s
#15 resolving provenance for metadata file
#15 DONE 0.0s
Migrations: already applied
API ready through LB: 7f6a80b69e32, f973eaec4d52
Replacing API: 7f6a80b69e32
Container lab3-check-a31f544685-api-2 Running
Container lab3-check-a31f544685-api-3 Creating
Container lab3-check-a31f544685-api-3 Created
Container lab3-check-a31f544685-api-3 Starting
Container lab3-check-a31f544685-api-3 Started
API ready through LB: ca9c7a52da66, f973eaec4d52
Replacing API: f973eaec4d52
Container lab3-check-a31f544685-api-3 Running
Container lab3-check-a31f544685-api-4 Creating
Container lab3-check-a31f544685-api-4 Created
Container lab3-check-a31f544685-api-4 Starting
Container lab3-check-a31f544685-api-4 Started
API ready through LB: 244bf0b910eb, ca9c7a52da66
PASS: all API containers replaced sequentially; stores and LB kept running
{"label": "sequential API replacement", "duration_seconds": 23.774, "successful_requests": 2131, "successful_writes": 1063, "errors": 11, "successful_rps": 89.64, "first_success_after_action_seconds": {"POST": 0.018, "GET": 0.004}, "maximum_success_gap_seconds": {"POST": 1.059, "GET": 2.034}, "action_start_seconds": 1.005, "action_end_seconds": 19.63, "p50_ms": 7.723, "p95_ms": 13.565, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["244bf0b910eb", "7f6a80b69e32", "ca9c7a52da66", "f973eaec4d52"]}
PASS: all lab3 grade-4 checks completed
Container lab3-check-a31f544685-lb-1 Stopping
Container lab3-check-a31f544685-lb-1 Stopped
Container lab3-check-a31f544685-lb-1 Removing
Container lab3-check-a31f544685-lb-1 Removed
Container lab3-check-a31f544685-api-3 Stopping
Container lab3-check-a31f544685-api-4 Stopping
Container lab3-check-a31f544685-api-4 Stopped
Container lab3-check-a31f544685-api-4 Removing
Container lab3-check-a31f544685-api-4 Removed
Container lab3-check-a31f544685-api-3 Stopped
Container lab3-check-a31f544685-api-3 Removing
Container lab3-check-a31f544685-api-3 Removed
Container lab3-check-a31f544685-migrate-1 Stopping
Container lab3-check-a31f544685-valkey-1 Stopping
Container lab3-check-a31f544685-migrate-1 Stopped
Container lab3-check-a31f544685-migrate-1 Removing
Container lab3-check-a31f544685-migrate-1 Removed
Container lab3-check-a31f544685-postgres-1 Stopping
Container lab3-check-a31f544685-valkey-1 Stopped
Container lab3-check-a31f544685-valkey-1 Removing
Container lab3-check-a31f544685-valkey-1 Removed
Container lab3-check-a31f544685-postgres-1 Stopped
Container lab3-check-a31f544685-postgres-1 Removing
Container lab3-check-a31f544685-postgres-1 Removed
Volume lab3-check-a31f544685_valkey_data Removing
Network lab3-check-a31f544685_default Removing
Volume lab3-check-a31f544685_postgres_data Removing
Volume lab3-check-a31f544685_valkey_data Removed
Volume lab3-check-a31f544685_postgres_data Removed
Network lab3-check-a31f544685_default Removed
Removed only temporary test containers and volumes: lab3-check-a31f544685
File diff suppressed because it is too large Load Diff
+533
View File
@@ -0,0 +1,533 @@
Command: ./scripts/check-docker.sh; project: lab4-check-5aad8e0065
Docker: 29.0.1 aarch64 CPUs=10 RAM=8217448448
Local .env and secrets are ready; values are not printed
#1 [internal] load local bake definitions
#1 reading from stdin 564B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 733B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
#3 DONE 0.0s
#4 [internal] load .dockerignore
#4 transferring context: 167B done
#4 DONE 0.0s
#5 [internal] load build context
#5 transferring context: 1.01kB done
#5 DONE 0.0s
#6 [1/8] FROM docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
#6 resolve docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3 done
#6 DONE 0.0s
#7 [4/8] RUN python -m pip install --no-cache-dir -r requirements.txt && python -m pip check
#7 CACHED
#8 [7/8] COPY migrations/ ./migrations/
#8 CACHED
#9 [2/8] WORKDIR /app
#9 CACHED
#10 [6/8] COPY src/ ./src/
#10 CACHED
#11 [3/8] COPY requirements.txt .
#11 CACHED
#12 [5/8] RUN groupadd --gid 10001 app && useradd --uid 10001 --gid app --no-create-home app
#12 CACHED
#13 [8/8] COPY research/ ./research/
#13 CACHED
#14 exporting to image
#14 exporting layers done
#14 exporting manifest sha256:644f5d408505f5982b75f18404860de692ee837b2b2e18f5c92b0b057081b7ec done
#14 exporting config sha256:726011343b32f5b6277348f2f8b38f95b8c31754ae0d21336375124a18aa2e0e done
#14 exporting attestation manifest sha256:1146b210a20471d58663fb48a3a24ba22986971d0cce8777f912886768f43306 done
#14 exporting manifest list sha256:a04b1bfe2aea8391c8f92dd952499559d766cc4b34091963fe57bcbaf6ec015d done
#14 naming to docker.io/library/mtusi-messenger:lab4 done
#14 unpacking to docker.io/library/mtusi-messenger:lab4 done
#14 DONE 0.1s
#15 resolving provenance for metadata file
#15 DONE 0.0s
mtusi-messenger:lab4 Built
Network lab4-check-5aad8e0065_default Creating
Network lab4-check-5aad8e0065_default Created
Network lab4-check-5aad8e0065_s3-internal Creating
Network lab4-check-5aad8e0065_s3-internal Created
Volume lab4-check-5aad8e0065_valkey_data Creating
Volume lab4-check-5aad8e0065_valkey_data Created
Volume lab4-check-5aad8e0065_postgres_data Creating
Volume lab4-check-5aad8e0065_postgres_data Created
Volume lab4-check-5aad8e0065_s3_data Creating
Volume lab4-check-5aad8e0065_s3_data Created
Container lab4-check-5aad8e0065-s3-storage-1 Creating
Container lab4-check-5aad8e0065-valkey-1 Creating
Container lab4-check-5aad8e0065-postgres-1 Creating
Container lab4-check-5aad8e0065-valkey-1 Created
Container lab4-check-5aad8e0065-s3-storage-1 Created
Container lab4-check-5aad8e0065-s3-1 Creating
Container lab4-check-5aad8e0065-postgres-1 Created
Container lab4-check-5aad8e0065-migrate-1 Creating
Container lab4-check-5aad8e0065-s3-1 Created
Container lab4-check-5aad8e0065-s3-bootstrap-1 Creating
Container lab4-check-5aad8e0065-migrate-1 Created
Container lab4-check-5aad8e0065-s3-bootstrap-1 Created
Container lab4-check-5aad8e0065-api-2 Creating
Container lab4-check-5aad8e0065-api-1 Creating
Container lab4-check-5aad8e0065-cleanup-1 Creating
Container lab4-check-5aad8e0065-api-1 Created
Container lab4-check-5aad8e0065-cleanup-1 Created
Container lab4-check-5aad8e0065-api-2 Created
Container lab4-check-5aad8e0065-lb-1 Creating
Container lab4-check-5aad8e0065-lb-1 Created
Container lab4-check-5aad8e0065-postgres-1 Starting
Container lab4-check-5aad8e0065-valkey-1 Starting
Container lab4-check-5aad8e0065-s3-storage-1 Starting
Container lab4-check-5aad8e0065-s3-storage-1 Started
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
Container lab4-check-5aad8e0065-valkey-1 Started
Container lab4-check-5aad8e0065-postgres-1 Started
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-migrate-1 Starting
Container lab4-check-5aad8e0065-migrate-1 Started
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
Container lab4-check-5aad8e0065-s3-1 Starting
Container lab4-check-5aad8e0065-s3-1 Started
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
Container lab4-check-5aad8e0065-migrate-1 Waiting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
Container lab4-check-5aad8e0065-valkey-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Waiting
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-migrate-1 Exited
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-migrate-1 Exited
Container lab4-check-5aad8e0065-valkey-1 Healthy
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
Container lab4-check-5aad8e0065-api-2 Starting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
Container lab4-check-5aad8e0065-cleanup-1 Starting
Container lab4-check-5aad8e0065-cleanup-1 Started
Container lab4-check-5aad8e0065-api-2 Started
Container lab4-check-5aad8e0065-api-1 Starting
Container lab4-check-5aad8e0065-api-1 Started
Container lab4-check-5aad8e0065-api-2 Waiting
Container lab4-check-5aad8e0065-api-1 Waiting
Container lab4-check-5aad8e0065-api-2 Healthy
Container lab4-check-5aad8e0065-api-1 Healthy
Container lab4-check-5aad8e0065-lb-1 Starting
Container lab4-check-5aad8e0065-lb-1 Started
API ready through LB: 19a474c69210, ca1f62dfc015
PASS: 2 real API containers, CPU/RAM limits, API non-root and no public ports; only LB published
Проверяем сервер: http://127.0.0.1:57722
/Users/vanek/Desktop/VNPlabs/VNPlab4/mtusi2026lab4/.venv/bin/python scripts/check_contract.py
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
/Users/vanek/Desktop/VNPlabs/VNPlab4/mtusi2026lab4/.venv/bin/python tests/smoke.py --base-url "http://127.0.0.1:57722"
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
test_chat_access (test_api.ApiTests.test_chat_access) ... ok
test_chat_pagination_and_cursor_owner (test_api.ApiTests.test_chat_pagination_and_cursor_owner) ... ok
test_cursor_cannot_be_used_for_another_chat_or_changed (test_api.ApiTests.test_cursor_cannot_be_used_for_another_chat_or_changed) ... ok
test_default_page_size (test_api.ApiTests.test_default_page_size) ... ok
test_duplicate_username (test_api.ApiTests.test_duplicate_username) ... ok
test_framework_errors_and_request_ids (test_api.ApiTests.test_framework_errors_and_request_ids) ... ok
test_identification_and_authentication_order (test_api.ApiTests.test_identification_and_authentication_order) ... ok
test_invalid_chat_members (test_api.ApiTests.test_invalid_chat_members) ... ok
test_invalid_chat_title (test_api.ApiTests.test_invalid_chat_title) ... ok
test_invalid_limits_and_cursors (test_api.ApiTests.test_invalid_limits_and_cursors) ... ok
test_invalid_message_fields (test_api.ApiTests.test_invalid_message_fields) ... ok
test_invalid_user_fields (test_api.ApiTests.test_invalid_user_fields) ... ok
test_malformed_json_and_encoding (test_api.ApiTests.test_malformed_json_and_encoding) ... ok
test_message_pagination_and_changing_limit (test_api.ApiTests.test_message_pagination_and_changing_limit) ... ok
test_unicode_and_text_preservation (test_api.ApiTests.test_unicode_and_text_preservation) ... ok
test_wrong_content_type (test_api.ApiTests.test_wrong_content_type) ... ok
test_access_and_authentication (test_attachments.AttachmentTests.test_access_and_authentication) ... ok
test_attachment_only_message_and_other_member (test_attachments.AttachmentTests.test_attachment_only_message_and_other_member) ... ok
test_chunked_limit_without_content_length (test_attachments.AttachmentTests.test_chunked_limit_without_content_length) ... ok
test_content_length_limit_before_body (test_attachments.AttachmentTests.test_content_length_limit_before_body) ... ok
test_corruption_and_mime_spoof (test_attachments.AttachmentTests.test_corruption_and_mime_spoof) ... ok
test_filename_rejected_before_storage (test_attachments.AttachmentTests.test_filename_rejected_before_storage) ... ok
test_foreign_and_unknown_attachment_ids (test_attachments.AttachmentTests.test_foreign_and_unknown_attachment_ids) ... ok
test_idempotency_includes_ordered_attachment_ids (test_attachments.AttachmentTests.test_idempotency_includes_ordered_attachment_ids) ... ok
test_invalid_attachment_lists (test_attachments.AttachmentTests.test_invalid_attachment_lists) ... ok
test_pixel_limit_before_decode (test_attachments.AttachmentTests.test_pixel_limit_before_decode) ... ok
test_png_and_jpeg_originals (test_attachments.AttachmentTests.test_png_and_jpeg_originals) ... ok
test_simultaneous_registration_of_one_username (test_concurrency.ConcurrencyTests.test_simultaneous_registration_of_one_username) ... ok
test_ten_clients_send_one_hundred_messages (test_concurrency.ConcurrencyTests.test_ten_clients_send_one_hundred_messages) ... ok
test_conflict_and_semantic_json_replay (test_idempotency.IdempotencyTests.test_conflict_and_semantic_json_replay) ... ok
test_key_format_and_missing_key (test_idempotency.IdempotencyTests.test_key_format_and_missing_key) ... ok
test_parallel_replay_across_replicas (test_idempotency.IdempotencyTests.test_parallel_replay_across_replicas) ... ok
test_replay_rechecks_session_and_access (test_idempotency.IdempotencyTests.test_replay_rechecks_session_and_access) ... ok
test_scope_is_user_and_chat (test_idempotency.IdempotencyTests.test_scope_is_user_and_chat) ... ok
test_logout_revokes_only_one_session (test_sessions.SessionTests.test_logout_revokes_only_one_session) ... ok
test_password_validation (test_sessions.SessionTests.test_password_validation) ... ok
test_sql_like_message_is_plain_text (test_sessions.SessionTests.test_sql_like_message_is_plain_text) ... ok
test_unicode_password_with_colon_and_private_response (test_sessions.SessionTests.test_unicode_password_with_colon_and_private_response) ... ok
test_wrong_password_and_basic_only_for_login (test_sessions.SessionTests.test_wrong_password_and_basic_only_for_login) ... ok
----------------------------------------------------------------------
Ran 39 tests in 9.593s
OK
PASS: 10 clients sent 100 messages; history contains exactly 100 unique messages.
Container lab4-check-5aad8e0065-postgres-1 Running
Container lab4-check-5aad8e0065-postgres-1 Running
PASS: migrations only in one-shot service; repeated execution preserves data
Container lab4-check-5aad8e0065-cleanup-1 Stopping
Container lab4-check-5aad8e0065-cleanup-1 Stopped
Container lab4-check-5aad8e0065-s3-storage-1 Running
Container lab4-check-5aad8e0065-s3-1 Running
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
Container lab4-check-5aad8e0065-s3-storage-1 Running
Container lab4-check-5aad8e0065-s3-1 Running
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
PASS: private S3, limited credentials, isolated administration, repeated bootstrap
Stopped API: ca1f62dfc015 container=ca1f62dfc0156fc235dbcd2f52b58eec2282e6ba8d3e056de11ca92758c69df0
API ready through LB: 19a474c69210
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
Container lab4-check-5aad8e0065-migrate-1 Starting
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Started
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-valkey-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Exited
Container lab4-check-5aad8e0065-valkey-1 Healthy
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
Container lab4-check-5aad8e0065-api-2 Starting
Container lab4-check-5aad8e0065-api-2 Started
API ready through LB: 19a474c69210, ca1f62dfc015
PASS: original downloaded through another replica after stopping uploader
PASS: post-PUT DB failure recorded; cleanup protects fresh, active and ready uploads
Container lab4-check-5aad8e0065-s3-1 Stopping
Container lab4-check-5aad8e0065-s3-1 Stopped
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
Container lab4-check-5aad8e0065-s3-1 Starting
Container lab4-check-5aad8e0065-s3-1 Started
API ready through LB: 19a474c69210, ca1f62dfc015
PASS: S3 outage gives API 503; recovery preserves original and session
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-migrate-1 Starting
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Started
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
Container lab4-check-5aad8e0065-migrate-1 Waiting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Exited
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
Container lab4-check-5aad8e0065-cleanup-1 Starting
Container lab4-check-5aad8e0065-cleanup-1 Started
PASS: 10 concurrent real PNG uploads; all checksums match; memory peaks: [215191552, 103378944]
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
Container lab4-check-5aad8e0065-migrate-1 Starting
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Started
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
Container lab4-check-5aad8e0065-valkey-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Waiting
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Exited
Container lab4-check-5aad8e0065-valkey-1 Healthy
Container lab4-check-5aad8e0065-migrate-1 Exited
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
/Users/vanek/Desktop/VNPlabs/VNPlab4/mtusi2026lab4/.venv/bin/python tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://127.0.0.1:57722"
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
PASS: both API, LB, PostgreSQL, Valkey, S3 and cleaner actually restarted; data and session preserved
PASS: SIGTERM with unfinished request: {"instance": "19a474c69210", "unfinished_request": true, "shutdown_seconds": 5.523, "new_connections_refused_seconds": 0.185, "exit_code": 0, "oom_killed": false}
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
Container lab4-check-5aad8e0065-migrate-1 Starting
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Started
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-valkey-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Waiting
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
Container lab4-check-5aad8e0065-valkey-1 Healthy
Container lab4-check-5aad8e0065-migrate-1 Exited
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
Container lab4-check-5aad8e0065-api-1 Starting
Container lab4-check-5aad8e0065-api-1 Started
API ready through LB: 19a474c69210, ca1f62dfc015
Container lab4-check-5aad8e0065-valkey-1 Stopping
Container lab4-check-5aad8e0065-valkey-1 Stopped
PASS: valkey outage: API live=200, ready=503, LB ready=503
Container lab4-check-5aad8e0065-valkey-1 Starting
Container lab4-check-5aad8e0065-valkey-1 Started
API ready through LB: 19a474c69210, ca1f62dfc015
Container lab4-check-5aad8e0065-postgres-1 Stopping
Container lab4-check-5aad8e0065-postgres-1 Stopped
PASS: postgres outage: API live=200, ready=503, LB ready=503
Container lab4-check-5aad8e0065-postgres-1 Starting
Container lab4-check-5aad8e0065-postgres-1 Started
API ready through LB: 19a474c69210, ca1f62dfc015
Container lab4-check-5aad8e0065-s3-1 Stopping
Container lab4-check-5aad8e0065-s3-1 Stopped
PASS: s3 outage: API live=200, ready=503, LB ready=503
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
Container lab4-check-5aad8e0065-s3-1 Starting
Container lab4-check-5aad8e0065-s3-1 Started
API ready through LB: 19a474c69210, ca1f62dfc015
Container lab4-check-5aad8e0065-postgres-1 Paused
Container lab4-check-5aad8e0065-postgres-1 Unpaused
PASS: paused PostgreSQL: SQL and direct readiness bounded; live works; unpause restores API without restart
/Users/vanek/Desktop/VNPlabs/VNPlab4/mtusi2026lab4/.venv/bin/python tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:57722"
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 110, "failed_attempts": 1, "first_recovery_seconds": 2.467}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-migrate-1 Starting
Container lab4-check-5aad8e0065-migrate-1 Started
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-valkey-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Waiting
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
Container lab4-check-5aad8e0065-valkey-1 Healthy
Container lab4-check-5aad8e0065-migrate-1 Exited
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
Container lab4-check-5aad8e0065-api-1 Starting
Container lab4-check-5aad8e0065-api-1 Started
API ready through LB: 19a474c69210, ca1f62dfc015
Stopped API: 19a474c69210 container=19a474c69210a7be1c6631830e8fa7f251cf6dd3890c7407843090383543ee75
{"label": "stop 19a474c69210", "duration_seconds": 16.012, "successful_requests": 1420, "successful_writes": 709, "errors": 4, "successful_rps": 88.68, "first_success_after_action_seconds": {"POST": 0.051, "GET": 0.004}, "maximum_success_gap_seconds": {"POST": 2.002, "GET": 1.995}, "action_start_seconds": 1.006, "action_end_seconds": 1.77, "p50_ms": 10.62, "p95_ms": 24.67, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["19a474c69210", "ca1f62dfc015"]}
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Starting
Container lab4-check-5aad8e0065-migrate-1 Started
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-valkey-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Waiting
Container lab4-check-5aad8e0065-valkey-1 Healthy
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-migrate-1 Exited
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
Container lab4-check-5aad8e0065-api-1 Starting
Container lab4-check-5aad8e0065-api-1 Started
API ready through LB: 19a474c69210, ca1f62dfc015
Stopped API: ca1f62dfc015 container=ca1f62dfc0156fc235dbcd2f52b58eec2282e6ba8d3e056de11ca92758c69df0
{"label": "stop ca1f62dfc015", "duration_seconds": 16.008, "successful_requests": 1466, "successful_writes": 732, "errors": 4, "successful_rps": 91.58, "first_success_after_action_seconds": {"POST": 0.018, "GET": 0.028}, "maximum_success_gap_seconds": {"POST": 2.002, "GET": 1.999}, "action_start_seconds": 1.002, "action_end_seconds": 1.832, "p50_ms": 9.397, "p95_ms": 18.616, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["19a474c69210", "ca1f62dfc015"]}
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
Container lab4-check-5aad8e0065-migrate-1 Starting
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Started
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-valkey-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Waiting
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
Container lab4-check-5aad8e0065-valkey-1 Healthy
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-migrate-1 Exited
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
Container lab4-check-5aad8e0065-api-2 Starting
Container lab4-check-5aad8e0065-api-2 Started
API ready through LB: 19a474c69210, ca1f62dfc015
Local .env and secrets are ready; values are not printed
#1 [internal] load local bake definitions
#1 reading from stdin 564B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 733B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
#3 DONE 0.0s
#4 [internal] load .dockerignore
#4 transferring context: 167B done
#4 DONE 0.0s
#5 [internal] load build context
#5 transferring context: 1.01kB done
#5 DONE 0.0s
#6 [1/8] FROM docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
#6 resolve docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3 0.0s done
#6 DONE 0.0s
#7 [3/8] COPY requirements.txt .
#7 CACHED
#8 [6/8] COPY src/ ./src/
#8 CACHED
#9 [2/8] WORKDIR /app
#9 CACHED
#10 [7/8] COPY migrations/ ./migrations/
#10 CACHED
#11 [5/8] RUN groupadd --gid 10001 app && useradd --uid 10001 --gid app --no-create-home app
#11 CACHED
#12 [4/8] RUN python -m pip install --no-cache-dir -r requirements.txt && python -m pip check
#12 CACHED
#13 [8/8] COPY research/ ./research/
#13 CACHED
#14 exporting to image
#14 exporting layers done
#14 exporting manifest sha256:644f5d408505f5982b75f18404860de692ee837b2b2e18f5c92b0b057081b7ec done
#14 exporting config sha256:726011343b32f5b6277348f2f8b38f95b8c31754ae0d21336375124a18aa2e0e done
#14 exporting attestation manifest sha256:f4a5559518ea1ef0e3747d6084c8ad8dbc8b16dc41cb00062303d0f895b9a235 0.0s done
#14 exporting manifest list sha256:93c838d55d79a0001346ecab28f72621d9540ba5c52eaeb1ca6020286dd05fd6 done
#14 naming to docker.io/library/mtusi-messenger:lab4 done
#14 unpacking to docker.io/library/mtusi-messenger:lab4 done
#14 DONE 0.1s
#15 resolving provenance for metadata file
#15 DONE 0.0s
mtusi-messenger:lab4 Built
Container lab4-check-5aad8e0065-postgres-1 Running
Migrations: already applied
API ready through LB: 19a474c69210, ca1f62dfc015
Replacing API: 19a474c69210
Container lab4-check-5aad8e0065-api-2 Running
Container lab4-check-5aad8e0065-api-3 Creating
Container lab4-check-5aad8e0065-api-3 Created
Container lab4-check-5aad8e0065-api-3 Starting
Container lab4-check-5aad8e0065-api-3 Started
API ready through LB: ca1f62dfc015, ed4f3c64a9f6
Replacing API: ca1f62dfc015
Container lab4-check-5aad8e0065-api-3 Running
Container lab4-check-5aad8e0065-api-4 Creating
Container lab4-check-5aad8e0065-api-4 Created
Container lab4-check-5aad8e0065-api-4 Starting
Container lab4-check-5aad8e0065-api-4 Started
API ready through LB: 63549c0d63ba, ed4f3c64a9f6
PASS: all API containers replaced sequentially; stores and LB kept running
{"label": "sequential API replacement", "duration_seconds": 24.869, "successful_requests": 2254, "successful_writes": 1128, "errors": 8, "successful_rps": 90.63, "first_success_after_action_seconds": {"POST": 0.01, "GET": 0.021}, "maximum_success_gap_seconds": {"POST": 1.04, "GET": 1.04}, "action_start_seconds": 1.009, "action_end_seconds": 21.862, "p50_ms": 8.907, "p95_ms": 16.321, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["19a474c69210", "63549c0d63ba", "ca1f62dfc015", "ed4f3c64a9f6"]}
PASS: all lab4 grade-4 checks completed
Container lab4-check-5aad8e0065-lb-1 Stopping
Container lab4-check-5aad8e0065-cleanup-1 Stopping
Container lab4-check-5aad8e0065-lb-1 Stopped
Container lab4-check-5aad8e0065-lb-1 Removing
Container lab4-check-5aad8e0065-lb-1 Removed
Container lab4-check-5aad8e0065-api-3 Stopping
Container lab4-check-5aad8e0065-api-4 Stopping
Container lab4-check-5aad8e0065-api-3 Stopped
Container lab4-check-5aad8e0065-api-3 Removing
Container lab4-check-5aad8e0065-api-3 Removed
Container lab4-check-5aad8e0065-api-4 Stopped
Container lab4-check-5aad8e0065-api-4 Removing
Container lab4-check-5aad8e0065-api-4 Removed
Container lab4-check-5aad8e0065-valkey-1 Stopping
Container lab4-check-5aad8e0065-valkey-1 Stopped
Container lab4-check-5aad8e0065-valkey-1 Removing
Container lab4-check-5aad8e0065-valkey-1 Removed
Container lab4-check-5aad8e0065-cleanup-1 Stopped
Container lab4-check-5aad8e0065-cleanup-1 Removing
Container lab4-check-5aad8e0065-cleanup-1 Removed
Container lab4-check-5aad8e0065-s3-bootstrap-1 Stopping
Container lab4-check-5aad8e0065-migrate-1 Stopping
Container lab4-check-5aad8e0065-migrate-1 Stopped
Container lab4-check-5aad8e0065-migrate-1 Removing
Container lab4-check-5aad8e0065-s3-bootstrap-1 Stopped
Container lab4-check-5aad8e0065-s3-bootstrap-1 Removing
Container lab4-check-5aad8e0065-s3-bootstrap-1 Removed
Container lab4-check-5aad8e0065-s3-1 Stopping
Container lab4-check-5aad8e0065-migrate-1 Removed
Container lab4-check-5aad8e0065-postgres-1 Stopping
Container lab4-check-5aad8e0065-postgres-1 Stopped
Container lab4-check-5aad8e0065-postgres-1 Removing
Container lab4-check-5aad8e0065-postgres-1 Removed
Container lab4-check-5aad8e0065-s3-1 Stopped
Container lab4-check-5aad8e0065-s3-1 Removing
Container lab4-check-5aad8e0065-s3-1 Removed
Container lab4-check-5aad8e0065-s3-storage-1 Stopping
Container lab4-check-5aad8e0065-s3-storage-1 Stopped
Container lab4-check-5aad8e0065-s3-storage-1 Removing
Container lab4-check-5aad8e0065-s3-storage-1 Removed
Volume lab4-check-5aad8e0065_valkey_data Removing
Network lab4-check-5aad8e0065_s3-internal Removing
Volume lab4-check-5aad8e0065_postgres_data Removing
Volume lab4-check-5aad8e0065_s3_data Removing
Network lab4-check-5aad8e0065_default Removing
Volume lab4-check-5aad8e0065_valkey_data Removed
Volume lab4-check-5aad8e0065_s3_data Removed
Volume lab4-check-5aad8e0065_postgres_data Removed
Network lab4-check-5aad8e0065_s3-internal Removed
Network lab4-check-5aad8e0065_default Removed
Removed only temporary test containers and volumes: lab4-check-5aad8e0065
+502
View File
@@ -0,0 +1,502 @@
{
"utc": "2026-11-12T15:23:27.573351+00:00",
"command": "./scripts/check-docker.sh",
"project": "lab5-check-d233766d34",
"host": "Darwin arm64",
"source_commit": "276dbdad8a34da4c5b57fbf1879ca169b581d75b",
"working_tree": true,
"configuration": {
"concurrency": 2,
"prefetch": 1,
"attempts": 5,
"timeout_seconds": 5,
"lease_seconds": 15,
"backoff_base_seconds": 0.15
},
"checks": {
"topology_and_limits": {
"passed": true,
"api_replicas": 2,
"broker": "Valkey Streams with AOF fsync always",
"worker_processes": 2
},
"public_smoke_and_schema": {
"passed": true
},
"student_api_regressions": {
"passed": true
},
"worker_offline_api_broker_restart": {
"passed": true,
"attachment_id": "63406ab0-cc3c-4fa2-a0aa-e5e3353d5ae3",
"job_id": "7f450ff7-cb2d-4189-a668-f1967dbd94e8",
"message_id": "bc37bafb-efe1-4481-b304-c725c6e8368d",
"variants": {
"thumbnail": {
"size": [
256,
160
],
"bytes": 5813
},
"crop": {
"size": [
128,
128
],
"bytes": 2462
},
"optimized": {
"size": [
320,
200
],
"bytes": 2462
}
}
},
"outbox_crash_before_publish": {
"passed": true,
"job_id": "a3973e2b-ff73-4bca-bc41-394f2b875d13",
"automatic_recovery": true
},
"outbox_crash_after_publish": {
"passed": true,
"job_id": "007cb1d1-4b3b-4d12-9b2b-91a4682e8b08",
"automatic_recovery": true
},
"worker_crash_after_objects": {
"passed": true,
"job_id": "21129707-7dd0-4a72-ac33-5691dae3194e",
"attempt": 2,
"canonical_objects": 4,
"duplicate_noop": true
},
"worker_crash_before_ack": {
"passed": true,
"job_id": "2eeba68e-3fc7-42e0-adf9-13ac01c6c982",
"attempt": 1,
"canonical_objects": 4,
"duplicate_noop": true
},
"bounded_retry_failed_registry_replay": {
"passed": true,
"job_id": "bf756bf3-c3ca-45d2-acdd-3e67eb882d4d",
"max_automatic_attempts": 5,
"audit": [
{
"event": "processing",
"attempt": 1,
"error_code": null,
"at": "2026-11-12T15:25:18.431558+00:00"
},
{
"event": "retry",
"attempt": 1,
"error_code": "storage_unavailable",
"at": "2026-11-12T15:25:18.432821+00:00"
},
{
"event": "processing",
"attempt": 2,
"error_code": null,
"at": "2026-11-12T15:25:18.692895+00:00"
},
{
"event": "retry",
"attempt": 2,
"error_code": "storage_unavailable",
"at": "2026-11-12T15:25:18.69618+00:00"
},
{
"event": "processing",
"attempt": 3,
"error_code": null,
"at": "2026-11-12T15:25:19.206061+00:00"
},
{
"event": "retry",
"attempt": 3,
"error_code": "storage_unavailable",
"at": "2026-11-12T15:25:19.208065+00:00"
},
{
"event": "processing",
"attempt": 4,
"error_code": null,
"at": "2026-11-12T15:25:19.97778+00:00"
},
{
"event": "retry",
"attempt": 4,
"error_code": "storage_unavailable",
"at": "2026-11-12T15:25:19.978984+00:00"
},
{
"event": "processing",
"attempt": 5,
"error_code": null,
"at": "2026-11-12T15:25:21.777757+00:00"
},
{
"event": "failed",
"attempt": 5,
"error_code": "retry_exhausted",
"at": "2026-11-12T15:25:21.78129+00:00"
},
{
"event": "operator_replay",
"attempt": 5,
"error_code": null,
"at": "2026-11-12T15:25:22.680791+00:00"
},
{
"event": "processing",
"attempt": 6,
"error_code": null,
"at": "2026-11-12T15:25:22.802027+00:00"
},
{
"event": "ready",
"attempt": 6,
"error_code": null,
"at": "2026-11-12T15:25:23.151043+00:00"
}
],
"ready_replay_rejected": true
},
"permanent_error": {
"passed": true,
"job_id": "c2056d32-08bf-4c8a-8980-2f8b6c2511c9",
"attempt": 1,
"error_code": "invalid_image"
},
"image_subprocess_timeout": {
"passed": true,
"job_id": "6e8efff5-d82d-45f4-af12-3b05dcdb0076",
"timeout_seconds": 5,
"task_memory_mb": 640,
"recovered": true
},
"image_subprocess_memory": {
"passed": true,
"job_id": "ff1e7e76-b82d-4ba1-8ca9-79f3d78a4dfe",
"timeout_seconds": 5,
"task_memory_mb": 640,
"recovered": true
},
"image_semantics": {
"passed": true,
"cases": [
{
"case": "small_alpha_on_white",
"variants": {
"thumbnail": {
"size": [
17,
9
],
"bytes": 635
},
"crop": {
"size": [
128,
128
],
"bytes": 883
},
"optimized": {
"size": [
17,
9
],
"bytes": 56
}
}
},
{
"case": "exif_orientation_6_and_floor",
"variants": {
"thumbnail": {
"size": [
160,
256
],
"bytes": 1270
},
"crop": {
"size": [
128,
128
],
"bytes": 886
},
"optimized": {
"size": [
201,
321
],
"bytes": 210
}
}
}
]
},
"staging_cleanup": {
"passed": true,
"active_preserved": true,
"orphan_removed": true,
"ready_removed": true,
"expired_failed_removed": true,
"expired_replay_rejected": true
},
"broker_offline_outbox": {
"passed": true,
"attachment_id": "3b3d3600-efcc-4840-ab76-07868501c21d",
"recovered": true
},
"public_persistence": {
"passed": true
},
"public_api_failover": {
"passed": true
}
},
"docker": "29.0.1 aarch64 CPUs=10 RAM=8217448448",
"containers": [
{
"id": "9fbe552bf1bd8c2be76b3434cc5662479ca8d328ca698813ac9a785bcafc0cfd",
"service": "api",
"image": "mtusi-messenger:lab5",
"memory_bytes": 402653184,
"nano_cpus": 1000000000,
"mounts": [
{
"destination": "/run/secrets/s3_secret_access_key",
"name": null
},
{
"destination": "/run/secrets/database_url",
"name": null
},
{
"destination": "/run/secrets/cache_url",
"name": null
},
{
"destination": "/run/secrets/s3_access_key_id",
"name": null
}
]
},
{
"id": "ebc82ed28847cbaf6f474088c68005f05d49469010b45ff3767fb0a944ead6e0",
"service": "api",
"image": "mtusi-messenger:lab5",
"memory_bytes": 402653184,
"nano_cpus": 1000000000,
"mounts": [
{
"destination": "/run/secrets/s3_access_key_id",
"name": null
},
{
"destination": "/run/secrets/s3_secret_access_key",
"name": null
},
{
"destination": "/run/secrets/database_url",
"name": null
},
{
"destination": "/run/secrets/cache_url",
"name": null
}
]
},
{
"id": "26a48bd7b3390a4cd99667d4d99d6342edce8e26accaee3b181a4d842203261a",
"service": "broker",
"image": "valkey/valkey:8.1.3",
"memory_bytes": 268435456,
"nano_cpus": 500000000,
"mounts": [
{
"destination": "/data",
"name": "lab5-check-d233766d34_broker_data"
},
{
"destination": "/run/secrets/broker_password",
"name": null
}
]
},
{
"id": "232fdc281b939bf905b4123102a6a6eb609a4ffcb730f6d03707beb07283d69c",
"service": "cleanup",
"image": "mtusi-messenger:lab5",
"memory_bytes": 402653184,
"nano_cpus": 1000000000,
"mounts": [
{
"destination": "/run/secrets/database_url",
"name": null
},
{
"destination": "/run/secrets/s3_access_key_id",
"name": null
},
{
"destination": "/run/secrets/s3_secret_access_key",
"name": null
}
]
},
{
"id": "e722801286bb112062f1da59af65dc21eddf5059ca086022e0d52f85a0f26154",
"service": "lb",
"image": "haproxy:3.2.6-alpine",
"memory_bytes": 67108864,
"nano_cpus": 500000000,
"mounts": [
{
"destination": "/usr/local/etc/haproxy/haproxy.cfg",
"name": null
}
]
},
{
"id": "ba806a4ce014a516ea867a516743de64bef0d449726bb544d265bf31b4055b1b",
"service": "migrate",
"image": "mtusi-messenger:lab5",
"memory_bytes": 402653184,
"nano_cpus": 1000000000,
"mounts": [
{
"destination": "/run/secrets/database_url",
"name": null
}
]
},
{
"id": "0fd62d960020290ba4089d05fdcc8da2bb0e2faf7528fc6ed3384e35236d4290",
"service": "postgres",
"image": "postgres:17.6-bookworm",
"memory_bytes": 536870912,
"nano_cpus": 1000000000,
"mounts": [
{
"destination": "/var/lib/postgresql/data",
"name": "lab5-check-d233766d34_postgres_data"
},
{
"destination": "/run/secrets/postgres_password",
"name": null
}
]
},
{
"id": "c658ec3ef1e14ca2f1ca8f1530e6f13b37b99ce8389fafeae84e841e65daa2fc",
"service": "publisher",
"image": "mtusi-messenger:lab5",
"memory_bytes": 402653184,
"nano_cpus": 1000000000,
"mounts": [
{
"destination": "/run/secrets/database_url",
"name": null
},
{
"destination": "/run/secrets/broker_url",
"name": null
}
]
},
{
"id": "bc944a3a03374052aba43f42767c253755a921090fd15ead92fa2a924507c941",
"service": "s3",
"image": "chrislusf/seaweedfs:3.85@sha256:49312939c00c01e5ee6afbd7d728b18027821d3764c35a797a72acd4fdf3296a",
"memory_bytes": 268435456,
"nano_cpus": 500000000,
"mounts": [
{
"destination": "/run/secrets/s3_config",
"name": null
},
{
"destination": "/data",
"name": "0496c4617fd5cac2843b4c391fc1eb42e15c90c4d9a35d092e71a6fd8815960f"
}
]
},
{
"id": "9e7382c34331cb65974fa3feb248105c8055fa25c251151f0f004b65fc388923",
"service": "s3-bootstrap",
"image": "mtusi-messenger:lab5",
"memory_bytes": 134217728,
"nano_cpus": 500000000,
"mounts": [
{
"destination": "/run/secrets/s3_root_password",
"name": null
},
{
"destination": "/run/secrets/s3_root_user",
"name": null
}
]
},
{
"id": "b9821d3ec48ce18b8255a446cf99b02d6ea9658de406a15cc0ace7ccf7d7350e",
"service": "s3-storage",
"image": "chrislusf/seaweedfs:3.85@sha256:49312939c00c01e5ee6afbd7d728b18027821d3764c35a797a72acd4fdf3296a",
"memory_bytes": 536870912,
"nano_cpus": 1000000000,
"mounts": [
{
"destination": "/data",
"name": "lab5-check-d233766d34_s3_data"
}
]
},
{
"id": "c8ddc9fede2be42d05e79147dfc7dd87efede3b67f3226afeee1161fcaa49428",
"service": "valkey",
"image": "valkey/valkey:8.1.3",
"memory_bytes": 134217728,
"nano_cpus": 500000000,
"mounts": [
{
"destination": "/data",
"name": "lab5-check-d233766d34_valkey_data"
}
]
},
{
"id": "9fee2e7832c5c11e81bb56df9d435ad090fdb1fddb85efbf014ea6019559ed75",
"service": "worker",
"image": "mtusi-messenger:lab5",
"memory_bytes": 1610612736,
"nano_cpus": 2000000000,
"mounts": [
{
"destination": "/run/secrets/broker_url",
"name": null
},
{
"destination": "/run/secrets/s3_access_key_id",
"name": null
},
{
"destination": "/run/secrets/s3_secret_access_key",
"name": null
},
{
"destination": "/run/secrets/database_url",
"name": null
}
]
}
],
"passed": true
}
+77
View File
@@ -0,0 +1,77 @@
PASS: topology_and_limits {"api_replicas": 2, "broker": "Valkey Streams with AOF fsync always", "worker_processes": 2}
/Users/vanek/Desktop/VNPlabs/VNPlab5/mtusi2026lab5/.venv/bin/python scripts/check_contract.py
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
/Users/vanek/Desktop/VNPlabs/VNPlab5/mtusi2026lab5/.venv/bin/python scripts/validate_openapi.py
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
/Users/vanek/Desktop/VNPlabs/VNPlab5/mtusi2026lab5/.venv/bin/python tests/smoke.py --base-url "http://127.0.0.1:57277"
PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.
PASS: public_smoke_and_schema {}
PASS: 10 clients sent 100 messages; history contains exactly 100 unique messages.
test_chat_access (test_api.ApiTests.test_chat_access) ... ok
test_chat_pagination_and_cursor_owner (test_api.ApiTests.test_chat_pagination_and_cursor_owner) ... ok
test_cursor_cannot_be_used_for_another_chat_or_changed (test_api.ApiTests.test_cursor_cannot_be_used_for_another_chat_or_changed) ... ok
test_default_page_size (test_api.ApiTests.test_default_page_size) ... ok
test_duplicate_username (test_api.ApiTests.test_duplicate_username) ... ok
test_framework_errors_and_request_ids (test_api.ApiTests.test_framework_errors_and_request_ids) ... ok
test_identification_and_authentication_order (test_api.ApiTests.test_identification_and_authentication_order) ... ok
test_invalid_chat_members (test_api.ApiTests.test_invalid_chat_members) ... ok
test_invalid_chat_title (test_api.ApiTests.test_invalid_chat_title) ... ok
test_invalid_limits_and_cursors (test_api.ApiTests.test_invalid_limits_and_cursors) ... ok
test_invalid_message_fields (test_api.ApiTests.test_invalid_message_fields) ... ok
test_invalid_user_fields (test_api.ApiTests.test_invalid_user_fields) ... ok
test_malformed_json_and_encoding (test_api.ApiTests.test_malformed_json_and_encoding) ... ok
test_message_pagination_and_changing_limit (test_api.ApiTests.test_message_pagination_and_changing_limit) ... ok
test_unicode_and_text_preservation (test_api.ApiTests.test_unicode_and_text_preservation) ... ok
test_wrong_content_type (test_api.ApiTests.test_wrong_content_type) ... ok
test_access_and_authentication (test_attachments.AttachmentTests.test_access_and_authentication) ... ok
test_attachment_only_message_and_other_member (test_attachments.AttachmentTests.test_attachment_only_message_and_other_member) ... ok
test_chunked_limit_without_content_length (test_attachments.AttachmentTests.test_chunked_limit_without_content_length) ... ok
test_content_length_limit_before_body (test_attachments.AttachmentTests.test_content_length_limit_before_body) ... ok
test_corruption_and_mime_spoof (test_attachments.AttachmentTests.test_corruption_and_mime_spoof) ... ok
test_filename_rejected_before_storage (test_attachments.AttachmentTests.test_filename_rejected_before_storage) ... ok
test_foreign_and_unknown_attachment_ids (test_attachments.AttachmentTests.test_foreign_and_unknown_attachment_ids) ... ok
test_idempotency_includes_ordered_attachment_ids (test_attachments.AttachmentTests.test_idempotency_includes_ordered_attachment_ids) ... ok
test_invalid_attachment_lists (test_attachments.AttachmentTests.test_invalid_attachment_lists) ... ok
test_pixel_limit_before_decode (test_attachments.AttachmentTests.test_pixel_limit_before_decode) ... ok
test_png_and_jpeg_originals (test_attachments.AttachmentTests.test_png_and_jpeg_originals) ... ok
test_simultaneous_registration_of_one_username (test_concurrency.ConcurrencyTests.test_simultaneous_registration_of_one_username) ... ok
test_ten_clients_send_one_hundred_messages (test_concurrency.ConcurrencyTests.test_ten_clients_send_one_hundred_messages) ... ok
test_conflict_and_semantic_json_replay (test_idempotency.IdempotencyTests.test_conflict_and_semantic_json_replay) ... ok
test_key_format_and_missing_key (test_idempotency.IdempotencyTests.test_key_format_and_missing_key) ... ok
test_parallel_replay_across_replicas (test_idempotency.IdempotencyTests.test_parallel_replay_across_replicas) ... ok
test_replay_rechecks_session_and_access (test_idempotency.IdempotencyTests.test_replay_rechecks_session_and_access) ... ok
test_scope_is_user_and_chat (test_idempotency.IdempotencyTests.test_scope_is_user_and_chat) ... ok
test_logout_revokes_only_one_session (test_sessions.SessionTests.test_logout_revokes_only_one_session) ... ok
test_password_validation (test_sessions.SessionTests.test_password_validation) ... ok
test_sql_like_message_is_plain_text (test_sessions.SessionTests.test_sql_like_message_is_plain_text) ... ok
test_unicode_password_with_colon_and_private_response (test_sessions.SessionTests.test_unicode_password_with_colon_and_private_response) ... ok
test_wrong_password_and_basic_only_for_login (test_sessions.SessionTests.test_wrong_password_and_basic_only_for_login) ... ok
----------------------------------------------------------------------
Ran 39 tests in 6.981s
OK
PASS: student_api_regressions {}
PASS: worker_offline_api_broker_restart {"attachment_id": "63406ab0-cc3c-4fa2-a0aa-e5e3353d5ae3", "job_id": "7f450ff7-cb2d-4189-a668-f1967dbd94e8", "message_id": "bc37bafb-efe1-4481-b304-c725c6e8368d", "variants": {"thumbnail": {"size": [256, 160], "bytes": 5813}, "crop": {"size": [128, 128], "bytes": 2462}, "optimized": {"size": [320, 200], "bytes": 2462}}}
PASS: outbox_crash_before_publish {"job_id": "a3973e2b-ff73-4bca-bc41-394f2b875d13", "automatic_recovery": true}
PASS: outbox_crash_after_publish {"job_id": "007cb1d1-4b3b-4d12-9b2b-91a4682e8b08", "automatic_recovery": true}
PASS: worker_crash_after_objects {"job_id": "21129707-7dd0-4a72-ac33-5691dae3194e", "attempt": 2, "canonical_objects": 4, "duplicate_noop": true}
PASS: worker_crash_before_ack {"job_id": "2eeba68e-3fc7-42e0-adf9-13ac01c6c982", "attempt": 1, "canonical_objects": 4, "duplicate_noop": true}
PASS: bounded_retry_failed_registry_replay {"job_id": "bf756bf3-c3ca-45d2-acdd-3e67eb882d4d", "max_automatic_attempts": 5, "audit": [{"event": "processing", "attempt": 1, "error_code": null, "at": "2026-11-12T15:25:18.431558+00:00"}, {"event": "retry", "attempt": 1, "error_code": "storage_unavailable", "at": "2026-11-12T15:25:18.432821+00:00"}, {"event": "processing", "attempt": 2, "error_code": null, "at": "2026-11-12T15:25:18.692895+00:00"}, {"event": "retry", "attempt": 2, "error_code": "storage_unavailable", "at": "2026-11-12T15:25:18.69618+00:00"}, {"event": "processing", "attempt": 3, "error_code": null, "at": "2026-11-12T15:25:19.206061+00:00"}, {"event": "retry", "attempt": 3, "error_code": "storage_unavailable", "at": "2026-11-12T15:25:19.208065+00:00"}, {"event": "processing", "attempt": 4, "error_code": null, "at": "2026-11-12T15:25:19.97778+00:00"}, {"event": "retry", "attempt": 4, "error_code": "storage_unavailable", "at": "2026-11-12T15:25:19.978984+00:00"}, {"event": "processing", "attempt": 5, "error_code": null, "at": "2026-11-12T15:25:21.777757+00:00"}, {"event": "failed", "attempt": 5, "error_code": "retry_exhausted", "at": "2026-11-12T15:25:21.78129+00:00"}, {"event": "operator_replay", "attempt": 5, "error_code": null, "at": "2026-11-12T15:25:22.680791+00:00"}, {"event": "processing", "attempt": 6, "error_code": null, "at": "2026-11-12T15:25:22.802027+00:00"}, {"event": "ready", "attempt": 6, "error_code": null, "at": "2026-11-12T15:25:23.151043+00:00"}], "ready_replay_rejected": true}
PASS: permanent_error {"job_id": "c2056d32-08bf-4c8a-8980-2f8b6c2511c9", "attempt": 1, "error_code": "invalid_image"}
PASS: image_subprocess_timeout {"job_id": "6e8efff5-d82d-45f4-af12-3b05dcdb0076", "timeout_seconds": 5, "task_memory_mb": 640, "recovered": true}
PASS: image_subprocess_memory {"job_id": "ff1e7e76-b82d-4ba1-8ca9-79f3d78a4dfe", "timeout_seconds": 5, "task_memory_mb": 640, "recovered": true}
PASS: image_semantics {"cases": [{"case": "small_alpha_on_white", "variants": {"thumbnail": {"size": [17, 9], "bytes": 635}, "crop": {"size": [128, 128], "bytes": 883}, "optimized": {"size": [17, 9], "bytes": 56}}}, {"case": "exif_orientation_6_and_floor", "variants": {"thumbnail": {"size": [160, 256], "bytes": 1270}, "crop": {"size": [128, 128], "bytes": 886}, "optimized": {"size": [201, 321], "bytes": 210}}}]}
PASS: staging_cleanup {"active_preserved": true, "orphan_removed": true, "ready_removed": true, "expired_failed_removed": true, "expired_replay_rejected": true}
PASS: broker_offline_outbox {"attachment_id": "3b3d3600-efcc-4840-ab76-07868501c21d", "recovered": true}
/Users/vanek/Desktop/VNPlabs/VNPlab5/mtusi2026lab5/.venv/bin/python tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://127.0.0.1:57277"
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
PASS: public_persistence {}
/Users/vanek/Desktop/VNPlabs/VNPlab5/mtusi2026lab5/.venv/bin/python tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:57277"
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 110, "failed_attempts": 1, "first_recovery_seconds": 2.797}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
PASS: public_api_failover {}
PASS: all lab5 grade-4 criteria completed
Removed isolated test stack: lab5-check-d233766d34
+140
View File
@@ -0,0 +1,140 @@
{
"utc": "2026-11-12T15:28:31.543303+00:00",
"command": ".venv/bin/python student-tests/verify_restart.py",
"passed": true,
"before": {
"9e5a9d3d2bf0b2e47ee6d85b40b1899881db0d4d91d9ec09e50fb36aca67833c": {
"service": "api",
"started_at": "2026-11-12T15:28:13.153166256Z",
"volumes": []
},
"83bab824b05accd2aa8829e83e272d72d4105644f8270cf3dc15f3412d569f5b": {
"service": "api",
"started_at": "2026-11-12T15:28:13.038450506Z",
"volumes": []
},
"6f72735cbaed44f82555dcd550199764ab53a8c61a63c88f25c612fd39b6baba": {
"service": "broker",
"started_at": "2026-11-12T15:27:46.698266716Z",
"volumes": [
"mtusi-lab5_broker_data"
]
},
"0f9060c7dc8473fa926c5affc560b3478b90c77771de9810ebba7858a30838d5": {
"service": "cleanup",
"started_at": "2026-11-12T15:28:13.036348173Z",
"volumes": []
},
"d381adfb7273a5dd2f22d361a6e1ff490e5c2cf44ff147a8253f7f940cd01de4": {
"service": "lb",
"started_at": "2026-11-12T15:28:18.76757655Z",
"volumes": []
},
"9906c3edca610723e6f09b64867fa71a543f18ef4b474804414e9f818a803115": {
"service": "postgres",
"started_at": "2026-11-12T15:27:46.700095216Z",
"volumes": [
"mtusi-lab5_postgres_data"
]
},
"d705dceffd740900bbad04ee520bb96247f286c7c5381734a28bc1c9e30f27e4": {
"service": "publisher",
"started_at": "2026-11-12T15:27:49.924416218Z",
"volumes": []
},
"cbe9063ff82960d9db58bc5713af90e4be3e72fdd1c08b0a92a93849dc6ad1aa": {
"service": "s3",
"started_at": "2026-11-12T15:28:09.808290754Z",
"volumes": [
"fe34bef7724beefe41335cd197f83aae91b16df70766cee6321bff12f765bc66"
]
},
"6d1b6c402e19a068a0f2f4b9c06d5abbec7e41b5d778a883e6d045be409f6d4c": {
"service": "s3-storage",
"started_at": "2026-11-12T15:27:46.699445508Z",
"volumes": [
"mtusi-lab5_s3_data"
]
},
"6d8c3e69c4b60acaf42e0259d367a0a3f2c06081a6349c284ddf9f2e9b5ba399": {
"service": "valkey",
"started_at": "2026-11-12T15:27:46.697619716Z",
"volumes": [
"mtusi-lab5_valkey_data"
]
},
"28fabe35a9a5b57e0a3a96bb4ae5308fbd2fa825470799180a2471b55400206e": {
"service": "worker",
"started_at": "2026-11-12T15:28:13.037172673Z",
"volumes": []
}
},
"after": {
"9e5a9d3d2bf0b2e47ee6d85b40b1899881db0d4d91d9ec09e50fb36aca67833c": {
"service": "api",
"started_at": "2026-11-12T15:28:33.373854585Z",
"volumes": []
},
"83bab824b05accd2aa8829e83e272d72d4105644f8270cf3dc15f3412d569f5b": {
"service": "api",
"started_at": "2026-11-12T15:28:33.338416001Z",
"volumes": []
},
"6f72735cbaed44f82555dcd550199764ab53a8c61a63c88f25c612fd39b6baba": {
"service": "broker",
"started_at": "2026-11-12T15:28:33.257433543Z",
"volumes": [
"mtusi-lab5_broker_data"
]
},
"0f9060c7dc8473fa926c5affc560b3478b90c77771de9810ebba7858a30838d5": {
"service": "cleanup",
"started_at": "2026-11-12T15:28:40.990298547Z",
"volumes": []
},
"d381adfb7273a5dd2f22d361a6e1ff490e5c2cf44ff147a8253f7f940cd01de4": {
"service": "lb",
"started_at": "2026-11-12T15:28:33.069501668Z",
"volumes": []
},
"9906c3edca610723e6f09b64867fa71a543f18ef4b474804414e9f818a803115": {
"service": "postgres",
"started_at": "2026-11-12T15:28:33.134879876Z",
"volumes": [
"mtusi-lab5_postgres_data"
]
},
"d705dceffd740900bbad04ee520bb96247f286c7c5381734a28bc1c9e30f27e4": {
"service": "publisher",
"started_at": "2026-11-12T15:28:40.90857463Z",
"volumes": []
},
"cbe9063ff82960d9db58bc5713af90e4be3e72fdd1c08b0a92a93849dc6ad1aa": {
"service": "s3",
"started_at": "2026-11-12T15:28:33.167460626Z",
"volumes": [
"fe34bef7724beefe41335cd197f83aae91b16df70766cee6321bff12f765bc66"
]
},
"6d1b6c402e19a068a0f2f4b9c06d5abbec7e41b5d778a883e6d045be409f6d4c": {
"service": "s3-storage",
"started_at": "2026-11-12T15:28:40.950084172Z",
"volumes": [
"mtusi-lab5_s3_data"
]
},
"6d8c3e69c4b60acaf42e0259d367a0a3f2c06081a6349c284ddf9f2e9b5ba399": {
"service": "valkey",
"started_at": "2026-11-12T15:28:33.205284001Z",
"volumes": [
"mtusi-lab5_valkey_data"
]
},
"28fabe35a9a5b57e0a3a96bb4ae5308fbd2fa825470799180a2471b55400206e": {
"service": "worker",
"started_at": "2026-11-12T15:28:33.299354043Z",
"volumes": []
}
},
"same_containers_and_volumes": true
}
+4
View File
@@ -0,0 +1,4 @@
/Users/vanek/Desktop/VNPlabs/VNPlab5/mtusi2026lab5/.venv/bin/python tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:8080"
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
PASS: all persistent services actually restarted; container IDs and volumes preserved
+51
View File
@@ -0,0 +1,51 @@
.venv/bin/python scripts/check_contract.py
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
.venv/bin/python scripts/validate_openapi.py
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
.venv/bin/python tests/smoke.py --base-url "http://localhost:8080"
PASS: lab6 public API scenario; infrastructure and grading evidence remain separate.
test_chat_access (test_api.ApiTests.test_chat_access) ... ok
test_chat_pagination_and_cursor_owner (test_api.ApiTests.test_chat_pagination_and_cursor_owner) ... ok
test_cursor_cannot_be_used_for_another_chat_or_changed (test_api.ApiTests.test_cursor_cannot_be_used_for_another_chat_or_changed) ... ok
test_default_page_size (test_api.ApiTests.test_default_page_size) ... ok
test_duplicate_username (test_api.ApiTests.test_duplicate_username) ... ok
test_framework_errors_and_request_ids (test_api.ApiTests.test_framework_errors_and_request_ids) ... ok
test_identification_and_authentication_order (test_api.ApiTests.test_identification_and_authentication_order) ... ok
test_invalid_chat_members (test_api.ApiTests.test_invalid_chat_members) ... ok
test_invalid_chat_title (test_api.ApiTests.test_invalid_chat_title) ... ok
test_invalid_limits_and_cursors (test_api.ApiTests.test_invalid_limits_and_cursors) ... ok
test_invalid_message_fields (test_api.ApiTests.test_invalid_message_fields) ... ok
test_invalid_user_fields (test_api.ApiTests.test_invalid_user_fields) ... ok
test_malformed_json_and_encoding (test_api.ApiTests.test_malformed_json_and_encoding) ... ok
test_message_pagination_and_changing_limit (test_api.ApiTests.test_message_pagination_and_changing_limit) ... ok
test_unicode_and_text_preservation (test_api.ApiTests.test_unicode_and_text_preservation) ... ok
test_wrong_content_type (test_api.ApiTests.test_wrong_content_type) ... ok
test_access_and_authentication (test_attachments.AttachmentTests.test_access_and_authentication) ... ok
test_attachment_only_message_and_other_member (test_attachments.AttachmentTests.test_attachment_only_message_and_other_member) ... ok
test_chunked_limit_without_content_length (test_attachments.AttachmentTests.test_chunked_limit_without_content_length) ... ok
test_content_length_limit_before_body (test_attachments.AttachmentTests.test_content_length_limit_before_body) ... ok
test_corruption_and_mime_spoof (test_attachments.AttachmentTests.test_corruption_and_mime_spoof) ... ok
test_filename_rejected_before_storage (test_attachments.AttachmentTests.test_filename_rejected_before_storage) ... ok
test_foreign_and_unknown_attachment_ids (test_attachments.AttachmentTests.test_foreign_and_unknown_attachment_ids) ... ok
test_idempotency_includes_ordered_attachment_ids (test_attachments.AttachmentTests.test_idempotency_includes_ordered_attachment_ids) ... ok
test_invalid_attachment_lists (test_attachments.AttachmentTests.test_invalid_attachment_lists) ... ok
test_pixel_limit_before_decode (test_attachments.AttachmentTests.test_pixel_limit_before_decode) ... ok
test_png_and_jpeg_originals (test_attachments.AttachmentTests.test_png_and_jpeg_originals) ... ok
test_simultaneous_registration_of_one_username (test_concurrency.ConcurrencyTests.test_simultaneous_registration_of_one_username) ... ok
test_ten_clients_send_one_hundred_messages (test_concurrency.ConcurrencyTests.test_ten_clients_send_one_hundred_messages) ... ok
test_conflict_and_semantic_json_replay (test_idempotency.IdempotencyTests.test_conflict_and_semantic_json_replay) ... ok
test_key_format_and_missing_key (test_idempotency.IdempotencyTests.test_key_format_and_missing_key) ... ok
test_parallel_replay_across_replicas (test_idempotency.IdempotencyTests.test_parallel_replay_across_replicas) ... ok
test_replay_rechecks_session_and_access (test_idempotency.IdempotencyTests.test_replay_rechecks_session_and_access) ... ok
test_scope_is_user_and_chat (test_idempotency.IdempotencyTests.test_scope_is_user_and_chat) ... ok
test_logout_revokes_only_one_session (test_sessions.SessionTests.test_logout_revokes_only_one_session) ... ok
test_password_validation (test_sessions.SessionTests.test_password_validation) ... ok
test_sql_like_message_is_plain_text (test_sessions.SessionTests.test_sql_like_message_is_plain_text) ... ok
test_unicode_password_with_colon_and_private_response (test_sessions.SessionTests.test_unicode_password_with_colon_and_private_response) ... ok
test_wrong_password_and_basic_only_for_login (test_sessions.SessionTests.test_wrong_password_and_basic_only_for_login) ... ok
----------------------------------------------------------------------
Ran 39 tests in 7.518s
OK
PASS: 10 clients sent 100 messages; history contains exactly 100 unique messages.
+102
View File
@@ -0,0 +1,102 @@
{
"utc": "2026-11-19T15:09:33.967862+00:00",
"checks": {
"retry_trace_and_logs": {
"passed": true,
"trace_id": "93e1177f9568f5bbfc0b6967a1324a84",
"request_id": "422b23e9-c8f9-4810-ad09-1cb99b6c9576",
"job_id": "fd4d7cce-b0dc-4edd-9938-61230afa7370",
"span_count": 59
},
"metrics_and_cardinality": {
"passed": true,
"series": 1192
},
"backend_outage": {
"passed": true,
"backend": "loki",
"successful_writes": 24,
"outage_seconds": 51.39,
"missing_request_logs": 6,
"tested_request_ids": [
"a5835a4b-40b6-4383-90a3-82d07b5bdcfb",
"ee8b61c0-9acc-4ff4-a0fd-f4a7d84e29a2",
"bb492157-8ffd-49a6-84ef-c2b17379bc8e",
"6a076038-13e0-415d-b961-254cb7d48194",
"59ab7cdd-f319-4828-a0b2-4bb9ecef9464",
"0b472396-a1f4-453c-aff2-0a8e2650f3e3",
"b0916ed0-b218-423a-ac39-1c6e904300e6",
"bcd32c58-953b-4759-ac8b-e87fc66c4fd4",
"0a065b55-3fac-46fc-9d92-c67f8ba6e409",
"11d46ead-2941-4279-aa86-0abf111bcaba",
"f6807acd-4e5f-4bef-9f35-93b2ac581390",
"fec97823-5700-4fab-a18a-8a98f11f5c4e",
"a03b0d52-34ea-4b0a-86f2-ac408a0a107c",
"6d44ee0d-f039-406b-b0f7-4ceb7743d020",
"efc0b1d3-e889-4ab4-a169-0369c4cb260b",
"0ae460c2-6b26-4f5c-8959-ecb48a61ca92",
"0f59b3eb-2187-412b-b569-c2373748020e",
"086464d0-5b79-4c9d-aa95-e08af7858c43",
"4c42e246-b973-4e79-8a37-be114b255b80",
"2065d6ba-c0f8-43a5-999f-55ec5299d855",
"8ccb2503-c32f-4b3d-9763-2d9455f13a99",
"979ef391-bc49-42af-95b8-194e028dd796",
"fe3a4f81-114e-4055-b160-7f47a15acfff",
"e6019f30-5b26-4bc1-a630-bcfbbeb6ba96"
],
"missing_request_ids": [
"a5835a4b-40b6-4383-90a3-82d07b5bdcfb",
"ee8b61c0-9acc-4ff4-a0fd-f4a7d84e29a2",
"bb492157-8ffd-49a6-84ef-c2b17379bc8e",
"6a076038-13e0-415d-b961-254cb7d48194",
"59ab7cdd-f319-4828-a0b2-4bb9ecef9464",
"0b472396-a1f4-453c-aff2-0a8e2650f3e3"
],
"collector_memory_samples": [
"{\"BlockIO\":\"0B / 0B\",\"CPUPerc\":\"0.12%\",\"Container\":\"6e21c63012d160a406e28a342d668de0be55107aa16228ed9d5dc15c34b447be\",\"ID\":\"6e21c63012d1\",\"MemPerc\":\"28.44%\",\"MemUsage\":\"72.8MiB / 256MiB\",\"Name\":\"mtusi-lab6-otel-collector-1\",\"NetIO\":\"7.32MB / 2.49MB\",\"PIDs\":\"15\"}",
"{\"BlockIO\":\"0B / 0B\",\"CPUPerc\":\"0.43%\",\"Container\":\"6e21c63012d160a406e28a342d668de0be55107aa16228ed9d5dc15c34b447be\",\"ID\":\"6e21c63012d1\",\"MemPerc\":\"27.52%\",\"MemUsage\":\"70.46MiB / 256MiB\",\"Name\":\"mtusi-lab6-otel-collector-1\",\"NetIO\":\"7.48MB / 2.53MB\",\"PIDs\":\"15\"}",
"{\"BlockIO\":\"0B / 0B\",\"CPUPerc\":\"0.42%\",\"Container\":\"6e21c63012d160a406e28a342d668de0be55107aa16228ed9d5dc15c34b447be\",\"ID\":\"6e21c63012d1\",\"MemPerc\":\"27.95%\",\"MemUsage\":\"71.56MiB / 256MiB\",\"Name\":\"mtusi-lab6-otel-collector-1\",\"NetIO\":\"7.64MB / 2.58MB\",\"PIDs\":\"15\"}",
"{\"BlockIO\":\"0B / 0B\",\"CPUPerc\":\"0.20%\",\"Container\":\"6e21c63012d160a406e28a342d668de0be55107aa16228ed9d5dc15c34b447be\",\"ID\":\"6e21c63012d1\",\"MemPerc\":\"27.96%\",\"MemUsage\":\"71.57MiB / 256MiB\",\"Name\":\"mtusi-lab6-otel-collector-1\",\"NetIO\":\"7.81MB / 2.62MB\",\"PIDs\":\"15\"}"
],
"collector_limit_bytes": 268435456,
"oom_killed": false,
"failed_exports_before": [
{
"metric": {
"__name__": "otelcol_exporter_send_failed_log_records_total",
"exporter": "otlphttp/loki",
"instance": "otel-collector:8888",
"job": "collector",
"otel_scope_name": "go.opentelemetry.io/collector/exporter/exporterhelper"
},
"value": [
1795100986.923,
"6"
]
}
],
"failed_exports_after": [
{
"metric": {
"__name__": "otelcol_exporter_send_failed_log_records_total",
"exporter": "otlphttp/loki",
"instance": "otel-collector:8888",
"job": "collector",
"otel_scope_name": "go.opentelemetry.io/collector/exporter/exporterhelper"
},
"value": [
1795101059.133,
"12"
]
}
]
},
"provisioned_dashboard_queries": {
"passed": true,
"panels": 19
}
},
"dashboard": "messenger-lab6",
"passed": true,
"duration_seconds": 85.23
}
+102
View File
@@ -0,0 +1,102 @@
{
"utc": "2026-11-19T15:09:33.967862+00:00",
"checks": {
"retry_trace_and_logs": {
"passed": true,
"trace_id": "93e1177f9568f5bbfc0b6967a1324a84",
"request_id": "422b23e9-c8f9-4810-ad09-1cb99b6c9576",
"job_id": "fd4d7cce-b0dc-4edd-9938-61230afa7370",
"span_count": 59
},
"metrics_and_cardinality": {
"passed": true,
"series": 1192
},
"backend_outage": {
"passed": true,
"backend": "loki",
"successful_writes": 24,
"outage_seconds": 51.39,
"missing_request_logs": 6,
"tested_request_ids": [
"a5835a4b-40b6-4383-90a3-82d07b5bdcfb",
"ee8b61c0-9acc-4ff4-a0fd-f4a7d84e29a2",
"bb492157-8ffd-49a6-84ef-c2b17379bc8e",
"6a076038-13e0-415d-b961-254cb7d48194",
"59ab7cdd-f319-4828-a0b2-4bb9ecef9464",
"0b472396-a1f4-453c-aff2-0a8e2650f3e3",
"b0916ed0-b218-423a-ac39-1c6e904300e6",
"bcd32c58-953b-4759-ac8b-e87fc66c4fd4",
"0a065b55-3fac-46fc-9d92-c67f8ba6e409",
"11d46ead-2941-4279-aa86-0abf111bcaba",
"f6807acd-4e5f-4bef-9f35-93b2ac581390",
"fec97823-5700-4fab-a18a-8a98f11f5c4e",
"a03b0d52-34ea-4b0a-86f2-ac408a0a107c",
"6d44ee0d-f039-406b-b0f7-4ceb7743d020",
"efc0b1d3-e889-4ab4-a169-0369c4cb260b",
"0ae460c2-6b26-4f5c-8959-ecb48a61ca92",
"0f59b3eb-2187-412b-b569-c2373748020e",
"086464d0-5b79-4c9d-aa95-e08af7858c43",
"4c42e246-b973-4e79-8a37-be114b255b80",
"2065d6ba-c0f8-43a5-999f-55ec5299d855",
"8ccb2503-c32f-4b3d-9763-2d9455f13a99",
"979ef391-bc49-42af-95b8-194e028dd796",
"fe3a4f81-114e-4055-b160-7f47a15acfff",
"e6019f30-5b26-4bc1-a630-bcfbbeb6ba96"
],
"missing_request_ids": [
"a5835a4b-40b6-4383-90a3-82d07b5bdcfb",
"ee8b61c0-9acc-4ff4-a0fd-f4a7d84e29a2",
"bb492157-8ffd-49a6-84ef-c2b17379bc8e",
"6a076038-13e0-415d-b961-254cb7d48194",
"59ab7cdd-f319-4828-a0b2-4bb9ecef9464",
"0b472396-a1f4-453c-aff2-0a8e2650f3e3"
],
"collector_memory_samples": [
"{\"BlockIO\":\"0B / 0B\",\"CPUPerc\":\"0.12%\",\"Container\":\"6e21c63012d160a406e28a342d668de0be55107aa16228ed9d5dc15c34b447be\",\"ID\":\"6e21c63012d1\",\"MemPerc\":\"28.44%\",\"MemUsage\":\"72.8MiB / 256MiB\",\"Name\":\"mtusi-lab6-otel-collector-1\",\"NetIO\":\"7.32MB / 2.49MB\",\"PIDs\":\"15\"}",
"{\"BlockIO\":\"0B / 0B\",\"CPUPerc\":\"0.43%\",\"Container\":\"6e21c63012d160a406e28a342d668de0be55107aa16228ed9d5dc15c34b447be\",\"ID\":\"6e21c63012d1\",\"MemPerc\":\"27.52%\",\"MemUsage\":\"70.46MiB / 256MiB\",\"Name\":\"mtusi-lab6-otel-collector-1\",\"NetIO\":\"7.48MB / 2.53MB\",\"PIDs\":\"15\"}",
"{\"BlockIO\":\"0B / 0B\",\"CPUPerc\":\"0.42%\",\"Container\":\"6e21c63012d160a406e28a342d668de0be55107aa16228ed9d5dc15c34b447be\",\"ID\":\"6e21c63012d1\",\"MemPerc\":\"27.95%\",\"MemUsage\":\"71.56MiB / 256MiB\",\"Name\":\"mtusi-lab6-otel-collector-1\",\"NetIO\":\"7.64MB / 2.58MB\",\"PIDs\":\"15\"}",
"{\"BlockIO\":\"0B / 0B\",\"CPUPerc\":\"0.20%\",\"Container\":\"6e21c63012d160a406e28a342d668de0be55107aa16228ed9d5dc15c34b447be\",\"ID\":\"6e21c63012d1\",\"MemPerc\":\"27.96%\",\"MemUsage\":\"71.57MiB / 256MiB\",\"Name\":\"mtusi-lab6-otel-collector-1\",\"NetIO\":\"7.81MB / 2.62MB\",\"PIDs\":\"15\"}"
],
"collector_limit_bytes": 268435456,
"oom_killed": false,
"failed_exports_before": [
{
"metric": {
"__name__": "otelcol_exporter_send_failed_log_records_total",
"exporter": "otlphttp/loki",
"instance": "otel-collector:8888",
"job": "collector",
"otel_scope_name": "go.opentelemetry.io/collector/exporter/exporterhelper"
},
"value": [
1795100986.923,
"6"
]
}
],
"failed_exports_after": [
{
"metric": {
"__name__": "otelcol_exporter_send_failed_log_records_total",
"exporter": "otlphttp/loki",
"instance": "otel-collector:8888",
"job": "collector",
"otel_scope_name": "go.opentelemetry.io/collector/exporter/exporterhelper"
},
"value": [
1795101059.133,
"12"
]
}
]
},
"provisioned_dashboard_queries": {
"passed": true,
"panels": 19
}
},
"dashboard": "messenger-lab6",
"passed": true,
"duration_seconds": 85.23
}
+306
View File
@@ -0,0 +1,306 @@
{
"status": "success",
"data": {
"resultType": "streams",
"result": [
{
"stream": {
"attachment_id": "0eb04e8c-8e7b-42a9-ad63-8fca3390d11c",
"deployment_environment_name": "lab",
"detected_level": "INFO",
"event": "attachment.accepted",
"flags": "3",
"job_id": "fd4d7cce-b0dc-4edd-9938-61230afa7370",
"scope_name": "messenger",
"scope_version": "6.0.0",
"service_instance_id": "166e3f521d4f:1",
"service_name": "messenger-api",
"service_version": "6.0.0",
"severity_number": "9",
"severity_text": "INFO",
"span_id": "e04c2c74d09fa066",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1",
"trace_id": "93e1177f9568f5bbfc0b6967a1324a84"
},
"values": [
[
"1795100974767540043",
"{\"timestamp\": \"2026-11-19T15:09:34.767497+00:00\", \"severity\": \"INFO\", \"service.name\": \"messenger-api\", \"event\": \"attachment.accepted\", \"trace_id\": \"93e1177f9568f5bbfc0b6967a1324a84\", \"span_id\": \"e04c2c74d09fa066\", \"job_id\": \"fd4d7cce-b0dc-4edd-9938-61230afa7370\", \"attachment_id\": \"0eb04e8c-8e7b-42a9-ad63-8fca3390d11c\"}"
]
]
},
{
"stream": {
"attempt": "1",
"deployment_environment_name": "lab",
"detected_level": "ERROR",
"error_code": "storage_unavailable",
"event": "job.retry",
"flags": "3",
"job_id": "fd4d7cce-b0dc-4edd-9938-61230afa7370",
"scope_name": "messenger",
"scope_version": "6.0.0",
"service_instance_id": "4a4dd93b6332:8",
"service_name": "messenger-worker",
"service_version": "6.0.0",
"severity_number": "17",
"severity_text": "ERROR",
"span_id": "f9ff281d3dbfaedd",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1",
"trace_id": "93e1177f9568f5bbfc0b6967a1324a84"
},
"values": [
[
"1795100977543203752",
"{\"timestamp\": \"2026-11-19T15:09:37.543167+00:00\", \"severity\": \"ERROR\", \"service.name\": \"messenger-worker\", \"event\": \"job.retry\", \"trace_id\": \"93e1177f9568f5bbfc0b6967a1324a84\", \"span_id\": \"f9ff281d3dbfaedd\", \"job_id\": \"fd4d7cce-b0dc-4edd-9938-61230afa7370\", \"attempt\": 1, \"error_code\": \"storage_unavailable\"}"
]
]
},
{
"stream": {
"attempt": "1",
"deployment_environment_name": "lab",
"detected_level": "ERROR",
"event": "job.finished",
"flags": "3",
"job_id": "fd4d7cce-b0dc-4edd-9938-61230afa7370",
"outcome": "retry",
"scope_name": "messenger",
"scope_version": "6.0.0",
"service_instance_id": "4a4dd93b6332:8",
"service_name": "messenger-worker",
"service_version": "6.0.0",
"severity_number": "17",
"severity_text": "ERROR",
"span_id": "2bb32cd3759889ae",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1",
"trace_id": "93e1177f9568f5bbfc0b6967a1324a84"
},
"values": [
[
"1795100977544205419",
"{\"timestamp\": \"2026-11-19T15:09:37.544179+00:00\", \"severity\": \"ERROR\", \"service.name\": \"messenger-worker\", \"event\": \"job.finished\", \"trace_id\": \"93e1177f9568f5bbfc0b6967a1324a84\", \"span_id\": \"2bb32cd3759889ae\", \"job_id\": \"fd4d7cce-b0dc-4edd-9938-61230afa7370\", \"attempt\": 1, \"outcome\": \"retry\"}"
]
]
},
{
"stream": {
"attempt": "2",
"deployment_environment_name": "lab",
"detected_level": "INFO",
"event": "job.finished",
"flags": "3",
"job_id": "fd4d7cce-b0dc-4edd-9938-61230afa7370",
"outcome": "ready",
"scope_name": "messenger",
"scope_version": "6.0.0",
"service_instance_id": "4a4dd93b6332:9",
"service_name": "messenger-worker",
"service_version": "6.0.0",
"severity_number": "9",
"severity_text": "INFO",
"span_id": "26b0193c185f8761",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1",
"trace_id": "93e1177f9568f5bbfc0b6967a1324a84"
},
"values": [
[
"1795100979222424711",
"{\"timestamp\": \"2026-11-19T15:09:39.222385+00:00\", \"severity\": \"INFO\", \"service.name\": \"messenger-worker\", \"event\": \"job.finished\", \"trace_id\": \"93e1177f9568f5bbfc0b6967a1324a84\", \"span_id\": \"26b0193c185f8761\", \"job_id\": \"fd4d7cce-b0dc-4edd-9938-61230afa7370\", \"attempt\": 2, \"outcome\": \"ready\"}"
]
]
},
{
"stream": {
"deployment_environment_name": "lab",
"detected_level": "INFO",
"duration_seconds": "0.009846",
"event": "http.request",
"flags": "3",
"method": "POST",
"request_id": "422b23e9-c8f9-4810-ad09-1cb99b6c9576",
"route": "/api/v1/chats/{chat_id}/attachments",
"scope_name": "messenger",
"scope_version": "6.0.0",
"service_instance_id": "166e3f521d4f:1",
"service_name": "messenger-api",
"service_version": "6.0.0",
"severity_number": "9",
"severity_text": "INFO",
"span_id": "e04c2c74d09fa066",
"status": "202",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1",
"trace_id": "93e1177f9568f5bbfc0b6967a1324a84"
},
"values": [
[
"1795100974767795501",
"{\"timestamp\": \"2026-11-19T15:09:34.767775+00:00\", \"severity\": \"INFO\", \"service.name\": \"messenger-api\", \"event\": \"http.request\", \"trace_id\": \"93e1177f9568f5bbfc0b6967a1324a84\", \"span_id\": \"e04c2c74d09fa066\", \"method\": \"POST\", \"route\": \"/api/v1/chats/{chat_id}/attachments\", \"status\": 202, \"duration_seconds\": 0.009846, \"request_id\": \"422b23e9-c8f9-4810-ad09-1cb99b6c9576\"}"
]
]
}
],
"stats": {
"summary": {
"bytesProcessedPerSecond": 80485548,
"linesProcessedPerSecond": 110946,
"totalBytesProcessed": 1706250,
"totalLinesProcessed": 2352,
"execTime": 0.021199,
"queueTime": 0.000413,
"subqueries": 0,
"totalEntriesReturned": 5,
"splits": 2,
"shards": 2,
"totalPostFilterLines": 5,
"totalStructuredMetadataBytesProcessed": 853184
},
"querier": {
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": false,
"chunk": {
"headChunkBytes": 0,
"headChunkLines": 0,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0,
"postFilterLines": 0,
"headChunkStructuredMetadataBytes": 0,
"decompressedStructuredMetadataBytes": 0
},
"chunkRefsFetchTime": 0,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0
}
},
"ingester": {
"totalReached": 2,
"totalChunksMatched": 2,
"totalBatches": 3,
"totalLinesSent": 5,
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": false,
"chunk": {
"headChunkBytes": 114091,
"headChunkLines": 228,
"decompressedBytes": 1592159,
"decompressedLines": 2124,
"compressedBytes": 190062,
"totalDuplicates": 0,
"postFilterLines": 5,
"headChunkStructuredMetadataBytes": 34104,
"decompressedStructuredMetadataBytes": 819080
},
"chunkRefsFetchTime": 180667,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0
}
},
"cache": {
"chunk": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"index": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"result": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"statsResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"volumeResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"seriesResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"labelResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"instantMetricResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
}
},
"index": {
"totalChunks": 0,
"postFilterChunks": 0,
"shardsDuration": 0,
"usedBloomFilters": false
}
}
}
}
+100
View File
@@ -0,0 +1,100 @@
{
"passed": true,
"panels": [
{
"panel": "Business API RPS",
"source": "prometheus",
"passed": true
},
{
"panel": "API 5xx / all business requests",
"source": "prometheus",
"passed": true
},
{
"panel": "API latency: aggregated histogram",
"source": "prometheus",
"passed": true
},
{
"panel": "Time from acceptance to ready: completed jobs",
"source": "prometheus",
"passed": true
},
{
"panel": "Job attempts by outcome",
"source": "prometheus",
"passed": true
},
{
"panel": "Worker processing p95",
"source": "prometheus",
"passed": true
},
{
"panel": "Outstanding jobs (DB source of truth)",
"source": "prometheus",
"passed": true
},
{
"panel": "Oldest unfinished job age",
"source": "prometheus",
"passed": true
},
{
"panel": "DB pool used / waiting",
"source": "prometheus",
"passed": true
},
{
"panel": "Worker busy / capacity",
"source": "prometheus",
"passed": true
},
{
"panel": "Process CPU: cores consumed",
"source": "prometheus",
"passed": true
},
{
"panel": "Process RSS",
"source": "prometheus",
"passed": true
},
{
"panel": "API availability SLO: >= 99% / 1h",
"source": "prometheus",
"passed": true
},
{
"panel": "Ready SLO: >= 95% within 30s / 1h",
"source": "prometheus",
"passed": true
},
{
"panel": "API burn rate (budget / hour)",
"source": "prometheus",
"passed": true
},
{
"panel": "Ready burn rate (budget / hour)",
"source": "prometheus",
"passed": true
},
{
"panel": "Local alert states",
"source": "prometheus",
"passed": true
},
{
"panel": "API / worker structured logs (click TraceID)",
"source": "loki",
"passed": true
},
{
"panel": "Recent real worker traces",
"source": "tempo",
"passed": true
}
]
}
+191
View File
@@ -0,0 +1,191 @@
{
"status": "success",
"data": {
"resultType": "streams",
"result": [
{
"stream": {
"deployment_environment_name": "lab",
"detected_level": "ERROR",
"duration_seconds": "2.003678",
"event": "http.request",
"flags": "3",
"method": "POST",
"request_id": "3c5e5b14-5825-41b8-a227-93cba420a532",
"route": "/api/v1/chats/{chat_id}/messages",
"scope_name": "messenger",
"scope_version": "6.0.0",
"service_instance_id": "2a50b728ce10:1",
"service_name": "messenger-api",
"service_version": "6.0.0",
"severity_number": "17",
"severity_text": "ERROR",
"span_id": "3cec0bb07faa77ff",
"status": "503",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1",
"trace_id": "f124d93f84dd25c5af978e78e2028646"
},
"values": [
[
"1795102249376144341",
"{\"timestamp\": \"2026-11-19T15:30:49.376082+00:00\", \"severity\": \"ERROR\", \"service.name\": \"messenger-api\", \"event\": \"http.request\", \"trace_id\": \"f124d93f84dd25c5af978e78e2028646\", \"span_id\": \"3cec0bb07faa77ff\", \"method\": \"POST\", \"route\": \"/api/v1/chats/{chat_id}/messages\", \"status\": 503, \"duration_seconds\": 2.003678, \"request_id\": \"3c5e5b14-5825-41b8-a227-93cba420a532\"}"
]
]
}
],
"stats": {
"summary": {
"bytesProcessedPerSecond": 61790648,
"linesProcessedPerSecond": 82173,
"totalBytesProcessed": 809854,
"totalLinesProcessed": 1077,
"execTime": 0.013106,
"queueTime": 0.000136,
"subqueries": 0,
"totalEntriesReturned": 1,
"splits": 0,
"shards": 1,
"totalPostFilterLines": 1,
"totalStructuredMetadataBytesProcessed": 400756
},
"querier": {
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": false,
"chunk": {
"headChunkBytes": 0,
"headChunkLines": 0,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0,
"postFilterLines": 0,
"headChunkStructuredMetadataBytes": 0,
"decompressedStructuredMetadataBytes": 0
},
"chunkRefsFetchTime": 0,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0
}
},
"ingester": {
"totalReached": 1,
"totalChunksMatched": 1,
"totalBatches": 2,
"totalLinesSent": 1,
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": false,
"chunk": {
"headChunkBytes": 23978,
"headChunkLines": 47,
"decompressedBytes": 785876,
"decompressedLines": 1030,
"compressedBytes": 95894,
"totalDuplicates": 0,
"postFilterLines": 1,
"headChunkStructuredMetadataBytes": 6976,
"decompressedStructuredMetadataBytes": 393780
},
"chunkRefsFetchTime": 72000,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0
}
},
"cache": {
"chunk": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"index": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"result": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"statsResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"volumeResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"seriesResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"labelResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"instantMetricResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
}
},
"index": {
"totalChunks": 0,
"postFilterChunks": 0,
"shardsDuration": 0,
"usedBloomFilters": false
}
}
}
}
+275
View File
@@ -0,0 +1,275 @@
{
"batches": [
{
"resource": {
"attributes": [
{
"key": "deployment.environment.name",
"value": {
"stringValue": "lab"
}
},
{
"key": "service.instance.id",
"value": {
"stringValue": "2a50b728ce10:1"
}
},
{
"key": "service.version",
"value": {
"stringValue": "6.0.0"
}
},
{
"key": "telemetry.sdk.language",
"value": {
"stringValue": "python"
}
},
{
"key": "telemetry.sdk.name",
"value": {
"stringValue": "opentelemetry"
}
},
{
"key": "telemetry.sdk.version",
"value": {
"stringValue": "1.45.1"
}
},
{
"key": "service.name",
"value": {
"stringValue": "messenger-api"
}
}
]
},
"scopeSpans": [
{
"scope": {
"name": "messenger",
"version": "6.0.0"
},
"spans": [
{
"traceId": "8STZP4TdJcWvl4544gKGRg==",
"spanId": "6YhkxV7CY4k=",
"parentSpanId": "zAWgCBMyg3I=",
"name": "postgresql.QUERY",
"kind": "SPAN_KIND_CLIENT",
"startTimeUnixNano": "1795102247372852382",
"endTimeUnixNano": "1795102247372984882",
"attributes": [
{
"key": "db.system.name",
"value": {
"stringValue": "postgresql"
}
},
{
"key": "db.operation.name",
"value": {
"stringValue": "QUERY"
}
}
],
"status": {}
},
{
"traceId": "8STZP4TdJcWvl4544gKGRg==",
"spanId": "9dARsClEhgI=",
"parentSpanId": "zAWgCBMyg3I=",
"name": "postgresql.SELECT",
"kind": "SPAN_KIND_CLIENT",
"startTimeUnixNano": "1795102247373056215",
"endTimeUnixNano": "1795102247373325590",
"attributes": [
{
"key": "db.system.name",
"value": {
"stringValue": "postgresql"
}
},
{
"key": "db.operation.name",
"value": {
"stringValue": "SELECT"
}
}
],
"status": {}
},
{
"traceId": "8STZP4TdJcWvl4544gKGRg==",
"spanId": "zAWgCBMyg3I=",
"parentSpanId": "POwLsH+qd/8=",
"name": "postgresql.transaction",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1795102247372781257",
"endTimeUnixNano": "1795102247373442673",
"status": {}
},
{
"traceId": "8STZP4TdJcWvl4544gKGRg==",
"spanId": "HOilfsT0R4s=",
"parentSpanId": "ZMWryXhMXeM=",
"name": "postgresql.QUERY",
"kind": "SPAN_KIND_CLIENT",
"startTimeUnixNano": "1795102247374040798",
"endTimeUnixNano": "1795102247374149048",
"attributes": [
{
"key": "db.system.name",
"value": {
"stringValue": "postgresql"
}
},
{
"key": "db.operation.name",
"value": {
"stringValue": "QUERY"
}
}
],
"status": {}
},
{
"traceId": "8STZP4TdJcWvl4544gKGRg==",
"spanId": "tNH67JGBzzQ=",
"parentSpanId": "ZMWryXhMXeM=",
"name": "postgresql.SELECT",
"kind": "SPAN_KIND_CLIENT",
"startTimeUnixNano": "1795102247374219298",
"endTimeUnixNano": "1795102247374661423",
"attributes": [
{
"key": "db.system.name",
"value": {
"stringValue": "postgresql"
}
},
{
"key": "db.operation.name",
"value": {
"stringValue": "SELECT"
}
}
],
"status": {}
},
{
"traceId": "8STZP4TdJcWvl4544gKGRg==",
"spanId": "pO76DIPXzps=",
"parentSpanId": "ZMWryXhMXeM=",
"name": "postgresql.SELECT",
"kind": "SPAN_KIND_CLIENT",
"startTimeUnixNano": "1795102247374726007",
"endTimeUnixNano": "1795102247374869840",
"attributes": [
{
"key": "db.system.name",
"value": {
"stringValue": "postgresql"
}
},
{
"key": "db.operation.name",
"value": {
"stringValue": "SELECT"
}
}
],
"status": {}
},
{
"traceId": "8STZP4TdJcWvl4544gKGRg==",
"spanId": "bxfNsCS1EJs=",
"parentSpanId": "ZMWryXhMXeM=",
"name": "postgresql.SELECT",
"kind": "SPAN_KIND_CLIENT",
"startTimeUnixNano": "1795102247374927173",
"endTimeUnixNano": "1795102249375042508",
"attributes": [
{
"key": "db.system.name",
"value": {
"stringValue": "postgresql"
}
},
{
"key": "db.operation.name",
"value": {
"stringValue": "SELECT"
}
},
{
"key": "error.type",
"value": {
"stringValue": "_WaitTimeout"
}
}
],
"status": {
"code": "STATUS_CODE_ERROR"
}
},
{
"traceId": "8STZP4TdJcWvl4544gKGRg==",
"spanId": "ZMWryXhMXeM=",
"parentSpanId": "POwLsH+qd/8=",
"name": "postgresql.transaction",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1795102247373987923",
"endTimeUnixNano": "1795102249375157091",
"attributes": [
{
"key": "error.type",
"value": {
"stringValue": "_WaitTimeout"
}
}
],
"status": {
"code": "STATUS_CODE_ERROR"
}
},
{
"traceId": "8STZP4TdJcWvl4544gKGRg==",
"spanId": "POwLsH+qd/8=",
"name": "POST /api/v1/chats/{chat_id}/messages",
"kind": "SPAN_KIND_SERVER",
"startTimeUnixNano": "1795102247372424840",
"endTimeUnixNano": "1795102249376209799",
"attributes": [
{
"key": "http.request.method",
"value": {
"stringValue": "POST"
}
},
{
"key": "http.route",
"value": {
"stringValue": "/api/v1/chats/{chat_id}/messages"
}
},
{
"key": "http.response.status_code",
"value": {
"intValue": "503"
}
}
],
"status": {
"code": "STATUS_CODE_ERROR"
}
}
]
}
]
}
]
}
+795
View File
@@ -0,0 +1,795 @@
{
"utc": "2026-11-19T15:09:44.028971+00:00",
"command": "./scripts/check-docker.sh",
"project": "lab5-check-c81d43a829",
"host": "Darwin arm64",
"source_commit": "470d9d6f1e4fe4f594e37c2ba59550ee5ca54fc1",
"working_tree": true,
"configuration": {
"concurrency": 2,
"prefetch": 1,
"attempts": 5,
"timeout_seconds": 5,
"lease_seconds": 15,
"backoff_base_seconds": 0.15
},
"checks": {
"topology_and_limits": {
"passed": true,
"api_replicas": 2,
"broker": "Valkey Streams with AOF fsync always",
"worker_processes": 2
},
"public_smoke_and_schema": {
"passed": true
},
"student_api_regressions": {
"passed": true
},
"worker_offline_api_broker_restart": {
"passed": true,
"attachment_id": "ef3471d3-538e-4d3f-8406-edfb2e78d803",
"job_id": "29387d95-7e9f-4d86-b645-024cb36049cf",
"message_id": "4ae76a83-fddf-4d6a-aa4c-1dffe8910540",
"variants": {
"thumbnail": {
"size": [
256,
160
],
"bytes": 5813
},
"crop": {
"size": [
128,
128
],
"bytes": 2462
},
"optimized": {
"size": [
320,
200
],
"bytes": 2462
}
}
},
"outbox_crash_before_publish": {
"passed": true,
"job_id": "630dde85-21c6-4a82-bcee-5ea7aafa21b8",
"automatic_recovery": true
},
"outbox_crash_after_publish": {
"passed": true,
"job_id": "5c9d0c57-07b5-41d9-951e-d07ed3ce2126",
"automatic_recovery": true
},
"worker_crash_after_objects": {
"passed": true,
"job_id": "e7e67187-45e2-49ad-b845-7c80952abaed",
"attempt": 2,
"canonical_objects": 4,
"duplicate_noop": true
},
"worker_crash_before_ack": {
"passed": true,
"job_id": "646cabc2-c06f-43a9-98ed-97a0ed430eef",
"attempt": 1,
"canonical_objects": 4,
"duplicate_noop": true
},
"bounded_retry_failed_registry_replay": {
"passed": true,
"job_id": "9f0223e7-c7f7-41ce-a6b6-a6b2a6827db7",
"max_automatic_attempts": 5,
"audit": [
{
"event": "processing",
"attempt": 1,
"error_code": null,
"at": "2026-11-19T15:11:43.910272+00:00"
},
{
"event": "retry",
"attempt": 1,
"error_code": "storage_unavailable",
"at": "2026-11-19T15:11:43.913931+00:00"
},
{
"event": "processing",
"attempt": 2,
"error_code": null,
"at": "2026-11-19T15:11:44.162093+00:00"
},
{
"event": "retry",
"attempt": 2,
"error_code": "storage_unavailable",
"at": "2026-11-19T15:11:44.163784+00:00"
},
{
"event": "processing",
"attempt": 3,
"error_code": null,
"at": "2026-11-19T15:11:44.672423+00:00"
},
{
"event": "retry",
"attempt": 3,
"error_code": "storage_unavailable",
"at": "2026-11-19T15:11:44.674937+00:00"
},
{
"event": "processing",
"attempt": 4,
"error_code": null,
"at": "2026-11-19T15:11:45.176057+00:00"
},
{
"event": "retry",
"attempt": 4,
"error_code": "storage_unavailable",
"at": "2026-11-19T15:11:45.17712+00:00"
},
{
"event": "processing",
"attempt": 5,
"error_code": null,
"at": "2026-11-19T15:11:45.941529+00:00"
},
{
"event": "failed",
"attempt": 5,
"error_code": "retry_exhausted",
"at": "2026-11-19T15:11:45.944804+00:00"
},
{
"event": "operator_replay",
"attempt": 5,
"error_code": null,
"at": "2026-11-19T15:11:47.029399+00:00"
},
{
"event": "processing",
"attempt": 6,
"error_code": null,
"at": "2026-11-19T15:11:47.203814+00:00"
},
{
"event": "ready",
"attempt": 6,
"error_code": null,
"at": "2026-11-19T15:11:47.601272+00:00"
}
],
"ready_replay_rejected": true
},
"permanent_error": {
"passed": true,
"job_id": "28022e30-0cd6-4df1-a41b-56905cf4feda",
"attempt": 1,
"error_code": "invalid_image"
},
"image_subprocess_timeout": {
"passed": true,
"job_id": "c0f7e1b2-59a1-496e-a60a-8ed3fdb19f59",
"timeout_seconds": 5,
"task_memory_mb": 640,
"recovered": true
},
"image_subprocess_memory": {
"passed": true,
"job_id": "f1e4caee-86d7-4185-9787-92687c0650a3",
"timeout_seconds": 5,
"task_memory_mb": 640,
"recovered": true
},
"image_semantics": {
"passed": true,
"cases": [
{
"case": "small_alpha_on_white",
"variants": {
"thumbnail": {
"size": [
17,
9
],
"bytes": 635
},
"crop": {
"size": [
128,
128
],
"bytes": 883
},
"optimized": {
"size": [
17,
9
],
"bytes": 56
}
}
},
{
"case": "exif_orientation_6_and_floor",
"variants": {
"thumbnail": {
"size": [
160,
256
],
"bytes": 1270
},
"crop": {
"size": [
128,
128
],
"bytes": 886
},
"optimized": {
"size": [
201,
321
],
"bytes": 210
}
}
}
]
},
"staging_cleanup": {
"passed": true,
"active_preserved": true,
"orphan_removed": true,
"ready_removed": true,
"expired_failed_removed": true,
"expired_replay_rejected": true
},
"broker_offline_outbox": {
"passed": true,
"attachment_id": "28321e5b-8b43-471a-973d-9b55cbc07b39",
"recovered": true
},
"public_persistence": {
"passed": true,
"restart_proof": {
"passed": true,
"before": {
"ea2b13aebb92a1ea2741de72078489e6b05ebcc41f1ca74a2604c0ad7d355983": {
"service": "api",
"started_at": "2026-11-19T15:10:30.841890222Z",
"volumes": []
},
"67eade4e281a4cfed3f55d87ab398097e12ee12155f82cbe19eb240a4e92b41a": {
"service": "api",
"started_at": "2026-11-19T15:10:30.809775805Z",
"volumes": []
},
"d675c92d195a1bb37db5860c3cf49a2a918eb3e98cfd9c855f8f89931138724b": {
"service": "broker",
"started_at": "2026-11-19T15:12:24.444290385Z",
"volumes": [
"lab5-check-c81d43a829_broker_data"
]
},
"f9b2e0b6e448dda6d845d9715d64e1216fc0f3f58c3e05ec22de30d305c2d3e6": {
"service": "cleanup",
"started_at": "2026-11-19T15:10:11.903050713Z",
"volumes": []
},
"e4abde6d7d326be2a1425d81667ab967dd655ce1eb39c1b0c1bd2f9956f3e91f": {
"service": "grafana",
"started_at": "2026-11-19T15:09:46.76721934Z",
"volumes": [
"lab5-check-c81d43a829_grafana_data"
]
},
"1e9b604ee9fe91572a4efd3b629a00819d86ab2b45ed1f04a1df66428cfe1920": {
"service": "lb",
"started_at": "2026-11-19T15:10:17.66400909Z",
"volumes": []
},
"487c1937cced6d3b86d6a67cae2b76fdfc2b534e6cf15b6cf2242b96b19c8af4": {
"service": "loki",
"started_at": "2026-11-19T15:09:46.088598506Z",
"volumes": [
"lab5-check-c81d43a829_loki_data"
]
},
"f31970f75546cfcf6c390fad538eb9a7ebc25d26c481caef09bf679a5af8e8d9": {
"service": "otel-collector",
"started_at": "2026-11-19T15:09:46.409972798Z",
"volumes": []
},
"82576012c7b7b5a5a352207679f4c327fe50cbd5c56337de33c16ec15b2b0501": {
"service": "postgres",
"started_at": "2026-11-19T15:09:46.08945334Z",
"volumes": [
"lab5-check-c81d43a829_postgres_data"
]
},
"6e305e5121dc3113f906c5689d1006091714d72b452acaabd07e3d366f1d13f6": {
"service": "prometheus",
"started_at": "2026-11-19T15:09:46.555234256Z",
"volumes": [
"lab5-check-c81d43a829_prometheus_data"
]
},
"fb35c350ab7fcff516b44e4f6933b1e293c896e6cee9bf29d06e71ba99a0f931": {
"service": "publisher",
"started_at": "2026-11-19T15:12:10.913909129Z",
"volumes": []
},
"8cdcd8bdac73e3bc827ef7bc5a77bad0696c1023cd330eb10206dcc6b29529cf": {
"service": "s3",
"started_at": "2026-11-19T15:10:08.720162878Z",
"volumes": [
"d6e81fdf6ec0a4bd30cb0461085c08461b951f8ae9c696222d04b21152ff6560"
]
},
"053f3a4b4e5cd9b28510f08e9bd8d07374c586af075cb62d5c5b49f62af5a6ac": {
"service": "s3-storage",
"started_at": "2026-11-19T15:09:46.086474548Z",
"volumes": [
"lab5-check-c81d43a829_s3_data"
]
},
"84473a25dbb67b410a0db8d8b0e029b84b264b3238da1961c7693f13d9e99634": {
"service": "tempo",
"started_at": "2026-11-19T15:09:46.086451215Z",
"volumes": [
"lab5-check-c81d43a829_tempo_data"
]
},
"7f3ff970cf57bedf3a8800a6bed5cc3a83f749dab07d9ac42425d14c7eafee80": {
"service": "valkey",
"started_at": "2026-11-19T15:09:46.08714634Z",
"volumes": [
"lab5-check-c81d43a829_valkey_data"
]
},
"521131dedb68623718c9d5d5c8dc8b0924dbfe6ec90acd3b21e08189a9a1a1e5": {
"service": "worker",
"started_at": "2026-11-19T15:12:21.892301301Z",
"volumes": []
}
},
"after": {
"ea2b13aebb92a1ea2741de72078489e6b05ebcc41f1ca74a2604c0ad7d355983": {
"service": "api",
"started_at": "2026-11-19T15:12:30.097473346Z",
"volumes": []
},
"67eade4e281a4cfed3f55d87ab398097e12ee12155f82cbe19eb240a4e92b41a": {
"service": "api",
"started_at": "2026-11-19T15:12:30.048159138Z",
"volumes": []
},
"d675c92d195a1bb37db5860c3cf49a2a918eb3e98cfd9c855f8f89931138724b": {
"service": "broker",
"started_at": "2026-11-19T15:12:28.168864679Z",
"volumes": [
"lab5-check-c81d43a829_broker_data"
]
},
"f9b2e0b6e448dda6d845d9715d64e1216fc0f3f58c3e05ec22de30d305c2d3e6": {
"service": "cleanup",
"started_at": "2026-11-19T15:12:35.75098096Z",
"volumes": []
},
"e4abde6d7d326be2a1425d81667ab967dd655ce1eb39c1b0c1bd2f9956f3e91f": {
"service": "grafana",
"started_at": "2026-11-19T15:12:28.016722095Z",
"volumes": [
"lab5-check-c81d43a829_grafana_data"
]
},
"1e9b604ee9fe91572a4efd3b629a00819d86ab2b45ed1f04a1df66428cfe1920": {
"service": "lb",
"started_at": "2026-11-19T15:12:27.799505637Z",
"volumes": []
},
"487c1937cced6d3b86d6a67cae2b76fdfc2b534e6cf15b6cf2242b96b19c8af4": {
"service": "loki",
"started_at": "2026-11-19T15:12:29.867573388Z",
"volumes": [
"lab5-check-c81d43a829_loki_data"
]
},
"f31970f75546cfcf6c390fad538eb9a7ebc25d26c481caef09bf679a5af8e8d9": {
"service": "otel-collector",
"started_at": "2026-11-19T15:12:29.704735929Z",
"volumes": []
},
"82576012c7b7b5a5a352207679f4c327fe50cbd5c56337de33c16ec15b2b0501": {
"service": "postgres",
"started_at": "2026-11-19T15:12:28.084774804Z",
"volumes": [
"lab5-check-c81d43a829_postgres_data"
]
},
"6e305e5121dc3113f906c5689d1006091714d72b452acaabd07e3d366f1d13f6": {
"service": "prometheus",
"started_at": "2026-11-19T15:12:28.052574179Z",
"volumes": [
"lab5-check-c81d43a829_prometheus_data"
]
},
"fb35c350ab7fcff516b44e4f6933b1e293c896e6cee9bf29d06e71ba99a0f931": {
"service": "publisher",
"started_at": "2026-11-19T15:12:35.770825668Z",
"volumes": []
},
"8cdcd8bdac73e3bc827ef7bc5a77bad0696c1023cd330eb10206dcc6b29529cf": {
"service": "s3",
"started_at": "2026-11-19T15:12:28.115603887Z",
"volumes": [
"d6e81fdf6ec0a4bd30cb0461085c08461b951f8ae9c696222d04b21152ff6560"
]
},
"053f3a4b4e5cd9b28510f08e9bd8d07374c586af075cb62d5c5b49f62af5a6ac": {
"service": "s3-storage",
"started_at": "2026-11-19T15:12:35.714312418Z",
"volumes": [
"lab5-check-c81d43a829_s3_data"
]
},
"84473a25dbb67b410a0db8d8b0e029b84b264b3238da1961c7693f13d9e99634": {
"service": "tempo",
"started_at": "2026-11-19T15:12:28.142091095Z",
"volumes": [
"lab5-check-c81d43a829_tempo_data"
]
},
"7f3ff970cf57bedf3a8800a6bed5cc3a83f749dab07d9ac42425d14c7eafee80": {
"service": "valkey",
"started_at": "2026-11-19T15:12:28.230646637Z",
"volumes": [
"lab5-check-c81d43a829_valkey_data"
]
},
"521131dedb68623718c9d5d5c8dc8b0924dbfe6ec90acd3b21e08189a9a1a1e5": {
"service": "worker",
"started_at": "2026-11-19T15:12:27.946355887Z",
"volumes": []
}
},
"same_containers_and_volumes": true
}
},
"public_api_failover": {
"passed": true
}
},
"docker": "29.0.1 aarch64 CPUs=10 RAM=8217448448",
"containers": [
{
"id": "ea2b13aebb92a1ea2741de72078489e6b05ebcc41f1ca74a2604c0ad7d355983",
"service": "api",
"image": "mtusi-messenger:lab6",
"memory_bytes": 402653184,
"nano_cpus": 1000000000,
"mounts": [
{
"destination": "/run/secrets/database_url",
"name": null
},
{
"destination": "/run/secrets/cache_url",
"name": null
},
{
"destination": "/run/secrets/s3_access_key_id",
"name": null
},
{
"destination": "/run/secrets/s3_secret_access_key",
"name": null
}
]
},
{
"id": "67eade4e281a4cfed3f55d87ab398097e12ee12155f82cbe19eb240a4e92b41a",
"service": "api",
"image": "mtusi-messenger:lab6",
"memory_bytes": 402653184,
"nano_cpus": 1000000000,
"mounts": [
{
"destination": "/run/secrets/cache_url",
"name": null
},
{
"destination": "/run/secrets/s3_access_key_id",
"name": null
},
{
"destination": "/run/secrets/s3_secret_access_key",
"name": null
},
{
"destination": "/run/secrets/database_url",
"name": null
}
]
},
{
"id": "d675c92d195a1bb37db5860c3cf49a2a918eb3e98cfd9c855f8f89931138724b",
"service": "broker",
"image": "valkey/valkey:8.1.3",
"memory_bytes": 268435456,
"nano_cpus": 500000000,
"mounts": [
{
"destination": "/data",
"name": "lab5-check-c81d43a829_broker_data"
},
{
"destination": "/run/secrets/broker_password",
"name": null
}
]
},
{
"id": "f9b2e0b6e448dda6d845d9715d64e1216fc0f3f58c3e05ec22de30d305c2d3e6",
"service": "cleanup",
"image": "mtusi-messenger:lab6",
"memory_bytes": 402653184,
"nano_cpus": 1000000000,
"mounts": [
{
"destination": "/run/secrets/s3_access_key_id",
"name": null
},
{
"destination": "/run/secrets/s3_secret_access_key",
"name": null
},
{
"destination": "/run/secrets/database_url",
"name": null
}
]
},
{
"id": "e4abde6d7d326be2a1425d81667ab967dd655ce1eb39c1b0c1bd2f9956f3e91f",
"service": "grafana",
"image": "grafana/grafana:12.0.2",
"memory_bytes": 402653184,
"nano_cpus": 1000000000,
"mounts": [
{
"destination": "/etc/grafana/provisioning",
"name": null
},
{
"destination": "/var/lib/grafana/dashboards",
"name": null
},
{
"destination": "/var/lib/grafana",
"name": "lab5-check-c81d43a829_grafana_data"
}
]
},
{
"id": "1e9b604ee9fe91572a4efd3b629a00819d86ab2b45ed1f04a1df66428cfe1920",
"service": "lb",
"image": "haproxy:3.2.6-alpine",
"memory_bytes": 67108864,
"nano_cpus": 500000000,
"mounts": [
{
"destination": "/usr/local/etc/haproxy/haproxy.cfg",
"name": null
}
]
},
{
"id": "487c1937cced6d3b86d6a67cae2b76fdfc2b534e6cf15b6cf2242b96b19c8af4",
"service": "loki",
"image": "grafana/loki:3.5.1",
"memory_bytes": 402653184,
"nano_cpus": 1000000000,
"mounts": [
{
"destination": "/etc/loki/config.yaml",
"name": null
},
{
"destination": "/loki",
"name": "lab5-check-c81d43a829_loki_data"
}
]
},
{
"id": "02572452b7c401e702c6ad13daac234d9e3c069bb791e44770a7f7fc11dca885",
"service": "migrate",
"image": "mtusi-messenger:lab6",
"memory_bytes": 402653184,
"nano_cpus": 1000000000,
"mounts": [
{
"destination": "/run/secrets/database_url",
"name": null
}
]
},
{
"id": "f31970f75546cfcf6c390fad538eb9a7ebc25d26c481caef09bf679a5af8e8d9",
"service": "otel-collector",
"image": "otel/opentelemetry-collector-contrib:0.123.0",
"memory_bytes": 268435456,
"nano_cpus": 1000000000,
"mounts": [
{
"destination": "/etc/otelcol/config.yaml",
"name": null
}
]
},
{
"id": "82576012c7b7b5a5a352207679f4c327fe50cbd5c56337de33c16ec15b2b0501",
"service": "postgres",
"image": "postgres:17.6-bookworm",
"memory_bytes": 536870912,
"nano_cpus": 1000000000,
"mounts": [
{
"destination": "/run/secrets/postgres_password",
"name": null
},
{
"destination": "/var/lib/postgresql/data",
"name": "lab5-check-c81d43a829_postgres_data"
}
]
},
{
"id": "6e305e5121dc3113f906c5689d1006091714d72b452acaabd07e3d366f1d13f6",
"service": "prometheus",
"image": "prom/prometheus:v3.4.1",
"memory_bytes": 402653184,
"nano_cpus": 1000000000,
"mounts": [
{
"destination": "/prometheus",
"name": "lab5-check-c81d43a829_prometheus_data"
},
{
"destination": "/etc/prometheus/prometheus.yaml",
"name": null
},
{
"destination": "/etc/prometheus/alerts.yaml",
"name": null
}
]
},
{
"id": "fb35c350ab7fcff516b44e4f6933b1e293c896e6cee9bf29d06e71ba99a0f931",
"service": "publisher",
"image": "mtusi-messenger:lab6",
"memory_bytes": 402653184,
"nano_cpus": 1000000000,
"mounts": [
{
"destination": "/run/secrets/database_url",
"name": null
},
{
"destination": "/run/secrets/broker_url",
"name": null
}
]
},
{
"id": "8cdcd8bdac73e3bc827ef7bc5a77bad0696c1023cd330eb10206dcc6b29529cf",
"service": "s3",
"image": "chrislusf/seaweedfs:3.85@sha256:49312939c00c01e5ee6afbd7d728b18027821d3764c35a797a72acd4fdf3296a",
"memory_bytes": 268435456,
"nano_cpus": 500000000,
"mounts": [
{
"destination": "/run/secrets/s3_config",
"name": null
},
{
"destination": "/data",
"name": "d6e81fdf6ec0a4bd30cb0461085c08461b951f8ae9c696222d04b21152ff6560"
}
]
},
{
"id": "e491085d55967c2cc9a4c907385fa87d8ee2e7d072ae77082a39ffde575cf0e8",
"service": "s3-bootstrap",
"image": "mtusi-messenger:lab6",
"memory_bytes": 134217728,
"nano_cpus": 500000000,
"mounts": [
{
"destination": "/run/secrets/s3_root_user",
"name": null
},
{
"destination": "/run/secrets/s3_root_password",
"name": null
}
]
},
{
"id": "053f3a4b4e5cd9b28510f08e9bd8d07374c586af075cb62d5c5b49f62af5a6ac",
"service": "s3-storage",
"image": "chrislusf/seaweedfs:3.85@sha256:49312939c00c01e5ee6afbd7d728b18027821d3764c35a797a72acd4fdf3296a",
"memory_bytes": 536870912,
"nano_cpus": 1000000000,
"mounts": [
{
"destination": "/data",
"name": "lab5-check-c81d43a829_s3_data"
}
]
},
{
"id": "84473a25dbb67b410a0db8d8b0e029b84b264b3238da1961c7693f13d9e99634",
"service": "tempo",
"image": "grafana/tempo:2.8.1",
"memory_bytes": 536870912,
"nano_cpus": 1000000000,
"mounts": [
{
"destination": "/etc/tempo/config.yaml",
"name": null
},
{
"destination": "/var/tempo",
"name": "lab5-check-c81d43a829_tempo_data"
}
]
},
{
"id": "7f3ff970cf57bedf3a8800a6bed5cc3a83f749dab07d9ac42425d14c7eafee80",
"service": "valkey",
"image": "valkey/valkey:8.1.3",
"memory_bytes": 134217728,
"nano_cpus": 500000000,
"mounts": [
{
"destination": "/data",
"name": "lab5-check-c81d43a829_valkey_data"
}
]
},
{
"id": "521131dedb68623718c9d5d5c8dc8b0924dbfe6ec90acd3b21e08189a9a1a1e5",
"service": "worker",
"image": "mtusi-messenger:lab6",
"memory_bytes": 1610612736,
"nano_cpus": 2000000000,
"mounts": [
{
"destination": "/run/secrets/s3_secret_access_key",
"name": null
},
{
"destination": "/run/secrets/database_url",
"name": null
},
{
"destination": "/run/secrets/broker_url",
"name": null
},
{
"destination": "/run/secrets/s3_access_key_id",
"name": null
}
]
}
],
"passed": true
}
File diff suppressed because one or more lines are too long
+1
View File
@@ -0,0 +1 @@
SUCCESS
+206
View File
@@ -0,0 +1,206 @@
{
"utc": "2026-11-19T15:11:37.994748+00:00",
"command": ".venv/bin/python student-tests/verify_restart.py",
"passed": true,
"before": {
"166e3f521d4f8b2206766859d89df193d0ebc44fb7b8c2d597da65ebefaaf1af": {
"service": "api",
"started_at": "2026-11-19T15:08:38.453676544Z",
"volumes": []
},
"863415a50886df6498a2c5d137298288e403eb8f628fd93088a93174bb744d8c": {
"service": "api",
"started_at": "2026-11-19T15:08:38.611590211Z",
"volumes": []
},
"ea51e5f3e0e7921a34c8678a17da799c5f440f0423fc5cc0d0ebc88b6be109b8": {
"service": "broker",
"started_at": "2026-11-19T15:00:33.36494925Z",
"volumes": [
"mtusi-lab6_broker_data"
]
},
"d24bddbf342d9bc0f3b43d8462e7d991d82e38c848d4ef762a8b0b1cbb5959ba": {
"service": "cleanup",
"started_at": "2026-11-19T15:08:38.446185961Z",
"volumes": []
},
"375a3083ca7919b26cdc327adf969072cd09d91f3dc664037b63fb890c91849a": {
"service": "grafana",
"started_at": "2026-11-19T15:00:33.781731Z",
"volumes": [
"mtusi-lab6_grafana_data"
]
},
"f56ed57943f2889033e15612d17d633e0b7bb3a9cceb9cb2ab336524117e6ffa": {
"service": "lb",
"started_at": "2026-11-19T15:00:58.904707679Z",
"volumes": []
},
"e107e6ede3c3b2be8a6995ef242f614dc6f592cf230f10d7862685696991ae71": {
"service": "loki",
"started_at": "2026-11-19T15:10:28.625123846Z",
"volumes": [
"mtusi-lab6_loki_data"
]
},
"6e21c63012d160a406e28a342d668de0be55107aa16228ed9d5dc15c34b447be": {
"service": "otel-collector",
"started_at": "2026-11-19T15:01:52.896095343Z",
"volumes": []
},
"1b3cdbc4689c6b855a435acab8112f86a3c9dc5e0207257c31e7aaa650635a0a": {
"service": "postgres",
"started_at": "2026-11-19T15:00:33.362963334Z",
"volumes": [
"mtusi-lab6_postgres_data"
]
},
"d54da34ab176f240c2519bb6fcc46417b5ea3d27de86e2ed05321675f14e1e23": {
"service": "prometheus",
"started_at": "2026-11-19T15:00:33.65938775Z",
"volumes": [
"mtusi-lab6_prometheus_data"
]
},
"0cd7cf668f46d355febf0dd912003289a65966692b21a759997e0a8d1c5d83eb": {
"service": "publisher",
"started_at": "2026-11-19T15:08:38.448713669Z",
"volumes": []
},
"b7b3fe9f4f328f57dc0df0a3af84d8426e431505ffd236da9d6cc143123380c9": {
"service": "s3",
"started_at": "2026-11-19T15:00:49.997656091Z",
"volumes": [
"a6b780554a2ebb58edb3e2f7b77fe7ee705ce9543ef0dbe0c8d9c4d80bc31da2"
]
},
"1b69e90a9f7108e2e5d2017365e82fa30495c42f5f93809d42b2fdfcc1f0a9a0": {
"service": "s3-storage",
"started_at": "2026-11-19T15:00:33.3498635Z",
"volumes": [
"mtusi-lab6_s3_data"
]
},
"7dc0833e289e6e708dff042076bf4753617ba5e61122a9a67dd61c37384f1c9b": {
"service": "tempo",
"started_at": "2026-11-19T15:08:38.454968128Z",
"volumes": [
"mtusi-lab6_tempo_data"
]
},
"f4c736f41e428fb83e0effb49e8a56d9efe23ea6bd979a16fdcf072af0b342b0": {
"service": "valkey",
"started_at": "2026-11-19T15:00:33.364171167Z",
"volumes": [
"mtusi-lab6_valkey_data"
]
},
"ef1c4fbd5187a8773b7b448cb5f0b0534e83bddb9b12a51f2ffcd7a326e5e729": {
"service": "worker",
"started_at": "2026-11-19T15:10:59.920254179Z",
"volumes": []
}
},
"after": {
"166e3f521d4f8b2206766859d89df193d0ebc44fb7b8c2d597da65ebefaaf1af": {
"service": "api",
"started_at": "2026-11-19T15:11:41.117466337Z",
"volumes": []
},
"863415a50886df6498a2c5d137298288e403eb8f628fd93088a93174bb744d8c": {
"service": "api",
"started_at": "2026-11-19T15:11:41.169749671Z",
"volumes": []
},
"ea51e5f3e0e7921a34c8678a17da799c5f440f0423fc5cc0d0ebc88b6be109b8": {
"service": "broker",
"started_at": "2026-11-19T15:11:40.171899503Z",
"volumes": [
"mtusi-lab6_broker_data"
]
},
"d24bddbf342d9bc0f3b43d8462e7d991d82e38c848d4ef762a8b0b1cbb5959ba": {
"service": "cleanup",
"started_at": "2026-11-19T15:11:47.626797257Z",
"volumes": []
},
"375a3083ca7919b26cdc327adf969072cd09d91f3dc664037b63fb890c91849a": {
"service": "grafana",
"started_at": "2026-11-19T15:11:39.799567837Z",
"volumes": [
"mtusi-lab6_grafana_data"
]
},
"f56ed57943f2889033e15612d17d633e0b7bb3a9cceb9cb2ab336524117e6ffa": {
"service": "lb",
"started_at": "2026-11-19T15:11:39.674464462Z",
"volumes": []
},
"e107e6ede3c3b2be8a6995ef242f614dc6f592cf230f10d7862685696991ae71": {
"service": "loki",
"started_at": "2026-11-19T15:11:40.282189378Z",
"volumes": [
"mtusi-lab6_loki_data"
]
},
"6e21c63012d160a406e28a342d668de0be55107aa16228ed9d5dc15c34b447be": {
"service": "otel-collector",
"started_at": "2026-11-19T15:11:39.989740337Z",
"volumes": []
},
"1b3cdbc4689c6b855a435acab8112f86a3c9dc5e0207257c31e7aaa650635a0a": {
"service": "postgres",
"started_at": "2026-11-19T15:11:40.043447628Z",
"volumes": [
"mtusi-lab6_postgres_data"
]
},
"d54da34ab176f240c2519bb6fcc46417b5ea3d27de86e2ed05321675f14e1e23": {
"service": "prometheus",
"started_at": "2026-11-19T15:11:39.859058378Z",
"volumes": [
"mtusi-lab6_prometheus_data"
]
},
"0cd7cf668f46d355febf0dd912003289a65966692b21a759997e0a8d1c5d83eb": {
"service": "publisher",
"started_at": "2026-11-19T15:11:47.594100007Z",
"volumes": []
},
"b7b3fe9f4f328f57dc0df0a3af84d8426e431505ffd236da9d6cc143123380c9": {
"service": "s3",
"started_at": "2026-11-19T15:11:40.13670567Z",
"volumes": [
"a6b780554a2ebb58edb3e2f7b77fe7ee705ce9543ef0dbe0c8d9c4d80bc31da2"
]
},
"1b69e90a9f7108e2e5d2017365e82fa30495c42f5f93809d42b2fdfcc1f0a9a0": {
"service": "s3-storage",
"started_at": "2026-11-19T15:11:47.65993734Z",
"volumes": [
"mtusi-lab6_s3_data"
]
},
"7dc0833e289e6e708dff042076bf4753617ba5e61122a9a67dd61c37384f1c9b": {
"service": "tempo",
"started_at": "2026-11-19T15:11:40.243825753Z",
"volumes": [
"mtusi-lab6_tempo_data"
]
},
"f4c736f41e428fb83e0effb49e8a56d9efe23ea6bd979a16fdcf072af0b342b0": {
"service": "valkey",
"started_at": "2026-11-19T15:11:40.087174462Z",
"volumes": [
"mtusi-lab6_valkey_data"
]
},
"ef1c4fbd5187a8773b7b448cb5f0b0534e83bddb9b12a51f2ffcd7a326e5e729": {
"service": "worker",
"started_at": "2026-11-19T15:11:40.21509467Z",
"volumes": []
}
},
"same_containers_and_volumes": true
}
+4
View File
@@ -0,0 +1,4 @@
/Users/vanek/Desktop/VNPlabs/VNPlab6/mtusi2026lab6/.venv/bin/python tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:8080"
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
PASS: all persistent services actually restarted; container IDs and volumes preserved
+322
View File
@@ -0,0 +1,322 @@
{
"passed": true,
"services": [
{
"service": "api",
"container_id": "33daefd247cfa9bdb99c0e9d10913c5305de5db293431162467a9fb0f55f3e4c",
"image": "mtusi-messenger:lab6",
"image_id": "sha256:a7932cb2a948db89797bf90f560ecf0046cac576c7fff97412aa55d8c4446a62",
"state": "running",
"health": "healthy",
"memory_limit_bytes": 402653184,
"nano_cpus": 1000000000,
"oom_killed": false,
"published_ports": {},
"volumes": []
},
{
"service": "api",
"container_id": "2a50b728ce1094a536622ed51528f1e359b5bbd51685bb3b368dfb9327a2f9d8",
"image": "mtusi-messenger:lab6",
"image_id": "sha256:a7932cb2a948db89797bf90f560ecf0046cac576c7fff97412aa55d8c4446a62",
"state": "running",
"health": "healthy",
"memory_limit_bytes": 402653184,
"nano_cpus": 1000000000,
"oom_killed": false,
"published_ports": {},
"volumes": []
},
{
"service": "broker",
"container_id": "ea51e5f3e0e7921a34c8678a17da799c5f440f0423fc5cc0d0ebc88b6be109b8",
"image": "valkey/valkey:8.1.3",
"image_id": "sha256:fea8b3e67b15729d4bb70589eb03367bab9ad1ee89c876f54327fc7c6e618571",
"state": "running",
"health": "healthy",
"memory_limit_bytes": 268435456,
"nano_cpus": 500000000,
"oom_killed": false,
"published_ports": {},
"volumes": [
"mtusi-lab6_broker_data"
]
},
{
"service": "cleanup",
"container_id": "e3328a92cd7b264a34d3ba2863ebd509ac105e68d507ce39062181cbe34e91bc",
"image": "mtusi-messenger:lab6",
"image_id": "sha256:a7932cb2a948db89797bf90f560ecf0046cac576c7fff97412aa55d8c4446a62",
"state": "running",
"health": null,
"memory_limit_bytes": 402653184,
"nano_cpus": 1000000000,
"oom_killed": false,
"published_ports": {},
"volumes": []
},
{
"service": "grafana",
"container_id": "9d373574ca2ee05f2e55ffc54a58266bfe13ac1c12d46c5275355a716fa8bd70",
"image": "grafana/grafana:12.0.2",
"image_id": "sha256:b5b59bfc7561634c2d7b136c4543d702ebcc94a3da477f21ff26f89ffd4214fa",
"state": "running",
"health": null,
"memory_limit_bytes": 402653184,
"nano_cpus": 1000000000,
"oom_killed": false,
"published_ports": {
"3000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "3000"
}
]
},
"volumes": [
"mtusi-lab6_grafana_data"
]
},
{
"service": "lb",
"container_id": "f56ed57943f2889033e15612d17d633e0b7bb3a9cceb9cb2ab336524117e6ffa",
"image": "haproxy:3.2.6-alpine",
"image_id": "sha256:3a819392f5a6af2a9203d1c63ff0feba74f0c940623a558511be06d7d85e361c",
"state": "running",
"health": "healthy",
"memory_limit_bytes": 67108864,
"nano_cpus": 500000000,
"oom_killed": false,
"published_ports": {
"8080/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "8080"
}
]
},
"volumes": []
},
{
"service": "loki",
"container_id": "e107e6ede3c3b2be8a6995ef242f614dc6f592cf230f10d7862685696991ae71",
"image": "grafana/loki:3.5.1",
"image_id": "sha256:a74594532eec4cc313401beedc4dd2708c43674c032084b1aeb87c14a5be1745",
"state": "running",
"health": null,
"memory_limit_bytes": 402653184,
"nano_cpus": 1000000000,
"oom_killed": false,
"published_ports": {},
"volumes": [
"mtusi-lab6_loki_data"
]
},
{
"service": "migrate",
"container_id": "a01249e7317b529181758de5244cd14133cc7b39ba5077ee3d844d240ca98cb2",
"image": "mtusi-messenger:lab6",
"image_id": "sha256:0189284e9808367735919f731c1169eb64ba34743d8f5dcaebf73a9f6788902c",
"state": "exited",
"health": null,
"memory_limit_bytes": 402653184,
"nano_cpus": 1000000000,
"oom_killed": false,
"published_ports": {},
"volumes": []
},
{
"service": "otel-collector",
"container_id": "6e21c63012d160a406e28a342d668de0be55107aa16228ed9d5dc15c34b447be",
"image": "otel/opentelemetry-collector-contrib:0.123.0",
"image_id": "sha256:e39311df1f3d941923c00da79ac7ba6269124a870ee87e3c3ad24d60f8aee4d2",
"state": "running",
"health": null,
"memory_limit_bytes": 268435456,
"nano_cpus": 1000000000,
"oom_killed": false,
"published_ports": {},
"volumes": []
},
{
"service": "postgres",
"container_id": "1b3cdbc4689c6b855a435acab8112f86a3c9dc5e0207257c31e7aaa650635a0a",
"image": "postgres:17.6-bookworm",
"image_id": "sha256:f3bd19c606e442c3d7bdfa8002e03fe260a1023351e0ea4598032022b68dd6e3",
"state": "running",
"health": "healthy",
"memory_limit_bytes": 536870912,
"nano_cpus": 1000000000,
"oom_killed": false,
"published_ports": {},
"volumes": [
"mtusi-lab6_postgres_data"
]
},
{
"service": "prometheus",
"container_id": "d54da34ab176f240c2519bb6fcc46417b5ea3d27de86e2ed05321675f14e1e23",
"image": "prom/prometheus:v3.4.1",
"image_id": "sha256:9abc6cf6aea7710d163dbb28d8eeb7dc5baef01e38fa4cd146a406dd9f07f70d",
"state": "running",
"health": null,
"memory_limit_bytes": 402653184,
"nano_cpus": 1000000000,
"oom_killed": false,
"published_ports": {},
"volumes": [
"mtusi-lab6_prometheus_data"
]
},
{
"service": "publisher",
"container_id": "02738dc58bbbadc9dc35910593e6a75bcb1794821b38f892d22f05c0f9df7772",
"image": "mtusi-messenger:lab6",
"image_id": "sha256:a7932cb2a948db89797bf90f560ecf0046cac576c7fff97412aa55d8c4446a62",
"state": "running",
"health": null,
"memory_limit_bytes": 402653184,
"nano_cpus": 1000000000,
"oom_killed": false,
"published_ports": {},
"volumes": []
},
{
"service": "s3",
"container_id": "b7b3fe9f4f328f57dc0df0a3af84d8426e431505ffd236da9d6cc143123380c9",
"image": "chrislusf/seaweedfs:3.85@sha256:49312939c00c01e5ee6afbd7d728b18027821d3764c35a797a72acd4fdf3296a",
"image_id": "sha256:49312939c00c01e5ee6afbd7d728b18027821d3764c35a797a72acd4fdf3296a",
"state": "running",
"health": "healthy",
"memory_limit_bytes": 268435456,
"nano_cpus": 500000000,
"oom_killed": false,
"published_ports": {},
"volumes": [
"a6b780554a2ebb58edb3e2f7b77fe7ee705ce9543ef0dbe0c8d9c4d80bc31da2"
]
},
{
"service": "s3-bootstrap",
"container_id": "4c9c5e1d677c45da83a1651c20809f58457588843791e01d5b6e22ab486c9467",
"image": "mtusi-messenger:lab6",
"image_id": "sha256:0189284e9808367735919f731c1169eb64ba34743d8f5dcaebf73a9f6788902c",
"state": "exited",
"health": null,
"memory_limit_bytes": 134217728,
"nano_cpus": 500000000,
"oom_killed": false,
"published_ports": {},
"volumes": []
},
{
"service": "s3-storage",
"container_id": "1b69e90a9f7108e2e5d2017365e82fa30495c42f5f93809d42b2fdfcc1f0a9a0",
"image": "chrislusf/seaweedfs:3.85@sha256:49312939c00c01e5ee6afbd7d728b18027821d3764c35a797a72acd4fdf3296a",
"image_id": "sha256:49312939c00c01e5ee6afbd7d728b18027821d3764c35a797a72acd4fdf3296a",
"state": "running",
"health": "healthy",
"memory_limit_bytes": 536870912,
"nano_cpus": 1000000000,
"oom_killed": false,
"published_ports": {},
"volumes": [
"mtusi-lab6_s3_data"
]
},
{
"service": "tempo",
"container_id": "7dc0833e289e6e708dff042076bf4753617ba5e61122a9a67dd61c37384f1c9b",
"image": "grafana/tempo:2.8.1",
"image_id": "sha256:bc9245fe3da4e63dc4c6862d9c2dad9bcd8be13d0ba4f7705fa6acda4c904d0e",
"state": "running",
"health": null,
"memory_limit_bytes": 536870912,
"nano_cpus": 1000000000,
"oom_killed": false,
"published_ports": {},
"volumes": [
"mtusi-lab6_tempo_data"
]
},
{
"service": "valkey",
"container_id": "f4c736f41e428fb83e0effb49e8a56d9efe23ea6bd979a16fdcf072af0b342b0",
"image": "valkey/valkey:8.1.3",
"image_id": "sha256:fea8b3e67b15729d4bb70589eb03367bab9ad1ee89c876f54327fc7c6e618571",
"state": "running",
"health": "healthy",
"memory_limit_bytes": 134217728,
"nano_cpus": 500000000,
"oom_killed": false,
"published_ports": {},
"volumes": [
"mtusi-lab6_valkey_data"
]
},
{
"service": "worker",
"container_id": "168b98028f54233914e74d5a5e4e0f75b68968230fce783e9835e768e1450813",
"image": "mtusi-messenger:lab6",
"image_id": "sha256:a7932cb2a948db89797bf90f560ecf0046cac576c7fff97412aa55d8c4446a62",
"state": "running",
"health": null,
"memory_limit_bytes": 1610612736,
"nano_cpus": 2000000000,
"oom_killed": false,
"published_ports": {},
"volumes": []
}
],
"grafana_version": "12.0.2",
"datasources": [
{
"uid": "loki",
"name": "Loki",
"type": "loki",
"url": "http://loki:3100",
"jsonData": {
"derivedFields": [
{
"datasourceUid": "tempo",
"matcherRegex": "\"trace_id\": \"([a-f0-9]{32})\"",
"name": "TraceID",
"url": "${__value.raw}"
}
]
}
},
{
"uid": "prometheus",
"name": "Prometheus",
"type": "prometheus",
"url": "http://prometheus:9090",
"jsonData": {
"timeInterval": "5s"
}
},
{
"uid": "tempo",
"name": "Tempo",
"type": "tempo",
"url": "http://tempo:3200",
"jsonData": {
"nodeGraph": {
"enabled": true
},
"tracesToLogs": {
"datasourceUid": "loki",
"filterByTraceID": true,
"spanEndTimeShift": "1m",
"spanStartTimeShift": "-1m",
"tags": [
{
"key": "service.name",
"value": "service_name"
}
]
}
}
}
]
}
+666
View File
@@ -0,0 +1,666 @@
{
"passed": true,
"metrics": [
{
"metric": {
"__name__": "otel_sdk_processor_log_queue_capacity",
"deployment_environment_name": "lab",
"exported_instance": "02738dc58bbb:1",
"exported_job": "messenger-publisher",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_log_processor/0",
"otel_component_type": "batching_log_processor",
"service_instance_id": "02738dc58bbb:1",
"service_name": "messenger-publisher",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"256"
]
},
{
"metric": {
"__name__": "otel_sdk_processor_log_queue_capacity",
"deployment_environment_name": "lab",
"exported_instance": "168b98028f54:8",
"exported_job": "messenger-worker",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_log_processor/0",
"otel_component_type": "batching_log_processor",
"service_instance_id": "168b98028f54:8",
"service_name": "messenger-worker",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"256"
]
},
{
"metric": {
"__name__": "otel_sdk_processor_log_queue_capacity",
"deployment_environment_name": "lab",
"exported_instance": "168b98028f54:9",
"exported_job": "messenger-worker",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_log_processor/0",
"otel_component_type": "batching_log_processor",
"service_instance_id": "168b98028f54:9",
"service_name": "messenger-worker",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"256"
]
},
{
"metric": {
"__name__": "otel_sdk_processor_log_queue_capacity",
"deployment_environment_name": "lab",
"exported_instance": "2a50b728ce10:1",
"exported_job": "messenger-api",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_log_processor/0",
"otel_component_type": "batching_log_processor",
"service_instance_id": "2a50b728ce10:1",
"service_name": "messenger-api",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"256"
]
},
{
"metric": {
"__name__": "otel_sdk_processor_log_queue_capacity",
"deployment_environment_name": "lab",
"exported_instance": "33daefd247cf:1",
"exported_job": "messenger-api",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_log_processor/0",
"otel_component_type": "batching_log_processor",
"service_instance_id": "33daefd247cf:1",
"service_name": "messenger-api",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"256"
]
},
{
"metric": {
"__name__": "otel_sdk_processor_log_queue_size",
"deployment_environment_name": "lab",
"exported_instance": "02738dc58bbb:1",
"exported_job": "messenger-publisher",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_log_processor/0",
"otel_component_type": "batching_log_processor",
"service_instance_id": "02738dc58bbb:1",
"service_name": "messenger-publisher",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"0"
]
},
{
"metric": {
"__name__": "otel_sdk_processor_log_queue_size",
"deployment_environment_name": "lab",
"exported_instance": "168b98028f54:8",
"exported_job": "messenger-worker",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_log_processor/0",
"otel_component_type": "batching_log_processor",
"service_instance_id": "168b98028f54:8",
"service_name": "messenger-worker",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"0"
]
},
{
"metric": {
"__name__": "otel_sdk_processor_log_queue_size",
"deployment_environment_name": "lab",
"exported_instance": "168b98028f54:9",
"exported_job": "messenger-worker",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_log_processor/0",
"otel_component_type": "batching_log_processor",
"service_instance_id": "168b98028f54:9",
"service_name": "messenger-worker",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"0"
]
},
{
"metric": {
"__name__": "otel_sdk_processor_log_queue_size",
"deployment_environment_name": "lab",
"exported_instance": "2a50b728ce10:1",
"exported_job": "messenger-api",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_log_processor/0",
"otel_component_type": "batching_log_processor",
"service_instance_id": "2a50b728ce10:1",
"service_name": "messenger-api",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"0"
]
},
{
"metric": {
"__name__": "otel_sdk_processor_log_queue_size",
"deployment_environment_name": "lab",
"exported_instance": "33daefd247cf:1",
"exported_job": "messenger-api",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_log_processor/0",
"otel_component_type": "batching_log_processor",
"service_instance_id": "33daefd247cf:1",
"service_name": "messenger-api",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"0"
]
},
{
"metric": {
"__name__": "otel_sdk_processor_span_queue_capacity",
"deployment_environment_name": "lab",
"exported_instance": "02738dc58bbb:1",
"exported_job": "messenger-publisher",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_span_processor/0",
"otel_component_type": "batching_span_processor",
"service_instance_id": "02738dc58bbb:1",
"service_name": "messenger-publisher",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"256"
]
},
{
"metric": {
"__name__": "otel_sdk_processor_span_queue_capacity",
"deployment_environment_name": "lab",
"exported_instance": "168b98028f54:8",
"exported_job": "messenger-worker",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_span_processor/0",
"otel_component_type": "batching_span_processor",
"service_instance_id": "168b98028f54:8",
"service_name": "messenger-worker",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"256"
]
},
{
"metric": {
"__name__": "otel_sdk_processor_span_queue_capacity",
"deployment_environment_name": "lab",
"exported_instance": "168b98028f54:9",
"exported_job": "messenger-worker",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_span_processor/0",
"otel_component_type": "batching_span_processor",
"service_instance_id": "168b98028f54:9",
"service_name": "messenger-worker",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"256"
]
},
{
"metric": {
"__name__": "otel_sdk_processor_span_queue_capacity",
"deployment_environment_name": "lab",
"exported_instance": "2a50b728ce10:1",
"exported_job": "messenger-api",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_span_processor/0",
"otel_component_type": "batching_span_processor",
"service_instance_id": "2a50b728ce10:1",
"service_name": "messenger-api",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"256"
]
},
{
"metric": {
"__name__": "otel_sdk_processor_span_queue_capacity",
"deployment_environment_name": "lab",
"exported_instance": "33daefd247cf:1",
"exported_job": "messenger-api",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_span_processor/0",
"otel_component_type": "batching_span_processor",
"service_instance_id": "33daefd247cf:1",
"service_name": "messenger-api",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"256"
]
},
{
"metric": {
"__name__": "otel_sdk_processor_span_queue_size",
"deployment_environment_name": "lab",
"exported_instance": "02738dc58bbb:1",
"exported_job": "messenger-publisher",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_span_processor/0",
"otel_component_type": "batching_span_processor",
"service_instance_id": "02738dc58bbb:1",
"service_name": "messenger-publisher",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"0"
]
},
{
"metric": {
"__name__": "otel_sdk_processor_span_queue_size",
"deployment_environment_name": "lab",
"exported_instance": "168b98028f54:8",
"exported_job": "messenger-worker",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_span_processor/0",
"otel_component_type": "batching_span_processor",
"service_instance_id": "168b98028f54:8",
"service_name": "messenger-worker",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"0"
]
},
{
"metric": {
"__name__": "otel_sdk_processor_span_queue_size",
"deployment_environment_name": "lab",
"exported_instance": "168b98028f54:9",
"exported_job": "messenger-worker",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_span_processor/0",
"otel_component_type": "batching_span_processor",
"service_instance_id": "168b98028f54:9",
"service_name": "messenger-worker",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"0"
]
},
{
"metric": {
"__name__": "otel_sdk_processor_span_queue_size",
"deployment_environment_name": "lab",
"exported_instance": "2a50b728ce10:1",
"exported_job": "messenger-api",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_span_processor/0",
"otel_component_type": "batching_span_processor",
"service_instance_id": "2a50b728ce10:1",
"service_name": "messenger-api",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"0"
]
},
{
"metric": {
"__name__": "otel_sdk_processor_span_queue_size",
"deployment_environment_name": "lab",
"exported_instance": "33daefd247cf:1",
"exported_job": "messenger-api",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_span_processor/0",
"otel_component_type": "batching_span_processor",
"service_instance_id": "33daefd247cf:1",
"service_name": "messenger-api",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"0"
]
},
{
"metric": {
"__name__": "otel_sdk_processor_log_processed_total",
"deployment_environment_name": "lab",
"exported_instance": "02738dc58bbb:1",
"exported_job": "messenger-publisher",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_log_processor/0",
"otel_component_type": "batching_log_processor",
"service_instance_id": "02738dc58bbb:1",
"service_name": "messenger-publisher",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"318"
]
},
{
"metric": {
"__name__": "otel_sdk_processor_log_processed_total",
"deployment_environment_name": "lab",
"exported_instance": "168b98028f54:9",
"exported_job": "messenger-worker",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_log_processor/0",
"otel_component_type": "batching_log_processor",
"service_instance_id": "168b98028f54:9",
"service_name": "messenger-worker",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"66"
]
},
{
"metric": {
"__name__": "otel_sdk_processor_log_processed_total",
"deployment_environment_name": "lab",
"exported_instance": "2a50b728ce10:1",
"exported_job": "messenger-api",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_log_processor/0",
"otel_component_type": "batching_log_processor",
"service_instance_id": "2a50b728ce10:1",
"service_name": "messenger-api",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"1296"
]
},
{
"metric": {
"__name__": "otel_sdk_processor_log_processed_total",
"deployment_environment_name": "lab",
"exported_instance": "33daefd247cf:1",
"exported_job": "messenger-api",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_log_processor/0",
"otel_component_type": "batching_log_processor",
"service_instance_id": "33daefd247cf:1",
"service_name": "messenger-api",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"1289"
]
},
{
"metric": {
"__name__": "otel_sdk_processor_span_processed_total",
"deployment_environment_name": "lab",
"exported_instance": "02738dc58bbb:1",
"exported_job": "messenger-publisher",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_span_processor/0",
"otel_component_type": "batching_span_processor",
"service_instance_id": "02738dc58bbb:1",
"service_name": "messenger-publisher",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"318"
]
},
{
"metric": {
"__name__": "otel_sdk_processor_span_processed_total",
"deployment_environment_name": "lab",
"exported_instance": "168b98028f54:9",
"exported_job": "messenger-worker",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_span_processor/0",
"otel_component_type": "batching_span_processor",
"service_instance_id": "168b98028f54:9",
"service_name": "messenger-worker",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"1320"
]
},
{
"metric": {
"__name__": "otel_sdk_processor_span_processed_total",
"deployment_environment_name": "lab",
"exported_instance": "2a50b728ce10:1",
"exported_job": "messenger-api",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_span_processor/0",
"otel_component_type": "batching_span_processor",
"service_instance_id": "2a50b728ce10:1",
"service_name": "messenger-api",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"13025"
]
},
{
"metric": {
"__name__": "otel_sdk_processor_span_processed_total",
"deployment_environment_name": "lab",
"exported_instance": "33daefd247cf:1",
"exported_job": "messenger-api",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_span_processor/0",
"otel_component_type": "batching_span_processor",
"service_instance_id": "33daefd247cf:1",
"service_name": "messenger-api",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"13012"
]
},
{
"metric": {
"__name__": "otel_sdk_processor_log_processed_total",
"deployment_environment_name": "lab",
"exported_instance": "168b98028f54:8",
"exported_job": "messenger-worker",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_log_processor/0",
"otel_component_type": "batching_log_processor",
"service_instance_id": "168b98028f54:8",
"service_name": "messenger-worker",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"54"
]
},
{
"metric": {
"__name__": "otel_sdk_processor_span_processed_total",
"deployment_environment_name": "lab",
"exported_instance": "168b98028f54:8",
"exported_job": "messenger-worker",
"instance": "otel-collector:9464",
"job": "messenger",
"otel_component_name": "batching_span_processor/0",
"otel_component_type": "batching_span_processor",
"service_instance_id": "168b98028f54:8",
"service_name": "messenger-worker",
"service_version": "6.0.0",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1"
},
"value": [
1795102653.838,
"1080"
]
}
],
"per_process_queue_capacity": 256
}
File diff suppressed because it is too large Load Diff
+246
View File
@@ -0,0 +1,246 @@
PHASE: baseline 120 seconds
PHASE: short_db_spike 4 seconds
PHASE: after_short_spike 60 seconds
PHASE: sustained_db_lock 90 seconds
PHASE: after_db_recovery 100 seconds
PHASE: worker_stopped 120 seconds
PHASE: after_worker_recovery 120 seconds
{
"utc": "2026-11-19T15:26:14.987289+00:00",
"hypothesis": "Multiwindow burn alerts ignore a 4s spike, detect sustained DB/worker degradation and resolve after recovery.",
"profile": {
"message_arrival_per_second": 2,
"image_arrival_per_second": 0.5,
"max_inflight": 16,
"sampling": 1.0,
"retries": 0,
"slo_window_seconds": 3600,
"api_slo": 0.99,
"ready_slo": 0.95,
"ready_deadline_seconds": 30
},
"docker": "29.0.1 aarch64 CPUs=10 RAM=8217448448",
"db_diagnosis": {
"trace_id": "f124d93f84dd25c5af978e78e2028646",
"request_id": "3c5e5b14-5825-41b8-a227-93cba420a532",
"slow_error_spans": [
{
"name": "postgresql.SELECT",
"status": {
"code": "STATUS_CODE_ERROR"
},
"duration_ms": 2000.115335,
"attributes": [
{
"key": "db.system.name",
"value": {
"stringValue": "postgresql"
}
},
{
"key": "db.operation.name",
"value": {
"stringValue": "SELECT"
}
},
{
"key": "error.type",
"value": {
"stringValue": "_WaitTimeout"
}
}
]
},
{
"name": "postgresql.transaction",
"status": {
"code": "STATUS_CODE_ERROR"
},
"duration_ms": 2001.169168,
"attributes": [
{
"key": "error.type",
"value": {
"stringValue": "_WaitTimeout"
}
}
]
},
{
"name": "POST /api/v1/chats/{chat_id}/messages",
"status": {
"code": "STATUS_CODE_ERROR"
},
"duration_ms": 2003.784959,
"attributes": [
{
"key": "http.request.method",
"value": {
"stringValue": "POST"
}
},
{
"key": "http.route",
"value": {
"stringValue": "/api/v1/chats/{chat_id}/messages"
}
},
{
"key": "http.response.status_code",
"value": {
"intValue": "503"
}
}
]
}
]
},
"short_spike_observed_error_ratio_peak": 0.06373490138007629,
"passed": true,
"false_positives": {
"baseline": 0,
"burn_alerts_after_short_spike": 0,
"single_window_alerts_after_short_spike": [
"APIHighErrorRate"
]
},
"start": 1795101974.987257,
"end": 1795102594.377363,
"duration_seconds": 619.3901059627533,
"phases": [
{
"name": "baseline",
"start": 1795101975.364327,
"planned_seconds": 120,
"active_at": 1795101975.364415,
"end": 1795102095.3712242
},
{
"name": "short_db_spike",
"start": 1795102095.371294,
"planned_seconds": 4,
"active_at": 1795102095.527463,
"end": 1795102099.381341
},
{
"name": "after_short_spike",
"start": 1795102099.381383,
"planned_seconds": 60,
"active_at": 1795102099.381435,
"end": 1795102159.386289
},
{
"name": "sustained_db_lock",
"start": 1795102159.386304,
"planned_seconds": 90,
"active_at": 1795102159.552147,
"end": 1795102249.3866298
},
{
"name": "after_db_recovery",
"start": 1795102252.4302201,
"planned_seconds": 100,
"active_at": 1795102252.430262,
"end": 1795102352.43217
},
{
"name": "worker_stopped",
"start": 1795102352.4322171,
"planned_seconds": 120,
"active_at": 1795102352.766314,
"end": 1795102472.4357738
},
{
"name": "after_worker_recovery",
"start": 1795102472.435835,
"planned_seconds": 120,
"active_at": 1795102474.322918,
"end": 1795102592.4362948
}
],
"skipped_arrivals": 0,
"phase_summary": {
"baseline": {
"requests": 301,
"5xx": 0,
"successful_rps": 2.5081891707489476,
"p50_ms": 16.607,
"p95_ms": 31.266,
"p99_ms": 35.193
},
"short_db_spike": {
"requests": 10,
"5xx": 4,
"successful_rps": 1.4962418299487805,
"p50_ms": 1670.714,
"p95_ms": 2017.188,
"p99_ms": 2017.188
},
"after_short_spike": {
"requests": 150,
"5xx": 0,
"successful_rps": 2.499795602583431,
"p50_ms": 16.475,
"p95_ms": 30.739,
"p99_ms": 32.763
},
"sustained_db_lock": {
"requests": 225,
"5xx": 176,
"successful_rps": 0.5444424728476727,
"p50_ms": 2011.805,
"p95_ms": 2022.545,
"p99_ms": 2025.758
},
"after_db_recovery": {
"requests": 250,
"5xx": 0,
"successful_rps": 2.4999512562719017,
"p50_ms": 16.283,
"p95_ms": 30.965,
"p99_ms": 33.443
},
"worker_stopped": {
"requests": 300,
"5xx": 0,
"successful_rps": 2.4999259036886072,
"p50_ms": 16.221,
"p95_ms": 31.277,
"p99_ms": 35.499
},
"after_worker_recovery": {
"requests": 300,
"5xx": 0,
"successful_rps": 2.4999904185900736,
"p50_ms": 13.134,
"p95_ms": 30.749,
"p99_ms": 35.272
}
},
"alert_timings": {
"APIAvailabilitySlowBurn": {
"detection_seconds": 60.28,
"resolution_after_restore_seconds": 65.27
},
"APIHighErrorRate": {
"detection_seconds": 25.11,
"resolution_after_restore_seconds": 35.15
},
"APIAvailabilityFastBurn": {
"detection_seconds": 40.18,
"resolution_after_restore_seconds": 35.15
},
"AttachmentQueueLag": {
"detection_seconds": 45.21,
"resolution_after_restore_seconds": 30.12
},
"AttachmentReadyFastBurn": {
"detection_seconds": 95.5,
"resolution_after_restore_seconds": 45.18
},
"AttachmentReadySlowBurn": {
"detection_seconds": 131.6,
"resolution_after_restore_seconds": 75.32
}
}
}
File diff suppressed because it is too large Load Diff
+130
View File
@@ -0,0 +1,130 @@
{
"base_commit": "470d9d6f1e4fe4f594e37c2ba59550ee5ca54fc1",
"working_tree": true,
"files": {
".dockerignore": "a092ac618bd90f555f48c85fc58cad3ac9f51f4de4379b242f51e14e3d468b38",
".env.example": "690df84f2c2f7200bd05a6ae325277f0771df9c57e581003746e2c20e8119c05",
".github/pull_request_template.lab5.md": "e3e9afa285870e2133b3dbc30fec881bae7418605789bdb1eaf497a2eb62914a",
".github/pull_request_template.md": "fd1a90691c2197f05c4aa0dd43aea9eda4be1e6abb0e1bb1430991228eda8e64",
".github/workflows/application.yml": "e9b1309023f630f7773c7bb7c525e68662245c1657e5b773a9393ec50d0d88a5",
".github/workflows/contracts.yml": "a45d7932ef14130aff8020bce79f043cccaf9692d697088a754d309a03f7d3e8",
".gitignore": "c26ab45b3a778a5376f6db4f624a51c5fdbf1a404f9a26df8613449277d95438",
"CONTRIBUTING.md": "2c01dd698cc9653dece49a94c9fd3f58d36e84764aaa643b46e26d7e599dc12e",
"COURSE.md": "c34d53cd082d194aa4b58f504c4e0348d25b6afec3d6a9c0963dd01c9c81e971",
"Dockerfile": "e2a683428301e5bf584fb0c067b447c22e2a088ff13ce147b7f7b8b2261cd8ea",
"Dockerfile.example": "fd7ac9491c0c3c663b083c41a762fb38bc2d212fa4a2ed3e6a0702af72108709",
"MAINTAINERS.md": "f2e7e9d59e06f260ccf3acc9baf79c4a3b66e1bc370ff6e21cbefb338210630a",
"Makefile": "c05bd07058ae0949dd4905afdae7674bc7e3e63863a6cdf8b72661140d445826",
"README.md": "583fa7625d728ead8a6ee2cd2de9f117ed3f4a1d9696c0bb9fb602fe549a31db",
"REPORT.lab5.md": "ce77ed0e6cc43a6283a9a2964a2508375496b274996222ae1ab5045b508f1d5f",
"REPORT.md": "4107c5e486c13d8e48eb0f2d37bbdabe267d857af56a17fe9ac879b5b652a6ea",
"compose.example.yaml": "9bfd9eacae286dd9bffe29d6fe349534552560f966c3c3e6e6540e92d7357afb",
"compose.yaml": "aed7a12a1ceb68bc729a102b7d76ab9354d426a136245921c37556f20a06a79c",
"contracts/README.md": "72df27a559f88ffa3fd8c3c79d3c32f6db2c5986a0602677c51994c434a3fc4f",
"contracts/attachment-states.md": "cf0519e137410e4b34ab632aadb8ad8a381d4b73033e27b354c58eeacbdf86d6",
"contracts/examples.json": "381cc18c03789e935b391d007fb5843964be9cfd9f7530cb6b1996ed4a1c4c37",
"contracts/infrastructure.json": "45264a3db653bb79de8d8fcb8d70733924f3cbfcba0fa1253cecbc6828bce217",
"contracts/jobs.schema.json": "9ccf9b11466c0c3a26a941520ac5ff543241fc4935e3a6eebb01ef303779b10d",
"contracts/observability.md": "882c120c0e153a06c9cc824a802e317c58eb6590bc3d3a576f2e0a96f8597b62",
"contracts/openapi.json": "2c4074e8fac6bed6337fbbc0aaed0ae2321a76ef30ea68afad80a1f39699535a",
"docs/runbook.md": "5b9b9529ecce1f0c510e9463783db3d5d0dd14f709d2a2d571c0094714067b49",
"docs/schema.md": "205d9a1524343b602f87b14f64ebc207372c747c1e38a3222d1c733c002093ec",
"examples/requests.http": "8c086fd540092b3e31a945a416632bcda6c4bea39253f86b076f791b362cf5e4",
"infra/README.md": "0e804fa51debba4c38d3cc84abf4704897665e0335b272060363f03aadd6354f",
"infra/grafana/dashboards/messenger.json": "ed102001c4002a3e63617216571979727c05b10866a6eb58c4ac795110402510",
"infra/grafana/provisioning/dashboards/provider.yaml": "d7aa7568978a55168509eb23f2522d081627cf3c1e141a356b2c6d8e3ff86f62",
"infra/grafana/provisioning/datasources/sources.yaml": "b89e3fa70a4432fac952a2bb7a55a0dbb6cae92787090a8d1b30f3468db56d82",
"infra/haproxy.cfg": "c80c30717494c8185eda0aa0437734e3b011bff3d213ca2fa2b2e0af83c88600",
"infra/otel/README.md": "0d6c93589f7b96b7045c1b82a261362697d25f14c9e1a4946b8ea8a977c1e842",
"infra/otel/alerts.yaml": "e7a1223c64b7040cbd76b754e19ed17a6d321aaf6da1996dafcd608b3ca669af",
"infra/otel/collector.yaml": "92c759d393d9669815131acca5c8df7ed1c71dc7c7818a7b445cefcb8c040379",
"infra/otel/loki.yaml": "a1b1e6af51a0f34a7b8356d6eab523b855731b292e129cbe6a480fe0551c2eeb",
"infra/otel/prometheus.yaml": "df8340a67cb44830d1d1a6c19b87acb22d00b8b85c2993ab37a28599b43829ff",
"infra/otel/tempo.yaml": "b7d437347fe6b344de2e6499426931330c6d41ce1ba2f39492e03069e2908a43",
"lab.json": "807dbb02b180ddb13dc898c117e09495e15a462e83f4b55133ba174f089594ce",
"migrations/001_initial.sql": "a319949c962c54c5ba537a284ff38f77f2ba04c5bbc748873889d49c2bf48a46",
"migrations/002_message_idempotency.sql": "8e484326b7861b8acf4b847d7126639bf6216e635012e1e969006e86caf902ed",
"migrations/003_attachments.sql": "9deda173c984675092869022715c32648aa7b91cf8f8641bb5daf68c6f871aba",
"migrations/004_attachment_jobs.sql": "d88a95701d0b7359a731fa81aff22b5b3e1948fb29aacdf1e7e27b9136d27427",
"migrations/005_observability.sql": "fb23df863aedb6686c185d59a2499134c09c2e6ad9859c2b7e1648027aa815ea",
"requirements-dev.txt": "4cefdb67007caf026d1613be7c519c01b4c8343db7c4f4b618677e01cf113525",
"requirements.txt": "861b5ad246e5cfa706a7cdcf7d3a4a9e9feca3261d89b387e1bb553244339ed6",
"research/README.lab2.md": "59606b7a6325ac75749963f45b6b367c93d5a3a97093cb8850587ee9ae92b5da",
"research/README.md": "cd14eeb3cfdb81a70269f4503f4e91383c99e482757bb462f9120f684c6dcca7",
"research/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"research/benchmark.py": "d66405d705f6941735efb7f297df2539ec73a31f3efa989723e4ed02f72af721",
"research/observations.py": "238b8da50a1c48656daf2fe424e60d0de46ae5a4743c17ac17dda064b555a7f6",
"research/run.py": "843a33bb642ebe0648ab8ccf551adfb2e05d9fe1127d8cbd83bf762f872b243b",
"research/slo.py": "7bf6848d94e483f65598681d39b908635d205893c89874fcbe14c229de8ca233",
"scripts/build.sh": "8f67493f0563b0dce0e0f327655c4838a21c04d9d887a03820798f4f56518c2e",
"scripts/check-docker.sh": "43e6d221f386af27eec2b49a18ed6c48c284331c7a06dcb5ee03e189e0bca3ba",
"scripts/check-lab6.sh": "eb8b12eb0b770a5503ffeb349cd3b79b2777e6c7b5bb6a41186d14ef3b1f2d7a",
"scripts/check.sh": "c0de27e3d079b503f6cf2a1606bc6d156b5ac35c61455582bab3d5ee809a17a6",
"scripts/check_contract.py": "7c4402c2a6ab80d3b4676e1866d6ecc87b6daea14c080bcb9be9a7e8b959d671",
"scripts/demo.sh": "8363935669d34bf56f1e170261d590c75dce9470308da90f6e700034a9bd01f3",
"scripts/init_local.py": "279f03cc5a4bc1735396594ec238c3946bc97a7343baf93f94e6abbe12f43e1f",
"scripts/migrate.sh": "8700e123e7f154b2b80926c83ffe384cd39572bb3e063d807dfa3c144c135913",
"scripts/replicas.py": "fa2700c2a486c5919cfa1095068307ed445a6268d8dfcf5b75ce15a1629509eb",
"scripts/research-lab2.sh": "081ea2b07c338b03e08eaba8316c1de4c9b49e0d83bdded7b4de956dc77352dc",
"scripts/research.sh": "84f34fcb0cf8338d8ff8749524db68aab0e2c7b328e2f4b45ad2cf980d179a66",
"scripts/restart.example.sh": "0328246ba75defbced992104e986fde973f63e3830d83251eced5953263e944b",
"scripts/restart.sh": "6fd3eb1e0e65df0f600d88cccce52068d60ea1f58a8a125b87622ec62b82a5de",
"scripts/rolling-update.sh": "dc09335e3e8989b42bbac75a9aec9e722ab234541361830bd924f544b45f489d",
"scripts/run-local.sh": "61d5021aaca500ae8ac4249d6d63a839edbd811df006a56ea27a916f9fcbeb71",
"scripts/start-worker.sh": "b5e8df907b645ab2d81799933f9e94cf160f4922ebdba21b9d1e52ee59c8d93f",
"scripts/start.sh": "5314c8d051541b330510d6f59d6aefebd5d3ee020464e14dd9f46c09af2f6c61",
"scripts/stop-one.example.sh": "1bcf34394520b89d95b7605ec56675be799eea920e83cfbba306d5dc0a6d9811",
"scripts/stop-one.sh": "fd6e58a604aaed3833c220f4e828ce52bd2c61a905a232a0c50da74b6a3b2edb",
"scripts/stop-worker.sh": "32ef73cfc0ef91e5acd437934a46e4567d83ab926247bf624be5aeb4ce3d3993",
"scripts/stop.sh": "db7ef5f1723566859b211f02f753c65657e6820f8135a4bc07ce6c544284df0d",
"scripts/validate_openapi.py": "4e981acbfc10dac58817bf922d5896e996798551fedc1b2385130e1cddb13d54",
"scripts/wait-observability.py": "af89dbdf9c0e4d9ae11bb7b80143fd8af08b254578159e695e660c55345693c7",
"scripts/wait-ready.sh": "c24e7a2297bbcbeb69451d17875a669b61a0b6b86e9a03fb33dabb2daea0dce7",
"src/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"src/__main__.py": "91819b838eef82165a9ef67aaf45cbb7a54d131f170076e1dde672d9a77e121b",
"src/app.py": "796aee198f9aa934bac388aa05e1cc2b067643db5f1bd2a843cc0618e1b9b09d",
"src/attachments.py": "9d4c4d3f76e489dfe3b96d89f892f7b9e9222e4ffec667d419df7dcc5850e7d5",
"src/auth.py": "75ad98d17940efb2e4f6dfc584b371a4edc6c397af4fb46f3666ee9e2d043866",
"src/bootstrap_s3.py": "ad8330225e4697873d2961f874cbdfbb69b66301c6d0ccf28d6999cc5b9c8191",
"src/broker.py": "13c5c025338ccd90eb74e48d8892aba4547adc33b71807fad021a99eb37d795e",
"src/cleanup.py": "839162ed4bf01db7734025b3b14e21a30449e2b7e5b6c3d60b39c9bd723c98b5",
"src/config.py": "ad9d1f94e796b82a7a7b9b93a98d94ca588decf02a65e588b34cab362ef0a6a0",
"src/db.py": "803446c82ebbfa11b33ef41c100b4b06204f7ba3fffcec8092e0e91f6b992882",
"src/errors.py": "f020bf3e5c9619c6ef47e1c8910a19e3331140eb9dd2bba74981e89e855e1149",
"src/faults.py": "b1ca87cf1c75a050c8572c9a3b05b569d2e459c7c5d4ce23efefbeffd2774e6b",
"src/healthcheck.py": "bb8c06c1836f601f77755de3aca4b1092294784d3c33d60bd27b41778cd130be",
"src/image_pipeline.py": "f1800c26913612739f919b6880e5b673e986c198cb4ea9509e105240f40f1fa8",
"src/job_admin.py": "51781bdeacf644c36118a9972eab125893d6bf65a3eaa1143d8eb3f3d24631e1",
"src/jobs.py": "9bbed284d1f61355d65e84271191242989b10f97bb803792907444908ea15165",
"src/migrate.py": "66b66b9d244925957ee8559dc734e872172da403c688c2bf20ae8312d7c3b260",
"src/models.py": "6f9adcc886e05c7c28c540393ca4a3d22083693f30e33a51a074306f511acc15",
"src/object_store.py": "177a9cd69e0a4f494a1cdafc92544cf4a0f150842a0b7d5e826b72c13b49532a",
"src/pagination.py": "ca07be71f21ba56873d77c9422530a7a5050679b57c055b88fcf547efc112558",
"src/passwords.py": "9cb66e8e841faee89868af041a793b0b511df49b85b8d5b1a27ca296165bf1de",
"src/publisher.py": "2f99db91efdf3f7e06383954fe54c30745af402efd07052bd372ed8e1ab78211",
"src/storage.py": "58c49912188944d739d8ebc9a0d834958b605b56f9e6296d59e9b99c2735961f",
"src/telemetry.py": "3151acbe5eed23c92cc2bf2a644bc0035a12be34fdec99737042487c1ef87af5",
"src/worker.py": "1215386583876119eb817934461b56edc5416369282af017b33e4a6a9d79da0a",
"student-tests/attachment_checks.py": "219648a7ea58d81ed2c709c1961d382558654c230db369f320d4e39f6b5ba4fb",
"student-tests/docker_lifecycle.py": "06a49931d151419eea5f78668a737bc17a5d31f0559d78e5429be882fd10046c",
"student-tests/helpers.py": "3723976953cb1212e6773e8fc9771148232e4d95ff665f049e5f47d23a97d7ba",
"student-tests/lab5_pipeline.py": "c23d50767759df0ea6642bff7a56b2f3cd0eb1010dd77b42cd941e15ca3650d3",
"student-tests/lab6_observability.py": "1a16f9e7315f980089dfb82c45e009ad3979c9754dd305b2628fe16c36e68b6c",
"student-tests/postgres_pause.py": "e81dee40fe1356940b7b8a65e2c9cf5cc1510adc6bf831dd235041044f654548",
"student-tests/promtool-slo.yaml": "f265001af6026fadc0677132a510f8dccf8ff1b5e1590ea156efec571040779e",
"student-tests/test_api.py": "5a60014b01ebc40a14c037ded532995ebb680c0aa5923a910460a8da181fc32f",
"student-tests/test_attachments.py": "8660c45842e9987b814f7075f3ae23c446c156cb7b755dd2734fed0b5b11af56",
"student-tests/test_concurrency.py": "ae48809fb1acef260abb04ba3598475a9456908d3193c90445458837547a9d84",
"student-tests/test_idempotency.py": "1f34c297029b182a0f66b99f007bc1dfebf7e6e7ca6fb5d695d0a8be37135d32",
"student-tests/test_sessions.py": "4d8fe299727266bf1ca1d7fd25b8ee3dd1780b10cce23e0af48bd47da95a852e",
"student-tests/verify_restart.py": "5c7d5197ede3d92351e07cc6768a7acc3ce204120e21f744ca48a98594def33f",
"tests/README.md": "e0bfa7e196797f6a5bf6e44d8ad2ffeb82651927ff5d8484eb1c2dda1ec7dec5",
"tests/client.py": "8d50976fc7c7bbb81e88a152f613c61ab25eaf53dba33c6a9dfab19f868d546d",
"tests/fixtures/README.md": "c001783760b4f87a7a8441a7ba55f5d080ff48e1397d3581d72a88d7691f96c2",
"tests/fixtures/sample.png": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
"tests/image_probe.py": "a9a4f89181c8e0fda6ce42e1a1af32fb42d8e3ae519e314207127c9b9b448b13",
"tests/load.py": "6d3511ea719c7a7e841083e73d435c86dcbfd9f4e5c2a07d62eb9614b01d9933",
"tests/resilience.py": "1f96663a174b3978d24eff66f2741f5f8c9b788d3242261d37a972720c52788b",
"tests/smoke.py": "52ac38eb343ff7d43e7543588e1da58ca06c9fce74b2b0d0bf0e4982bc23c1db"
}
}
+12
View File
@@ -0,0 +1,12 @@
{
"passed": true,
"trace_id": "93e1177f9568f5bbfc0b6967a1324a84",
"stored_span_count": 59,
"after_full_restart": true,
"prometheus_series": 4,
"datasources": [
"loki",
"prometheus",
"tempo"
]
}
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+248
View File
@@ -0,0 +1,248 @@
{
"status": "success",
"data": {
"resultType": "streams",
"result": [
{
"stream": {
"attachment_id": "d0c6d064-5a57-47eb-a5b1-c8d2f7a9870f",
"deployment_environment_name": "lab",
"detected_level": "INFO",
"event": "attachment.accepted",
"flags": "3",
"job_id": "03320707-a810-4a0e-a72d-b10ef7734c32",
"scope_name": "messenger",
"scope_version": "6.0.0",
"service_instance_id": "33daefd247cf:1",
"service_name": "messenger-api",
"service_version": "6.0.0",
"severity_number": "9",
"severity_text": "INFO",
"span_id": "65e8265a35af1c1d",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1",
"trace_id": "2b1592ca18aac08bbd14b3f1825f0816"
},
"values": [
[
"1795102383403137083",
"{\"timestamp\": \"2026-11-19T15:33:03.403092+00:00\", \"severity\": \"INFO\", \"service.name\": \"messenger-api\", \"event\": \"attachment.accepted\", \"trace_id\": \"2b1592ca18aac08bbd14b3f1825f0816\", \"span_id\": \"65e8265a35af1c1d\", \"job_id\": \"03320707-a810-4a0e-a72d-b10ef7734c32\", \"attachment_id\": \"d0c6d064-5a57-47eb-a5b1-c8d2f7a9870f\"}"
]
]
},
{
"stream": {
"attempt": "1",
"deployment_environment_name": "lab",
"detected_level": "INFO",
"event": "job.finished",
"flags": "3",
"job_id": "03320707-a810-4a0e-a72d-b10ef7734c32",
"outcome": "ready",
"scope_name": "messenger",
"scope_version": "6.0.0",
"service_instance_id": "168b98028f54:8",
"service_name": "messenger-worker",
"service_version": "6.0.0",
"severity_number": "9",
"severity_text": "INFO",
"span_id": "ba2d3f8e53a54fd2",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1",
"trace_id": "2b1592ca18aac08bbd14b3f1825f0816"
},
"values": [
[
"1795102477984404627",
"{\"timestamp\": \"2026-11-19T15:34:37.984358+00:00\", \"severity\": \"INFO\", \"service.name\": \"messenger-worker\", \"event\": \"job.finished\", \"trace_id\": \"2b1592ca18aac08bbd14b3f1825f0816\", \"span_id\": \"ba2d3f8e53a54fd2\", \"job_id\": \"03320707-a810-4a0e-a72d-b10ef7734c32\", \"attempt\": 1, \"outcome\": \"ready\"}"
]
]
},
{
"stream": {
"deployment_environment_name": "lab",
"detected_level": "INFO",
"duration_seconds": "0.024863",
"event": "http.request",
"flags": "3",
"method": "POST",
"request_id": "bad9f74b-1237-475e-b7a9-13e04b9457fb",
"route": "/api/v1/chats/{chat_id}/attachments",
"scope_name": "messenger",
"scope_version": "6.0.0",
"service_instance_id": "33daefd247cf:1",
"service_name": "messenger-api",
"service_version": "6.0.0",
"severity_number": "9",
"severity_text": "INFO",
"span_id": "65e8265a35af1c1d",
"status": "202",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1",
"trace_id": "2b1592ca18aac08bbd14b3f1825f0816"
},
"values": [
[
"1795102383403584208",
"{\"timestamp\": \"2026-11-19T15:33:03.403545+00:00\", \"severity\": \"INFO\", \"service.name\": \"messenger-api\", \"event\": \"http.request\", \"trace_id\": \"2b1592ca18aac08bbd14b3f1825f0816\", \"span_id\": \"65e8265a35af1c1d\", \"method\": \"POST\", \"route\": \"/api/v1/chats/{chat_id}/attachments\", \"status\": 202, \"duration_seconds\": 0.024863, \"request_id\": \"bad9f74b-1237-475e-b7a9-13e04b9457fb\"}"
]
]
}
],
"stats": {
"summary": {
"bytesProcessedPerSecond": 99017520,
"linesProcessedPerSecond": 135509,
"totalBytesProcessed": 2015283,
"totalLinesProcessed": 2758,
"execTime": 0.020353,
"queueTime": 0.000121,
"subqueries": 0,
"totalEntriesReturned": 3,
"splits": 0,
"shards": 1,
"totalPostFilterLines": 3,
"totalStructuredMetadataBytesProcessed": 1011858
},
"querier": {
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": false,
"chunk": {
"headChunkBytes": 0,
"headChunkLines": 0,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0,
"postFilterLines": 0,
"headChunkStructuredMetadataBytes": 0,
"decompressedStructuredMetadataBytes": 0
},
"chunkRefsFetchTime": 0,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0
}
},
"ingester": {
"totalReached": 1,
"totalChunksMatched": 2,
"totalBatches": 2,
"totalLinesSent": 3,
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": false,
"chunk": {
"headChunkBytes": 39416,
"headChunkLines": 85,
"decompressedBytes": 1975867,
"decompressedLines": 2673,
"compressedBytes": 241323,
"totalDuplicates": 0,
"postFilterLines": 3,
"headChunkStructuredMetadataBytes": 12008,
"decompressedStructuredMetadataBytes": 999850
},
"chunkRefsFetchTime": 293083,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0
}
},
"cache": {
"chunk": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"index": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"result": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"statsResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"volumeResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"seriesResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"labelResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"instantMetricResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
}
},
"index": {
"totalChunks": 0,
"postFilterChunks": 0,
"shardsDuration": 0,
"usedBloomFilters": false
}
}
}
}
File diff suppressed because it is too large Load Diff
+76
View File
@@ -0,0 +1,76 @@
{
"utc": "2026-11-06T07:27:40.514878+00:00",
"project": "lab4-pg-pause-30f1d978ce",
"command": ".venv/bin/python student-tests/postgres_pause.py",
"passed": true,
"sql_without_server_timeout": {
"elapsed_seconds": 2.001,
"connection_closed": true
},
"direct_ready": {
"status": 503,
"elapsed_seconds": 2.012
},
"direct_live": {
"status": 200,
"elapsed_seconds": 0.006
},
"repeated_ready": [
{
"status": 503,
"elapsed_seconds": 2.11
},
{
"status": 503,
"elapsed_seconds": 2.011
},
{
"status": 503,
"elapsed_seconds": 2.014
},
{
"status": 503,
"elapsed_seconds": 2.097
},
{
"status": 503,
"elapsed_seconds": 2.194
},
{
"status": 503,
"elapsed_seconds": 2.013
},
{
"status": 503,
"elapsed_seconds": 2.299
},
{
"status": 503,
"elapsed_seconds": 2.203
},
{
"status": 503,
"elapsed_seconds": 2.013
},
{
"status": 503,
"elapsed_seconds": 2.094
}
],
"ready_after_12_seconds": {
"status": 503,
"elapsed_seconds": 0.009
},
"live_after_repeated_requests": {
"status": 200,
"elapsed_seconds": 0.006
},
"paused_seconds": 12.565,
"recovered_ready": {
"status": 200,
"elapsed_seconds": 0.009
},
"recovery_seconds": 2.874,
"api_not_restarted": true,
"existing_session_and_message_preserved": true
}
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+32
View File
@@ -0,0 +1,32 @@
Research project: lab2-research-85021a09d9
Measuring two indexes on the same 100000 messages, three repeats...
{
"chat_only": {
"insert_100000_median_ms": 1386.7587500000127,
"index_bytes": 671744,
"wal_bytes_median": 38822560,
"queries": 180,
"errors": 0,
"concurrency": 1,
"p50_ms": 1.6852079999978287,
"p95_ms": 7.566457999985232,
"p99_ms": 46.40233399999261,
"successful_sql_rps": 282.01364182715855
},
"history": {
"insert_100000_median_ms": 1404.4910420000178,
"index_bytes": 7315456,
"wal_bytes_median": 43108512,
"queries": 180,
"errors": 0,
"concurrency": 1,
"p50_ms": 0.22610399997802233,
"p95_ms": 0.29704099995342403,
"p99_ms": 0.3384169999662845,
"successful_sql_rps": 4322.810483886352
}
}
PASS: backup 0.54s; restore 0.68s; 100000 messages, all hashes and 12 samples match
PASS: restored into a different container and fresh volume; one post-backup commit is absent as expected
Results: evidence/research.json, evidence/query-plans.json; dump stays outside Git in backups/
Removed only temporary research project: lab2-research-85021a09d9
+618
View File
@@ -0,0 +1,618 @@
{
"uid": "messenger-lab6",
"title": "Messenger — Lab 6: signals and SLO",
"schemaVersion": 41,
"version": 1,
"refresh": "5s",
"time": {
"from": "now-30m",
"to": "now"
},
"tags": [
"lab6",
"slo"
],
"panels": [
{
"id": 1,
"title": "Business API RPS",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 0,
"w": 12,
"h": 8
},
"fieldConfig": {
"defaults": {
"unit": "reqps"
},
"overrides": []
},
"description": "",
"targets": [
{
"refId": "A",
"expr": "sum by (method, route) (rate(messenger_http_requests_total[1m]))",
"legendFormat": "{{method}} {{route}}"
}
]
},
{
"id": 2,
"title": "API 5xx / all business requests",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 0,
"w": 12,
"h": 8
},
"fieldConfig": {
"defaults": {
"unit": "percentunit"
},
"overrides": []
},
"description": "",
"targets": [
{
"refId": "A",
"expr": "lab6:api_bad_ratio:1m",
"legendFormat": "API 5xx / all business requests"
}
]
},
{
"id": 3,
"title": "API latency: aggregated histogram",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 8,
"w": 12,
"h": 8
},
"fieldConfig": {
"defaults": {
"unit": "s"
},
"overrides": []
},
"description": "Seconds to response headers; excludes health. Sum buckets across replicas before percentile.",
"targets": [
{
"refId": "A",
"expr": "histogram_quantile(0.5, sum by (le) (rate(messenger_http_duration_seconds_bucket[1m])))",
"legendFormat": "p50"
},
{
"refId": "B",
"expr": "histogram_quantile(0.95, sum by (le) (rate(messenger_http_duration_seconds_bucket[1m])))",
"legendFormat": "p95"
},
{
"refId": "C",
"expr": "histogram_quantile(0.99, sum by (le) (rate(messenger_http_duration_seconds_bucket[1m])))",
"legendFormat": "p99"
}
]
},
{
"id": 4,
"title": "Time from acceptance to ready: completed jobs",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 8,
"w": 12,
"h": 8
},
"fieldConfig": {
"defaults": {
"unit": "s"
},
"overrides": []
},
"description": "Success histogram only. SLO below also includes jobs that never complete.",
"targets": [
{
"refId": "A",
"expr": "histogram_quantile(0.5, sum by (le) (rate(messenger_attachment_ready_duration_seconds_bucket[2m])))",
"legendFormat": "p50"
},
{
"refId": "B",
"expr": "histogram_quantile(0.95, sum by (le) (rate(messenger_attachment_ready_duration_seconds_bucket[2m])))",
"legendFormat": "p95"
},
{
"refId": "C",
"expr": "histogram_quantile(0.99, sum by (le) (rate(messenger_attachment_ready_duration_seconds_bucket[2m])))",
"legendFormat": "p99"
}
]
},
{
"id": 5,
"title": "Job attempts by outcome",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 16,
"w": 12,
"h": 8
},
"fieldConfig": {
"defaults": {
"unit": "ops"
},
"overrides": []
},
"description": "",
"targets": [
{
"refId": "A",
"expr": "sum by (outcome) (rate(messenger_job_attempts_total[1m]))",
"legendFormat": "{{outcome}}"
}
]
},
{
"id": 6,
"title": "Worker processing p95",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 16,
"w": 12,
"h": 8
},
"fieldConfig": {
"defaults": {
"unit": "s"
},
"overrides": []
},
"description": "",
"targets": [
{
"refId": "A",
"expr": "histogram_quantile(0.95, sum by (le, outcome) (rate(messenger_job_duration_seconds_bucket[1m])))",
"legendFormat": "{{outcome}}"
}
]
},
{
"id": 7,
"title": "Outstanding jobs (DB source of truth)",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 24,
"w": 12,
"h": 8
},
"fieldConfig": {
"defaults": {
"unit": "short"
},
"overrides": []
},
"description": "Queued + processing durable jobs, including retry/outbox; not Redis XLEN.",
"targets": [
{
"refId": "A",
"expr": "max(messenger_queue_depth)",
"legendFormat": "Outstanding jobs (DB source of truth)"
}
]
},
{
"id": 8,
"title": "Oldest unfinished job age",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 24,
"w": 12,
"h": 8
},
"fieldConfig": {
"defaults": {
"unit": "s"
},
"overrides": []
},
"description": "",
"targets": [
{
"refId": "A",
"expr": "max(messenger_queue_oldest_age_seconds)",
"legendFormat": "Oldest unfinished job age"
}
]
},
{
"id": 9,
"title": "DB pool used / waiting",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 32,
"w": 12,
"h": 8
},
"fieldConfig": {
"defaults": {
"unit": "short"
},
"overrides": []
},
"description": "",
"targets": [
{
"refId": "A",
"expr": "sum by (service_name, state) (messenger_db_pool)",
"legendFormat": "{{service_name}} {{state}}"
}
]
},
{
"id": 10,
"title": "Worker busy / capacity",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 32,
"w": 12,
"h": 8
},
"fieldConfig": {
"defaults": {
"unit": "short"
},
"overrides": []
},
"description": "",
"targets": [
{
"refId": "A",
"expr": "sum(messenger_worker_busy) or vector(0)",
"legendFormat": "busy"
},
{
"refId": "B",
"expr": "sum(messenger_worker_capacity) or vector(0)",
"legendFormat": "capacity"
}
]
},
{
"id": 11,
"title": "Process CPU: cores consumed",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 40,
"w": 12,
"h": 8
},
"fieldConfig": {
"defaults": {
"unit": "short"
},
"overrides": []
},
"description": "",
"targets": [
{
"refId": "A",
"expr": "sum by (service_name) (rate(messenger_process_cpu_time_seconds_total[1m]))",
"legendFormat": "{{service_name}}"
}
]
},
{
"id": 12,
"title": "Process RSS",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 40,
"w": 12,
"h": 8
},
"fieldConfig": {
"defaults": {
"unit": "bytes"
},
"overrides": []
},
"description": "",
"targets": [
{
"refId": "A",
"expr": "sum by (service_name) (messenger_process_memory_bytes)",
"legendFormat": "{{service_name}}"
}
]
},
{
"id": 13,
"title": "API availability SLO: >= 99% / 1h",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 48,
"w": 12,
"h": 8
},
"fieldConfig": {
"defaults": {
"unit": "percentunit"
},
"overrides": []
},
"description": "5xx bad; all instrumented business requests denominator; 4xx not server failures.",
"targets": [
{
"refId": "A",
"expr": "1-lab6:api_bad_ratio:1h",
"legendFormat": "API availability SLO: >= 99% / 1h"
}
]
},
{
"id": 14,
"title": "Ready SLO: >= 95% within 30s / 1h",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 48,
"w": 12,
"h": 8
},
"fieldConfig": {
"defaults": {
"unit": "percentunit"
},
"overrides": []
},
"description": "Acceptance cohorts mature after 30s. Failed, stuck and late jobs are BAD. No ready-only bias.",
"targets": [
{
"refId": "A",
"expr": "1-lab6:ready_bad_ratio:1h",
"legendFormat": "Ready SLO: >= 95% within 30s / 1h"
}
]
},
{
"id": 15,
"title": "API burn rate (budget / hour)",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 56,
"w": 12,
"h": 8
},
"fieldConfig": {
"defaults": {
"unit": "short"
},
"overrides": []
},
"description": "Fast: 2m AND 30s > 6x, for 15s. Slow: 6m AND 1m > 3x, for 30s.",
"targets": [
{
"refId": "A",
"expr": "lab6:api_bad_ratio:30s/0.01",
"legendFormat": "30s"
},
{
"refId": "B",
"expr": "lab6:api_bad_ratio:2m/0.01",
"legendFormat": "2m"
},
{
"refId": "C",
"expr": "lab6:api_bad_ratio:1m/0.01",
"legendFormat": "1m"
},
{
"refId": "D",
"expr": "lab6:api_bad_ratio:6m/0.01",
"legendFormat": "6m"
}
]
},
{
"id": 16,
"title": "Ready burn rate (budget / hour)",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 56,
"w": 12,
"h": 8
},
"fieldConfig": {
"defaults": {
"unit": "short"
},
"overrides": []
},
"description": "Fast: 2m AND 30s > 6x, for 15s. Slow: 6m AND 1m > 3x, for 30s.",
"targets": [
{
"refId": "A",
"expr": "lab6:ready_bad_ratio:30s/0.05",
"legendFormat": "30s"
},
{
"refId": "B",
"expr": "lab6:ready_bad_ratio:2m/0.05",
"legendFormat": "2m"
},
{
"refId": "C",
"expr": "lab6:ready_bad_ratio:1m/0.05",
"legendFormat": "1m"
},
{
"refId": "D",
"expr": "lab6:ready_bad_ratio:6m/0.05",
"legendFormat": "6m"
}
]
},
{
"id": 17,
"title": "Local alert states",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 64,
"w": 12,
"h": 8
},
"fieldConfig": {
"defaults": {
"unit": "short"
},
"overrides": []
},
"description": "",
"targets": [
{
"refId": "A",
"expr": "ALERTS{alertstate=~\"pending|firing\"}",
"legendFormat": "{{alertname}} {{alertstate}}"
}
]
},
{
"id": 18,
"title": "API / worker structured logs (click TraceID)",
"type": "logs",
"datasource": {
"type": "loki",
"uid": "loki"
},
"gridPos": {
"x": 12,
"y": 64,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "{service_name=~\"messenger-api|messenger-worker\"} | json"
}
],
"options": {
"showTime": true,
"wrapLogMessage": true
}
},
{
"id": 19,
"title": "Recent real worker traces",
"type": "table",
"datasource": {
"type": "tempo",
"uid": "tempo"
},
"gridPos": {
"x": 0,
"y": 72,
"w": 24,
"h": 9
},
"targets": [
{
"refId": "A",
"queryType": "traceql",
"query": "{ resource.service.name = \"messenger-worker\" }",
"limit": 20,
"tableType": "traces",
"filters": []
}
]
}
],
"annotations": {
"list": []
}
}
@@ -0,0 +1,9 @@
apiVersion: 1
providers:
- name: messenger
folder: Lab 6
type: file
disableDeletion: false
updateIntervalSeconds: 10
options:
path: /var/lib/grafana/dashboards
@@ -0,0 +1,35 @@
apiVersion: 1
datasources:
- name: Prometheus
uid: prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
jsonData:
timeInterval: 5s
- name: Loki
uid: loki
type: loki
access: proxy
url: http://loki:3100
jsonData:
derivedFields:
- name: TraceID
matcherRegex: '"trace_id": "([a-f0-9]{32})"'
datasourceUid: tempo
url: '$${__value.raw}'
- name: Tempo
uid: tempo
type: tempo
access: proxy
url: http://tempo:3200
jsonData:
tracesToLogs:
datasourceUid: loki
spanStartTimeShift: '-1m'
spanEndTimeShift: '1m'
filterByTraceID: true
tags: [{key: service.name, value: service_name}]
nodeGraph:
enabled: true
+40
View File
@@ -0,0 +1,40 @@
global
log stdout format raw local0
maxconn 256
defaults
mode http
log global
option dontlognull
option http-server-close
timeout connect 1s
timeout client 40s
timeout server 60s
timeout http-request 5s
timeout check 1s
retries 1
resolvers docker
nameserver dns 127.0.0.11:53
accepted_payload_size 8192
resolve_retries 3
timeout resolve 1s
timeout retry 1s
hold valid 1s
hold obsolete 1s
hold nx 1s
hold timeout 1s
hold refused 1s
hold other 1s
frontend public
bind :8080
default_backend api
backend api
balance roundrobin
option httpchk
http-check send meth GET uri /health/ready ver HTTP/1.1 hdr Host localhost
http-check expect status 200
default-server inter 1s fastinter 1s downinter 1s fall 2 rise 2
server-template replica 10 api:8080 check resolvers docker resolve-prefer ipv4 init-addr none
+113
View File
@@ -0,0 +1,113 @@
groups:
- name: messenger-slo
interval: 5s
rules:
- record: lab6:api_bad_ratio:30s
expr: (sum(rate(messenger_http_requests_total{status_class="5xx"}[30s])) or vector(0))
/ sum(rate(messenger_http_requests_total[30s]))
- record: lab6:ready_bad_ratio:30s
expr: sum(rate(messenger_ready_deadlines_total{outcome="bad"}[30s])) / sum(rate(messenger_ready_deadlines_total[30s]))
- record: lab6:api_bad_ratio:1m
expr: (sum(rate(messenger_http_requests_total{status_class="5xx"}[1m])) or vector(0))
/ sum(rate(messenger_http_requests_total[1m]))
- record: lab6:ready_bad_ratio:1m
expr: sum(rate(messenger_ready_deadlines_total{outcome="bad"}[1m])) / sum(rate(messenger_ready_deadlines_total[1m]))
- record: lab6:api_bad_ratio:2m
expr: (sum(rate(messenger_http_requests_total{status_class="5xx"}[2m])) or vector(0))
/ sum(rate(messenger_http_requests_total[2m]))
- record: lab6:ready_bad_ratio:2m
expr: sum(rate(messenger_ready_deadlines_total{outcome="bad"}[2m])) / sum(rate(messenger_ready_deadlines_total[2m]))
- record: lab6:api_bad_ratio:6m
expr: (sum(rate(messenger_http_requests_total{status_class="5xx"}[6m])) or vector(0))
/ sum(rate(messenger_http_requests_total[6m]))
- record: lab6:ready_bad_ratio:6m
expr: sum(rate(messenger_ready_deadlines_total{outcome="bad"}[6m])) / sum(rate(messenger_ready_deadlines_total[6m]))
- record: lab6:api_bad_ratio:1h
expr: (sum(rate(messenger_http_requests_total{status_class="5xx"}[1h])) or vector(0))
/ sum(rate(messenger_http_requests_total[1h]))
- record: lab6:ready_bad_ratio:1h
expr: sum(rate(messenger_ready_deadlines_total{outcome="bad"}[1h])) / sum(rate(messenger_ready_deadlines_total[1h]))
- alert: APIAvailabilityFastBurn
expr: (lab6:api_bad_ratio:2m > 6 * 0.01) and (lab6:api_bad_ratio:30s > 6 * 0.01)
and (sum(increase(messenger_http_requests_total[2m])) >= 30) and (sum(increase(messenger_http_requests_total[30s]))
>= 2)
for: 15s
labels:
severity: critical
slo: api
annotations:
summary: 'api: 6x budget burn in both 2m and 30s'
description: Budget 0.01; rolling SLO window 1h. Requires real traffic, not
health requests.
runbook_url: https://gitea.privcloud.tech/Savelev_Ivan/mtusi2026lab6/src/branch/main/docs/runbook.md#api
runbook: docs/runbook.md#api
- alert: APIAvailabilitySlowBurn
expr: (lab6:api_bad_ratio:6m > 3 * 0.01) and (lab6:api_bad_ratio:1m > 3 * 0.01)
and (sum(increase(messenger_http_requests_total[6m])) >= 30) and (sum(increase(messenger_http_requests_total[1m]))
>= 2)
for: 30s
labels:
severity: warning
slo: api
annotations:
summary: 'api: 3x budget burn in both 6m and 1m'
description: Budget 0.01; rolling SLO window 1h. Requires real traffic, not
health requests.
runbook_url: https://gitea.privcloud.tech/Savelev_Ivan/mtusi2026lab6/src/branch/main/docs/runbook.md#api
runbook: docs/runbook.md#api
- alert: AttachmentReadyFastBurn
expr: (lab6:ready_bad_ratio:2m > 6 * 0.05) and (lab6:ready_bad_ratio:30s > 6 *
0.05) and (sum(increase(messenger_ready_deadlines_total[2m])) >= 5) and (sum(increase(messenger_ready_deadlines_total[30s]))
>= 2)
for: 15s
labels:
severity: critical
slo: ready
annotations:
summary: 'ready: 6x budget burn in both 2m and 30s'
description: Budget 0.05; rolling SLO window 1h. Requires real traffic, not
health requests.
runbook_url: https://gitea.privcloud.tech/Savelev_Ivan/mtusi2026lab6/src/branch/main/docs/runbook.md#ready
runbook: docs/runbook.md#ready
- alert: AttachmentReadySlowBurn
expr: (lab6:ready_bad_ratio:6m > 3 * 0.05) and (lab6:ready_bad_ratio:1m > 3 *
0.05) and (sum(increase(messenger_ready_deadlines_total[6m])) >= 5) and (sum(increase(messenger_ready_deadlines_total[1m]))
>= 2)
for: 30s
labels:
severity: warning
slo: ready
annotations:
summary: 'ready: 3x budget burn in both 6m and 1m'
description: Budget 0.05; rolling SLO window 1h. Requires real traffic, not
health requests.
runbook_url: https://gitea.privcloud.tech/Savelev_Ivan/mtusi2026lab6/src/branch/main/docs/runbook.md#ready
runbook: docs/runbook.md#ready
- alert: APIHighErrorRate
expr: lab6:api_bad_ratio:30s > 0.05
for: 5s
labels:
severity: warning
annotations:
summary: API 5xx > 5% for 30s (single-window comparator)
runbook: docs/runbook.md#api
runbook_url: https://gitea.privcloud.tech/Savelev_Ivan/mtusi2026lab6/src/branch/main/docs/runbook.md#api
- alert: AttachmentQueueLag
expr: (max(messenger_queue_oldest_age_seconds) > 20) and (max(messenger_queue_depth)
> 0)
for: 10s
labels:
severity: warning
annotations:
summary: Oldest unfinished job exceeds 20s; queue not empty
runbook: docs/runbook.md#ready
runbook_url: https://gitea.privcloud.tech/Savelev_Ivan/mtusi2026lab6/src/branch/main/docs/runbook.md#ready
- alert: TelemetryCollectorUnavailable
expr: up{job="collector"} == 0
for: 15s
labels:
severity: warning
annotations:
summary: Collector scrape unavailable; SLI may be incomplete
runbook: docs/runbook.md#telemetry
runbook_url: https://gitea.privcloud.tech/Savelev_Ivan/mtusi2026lab6/src/branch/main/docs/runbook.md#telemetry
+74
View File
@@ -0,0 +1,74 @@
extensions:
health_check:
endpoint: 0.0.0.0:13133
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
processors:
memory_limiter:
check_interval: 1s
limit_mib: 192
spike_limit_mib: 32
batch:
timeout: 1s
send_batch_size: 64
send_batch_max_size: 128
exporters:
prometheus:
endpoint: 0.0.0.0:9464
metric_expiration: 30s
resource_to_telemetry_conversion:
enabled: true
otlphttp/loki:
endpoint: http://loki:3100/otlp
timeout: 2s
sending_queue:
enabled: true
num_consumers: 2
queue_size: 64
retry_on_failure:
initial_interval: 1s
max_interval: 5s
max_elapsed_time: 15s
otlp/tempo:
endpoint: tempo:4317
tls:
insecure: true
timeout: 2s
sending_queue:
enabled: true
num_consumers: 2
queue_size: 64
retry_on_failure:
initial_interval: 1s
max_interval: 5s
max_elapsed_time: 15s
service:
extensions: [health_check]
telemetry:
metrics:
readers:
- pull:
exporter:
prometheus:
host: 0.0.0.0
port: 8888
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlp/tempo]
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [prometheus]
logs:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlphttp/loki]
+37
View File
@@ -0,0 +1,37 @@
auth_enabled: false
server:
http_listen_port: 3100
common:
path_prefix: /loki
replication_factor: 1
ring:
kvstore:
store: inmemory
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules
schema_config:
configs:
- from: 2024-01-01
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
limits_config:
allow_structured_metadata: true
retention_period: 168h
otlp_config:
resource_attributes:
ignore_defaults: true
attributes_config:
- action: index_label
attributes: [service.name, deployment.environment.name]
- action: structured_metadata
attributes: [service.instance.id, service.version]
compactor:
working_directory: /loki/compactor
retention_enabled: true
delete_request_store: filesystem
+15
View File
@@ -0,0 +1,15 @@
global:
scrape_interval: 5s
evaluation_interval: 5s
rule_files:
- /etc/prometheus/alerts.yaml
scrape_configs:
- job_name: messenger
static_configs:
- targets: [otel-collector:9464]
- job_name: collector
static_configs:
- targets: [otel-collector:8888]
- job_name: prometheus
static_configs:
- targets: [localhost:9090]
+20
View File
@@ -0,0 +1,20 @@
server:
http_listen_port: 3200
distributor:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
ingester:
max_block_duration: 1m
compactor:
compaction:
block_retention: 168h
storage:
trace:
backend: local
wal:
path: /var/tempo/wal
local:
path: /var/tempo/blocks
+30
View File
@@ -0,0 +1,30 @@
CREATE TABLE users (
id uuid PRIMARY KEY,
username varchar(32) NOT NULL UNIQUE CHECK (username ~ '^[a-z0-9_]{3,32}$'),
display_name varchar(100) NOT NULL CHECK (char_length(display_name) >= 1),
password_hash text NOT NULL,
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE TABLE chats (
id uuid PRIMARY KEY,
title varchar(100) NOT NULL CHECK (char_length(title) >= 1),
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE TABLE chat_members (
chat_id uuid NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
user_id uuid NOT NULL REFERENCES users(id),
PRIMARY KEY (chat_id, user_id)
);
CREATE INDEX chat_members_user_idx ON chat_members (user_id, chat_id);
CREATE TABLE messages (
id uuid PRIMARY KEY,
chat_id uuid NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
sender_id uuid NOT NULL,
text text NOT NULL CHECK (char_length(text) BETWEEN 1 AND 4000 AND text !~ '^\s*$'),
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
FOREIGN KEY (chat_id, sender_id) REFERENCES chat_members(chat_id, user_id)
);
CREATE INDEX messages_history_idx ON messages (chat_id, created_at, id);
+10
View File
@@ -0,0 +1,10 @@
-- Добавочная миграция: существующие данные и таблицы lab2 сохраняются.
CREATE TABLE message_idempotency (
user_id uuid NOT NULL REFERENCES users(id),
chat_id uuid NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
key varchar(128) NOT NULL CHECK (key ~ '^[A-Za-z0-9._:-]{1,128}$'),
request_text text NOT NULL,
message_id uuid NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
expires_at timestamptz NOT NULL,
PRIMARY KEY (user_id, chat_id, key)
);
+34
View File
@@ -0,0 +1,34 @@
-- Только добавления к lab3; старые сообщения и ключи сохраняются.
CREATE TABLE attachments (
id uuid PRIMARY KEY,
chat_id uuid NOT NULL REFERENCES chats(id),
owner_id uuid NOT NULL,
object_key text NOT NULL UNIQUE,
filename varchar(255) NOT NULL,
content_type text NOT NULL CHECK (content_type IN ('image/png', 'image/jpeg')),
size_bytes integer NOT NULL CHECK (size_bytes BETWEEN 1 AND 10485760),
sha256 char(64) NOT NULL CHECK (sha256 ~ '^[a-f0-9]{64}$'),
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
FOREIGN KEY (chat_id, owner_id) REFERENCES chat_members(chat_id, user_id)
);
CREATE INDEX attachments_chat_idx ON attachments(chat_id);
ALTER TABLE messages DROP CONSTRAINT messages_text_check;
ALTER TABLE messages ADD CONSTRAINT messages_text_check CHECK (char_length(text) <= 4000);
CREATE TABLE message_attachments (
message_id uuid NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
attachment_id uuid NOT NULL REFERENCES attachments(id),
position integer NOT NULL CHECK (position BETWEEN 0 AND 9),
PRIMARY KEY (message_id, attachment_id),
UNIQUE (message_id, position)
);
ALTER TABLE message_idempotency
ADD COLUMN request_attachment_ids uuid[] NOT NULL DEFAULT '{}';
-- Фиксируется ДО S3 PUT. После COMMIT ready-метаданных удаляется атомарно с ними.
CREATE TABLE attachment_uploads (
id uuid PRIMARY KEY,
object_key text NOT NULL UNIQUE,
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE INDEX attachment_uploads_created_idx ON attachment_uploads(created_at);
+42
View File
@@ -0,0 +1,42 @@
-- Lab5: существующие ready-вложения lab4 остаются доступными.
ALTER TABLE attachments
ADD COLUMN status text NOT NULL DEFAULT 'ready' CHECK (status IN ('queued','processing','ready','failed')),
ADD COLUMN variants jsonb NOT NULL DEFAULT '[]',
ADD COLUMN error_code text,
ADD COLUMN updated_at timestamptz NOT NULL DEFAULT clock_timestamp();
UPDATE attachments SET updated_at=created_at;
CREATE TABLE attachment_jobs (
id uuid PRIMARY KEY,
attachment_id uuid NOT NULL UNIQUE REFERENCES attachments(id),
pipeline_version integer NOT NULL DEFAULT 1 CHECK (pipeline_version=1),
staging_key text NOT NULL UNIQUE,
attempt integer NOT NULL DEFAULT 1 CHECK (attempt>=1),
max_attempts integer NOT NULL CHECK (max_attempts>=1),
replay_count integer NOT NULL DEFAULT 0,
lease_token uuid,
lease_until timestamptz,
next_attempt_at timestamptz NOT NULL DEFAULT clock_timestamp(),
last_error text,
staging_deleted_at timestamptz,
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE INDEX attachment_jobs_lease_idx ON attachment_jobs(lease_until) WHERE lease_until IS NOT NULL;
CREATE TABLE attachment_outbox (
job_id uuid NOT NULL REFERENCES attachment_jobs(id),
attempt integer NOT NULL,
available_at timestamptz NOT NULL DEFAULT clock_timestamp(),
published_at timestamptz,
PRIMARY KEY(job_id, attempt)
);
CREATE INDEX attachment_outbox_due_idx ON attachment_outbox(available_at) WHERE published_at IS NULL;
CREATE TABLE attachment_job_events (
id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
job_id uuid NOT NULL REFERENCES attachment_jobs(id),
attempt integer NOT NULL,
event text NOT NULL,
error_code text,
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
+9
View File
@@ -0,0 +1,9 @@
-- Context belongs to the durable job, not the lifetime of the HTTP process.
ALTER TABLE attachment_jobs
ADD COLUMN traceparent text,
ADD COLUMN tracestate text,
ADD COLUMN ready_at timestamptz;
-- Existing completed jobs predate instrumentation; preserve their actual update time.
UPDATE attachment_jobs j SET ready_at=a.updated_at
FROM attachments a WHERE a.id=j.attachment_id AND a.status='ready';
CREATE INDEX attachment_jobs_created_idx ON attachment_jobs(created_at);
+50
View File
@@ -0,0 +1,50 @@
# Основные библиотеки
fastapi==0.142.4
uvicorn==0.54.0
pydantic==2.13.5
# Их зависимости
annotated-doc==0.0.5
annotated-types==0.8.0
anyio==4.15.1
click==8.5.0
h11==0.16.0
idna==3.20
opentelemetry-api==1.45.1
pydantic-core==2.46.5
starlette==1.7.0
typing-extensions==4.16.0
typing-inspection==0.4.4
# PostgreSQL, сессии и Argon2id
psycopg[binary,pool]==3.3.6
redis==8.1.0
argon2-cffi==25.1.0
# Транзитивные зависимости новых библиотек
psycopg-binary==3.3.6
psycopg-pool==3.3.3
argon2-cffi-bindings==26.1.0
cffi==2.1.1
pycparser==3.11
# S3 и проверка настоящих PNG/JPEG (без изменения оригинала)
minio==7.2.20
Pillow==13.0.0
certifi==2026.10.23
pycryptodome==4.3.0
urllib3==2.8.0
# Explicit OpenTelemetry SDK (all signals), no automatic header/SQL/body capture.
opentelemetry-sdk==1.45.1
opentelemetry-exporter-otlp-proto-http==1.45.1
opentelemetry-semantic-conventions==0.66b1
opentelemetry-exporter-otlp-proto-common==1.45.1
opentelemetry-proto==1.45.1
opentelemetry-exporter-http-transport==0.66b1
opentelemetry-exporter-otlp-common==0.66b1
googleapis-common-protos==1.76.0
protobuf==7.36.2
requests==2.34.2
charset-normalizer==3.5.2
psutil==7.2.2
+12
View File
@@ -0,0 +1,12 @@
# Трек 2 на оценку 5
Запуск из корня: `./scripts/research-lab2.sh`. Нужны Docker Compose, Python 3 и интернет для первой сборки. Скрипт собирает API-образ и использует отдельный Compose project с новыми volumes. Текущий стенд не участвует.
- run.py управляет контейнерами, pg_dump, pg_restore, файлами evidence и cleanup.
- benchmark.py выполняет SQL и сверку внутри API-образа. Перед заполнением требует пустую БД. Исследовательский флаг устанавливает только драйвер.
Датасет: 100000 сообщений в 4 чатах (70000/20000/9000/1000). Сравниваются индекс chat_id и индекс (chat_id,created_at,id), по 3 повтора с одинаковыми ограничениями и записями. На каждый чат — страницы около 10/50/90%, один прогрев и пять измерений, 51 строка. Измеряется INSERT SELECT с COMMIT, WAL и размер индекса. EXPLAIN и основные цифры сохраняются в evidence.
После dump создаётся ещё одно сообщение. Backup восстанавливается в другой контейнер и свежий volume. Сравниваются все счётчики, 12 полных записей и SHA-256 всех сообщений. Последующая запись ожидаемо отсутствует в восстановленном снимке.
Файлы research.json, query-plans.json и research.txt обновляются автоматически. Dump остаётся в игнорируемом backups/. Временные исследовательские volumes удаляются после проверки. Методика, фактические результаты и ограничения лабораторной 2 описаны в [REPORT.lab2.md](../REPORT.lab2.md).
+37
View File
@@ -0,0 +1,37 @@
# Лабораторная 6, оценка 5, трек 2: SLO и обнаружение инцидентов
После `./scripts/start.sh`: `./scripts/research.sh` (около 11 минут).
Полный набор проверок вместе с сохранённым минимумом lab5:
`./scripts/check-lab6.sh` (около 17–20 минут при готовых образах).
Эксперимент намеренно блокирует messages и останавливает worker своего локального
синтетического стенда. Отказы ограничены по времени, worker восстанавливается
в `finally`. Нужны Docker Compose, Python 3.11 и `.venv` с корневыми requirements.
Гипотеза: alert по двум окнам не сработает на 4s всплеск, но обнаружит длительную
деградацию БД и worker; после восстановления станет inactive. Сравнение — простой
5%/30s alert. Параметры заранее заданы в `infra/otel/alerts.yaml` и `docs/runbook.md`;
эксперимент их не меняет.
Профиль: 2 POST сообщения/сек и 0.5 PNG upload/сек, fixed arrival, максимум
16 in-flight, без retries. Jobs реально обрабатываются через S3 и broker.
Baseline 120s → DB spike 4s → recovery 60s → DB lock 90s → recovery 100s →
worker stopped 120s → recovery 120s. Состояния alerts и backlog читаются каждые 5s
из Prometheus. SQL/Docker создают и снимают воздействие; причина определяется
по сохранённым metrics/logs/traces, а не по выводу команды внедрения отказа.
Результаты: `evidence/lab6-slo-research.json` (каждый запрос без body/credentials,
фазы, detection/resolution, p50/p95/p99, RPS), `lab6-slo-series.json`
(CPU/RSS, pool, saturation и остальные временные ряды),
`lab6-db-incident-trace.json`, `lab6-db-incident-logs.json`,
`lab6-worker-incident-trace.json`, `lab6-worker-incident-logs.json`.
`student-tests/promtool-slo.yaml` дополнительно проверяет правила на повторяемых
синтетических временных рядах: всплеск, длительный отказ, recovery, stalled jobs,
нет трафика. Это дополнение к live эксперименту, не замена ему.
Ограничения: один Docker-host, небольшие PNG, один live прогон каждого отказа;
шаг измерения alerts 5s, задержки SDK export/scrape. Часовой SLO задан, но
эксперимент короче часа: результат проверяет детектор, а не долгосрочную надёжность.
False-positive count нельзя переносить на другие нагрузки/окна.
Историческое исследование lab2: `run.py`, `benchmark.py`, `scripts/research-lab2.sh`,
[README.lab2.md](README.lab2.md).
View File
+168
View File
@@ -0,0 +1,168 @@
"""SQL-эксперимент запускается только в отдельном research Compose project."""
from datetime import datetime, timedelta, timezone
import hashlib
import json
import os
import secrets
import statistics
import sys
import time
from uuid import UUID
import psycopg
from psycopg.conninfo import make_conninfo
from psycopg.rows import dict_row
from src import db, migrate, passwords
from src.config import secret
SIZES = [70000, 20000, 9000, 1000]
START = datetime(2026, 1, 1, tzinfo=timezone.utc)
QUERY = """SELECT * FROM messages WHERE chat_id=%s AND (created_at, id)>(%s, %s)
ORDER BY created_at, id LIMIT %s"""
def identifier(value):
return UUID(hashlib.md5(value.encode(), usedforsecurity=False).hexdigest())
def connect(restored=False):
url = secret("DATABASE_URL")
if restored:
url = make_conninfo(url, host="postgres-restore")
return psycopg.connect(url, autocommit=True, row_factory=dict_row,
options="-c timezone=UTC -c statement_timeout=120000")
def snapshot(conn):
counts = {}
for table in ("users", "chats", "chat_members", "messages", "schema_migrations"):
# Имена таблиц — фиксированный список выше, не входные данные клиента.
counts[table] = conn.execute("SELECT count(*) AS n FROM " + table).fetchone()["n"]
digest = hashlib.sha256()
for row in conn.execute("SELECT * FROM messages ORDER BY id"):
digest.update((json.dumps(row, default=str, sort_keys=True, ensure_ascii=False) + "\n").encode())
samples = []
offset = 0
for size in SIZES:
for number in (offset + 1, offset + size // 2, offset + size):
row = conn.execute("SELECT * FROM messages WHERE id=%s", (identifier(f"message:{number}"),)).fetchone()
samples.append(json.loads(json.dumps(row, default=str, sort_keys=True, ensure_ascii=False)))
offset += size
return {"counts": counts, "messages_sha256": digest.hexdigest(), "samples": samples}
def benchmark():
db.pool.open(wait=True, timeout=15)
try:
migrate.apply()
finally:
db.pool.close()
results, plans = [], []
with connect() as conn:
assert conn.execute("SELECT count(*) AS n FROM users").fetchone()["n"] == 0, "Research requires an empty database"
encoded = passwords.hash_password(secrets.token_urlsafe(24))
for n in range(1, 11):
conn.execute("INSERT INTO users(id,username,display_name,password_hash) VALUES(%s,%s,%s,%s)",
(identifier(f"user:{n}"), f"research_{n}", "Synthetic research user", encoded))
for n in range(4):
chat = identifier(f"chat:{n}")
conn.execute("INSERT INTO chats(id,title) VALUES(%s,%s)", (chat, f"Research chat {n}"))
for user in (1, 2):
conn.execute("INSERT INTO chat_members(chat_id,user_id) VALUES(%s,%s)", (chat, identifier(f"user:{user}")))
variants = [("chat_only", "chat_id"), ("history", "chat_id, created_at, id")]
for repeat in range(3):
order = variants if repeat % 2 == 0 else variants[::-1]
for variant, columns in order:
conn.execute("TRUNCATE messages")
conn.execute("DROP INDEX IF EXISTS messages_history_idx")
conn.execute("DROP INDEX IF EXISTS messages_chat_idx")
index = "messages_history_idx" if variant == "history" else "messages_chat_idx"
conn.execute("CREATE INDEX " + index + " ON messages (" + columns + ")")
lsn = conn.execute("SELECT pg_current_wal_insert_lsn() AS lsn").fetchone()["lsn"]
started = time.perf_counter()
conn.execute("""INSERT INTO messages(id,chat_id,sender_id,text,created_at)
SELECT md5('message:'||n)::uuid,
md5('chat:'||(CASE WHEN n<=70000 THEN 0 WHEN n<=90000 THEN 1 WHEN n<=99000 THEN 2 ELSE 3 END))::uuid,
md5('user:'||(1+n%2))::uuid,
'Synthetic message '||n||' '||repeat('x',80),
timestamptz '2026-01-01 00:00:00+00' + n*interval '1 microsecond'
FROM generate_series(1,100000) AS n""")
insert_ms = (time.perf_counter() - started) * 1000
wal = conn.execute("SELECT pg_wal_lsn_diff(pg_current_wal_insert_lsn(), %s) AS bytes", (lsn,)).fetchone()["bytes"]
conn.execute("VACUUM (ANALYZE) messages")
index_bytes = conn.execute("SELECT pg_relation_size(%s::regclass) AS bytes", (index,)).fetchone()["bytes"]
result = {"variant": variant, "repeat": repeat + 1, "insert_100000_ms": insert_ms,
"wal_bytes": int(wal), "index_bytes": index_bytes, "queries": []}
offset = 0
for chat, size in enumerate(SIZES):
for fraction in (0.1, 0.5, 0.9):
number = offset + int(size * fraction)
params = (identifier(f"chat:{chat}"), START + timedelta(microseconds=number), identifier(f"message:{number}"), 51)
conn.execute(QUERY, params).fetchall() # один прогрев именно этой страницы
times = []
for _ in range(5):
started = time.perf_counter()
rows = conn.execute(QUERY, params).fetchall()
times.append((time.perf_counter() - started) * 1000)
assert len(rows) == 51
plan = conn.execute("EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) " + QUERY, params).fetchone()["QUERY PLAN"][0]
result["queries"].append({"chat_size": size, "position": fraction, "latency_ms": times,
"explain_execution_ms": plan["Execution Time"]})
if repeat == 0:
plans.append({"variant": variant, "chat_size": size, "position": fraction, "plan": plan})
offset += size
results.append(result)
# Последняя итерация заканчивается production-индексом history.
assert conn.execute("SELECT to_regclass('messages_history_idx') AS name").fetchone()["name"]
baseline = snapshot(conn)
assert baseline["counts"]["messages"] == 100000
summary = {}
for variant, _ in variants:
trials = [row for row in results if row["variant"] == variant]
latencies = sorted(t for row in trials for q in row["queries"] for t in q["latency_ms"])
summary[variant] = {
"insert_100000_median_ms": statistics.median(row["insert_100000_ms"] for row in trials),
"index_bytes": trials[-1]["index_bytes"],
"wal_bytes_median": statistics.median(row["wal_bytes"] for row in trials),
"queries": len(latencies), "errors": 0, "concurrency": 1,
"p50_ms": statistics.median(latencies),
"p95_ms": latencies[int(0.95 * (len(latencies) - 1))],
"p99_ms": latencies[int(0.99 * (len(latencies) - 1))],
"successful_sql_rps": len(latencies) / (sum(latencies) / 1000),
}
return {"postgres": conn.execute("SELECT version() AS version").fetchone()["version"],
"messages": 100000, "chat_sizes": SIZES, "repeats": 3, "warmup_per_case": 1,
"measurements_per_case": 5, "page_size": 51, "query": QUERY,
"summary": summary, "trials": results, "plans": plans, "snapshot": baseline}
def after_backup():
with connect() as conn:
message_id = identifier("after-backup")
conn.execute("INSERT INTO messages(id,chat_id,sender_id,text) VALUES(%s,%s,%s,%s)",
(message_id, identifier("chat:0"), identifier("user:1"), "Committed AFTER backup"))
return {"id": str(message_id), "source_messages": conn.execute("SELECT count(*) AS n FROM messages").fetchone()["n"]}
def verify():
expected = json.load(sys.stdin)
with connect(restored=True) as restored, connect() as source:
actual = snapshot(restored)
assert actual == expected, "Restored data differs from the snapshot"
marker = (identifier("after-backup"),)
assert restored.execute("SELECT 1 FROM messages WHERE id=%s", marker).fetchone() is None
assert source.execute("SELECT 1 FROM messages WHERE id=%s", marker).fetchone() is not None
return {"counts": actual["counts"], "messages_sha256": actual["messages_sha256"],
"samples_verified": len(actual["samples"]), "all_messages_match": True,
"after_backup_message_missing_in_restore": True,
"source_messages": source.execute("SELECT count(*) AS n FROM messages").fetchone()["n"]}
if __name__ == "__main__":
if os.environ.get("LAB2_RESEARCH") != "1":
raise SystemExit("Use ./scripts/research.sh: an isolated research database is required")
operation = sys.argv[1] if len(sys.argv) > 1 else "benchmark"
result = {"benchmark": benchmark, "after-backup": after_backup, "verify": verify}[operation]()
print(json.dumps(result, ensure_ascii=False, default=str))
+111
View File
@@ -0,0 +1,111 @@
"""Read stored observations through Grafana's provisioned datasource proxy."""
import json
import os
from pathlib import Path
import subprocess
import time
from urllib.parse import urlencode
from urllib.request import urlopen
ROOT = Path(__file__).resolve().parents[1]
GRAFANA = os.environ.get('GRAFANA_URL')
if not GRAFANA:
# Also supports an ephemeral host port (GRAFANA_PORT=0) in isolated checks.
address = subprocess.check_output(['docker','compose','port','grafana','3000'], cwd=ROOT, text=True).strip()
GRAFANA = 'http://' + address
GRAFANA = GRAFANA.rstrip('/')
def get(path, **params):
url = GRAFANA + path + ('?' + urlencode(params) if params else '')
with urlopen(url, timeout=15) as response:
return json.load(response)
def proxy(source, path, **params):
return get('/api/datasources/proxy/uid/' + source + path, **params)
def query(expr):
result = proxy('prometheus', '/api/v1/query', query=expr)
assert result['status'] == 'success', result
return result['data']['result']
def scalar(expr, default=0):
result = query(expr)
return float(result[0]['value'][1]) if result else default
def logs(expr, start=None, end=None, limit=1000):
return proxy('loki', '/loki/api/v1/query_range', query=expr,
start=str(int((start or time.time()-600)*1e9)),
end=str(int((end or time.time())*1e9)), limit=limit, direction='forward')
def trace(trace_id):
return proxy('tempo', '/api/traces/' + trace_id)
def compose(*args, check=True, timeout=180):
return subprocess.run(['docker', 'compose', *args], cwd=ROOT, text=True,
capture_output=True, check=check, timeout=timeout)
def sql(statement):
return compose('exec', '-T', 'postgres', 'psql', '-U', 'messenger', '-d', 'messenger', '-At', '-c', statement).stdout.strip()
def wait(predicate, timeout=60):
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
try:
result = predicate()
if result:
return result
except (OSError, ValueError):
pass
time.sleep(1)
raise AssertionError('Timed out waiting for observed condition')
def save(name, value):
path = ROOT / 'evidence' / name
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(value, ensure_ascii=False, indent=2) + '\n')
def spans(value):
result = []
def walk(item, service=''):
if isinstance(item, dict):
if 'resource' in item:
for attr in item['resource'].get('attributes', []):
if attr['key'] == 'service.name':
service = attr['value']['stringValue']
if 'spanId' in item and 'name' in item:
result.append({'service': service, **item})
for child in item.values():
walk(child, service)
elif isinstance(item, list):
for child in item:
walk(child, service)
walk(value)
return result
def check_dashboard():
dashboard = get('/api/dashboards/uid/messenger-lab6')['dashboard']
checked = []
for panel in dashboard['panels']:
source = panel.get('datasource', {}).get('uid')
for target in panel.get('targets', []):
if source == 'prometheus':
query(target['expr'])
elif source == 'loki':
assert logs(target['expr'])['status'] == 'success'
elif source == 'tempo':
# Grafana v12 routes traceql frontend searches to this proxy endpoint.
assert proxy('tempo', '/api/search', q=target['query'], limit=20).get('traces')
checked.append({'panel': panel['title'], 'source': source, 'passed': True})
return {'passed': True, 'panels': checked}
+99
View File
@@ -0,0 +1,99 @@
"""Создаёт изолированный стенд, измеряет SQL, делает dump и restore; пишет реальные результаты."""
from datetime import datetime, timezone
import json
import os
from pathlib import Path
import platform
import subprocess
import sys
import time
from uuid import uuid4
ROOT = Path(__file__).resolve().parents[1]
project = "lab2-research-" + uuid4().hex[:10]
env = dict(os.environ, COMPOSE_PROJECT_NAME=project)
command = ["docker", "compose", "--profile", "research"]
def compose(*args, stdin=None, stdout=subprocess.PIPE):
return subprocess.run([*command, *args], cwd=ROOT, env=env, input=stdin,
stdout=stdout, stderr=subprocess.PIPE, check=True, timeout=600)
def run_module(operation, stdin=None):
result = compose("run", "--rm", "--no-deps", "-T", "-e", "LAB2_RESEARCH=1", "api", "python", "-m", "research.benchmark", operation, stdin=stdin)
return json.loads(result.stdout)
def wait_database(service):
for _ in range(60):
cid = compose("ps", "-q", service).stdout.decode().strip()
state = subprocess.run(["docker", "inspect", "--format", "{{.State.Health.Status}}", cid], capture_output=True, text=True, check=True)
if state.stdout.strip() == "healthy":
return
time.sleep(1)
raise RuntimeError(service + " did not become healthy")
def main():
evidence = ROOT / "evidence"
evidence.mkdir(exist_ok=True)
backups = ROOT / "backups"
backups.mkdir(exist_ok=True)
dump = backups / (project + ".dump")
metadata = {"utc": datetime.now(timezone.utc).isoformat(), "project": project,
"command": "./scripts/research.sh", "host": platform.system() + " " + platform.machine(),
"commit": subprocess.run(["git", "rev-parse", "--short", "HEAD"], cwd=ROOT, capture_output=True, text=True).stdout.strip() + " + working tree",
"postgres_limits": {"cpus": 1, "memory_mib": 512}, "api_runner_limits": {"cpus": 1, "memory_mib": 256}}
started = time.perf_counter()
try:
print("Research project:", project, flush=True)
compose("up", "-d", "postgres", "postgres-restore")
wait_database("postgres")
wait_database("postgres-restore")
print("Measuring two indexes on the same 100000 messages, three repeats...", flush=True)
result = run_module("benchmark")
plans = result.pop("plans")
(evidence / "query-plans.json").write_text(json.dumps(plans, ensure_ascii=False, indent=2))
source_id = compose("ps", "-q", "postgres").stdout.decode().strip()
restore_id = compose("ps", "-q", "postgres-restore").stdout.decode().strip()
assert source_id != restore_id
for label, cid in (("source", source_id), ("restored", restore_id)):
inspected = json.loads(subprocess.run(["docker", "inspect", cid], capture_output=True, check=True).stdout)[0]
metadata[label] = {"container": cid, "image": inspected["Image"],
"volumes": [m["Name"] for m in inspected["Mounts"] if m["Type"] == "volume"]}
assert metadata["source"]["volumes"] != metadata["restored"]["volumes"]
metadata["docker"] = subprocess.run(["docker", "info", "--format", "{{.ServerVersion}} {{.Architecture}} CPUs={{.NCPU}} RAM={{.MemTotal}}"], capture_output=True, text=True, check=True).stdout.strip()
metadata["stats_after_queries"] = subprocess.run(["docker", "stats", "--no-stream", "--format", "{{json .}}", source_id], capture_output=True, text=True, check=True).stdout.strip()
backup_started = time.perf_counter()
with dump.open("wb") as output:
compose("exec", "-T", "postgres", "pg_dump", "-U", "messenger", "-d", "messenger", "--format=custom", stdout=output)
backup_seconds = time.perf_counter() - backup_started
marker = run_module("after-backup")
restore_started = time.perf_counter()
# Dump подаётся через stdin, без вывода данных или credentials в журнал.
with dump.open("rb") as input_file:
subprocess.run([*command, "exec", "-T", "postgres-restore", "pg_restore", "-U", "messenger", "-d", "messenger",
"--exit-on-error", "--single-transaction", "--no-owner"], cwd=ROOT, env=env, stdin=input_file,
stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, check=True, timeout=300)
restore_seconds = time.perf_counter() - restore_started
verification_started = time.perf_counter()
verification = run_module("verify", stdin=json.dumps(result["snapshot"]).encode())
result.update(metadata)
result["backup"] = {"file": "backups/" + dump.name, "bytes": dump.stat().st_size, "seconds": backup_seconds}
result["restore"] = {"seconds": restore_seconds, "verification_seconds": time.perf_counter() - verification_started,
"verification": verification, "post_backup_insert": marker}
result["total_seconds"] = time.perf_counter() - started
(evidence / "research.json").write_text(json.dumps(result, ensure_ascii=False, indent=2))
print(json.dumps(result["summary"], ensure_ascii=False, indent=2), flush=True)
print(f"PASS: backup {backup_seconds:.2f}s; restore {restore_seconds:.2f}s; 100000 messages, all hashes and 12 samples match", flush=True)
print("PASS: restored into a different container and fresh volume; one post-backup commit is absent as expected", flush=True)
print("Results: evidence/research.json, evidence/query-plans.json; dump stays outside Git in backups/", flush=True)
finally:
compose("down", "--volumes", "--timeout", "10")
print("Removed only temporary research project:", project, flush=True)
if __name__ == "__main__":
main()
+207
View File
@@ -0,0 +1,207 @@
"""Track 2: real-time SLO alert experiment against the current synthetic lab6 stack."""
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime, timezone
import json
import os
from pathlib import Path
import statistics
import subprocess
import sys
import threading
import time
from uuid import uuid4
ROOT = Path(__file__).resolve().parents[1]
sys.path[:0] = [str(ROOT / 'tests'), str(ROOT)]
from client import Client
from research.observations import compose, logs, proxy, query, save, spans, sql, trace, wait
def worker_evidence(requests, phases, started):
stopped = next(p for p in phases if p['name'] == 'worker_stopped')
restored = next(p for p in phases if p['name'] == 'after_worker_recovery')
upload = next(r for r in requests if r['kind'] == 'upload' and r['status'] == 202
and stopped['active_at'] + 30 < r['utc'] < stopped['end'] - 30)
stored = trace(upload['trace_id'])
chain = spans(stored)
consumers = [s for s in chain if s['name'] == 'attachments process']
assert consumers, 'Accepted upload must eventually have a real consumer span'
consumed_at = min(int(s['startTimeUnixNano']) / 1e9 for s in consumers)
assert consumed_at >= restored['active_at'] - 2, 'Job was consumed while worker was stopped'
assert {'s3.get_object', 's3.put_object', 'postgresql.UPDATE'} <= {s['name'] for s in chain}
save('lab6-worker-incident-trace.json', stored)
save('lab6-worker-incident-logs.json', logs(
'{service_name=~"messenger-api|messenger-worker"} |= "' + upload['trace_id'] + '"', start=started))
return {'trace_id': upload['trace_id'], 'request_id': upload['request_id'],
'accept_to_consume_seconds': round(consumed_at - upload['utc'], 2),
'consumer_after_restore': True}
def main():
client = Client()
alice, _, _, chat = client.bootstrap()
png = (ROOT / 'tests/fixtures/sample.png').read_bytes()
started = time.time()
stop = threading.Event()
lock = threading.Lock()
requests, samples, phases = [], [], []
skipped = 0
phase_name = 'baseline'
pending = set()
def send(kind):
now = time.time()
name_at_start = phase_name
try:
if kind == 'message':
r = client.call('POST', '/api/v1/chats/' + chat['id'] + '/messages', [201,500,503],
actor=alice,body={'text':'synthetic SLO experiment'},headers={'Idempotency-Key':uuid4().hex},schema='Message')
else:
r = client.call('POST', '/api/v1/chats/' + chat['id'] + '/attachments?filename=synthetic.png', [202,500,503],
actor=alice,raw=png,headers={'Content-Type':'image/png'},schema='Attachment')
row = {'utc':now,'phase':name_at_start,'kind':kind,'status':r['status'],
'duration_seconds':r['elapsed'],'trace_id':r['headers'].get('X-Trace-Id',''),
'request_id':r['headers']['X-Request-Id']}
except Exception as error:
row = {'utc':now,'phase':name_at_start,'kind':kind,'status':0,'error_type':type(error).__name__,
'duration_seconds':time.time()-now}
with lock:
requests.append(row)
def load():
nonlocal skipped
tick = 0
next_tick = time.monotonic()
with ThreadPoolExecutor(max_workers=16) as pool:
while not stop.is_set():
pending.difference_update({f for f in pending if f.done()})
kinds = ['message'] + (['upload'] if tick % 4 == 0 else [])
for kind in kinds:
if len(pending) >= 16:
skipped += 1
else:
pending.add(pool.submit(send, kind))
tick += 1
next_tick += .5
stop.wait(max(0,next_tick-time.monotonic()))
def phase(name, seconds, action=None):
nonlocal phase_name
phase_name = name
row = {'name':name,'start':time.time(),'planned_seconds':seconds}
print('PHASE:', name, seconds, 'seconds', flush=True)
if action:
action()
row['active_at'] = time.time()
deadline = row['start'] + seconds
while time.time() < deadline:
alerts = proxy('prometheus','/api/v1/alerts')['data']['alerts']
samples.append({'utc':time.time(),'phase':name,'alerts':[
{'name':a['labels']['alertname'],'state':a['state'],'activeAt':a['activeAt']} for a in alerts],
'queue_depth':query('max(messenger_queue_depth)'),
'queue_age':query('max(messenger_queue_oldest_age_seconds)'),
'api_bad_30s':query('lab6:api_bad_ratio:30s'),
'api_bad_2m':query('lab6:api_bad_ratio:2m')})
time.sleep(min(5,max(0,deadline-time.time())))
row['end'] = time.time()
phases.append(row)
def db_lock(seconds):
# Named session and bounded pg_sleep: the fault self-releases even if the driver exits.
p = subprocess.Popen(['docker','compose','exec','-T','-e','PGAPPNAME=lab6-incident','postgres',
'psql','-v','ON_ERROR_STOP=1','-U','messenger','-d','messenger','-c',
f"BEGIN; LOCK TABLE messages IN ACCESS EXCLUSIVE MODE; SELECT pg_sleep({seconds}); COMMIT;"],
cwd=ROOT, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
wait(lambda: sql("SELECT count(*) FROM pg_stat_activity WHERE application_name='lab6-incident' AND wait_event='PgSleep'") == '1',timeout=15)
return p
thread = threading.Thread(target=load)
result = {'utc':datetime.now(timezone.utc).isoformat(),'hypothesis':
'Multiwindow burn alerts ignore a 4s spike, detect sustained DB/worker degradation and resolve after recovery.',
'profile':{'message_arrival_per_second':2,'image_arrival_per_second':.5,'max_inflight':16,
'sampling':1.0,'retries':0,'slo_window_seconds':3600,'api_slo':.99,'ready_slo':.95,'ready_deadline_seconds':30},
'docker':subprocess.check_output(['docker','info','--format','{{.ServerVersion}} {{.Architecture}} CPUs={{.NCPU}} RAM={{.MemTotal}}'],text=True).strip()}
thread.start()
try:
phase('baseline',120)
phase('short_db_spike',4,lambda:db_lock(4))
phase('after_short_spike',60)
phase('sustained_db_lock',90,lambda:db_lock(90))
# Read only saved signals to retain the diagnosis: error trace + correlated log.
failing = next(r for r in reversed(requests) if r['status'] >= 500 and r.get('trace_id'))
time.sleep(3)
saved_trace = trace(failing['trace_id'])
save('lab6-db-incident-trace.json',saved_trace)
save('lab6-db-incident-logs.json',logs('{service_name="messenger-api"} |= "'+failing['trace_id']+'"',start=started))
result['db_diagnosis'] = {'trace_id':failing['trace_id'],'request_id':failing['request_id'],
'slow_error_spans':[{'name':s['name'],'status':s.get('status'),
'duration_ms':(int(s['endTimeUnixNano'])-int(s['startTimeUnixNano']))/1e6,
'attributes':s.get('attributes',[])} for s in spans(saved_trace) if s.get('status',{}).get('code') in [2,'STATUS_CODE_ERROR']]}
phase('after_db_recovery',100)
phase('worker_stopped',120,lambda:compose('stop','--timeout','5','worker'))
phase('after_worker_recovery',120,lambda:compose('start','worker'))
result['worker_diagnosis'] = worker_evidence(requests, phases, started)
assert all(r['status'] in {201,202,500,503} for r in requests), 'Transport/client errors occurred'
baseline_firing = {a['name'] for s in samples if s['phase']=='baseline' for a in s['alerts'] if a['state']=='firing'}
assert not baseline_firing, baseline_firing
spike_firing = {a['name'] for s in samples if s['phase'] in {'short_db_spike','after_short_spike'} for a in s['alerts'] if a['state']=='firing'}
assert not any('Burn' in a for a in spike_firing), spike_firing
spike_peak = max([float(v['value'][1]) for s in samples
if s['phase'] in {'short_db_spike','after_short_spike'} for v in s['api_bad_30s']] or [0])
assert spike_peak > 0, 'The real short spike must be visible in the stored error ratio'
result['short_spike_observed_error_ratio_peak'] = spike_peak
firing = {a['name'] for s in samples for a in s['alerts'] if a['state']=='firing'}
expected = {'APIAvailabilityFastBurn','APIAvailabilitySlowBurn','AttachmentReadyFastBurn',
'AttachmentReadySlowBurn','AttachmentQueueLag','APIHighErrorRate'}
assert expected <= firing, expected-firing
assert not [a for a in samples[-1]['alerts'] if a['state']=='firing'], samples[-1]
result['passed']=True
result['false_positives']={'baseline':len(baseline_firing),'burn_alerts_after_short_spike':0,
'single_window_alerts_after_short_spike':sorted(spike_firing)}
finally:
stop.set(); thread.join(timeout=20)
# End the bounded lock early on interruption; restore the worker in all cases.
sql("SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE application_name='lab6-incident'")
compose('start','worker')
ended=time.time()
result.update({'start':started,'end':ended,'duration_seconds':ended-started,'phases':phases,
'samples':samples,'requests':requests,'skipped_arrivals':skipped})
summary={}
for p in phases:
rows=[r for r in requests if p['start']<=r['utc']<p['end']]
latencies=sorted(r['duration_seconds']*1000 for r in rows)
summary[p['name']]={'requests':len(rows),'5xx':sum(r['status']>=500 for r in rows),
'successful_rps':sum(200<=r['status']<300 for r in rows)/(p['end']-p['start']),
**{f'p{q}_ms':round(latencies[min(len(latencies)-1,int(len(latencies)*q/100))],3)
for q in [50,95,99] if latencies}}
result['phase_summary']=summary
# Actual detection and resolution times, measured from observed Prometheus states.
timings={}
for prefix,incident,recovery in [('API','sustained_db_lock','after_db_recovery'),('Attachment','worker_stopped','after_worker_recovery')]:
ip=next((p for p in phases if p['name']==incident),None)
rp=next((p for p in phases if p['name']==recovery),None)
if not ip or not rp: continue
names={a['name'] for s in samples for a in s['alerts'] if a['name'].startswith(prefix) and a['state']=='firing'}
for name in names:
fired=[s['utc'] for s in samples if s['utc']>=ip['start'] and any(a['name']==name and a['state']=='firing' for a in s['alerts'])]
if not fired: continue
resolved=next((s['utc'] for s in samples if s['utc']>=max(rp['active_at'],fired[0]) and not any(a['name']==name and a['state']=='firing' for a in s['alerts'])),None)
timings[name]={'detection_seconds':round(fired[0]-ip['active_at'],2),
'resolution_after_restore_seconds':round(resolved-rp['active_at'],2) if resolved else None}
result['alert_timings']=timings
expressions={'http_rps':'sum(rate(messenger_http_requests_total[1m]))',
'http_p95':'histogram_quantile(0.95,sum by(le)(rate(messenger_http_duration_seconds_bucket[1m])))',
'queue_depth':'max(messenger_queue_depth)','queue_age':'max(messenger_queue_oldest_age_seconds)',
'db_pool':'sum by(state)(messenger_db_pool)','worker_busy':'sum(messenger_worker_busy)',
'worker_capacity':'sum(messenger_worker_capacity)',
'cpu':'sum by(service_name)(rate(messenger_process_cpu_time_seconds_total[1m]))',
'rss':'sum by(service_name)(messenger_process_memory_bytes)',
'api_bad':'lab6:api_bad_ratio:2m','ready_bad':'lab6:ready_bad_ratio:2m','alerts':'ALERTS{alertstate="firing"}'}
series={name:proxy('prometheus','/api/v1/query_range',query=expr,start=started,end=ended,step=5) for name,expr in expressions.items()}
save('lab6-slo-series.json',series)
save('lab6-slo-research.json',result)
print(json.dumps({k:v for k,v in result.items() if k not in {'requests','samples'}},ensure_ascii=False,indent=2),flush=True)
if __name__=='__main__':
main()
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
set -euo pipefail
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$project_dir"
python3 scripts/init_local.py
docker compose build "$@" api
+11
View File
@@ -0,0 +1,11 @@
#!/usr/bin/env bash
set -euo pipefail
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$project_dir"
if [[ ! -x .venv/bin/python ]]; then
echo "Создай .venv и установи requirements.txt по REPORT.md" >&2
exit 1
fi
./scripts/build.sh
mkdir -p evidence
.venv/bin/python -u student-tests/lab5_pipeline.py 2>&1 | sed -E 's/[[:blank:]]+$//' | tee evidence/lab5-checks.txt
+29
View File
@@ -0,0 +1,29 @@
#!/usr/bin/env bash
set -euo pipefail
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$project_dir"
mkdir -p evidence
./scripts/build.sh
# Inherited grade-4 lab5 criteria, on a separate temporary project and fresh volumes.
LAB5_EVIDENCE_FILE=evidence/lab6-lab5-regression.json \
.venv/bin/python -u student-tests/lab5_pipeline.py 2>&1 | tee evidence/lab6-lab5-regression.txt
./scripts/start.sh
make check validate test PYTHON=.venv/bin/python 2>&1 | tee evidence/lab6-api-checks.txt
.venv/bin/python -m unittest discover -s student-tests -v 2>&1 | tee -a evidence/lab6-api-checks.txt
docker run --rm --entrypoint /bin/promtool -v "$project_dir:/work:ro" -w /work \
prom/prometheus:v3.4.1 test rules student-tests/promtool-slo.yaml | tee evidence/lab6-promtool.txt
# Only this short check enables the already inherited one-shot worker failpoint.
trap 'ENABLE_FAULT_INJECTION=0 docker compose up -d --no-deps worker >/dev/null 2>&1' EXIT
ENABLE_FAULT_INJECTION=1 docker compose up -d --no-deps worker
.venv/bin/python -u student-tests/lab6_observability.py 2>&1 | tee evidence/lab6-checks.txt
ENABLE_FAULT_INJECTION=0 docker compose up -d --no-deps worker
.venv/bin/python -u student-tests/verify_restart.py 2>&1 | tee evidence/lab6-restart.txt
.venv/bin/python - <<'PY'
import json
from research.observations import ROOT, spans, trace, wait
evidence=json.loads((ROOT/'evidence/lab6-checks.json').read_text())
trace_id=evidence['checks']['retry_trace_and_logs']['trace_id']
wait(lambda: len(spans(trace(trace_id))) >= 10)
print('PASS: trace remains queryable after restarting all telemetry stores with their volumes')
PY
.venv/bin/python -u research/slo.py 2>&1 | tee evidence/lab6-slo-research.txt
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env bash
set -euo pipefail
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$project_dir"
if [[ ! -x .venv/bin/python ]]; then
echo "Не хватает .venv" >&2
echo "python3 -m venv .venv" >&2
echo ".venv/bin/python -m pip install -r requirements.txt" >&2
exit 1
fi
export BASE_URL="${BASE_URL:-http://localhost:8080}"
echo "Проверяем сервер: $BASE_URL"
make check test PYTHON="$project_dir/.venv/bin/python" BASE_URL="$BASE_URL"
.venv/bin/python -m unittest discover -s student-tests -v
+28
View File
@@ -0,0 +1,28 @@
#!/usr/bin/env bash
set -euo pipefail
base_url="${BASE_URL:-http://localhost:8080}"
suffix="$(date +%s)_$RANDOM"
# Только синтетические пароли; токены остаются в памяти shell и не выводятся.
password="Demo-password-$suffix"
register() {
local username="$1" display_name="$2"
jq -nc --arg u "$username" --arg d "$display_name" --arg p "$password" \
'{username:$u, display_name:$d, password:$p}' |
curl -fsS -H 'Content-Type: application/json' --data-binary @- "$base_url/api/v1/users" | jq -er '.id'
}
login() {
curl -fsS -X POST -u "$1:$password" "$base_url/api/v1/auth/sessions" | jq -er '.token'
}
alice_id="$(register "alice_$suffix" Алиса)"
bob_id="$(register "bob_$suffix" Боб)"
alice_token="$(login "alice_$suffix")"
bob_token="$(login "bob_$suffix")"
chat_id="$(jq -nc --arg a "$alice_id" --arg b "$bob_id" '{title:"Учебный чат",member_ids:[$a,$b]}' |
curl -fsS -H "Authorization: Bearer $alice_token" -H 'Content-Type: application/json' \
--data-binary @- "$base_url/api/v1/chats" | jq -er '.id')"
for token in "$alice_token" "$bob_token"; do
curl -fsS -H "Authorization: Bearer $token" -H 'Content-Type: application/json' \
-H "Idempotency-Key: demo-$RANDOM-$RANDOM" \
-d '{"text":"Привет из lab5! 👋"}' "$base_url/api/v1/chats/$chat_id/messages" | jq .
done
curl -fsS -H "Authorization: Bearer $bob_token" "$base_url/api/v1/chats/$chat_id/messages" | jq .
+62
View File
@@ -0,0 +1,62 @@
"""Создаёт локальное окружение и secrets; существующие значения сохраняет."""
from pathlib import Path
import json
import os
import re
import secrets
import shutil
from urllib.parse import quote
root = Path(__file__).resolve().parents[1]
if not (root / ".env").exists():
shutil.copyfile(root / ".env.example", root / ".env")
directory = root / "secrets"
directory.mkdir(exist_ok=True, mode=0o700)
directory.chmod(0o700)
password_file = directory / "postgres_password"
if not password_file.exists():
password_file.write_text(secrets.token_hex(24))
password = password_file.read_text().strip()
values = {
"database_url": f"postgresql://messenger:{quote(password, safe='')}@postgres:5432/messenger",
"cache_url": "redis://valkey:6379/0",
"broker_password": secrets.token_hex(24),
"s3_root_user": "root-" + secrets.token_hex(8),
"s3_root_password": secrets.token_urlsafe(32),
"s3_access_key_id": "api-" + secrets.token_hex(8),
"s3_secret_access_key": secrets.token_urlsafe(32),
}
for name, value in values.items():
path = directory / name
if not path.exists():
path.write_text(value)
broker_url = directory / "broker_url"
if not broker_url.exists():
broker_url.write_text("redis://:" + (directory / "broker_password").read_text().strip() + "@broker:6379/0")
broker_url.chmod(0o444)
dotenv = dict(line.split("=", 1) for line in (root / ".env").read_text().splitlines()
if line.strip() and not line.lstrip().startswith("#") and "=" in line)
bucket = os.environ.get("S3_BUCKET", dotenv.get("S3_BUCKET", "messenger-attachments"))
if re.fullmatch(r"[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]", bucket) is None:
raise ValueError("Недопустимое имя S3 bucket")
def local_secret(name):
return (directory / name).read_text().strip()
config = {"accounts": [{"id": "messenger-api", "displayName": "messenger-api"}], "identities": [
{"name": "bootstrap", "credentials": [{"accessKey": local_secret("s3_root_user"),
"secretKey": local_secret("s3_root_password")}], "actions": ["Admin"]},
{"name": "messenger-api", "account": {"id": "messenger-api", "displayName": "messenger-api"},
"credentials": [{"accessKey": local_secret("s3_access_key_id"), "secretKey": local_secret("s3_secret_access_key")}],
"actions": ["Read:" + bucket, "List:" + bucket, "Write:" + bucket + "/attachments/*"]},
]}
config_file = directory / "s3_config"
encoded_config = json.dumps(config)
if not config_file.exists() or config_file.read_text() != encoded_config:
temporary = directory / "s3_config.tmp"
temporary.write_text(encoded_config)
temporary.chmod(0o444)
temporary.replace(config_file)
# Каталог 0700 защищает файлы на хосте; 0444 позволяет USER 10001 читать bind-mounted secret.
for name in ("postgres_password", "s3_config", *values):
(directory / name).chmod(0o444)
print("Local .env and secrets are ready; values are not printed")
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
set -euo pipefail
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$project_dir"
./scripts/build.sh
docker compose run --rm migrate
+90
View File
@@ -0,0 +1,90 @@
"""Управление репликами только текущего Compose project через Docker CLI."""
import argparse
import json
import os
from pathlib import Path
import subprocess
import time
from urllib.request import urlopen
ROOT = Path(__file__).resolve().parents[1]
def command(*args):
return subprocess.check_output(args, cwd=ROOT, text=True, timeout=120).strip()
def replicas():
ids = command("docker", "compose", "ps", "-q", "api").split()
return json.loads(command("docker", "inspect", *ids)) if ids else []
def base_url():
config = json.loads(command("docker", "compose", "config", "--format", "json"))
port = config["services"]["lb"]["ports"][0]["published"]
return os.environ.get("BASE_URL", "http://127.0.0.1:" + str(port)).rstrip("/")
def wait_ready():
url = base_url() + "/health/ready"
deadline = time.monotonic() + 90
seen = set()
while time.monotonic() < deadline:
current = replicas()
expected = {info["Config"]["Hostname"] for info in current}
if expected and all(info["State"].get("Health", {}).get("Status") == "healthy" for info in current):
try:
with urlopen(url, timeout=2) as response:
if response.status == 200:
seen.add(response.headers.get("X-Instance-Id"))
if expected <= seen:
print("API ready through LB:", ", ".join(sorted(expected)), flush=True)
return
except OSError:
pass
time.sleep(0.2)
raise RuntimeError("Не все работающие API-реплики готовы через LB за 90 секунд")
def stop_one():
target = os.environ.get("TARGET_INSTANCE_ID")
if not target:
raise RuntimeError("Задайте TARGET_INSTANCE_ID из X-Instance-Id")
current = replicas()
matches = [info for info in current if info["Config"]["Hostname"] == target]
if len(current) < 2 or len(matches) != 1:
raise RuntimeError("Нужны минимум две работающие реплики и ровно одна выбранная")
selected = matches[0]
command("docker", "stop", "--time", "8", selected["Id"])
print("Stopped API:", target, "container=" + selected["Id"], flush=True)
def rolling():
wait_ready()
originals = replicas()
count = len(originals)
if count < 2:
raise RuntimeError("Для поочерёдной замены нужны минимум две API-реплики")
for old in originals:
active = replicas()
healthy = [info for info in active if info["Id"] != old["Id"]
and info["State"].get("Health", {}).get("Status") == "healthy"]
if not healthy:
raise RuntimeError("Нет готовой реплики, которая продолжит обслуживать запросы")
print("Replacing API:", old["Config"]["Hostname"], flush=True)
command("docker", "stop", "--time", "8", old["Id"])
command("docker", "rm", old["Id"])
# --no-recreate сохраняет остальные процессы; Compose создаёт только недостающий.
command("docker", "compose", "up", "-d", "--no-deps", "--no-recreate", "--scale", f"api={count}", "api")
wait_ready()
assert len(replicas()) == count
final = replicas()
assert not ({info["Id"] for info in originals} & {info["Id"] for info in final})
print("PASS: all API containers replaced sequentially; stores and LB kept running", flush=True)
if __name__ == "__main__":
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("action", choices=["wait", "stop", "rolling"])
action = parser.parse_args().action
{"wait": wait_ready, "stop": stop_one, "rolling": rolling}[action]()
+7
View File
@@ -0,0 +1,7 @@
#!/usr/bin/env bash
set -euo pipefail
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$project_dir"
./scripts/build.sh
mkdir -p evidence
python3 -u research/run.py 2>&1 | tee evidence/research.txt
+7
View File
@@ -0,0 +1,7 @@
#!/usr/bin/env bash
set -euo pipefail
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$project_dir"
python3 scripts/wait-observability.py
mkdir -p evidence
.venv/bin/python -u research/slo.py 2>&1 | tee evidence/lab6-slo-research.txt
+7
View File
@@ -0,0 +1,7 @@
#!/usr/bin/env bash
set -euo pipefail
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$project_dir"
docker compose restart --timeout 8 postgres valkey s3-storage s3 broker api lb cleanup publisher worker otel-collector prometheus loki tempo grafana
./scripts/wait-ready.sh
python3 scripts/wait-observability.py
+7
View File
@@ -0,0 +1,7 @@
#!/usr/bin/env bash
set -euo pipefail
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$project_dir"
# Сначала собираем образ и применяем только обратно совместимые миграции.
./scripts/migrate.sh
exec python3 scripts/replicas.py rolling
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env bash
set -euo pipefail
# Локальный стенд: HAProxy, две API-реплики, PostgreSQL и Valkey.
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
exec "$project_dir/scripts/start.sh"
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env bash
set -euo pipefail
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$project_dir"
docker compose start worker
+8
View File
@@ -0,0 +1,8 @@
#!/usr/bin/env bash
set -euo pipefail
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$project_dir"
python3 scripts/init_local.py
docker compose up -d --build lb cleanup publisher worker otel-collector prometheus grafana
./scripts/wait-ready.sh
python3 scripts/wait-observability.py
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env bash
set -euo pipefail
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$project_dir"
exec python3 scripts/replicas.py stop
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env bash
set -euo pipefail
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$project_dir"
docker compose stop --timeout 8 worker
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
set -euo pipefail
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$project_dir"
docker compose down --timeout 10
echo "Контейнеры удалены; именованные volumes сохранены"
+26
View File
@@ -0,0 +1,26 @@
"""Readiness of provisioned telemetry, queried through the local Grafana proxy."""
from pathlib import Path
import sys
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
from research.observations import get, proxy, query, wait
def ready():
assert get('/api/health')['database'] == 'ok'
assert get('/api/dashboards/uid/messenger-lab6')['dashboard']['uid'] == 'messenger-lab6'
assert len(get('/api/datasources')) == 3
assert len(query('up == 1')) == 3
proxy('loki', '/loki/api/v1/labels')
proxy('tempo', '/api/search', limit=1)
return True
if __name__ == '__main__':
# Assertion failures are readiness failures, not successful probes.
def probe():
try:
return ready()
except (AssertionError, OSError, ValueError):
return False
wait(probe, timeout=90)
print('PASS: Collector scrape, stores, Grafana datasources and dashboard are ready')
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env bash
set -euo pipefail
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$project_dir"
exec python3 scripts/replicas.py wait
View File
+16
View File
@@ -0,0 +1,16 @@
"""Общий запуск"""
import os
import uvicorn
if __name__ == "__main__":
uvicorn.run(
"src.app:app",
host=os.environ.get("APP_HOST", "0.0.0.0"),
port=int(os.environ.get("APP_PORT", "8080")),
workers=1,
access_log=False,
timeout_graceful_shutdown=5,
)
+222
View File
@@ -0,0 +1,222 @@
"""HTTP-адреса, определение пользователя и обработка запросов"""
from contextlib import asynccontextmanager
import json
import logging
import re
from time import perf_counter
from uuid import UUID, uuid4
from urllib.parse import quote
from fastapi import FastAPI, Request, Response
from fastapi.responses import StreamingResponse
from starlette.concurrency import run_in_threadpool
from fastapi.exceptions import RequestValidationError
from starlette.exceptions import HTTPException
from src import attachments, auth, db, migrate, object_store, passwords, storage, telemetry
from src.errors import ApiError, error_response
from src.models import CreateChat, CreateMessage, CreateUser
from src.config import INSTANCE_ID
@asynccontextmanager
async def lifespan(app):
telemetry.setup('messenger-api', db.pool)
# Expose zeros before the first error. Otherwise rate() cannot see a short
# burst that finishes before the first scrape of a new status-class series.
for route in app.routes:
if getattr(route, 'path', '').startswith('/api/v1/'):
for method in route.methods:
for status_class in ('2xx', '3xx', '4xx', '5xx'):
telemetry.http_requests.add(0, {'method': method, 'route': route.path,
'status_class': status_class})
await run_in_threadpool(db.pool.open, wait=True, timeout=15)
try:
await run_in_threadpool(migrate.check)
yield
finally:
await run_in_threadpool(db.pool.close)
await run_in_threadpool(telemetry.shutdown)
app = FastAPI(title="Учебный мессенджер", redirect_slashes=False, lifespan=lifespan,
telemetry={'auto_configure': False, 'tracing': False, 'metrics': False,
'logs': False, 'operation_spans': False})
logger = logging.getLogger("uvicorn.error")
@app.middleware("http")
async def handle_request(request: Request, call_next):
path = request.url.path
route = next((route.path for route in app.routes
if hasattr(route, 'path_regex') and route.path_regex.fullmatch(path)), 'unmatched')
method = request.method if request.method in {'GET', 'POST', 'DELETE', 'PUT', 'PATCH', 'HEAD', 'OPTIONS'} else 'OTHER'
attrs = {'method': method, 'route': route}
started = perf_counter()
if path.startswith('/health/') or not path.startswith('/api/v1/'):
return await process_request(request, call_next)
with telemetry.span(method + ' ' + route, kind=telemetry.SpanKind.SERVER, parent=request.headers,
attributes={'http.request.method': method, 'http.route': route}) as current:
response = await process_request(request, call_next)
current.set_attribute('http.response.status_code', response.status_code)
if response.status_code >= 500:
current.set_status(telemetry.Status(telemetry.StatusCode.ERROR))
context = current.get_span_context()
if context.is_valid:
response.headers['X-Trace-Id'] = format(context.trace_id, '032x')
elapsed = perf_counter() - started
attrs['status_class'] = str(response.status_code // 100) + 'xx'
telemetry.http_requests.add(1, attrs)
telemetry.http_duration.record(elapsed, attrs)
telemetry.event('http.request', 'ERROR' if response.status_code >= 500 else 'INFO',
method=method, route=route, status=response.status_code, duration_seconds=round(elapsed, 6),
request_id=request.state.request_id)
return response
async def process_request(request: Request, call_next):
request.state.request_id = str(uuid4())
started = perf_counter()
path = request.url.path
try:
is_registration = request.method == "POST" and path == "/api/v1/users"
is_login = request.method == "POST" and path == "/api/v1/auth/sessions"
is_upload = request.method == "POST" and re.fullmatch(r"/api/v1/chats/[^/]+/attachments", path)
if is_login:
request.state.user = await run_in_threadpool(auth.basic_user, request.headers.get("Authorization"))
if await request.body():
raise ApiError(400, "invalid_request", "Вход через Basic не принимает тело запроса")
elif path.startswith("/api/v1/") and not is_registration:
request.state.user = await run_in_threadpool(auth.current_user, request.headers.get("Authorization"))
if request.method == "POST" and path.startswith("/api/v1/") and not is_login and not is_upload:
content_type = request.headers.get("Content-Type", "").split(";", 1)[0].strip().lower()
if content_type != "application/json":
raise ApiError(415, "unsupported_media_type", "Ожидается application/json")
try:
json.loads((await request.body()).decode("utf-8"))
except (ValueError, RecursionError):
raise ApiError(400, "invalid_request", "Ожидается корректный JSON в UTF-8")
response = await call_next(request)
except ApiError as error:
response = error_response(request, error)
except Exception as error:
# Не печатаем тело запроса или текст исключения: там могут быть сообщения.
telemetry.event('http.exception', 'ERROR', error_type=type(error).__name__, request_id=request.state.request_id)
response = error_response(request, ApiError(500, "internal_error", "Внутренняя ошибка сервера"))
response.headers["X-Request-Id"] = request.state.request_id
response.headers["X-Instance-Id"] = INSTANCE_ID
return response
@app.exception_handler(ApiError)
def handle_api_error(request: Request, error: ApiError):
return error_response(request, error)
@app.exception_handler(RequestValidationError)
def handle_validation_error(request: Request, error: RequestValidationError):
# Контракт требует 400 и наш формат ошибки, а не стандартный ответ FastAPI.
return error_response(request, ApiError(400, "invalid_request", "Неверные поля или параметры запроса"))
@app.exception_handler(HTTPException)
def handle_http_error(request: Request, error: HTTPException):
code = "not_found" if error.status_code == 404 else "invalid_request"
message = "Адрес не найден" if error.status_code == 404 else "Недопустимый HTTP-запрос"
response = error_response(request, ApiError(error.status_code, code, message))
if error.headers:
response.headers.update(error.headers)
return response
@app.get("/health/live")
def health_live():
return {"status": "ok"}
@app.get("/health/ready")
def health_ready():
db.ready()
auth.ready()
object_store.ready()
return {"status": "ok"}
@app.post("/api/v1/users", status_code=201)
def create_user(data: CreateUser):
return storage.create_user(data.username, data.display_name, passwords.hash_password(data.password))
@app.get("/api/v1/users/me")
def current_user(request: Request):
return request.state.user
@app.post("/api/v1/chats", status_code=201)
def create_chat(request: Request, data: CreateChat):
return storage.create_chat(request.state.user["id"], data.title, data.member_ids)
@app.get("/api/v1/chats")
def list_chats(request: Request, limit: str = "50", cursor: str | None = None):
user_id = request.state.user["id"]
return storage.list_chats(user_id, limit, cursor)
@app.get("/api/v1/chats/{chat_id}")
def get_chat(request: Request, chat_id: UUID):
return storage.get_chat(request.state.user["id"], str(chat_id))
@app.post("/api/v1/chats/{chat_id}/messages", status_code=201)
def create_message(request: Request, chat_id: UUID, data: CreateMessage):
keys = request.headers.getlist("Idempotency-Key")
if len(keys) != 1 or re.fullmatch(r"[A-Za-z0-9._:-]{1,128}", keys[0]) is None:
raise ApiError(400, "invalid_request", "Нужен Idempotency-Key: 1–128 символов ASCII [A-Za-z0-9._:-]")
return storage.create_message(request.state.user["id"], str(chat_id), data.text, keys[0], data.attachment_ids)
@app.post("/api/v1/chats/{chat_id}/attachments", status_code=202)
async def upload_attachment(request: Request, chat_id: UUID, filename: str):
return await attachments.upload(request, str(chat_id), filename)
@app.get("/api/v1/attachments/{attachment_id}")
def get_attachment(request: Request, attachment_id: UUID):
return attachments.public_attachment(attachments.find(request.state.user["id"], str(attachment_id)))
@app.get("/api/v1/attachments/{attachment_id}/content")
def download_attachment(request: Request, attachment_id: UUID, variant: str = "original"):
row = attachments.find(request.state.user["id"], str(attachment_id))
if row["status"] != "ready":
raise ApiError(409, "conflict", "Вложение не готово")
info = row
key = row["object_key"]
if variant != "original":
info = next((item for item in row["variants"] if item["name"] == variant), None)
if info is None:
raise ApiError(404, "not_found", "Вариант вложения не найден")
key = row["object_key"].rsplit("/", 1)[0] + "/" + variant
response = object_store.get(key)
return StreamingResponse(attachments.chunks(response), media_type=info["content_type"], headers={
"Content-Length": str(info["size_bytes"]), "X-Content-Type-Options": "nosniff",
"Content-Disposition": "attachment; filename*=UTF-8''" + quote(row["filename"], safe=""),
})
@app.get("/api/v1/chats/{chat_id}/messages")
def list_messages(request: Request, chat_id: UUID, limit: str = "50", cursor: str | None = None):
return storage.list_messages(request.state.user["id"], str(chat_id), limit, cursor)
@app.post("/api/v1/auth/sessions", status_code=201)
def create_session(request: Request):
return auth.create_session(request.state.user)
@app.delete("/api/v1/auth/sessions/current", status_code=204)
def delete_session(request: Request):
auth.logout(request.headers.get("Authorization"))
return Response(status_code=204)
+154
View File
@@ -0,0 +1,154 @@
"""Приём и проверка -> durable S3 staging + PostgreSQL outbox -> 202."""
import asyncio
import hashlib
import os
from tempfile import TemporaryFile
import unicodedata
from uuid import UUID, uuid4
import warnings
from PIL import Image, UnidentifiedImageError
from starlette.concurrency import run_in_threadpool
from starlette.requests import ClientDisconnect
from src import db, jobs, object_store, storage, telemetry
from src.errors import ApiError
MAX_BYTES = int(os.environ.get("MAX_UPLOAD_BYTES", "10485760"))
MAX_PIXELS = int(os.environ.get("MAX_IMAGE_PIXELS", "40000000"))
if not 1 <= MAX_BYTES <= 10485760 or not 1 <= MAX_PIXELS <= 40000000:
raise ValueError("Лимиты загрузки должны быть положительными и не превышать контракт")
Image.MAX_IMAGE_PIXELS = MAX_PIXELS
warnings.filterwarnings("error", category=Image.DecompressionBombWarning)
# Максимум одно полное декодирование на реплику, независимо от числа клиентов.
upload_slot = asyncio.Semaphore(1)
def public_attachment(row):
return {"id": str(row["id"]), "chat_id": str(row["chat_id"]), "owner_id": str(row["owner_id"]),
"filename": row["filename"], "content_type": row["content_type"],
"size_bytes": row["size_bytes"], "sha256": row["sha256"], "status": row["status"],
"variants": row["variants"], "error_code": row["error_code"], "created_at": storage.utc(row["created_at"]),
"updated_at": storage.utc(row["updated_at"])}
def validate_filename(filename):
if not 1 <= len(filename) <= 255 or not filename.strip() or any(
char in "/\\" or unicodedata.category(char) == "Cc" for char in filename
):
raise ApiError(400, "invalid_request", "Недопустимое имя файла")
try:
filename.encode("utf-8")
except UnicodeError:
raise ApiError(400, "invalid_request", "Недопустимое имя файла") from None
def validate_image(file, declared_type):
try:
file.seek(0)
with Image.open(file, formats=["PNG", "JPEG"]) as image:
true_type = {"PNG": "image/png", "JPEG": "image/jpeg"}[image.format]
if image.width * image.height > MAX_PIXELS:
raise ApiError(413, "payload_too_large", "Слишком много пикселей")
if true_type != declared_type:
raise ApiError(415, "unsupported_media_type", "MIME не совпадает с содержимым")
image.verify()
# verify проверяет структуру/CRC, load дополнительно обнаруживает обрыв pixel data.
file.seek(0)
with Image.open(file, formats=["PNG", "JPEG"]) as image:
image.load()
return true_type
except (Image.DecompressionBombError, Image.DecompressionBombWarning):
raise ApiError(413, "payload_too_large", "Слишком много пикселей") from None
except (UnidentifiedImageError, OSError, ValueError, SyntaxError):
raise ApiError(415, "unsupported_media_type", "Повреждённый PNG/JPEG или другой формат") from None
def lock_upload(conn, key, blocking=True):
function = "pg_advisory_xact_lock" if blocking else "pg_try_advisory_xact_lock"
return conn.execute(f"SELECT {function}(hashtextextended(%s, 4)) AS locked", (key,)).fetchone()["locked"]
def save(user_id, chat_id, filename, content_type, file, size, sha256):
attachment_id, job_id = uuid4(), uuid4()
staging_key = "attachments/staging/" + attachment_id.hex
final_key = "attachments/" + attachment_id.hex + "/v1/original"
# Журнал фиксируется до PUT: crash до записи job оставит управляемый orphan.
with db.connection() as conn:
storage.require_member(conn, user_id, chat_id)
conn.execute("INSERT INTO attachment_uploads(id, object_key) VALUES (%s, %s)", (attachment_id, staging_key))
with db.connection() as conn:
conn.execute("SET LOCAL idle_in_transaction_session_timeout='30s'")
lock_upload(conn, staging_key)
if not conn.execute("SELECT 1 FROM attachment_uploads WHERE id=%s", (attachment_id,)).fetchone():
raise ApiError(503, "unavailable", "Загрузка больше не активна")
object_store.put(staging_key, file, size, content_type)
row = conn.execute("""INSERT INTO attachments(id,chat_id,owner_id,object_key,filename,
content_type,size_bytes,sha256,status) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,'queued') RETURNING *""",
(attachment_id, UUID(chat_id), UUID(user_id), final_key, filename, content_type, size, sha256)).fetchone()
context = telemetry.carrier()
conn.execute("""INSERT INTO attachment_jobs(id,attachment_id,staging_key,max_attempts,traceparent,tracestate)
VALUES (%s,%s,%s,%s,%s,%s)""", (job_id, attachment_id, staging_key, jobs.MAX_ATTEMPTS,
context.get('traceparent'), context.get('tracestate')))
conn.execute("INSERT INTO attachment_outbox(job_id,attempt) VALUES (%s,1)", (job_id,))
conn.execute("DELETE FROM attachment_uploads WHERE id=%s", (attachment_id,))
result = public_attachment(row)
# Снимок queued даже если worker успеет закончить до отправки ответа.
telemetry.event('attachment.accepted', job_id=str(job_id), attachment_id=str(attachment_id))
return result
async def upload(request, chat_id, filename):
user_id = request.state.user["id"]
await run_in_threadpool(storage.get_chat, user_id, chat_id)
validate_filename(filename)
content_type = request.headers.get("Content-Type", "").split(";", 1)[0].strip().lower()
if content_type not in {"image/png", "image/jpeg"}:
raise ApiError(415, "unsupported_media_type", "Ожидаются сырые bytes PNG/JPEG")
length = request.headers.get("Content-Length")
if length is not None:
if not length.isascii() or not length.isdecimal():
raise ApiError(400, "invalid_request", "Неверный Content-Length")
if len(length) > 10 or int(length) > MAX_BYTES:
raise ApiError(413, "payload_too_large", "Файл больше допустимого размера")
try:
await asyncio.wait_for(upload_slot.acquire(), timeout=20)
except TimeoutError:
raise ApiError(503, "unavailable", "Загрузки заняты; повторите позднее") from None
try:
with TemporaryFile() as file:
size = 0
digest = hashlib.sha256()
try:
async with asyncio.timeout(30):
async for chunk in request.stream():
size += len(chunk)
if size > MAX_BYTES:
raise ApiError(413, "payload_too_large", "Файл больше допустимого размера")
digest.update(chunk)
await run_in_threadpool(file.write, chunk)
except (TimeoutError, ClientDisconnect):
raise ApiError(400, "invalid_request", "Загрузка прервана или истёк таймаут") from None
true_type = await run_in_threadpool(validate_image, file, content_type)
return await run_in_threadpool(save, user_id, chat_id, filename, true_type, file, size, digest.hexdigest())
finally:
upload_slot.release()
def find(user_id, attachment_id):
with db.connection() as conn:
row = conn.execute("""SELECT a.* FROM attachments a JOIN chat_members m ON m.chat_id=a.chat_id
WHERE a.id=%s AND m.user_id=%s""", (UUID(attachment_id), UUID(user_id))).fetchone()
if row is None:
raise ApiError(404, "not_found", "Вложение не найдено")
return row
def chunks(response):
try:
while chunk := response.read(65536):
yield chunk
finally:
response.close()
response.release_conn()
+95
View File
@@ -0,0 +1,95 @@
"""Basic только для входа; Bearer-сессии с TTL и AOF находятся в Valkey."""
import base64
import binascii
from contextlib import contextmanager
from datetime import datetime, timedelta, timezone
import hashlib
import json
import secrets
import redis
from src import passwords, storage
from src.config import SESSION_TTL, secret
from src.errors import ApiError
sessions = redis.Redis.from_url(
secret("CACHE_URL"), decode_responses=True,
socket_connect_timeout=0.5, socket_timeout=0.5, max_connections=10,
retry=redis.retry.Retry(redis.backoff.NoBackoff(), 0),
)
@contextmanager
def available():
try:
yield
except redis.RedisError:
raise ApiError(503, "unavailable", "Valkey временно недоступен") from None
def unauthorized(basic=False):
challenge = 'Basic realm="messenger", charset="UTF-8"' if basic else 'Bearer realm="messenger"'
return ApiError(401, "unauthorized", "Неверная или отсутствующая идентификация", {"WWW-Authenticate": challenge})
def credential(header, scheme):
parts = (header or "").split()
if len(parts) != 2 or parts[0].lower() != scheme.lower():
raise unauthorized(scheme == "Basic")
return parts[1]
def basic_user(header):
value = credential(header, "Basic")
try:
decoded = base64.b64decode(value, validate=True).decode("utf-8")
username, password = decoded.split(":", 1)
except (ValueError, UnicodeError, binascii.Error):
raise unauthorized(True) from None
user = storage.find_login(username)
if user is None or not passwords.matches(user["password_hash"], password):
raise unauthorized(True)
return storage.public_user(user)
def session_key(token):
return "session:" + hashlib.sha256(token.encode()).hexdigest()
def create_session(user):
token = secrets.token_urlsafe(32)
expires = datetime.now(timezone.utc) + timedelta(seconds=SESSION_TTL)
value = {"user_id": user["id"], "expires_at": storage.utc(expires)}
with available():
sessions.set(session_key(token), json.dumps(value), ex=SESSION_TTL)
return {"token": token, "token_type": "Bearer", "expires_at": value["expires_at"], "user": user}
def current_user(header):
token = credential(header, "Bearer")
with available():
value = sessions.get(session_key(token))
if value is None:
raise unauthorized()
data = json.loads(value)
if datetime.fromisoformat(data["expires_at"].replace("Z", "+00:00")) <= datetime.now(timezone.utc):
raise unauthorized()
user = storage.find_user(data["user_id"])
if user is None:
raise unauthorized()
return user
def logout(header):
token = credential(header, "Bearer")
with available():
removed = sessions.delete(session_key(token))
if not removed:
raise unauthorized()
def ready():
with available():
sessions.ping()
+38
View File
@@ -0,0 +1,38 @@
"""Идемпотентное создание приватного bucket отдельной административной учёткой."""
import os
import time
from urllib.parse import urlsplit
from minio import Minio
from minio.error import MinioException, S3Error
import urllib3
from src.config import secret
def bootstrap():
endpoint = urlsplit(os.environ.get("S3_ENDPOINT", "http://s3:9000"))
client = Minio(endpoint.netloc, access_key=secret("S3_ROOT_USER"), secret_key=secret("S3_ROOT_PASSWORD"),
secure=endpoint.scheme == "https", region=os.environ.get("S3_REGION", "us-east-1"),
http_client=urllib3.PoolManager(timeout=urllib3.Timeout(connect=1, read=3), retries=False))
bucket = os.environ.get("S3_BUCKET", "messenger-attachments")
deadline = time.monotonic() + 60
while True:
try:
if not client.bucket_exists(bucket):
client.make_bucket(bucket)
print("Private bucket is ready; static IAM permits only authenticated accounts", flush=True)
return
except S3Error as error:
if error.code in {"BucketAlreadyOwnedByYou", "BucketAlreadyExists"}:
return
if time.monotonic() >= deadline:
raise RuntimeError("S3 bootstrap failed") from None
except (MinioException, urllib3.exceptions.HTTPError, OSError):
if time.monotonic() >= deadline:
raise RuntimeError("S3 bootstrap timed out") from None
time.sleep(1)
if __name__ == "__main__":
bootstrap()
+46
View File
@@ -0,0 +1,46 @@
"""Один durable Stream, consumer group; ACK только после эффекта/устойчивого retry."""
import json
import redis
from src.config import secret
from src import telemetry
STREAM = 'attachments:v1'
GROUP = 'image-workers'
def connect():
return redis.Redis.from_url(secret('BROKER_URL'), decode_responses=True,
socket_connect_timeout=2, socket_timeout=3,
retry_on_timeout=False)
def bootstrap(client):
try:
client.xgroup_create(STREAM, GROUP, id='0', mkstream=True)
except redis.ResponseError as error:
if 'BUSYGROUP' not in str(error):
raise
def publish(client, job):
payload = {'schema_version': 1, 'job_id': str(job['id']), 'type': 'attachment.process.v1',
'attachment_id': str(job['attachment_id']), 'pipeline_version': job['pipeline_version'],
'attempt': job['attempt'], 'created_at': job['created_at'].isoformat().replace('+00:00', 'Z')}
with telemetry.span('attachments publish', kind=telemetry.SpanKind.PRODUCER,
parent={key: job[key] for key in ('traceparent', 'tracestate') if job.get(key)},
attributes={'messaging.system': 'redis', 'messaging.destination.name': STREAM,
'messaging.operation.type': 'send', 'job.attempt': job['attempt']}):
payload.update(telemetry.carrier())
result = client.xadd(STREAM, {'job': json.dumps(payload)})
telemetry.event('job.published', job_id=str(job['id']), attempt=job['attempt'])
return result
def acknowledge(client, message_id):
# Единственная группа: подтверждённые записи можно удалить. Pending не trim-им.
with client.pipeline(transaction=True) as pipe:
pipe.xack(STREAM, GROUP, message_id)
pipe.xdel(STREAM, message_id)
pipe.execute()
+87
View File
@@ -0,0 +1,87 @@
"""Очистка orphan PUT и staging только ready/просроченных failed; текущую работу не трогает."""
import argparse
import json
import logging
import os
import time
from src import db, object_store
from src.attachments import lock_upload
GRACE_SECONDS = int(os.environ.get("UPLOAD_CLEANUP_GRACE_SECONDS", "3600"))
if GRACE_SECONDS < 60:
raise ValueError("Безопасный grace period должен быть не меньше 60 секунд")
def cleanup_orphans():
result = {"removed": 0, "kept_ready": 0, "skipped_active": 0}
with db.connection() as conn:
candidates = conn.execute("""SELECT id, object_key FROM attachment_uploads
WHERE created_at < clock_timestamp() - %s * interval '1 second'
ORDER BY created_at LIMIT 100""", (GRACE_SECONDS,)).fetchall()
for row in candidates:
with db.connection() as conn:
if not lock_upload(conn, row["object_key"], blocking=False):
result["skipped_active"] += 1
continue
pending = conn.execute("""SELECT 1 FROM attachment_uploads WHERE id=%s
AND created_at < clock_timestamp() - %s * interval '1 second' FOR UPDATE""",
(row["id"], GRACE_SECONDS)).fetchone()
if not pending:
continue
ready = conn.execute("SELECT 1 FROM attachments WHERE object_key=%s", (row["object_key"],)).fetchone()
if ready:
result["kept_ready"] += 1
else:
object_store.remove(row["object_key"])
result["removed"] += 1
conn.execute("DELETE FROM attachment_uploads WHERE id=%s", (row["id"],))
return result
RETENTION = int(os.environ.get("STAGING_RETENTION_SECONDS", "86400"))
if RETENTION < 60:
raise ValueError("STAGING_RETENTION_SECONDS должен быть >=60")
def cleanup_staging():
removed = 0
with db.connection() as conn:
rows = conn.execute("""SELECT j.*,a.status,a.object_key FROM attachment_jobs j
JOIN attachments a ON a.id=j.attachment_id WHERE j.staging_deleted_at IS NULL
AND (a.status='ready' OR (a.status='failed' AND a.updated_at<clock_timestamp()-%s*interval '1 second'))
LIMIT 20 FOR UPDATE OF j,a SKIP LOCKED""", (RETENTION,)).fetchall()
for job in rows:
object_store.remove(job['staging_key'])
if job['status'] == 'failed':
prefix = job['object_key'].rsplit('/', 1)[0] + '/'
for variant in ('original', 'thumbnail', 'crop', 'optimized'):
object_store.remove(prefix + variant)
conn.execute('UPDATE attachment_jobs SET staging_deleted_at=clock_timestamp() WHERE id=%s', (job['id'],))
removed += 1
return removed
def cleanup_once():
return {**cleanup_orphans(), "staging_removed": cleanup_staging()}
if __name__ == "__main__":
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--once", action="store_true")
args = parser.parse_args()
db.pool.open(wait=True, timeout=15)
try:
while True:
try:
print(json.dumps(cleanup_once()), flush=True)
except Exception as error:
if args.once:
raise
logging.error("Cleanup postponed: %s", type(error).__name__)
if args.once:
break
time.sleep(30)
finally:
db.pool.close()
+20
View File
@@ -0,0 +1,20 @@
"""Настройки и чтение локальных Docker secrets."""
import os
import socket
from pathlib import Path
def secret(name):
return Path(os.environ[name + "_FILE"]).read_text().strip()
SESSION_TTL = int(os.environ.get("SESSION_TTL_SECONDS", "3600"))
if SESSION_TTL < 1:
raise ValueError("SESSION_TTL_SECONDS должен быть положительным")
if os.environ.get("SESSION_STORE", "valkey") != "valkey":
raise ValueError("В этой реализации SESSION_STORE=valkey")
INSTANCE_ID = socket.gethostname()
IDEMPOTENCY_TTL = int(os.environ.get("IDEMPOTENCY_TTL_SECONDS", "86400"))
if IDEMPOTENCY_TTL < 86400:
raise ValueError("IDEMPOTENCY_TTL_SECONDS должен быть не меньше 86400")
+63
View File
@@ -0,0 +1,63 @@
"""Ограниченный пул; каждый with connection() — отдельная транзакция."""
from contextlib import contextmanager
import psycopg
from psycopg.rows import dict_row
from psycopg_pool import ConnectionPool, PoolTimeout, TooManyRequests
from src.config import secret
from src.errors import ApiError
from src import telemetry
IO_TIMEOUT = 2.0
class TimedConnection(psycopg.Connection):
def execute(self, query, params=None, **kwargs):
# Only the SQL verb is allowed: never statement text or parameters.
verb = query.lstrip().split(None, 1)[0].upper() if isinstance(query, str) and query.strip() else 'QUERY'
if verb not in {'SELECT', 'INSERT', 'UPDATE', 'DELETE', 'SET'}:
verb = 'QUERY'
with telemetry.span('postgresql.' + verb, kind=telemetry.SpanKind.CLIENT,
attributes={'db.system.name': 'postgresql', 'db.operation.name': verb}, require_parent=True):
return super().execute(query, params, **kwargs)
def wait(self, gen, interval=0.1, timeout=None):
# statement_timeout работает на сервере и не помогает при docker pause.
# Ограничиваем само ожидание I/O: проверку пула, SQL, COMMIT и ROLLBACK.
timeout = IO_TIMEOUT if timeout is None else min(timeout, IO_TIMEOUT)
try:
return super().wait(gen, interval=interval, timeout=timeout)
except psycopg.OperationalError:
# Не ждём отмены/ROLLBACK от зависшего сервера. Пул заменит соединение.
self.close()
raise
pool = ConnectionPool(
conninfo=secret("DATABASE_URL"), open=False,
connection_class=TimedConnection,
min_size=1, max_size=5, max_waiting=20, timeout=2,
check=ConnectionPool.check_connection,
kwargs={
"row_factory": dict_row,
"connect_timeout": 2,
"options": "-c timezone=UTC -c statement_timeout=3000 -c lock_timeout=2000 -c idle_in_transaction_session_timeout=5000",
},
)
@contextmanager
def connection():
try:
with telemetry.span('postgresql.transaction', require_parent=True):
with pool.connection() as conn:
yield conn
except (psycopg.OperationalError, PoolTimeout, TooManyRequests):
raise ApiError(503, "unavailable", "PostgreSQL временно недоступен") from None
def ready():
with connection() as conn:
conn.execute("SELECT 1")
+27
View File
@@ -0,0 +1,27 @@
"""общий формат ошибок"""
from fastapi import Request
from fastapi.responses import JSONResponse
class ApiError(Exception):
def __init__(self, status: int, code: str, message: str, headers=None):
super().__init__(message)
self.status = status
self.code = code
self.message = message
self.headers = headers or {}
def error_response(request: Request, error: ApiError):
return JSONResponse(
status_code=error.status,
content={
"error": {
"code": error.code,
"message": error.message,
"request_id": request.state.request_id,
}
},
headers={"X-Request-Id": request.state.request_id, **error.headers},
)

Some files were not shown because too many files have changed in this diff Show More