ЛР 5 — БВТ2304 — Савельев — на 4 #59
+11
-20
@@ -1,20 +1,11 @@
|
||||
.git
|
||||
.github
|
||||
.env
|
||||
.env.*
|
||||
!.env.example
|
||||
secrets
|
||||
certs
|
||||
volumes
|
||||
data
|
||||
backups
|
||||
*.key
|
||||
*.pem
|
||||
*.p12
|
||||
*.pfx
|
||||
*.jks
|
||||
.venv
|
||||
__pycache__
|
||||
node_modules
|
||||
evidence
|
||||
.DS_Store
|
||||
*
|
||||
!Dockerfile
|
||||
!requirements.txt
|
||||
!src/
|
||||
!src/**
|
||||
!migrations/
|
||||
!migrations/**
|
||||
!research/
|
||||
!research/**
|
||||
**/__pycache__/
|
||||
**/*.py[cod]
|
||||
|
||||
+10
-2
@@ -3,11 +3,12 @@
|
||||
APP_HOST=0.0.0.0
|
||||
APP_PORT=8080
|
||||
APP_ENV=lab
|
||||
HOST_PORT=8080
|
||||
SESSION_TTL_SECONDS=3600
|
||||
# Choose credentials locally. File paths contain no secret values.
|
||||
DATABASE_URL_FILE=/run/secrets/database_url
|
||||
# Optional on 3, required use on 4 in lab2; Redis OR Valkey:
|
||||
SESSION_STORE=postgres
|
||||
# Shared sessions carried over from lab4 use Valkey.
|
||||
SESSION_STORE=valkey
|
||||
CACHE_URL_FILE=/run/secrets/cache_url
|
||||
API_REPLICAS=2
|
||||
IDEMPOTENCY_TTL_SECONDS=86400
|
||||
@@ -18,8 +19,15 @@ S3_ACCESS_KEY_ID_FILE=/run/secrets/s3_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE=/run/secrets/s3_secret_access_key
|
||||
MAX_UPLOAD_BYTES=10485760
|
||||
MAX_IMAGE_PIXELS=40000000
|
||||
UPLOAD_CLEANUP_GRACE_SECONDS=3600
|
||||
BROKER_URL_FILE=/run/secrets/broker_url
|
||||
WORKER_CONCURRENCY=2
|
||||
JOB_MAX_ATTEMPTS=5
|
||||
JOB_TIMEOUT_SECONDS=60
|
||||
STAGING_RETENTION_SECONDS=86400
|
||||
JOB_LEASE_SECONDS=90
|
||||
# Lease продлевается каждые min(10, lease/3) секунд. Общий timeout независим.
|
||||
# Для долгих задач JOB_TIMEOUT_SECONDS может быть больше JOB_LEASE_SECONDS.
|
||||
JOB_TASK_TIMEOUT_SECONDS=180
|
||||
JOB_BACKOFF_SECONDS=2
|
||||
WORKER_TASK_MEMORY_MB=640
|
||||
|
||||
@@ -1,18 +1,58 @@
|
||||
## Лабораторная 5
|
||||
|
||||
Автор / группа: TODO
|
||||
Целевая оценка (3/4/5): TODO
|
||||
Трек 5, если заявлен: TODO
|
||||
Автор / группа: Савельев Иван / БВТ2304
|
||||
Целевая оценка (3/4/5): **4** — выполнены требования уровней 3 и 4.
|
||||
Трек 5, если заявлен: не заявлен.
|
||||
|
||||
Что реализовано и что перенесено: TODO
|
||||
Команды запуска из чистого clone: TODO
|
||||
Результаты проверок и ссылка на REPORT.md: TODO
|
||||
Ограничения / известные ошибки: TODO
|
||||
Источники и использование ИИ: TODO
|
||||
### Что реализовано и что перенесено
|
||||
|
||||
- [ ] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
|
||||
- [ ] Контракт и публичные тесты не ослаблены.
|
||||
- [ ] make check и make test выполнены; приложен реальный вывод.
|
||||
- [ ] Сценарии отказов/безопасности своего уровня воспроизведены.
|
||||
- [ ] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
|
||||
- [ ] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
|
||||
Из lab4 перенесены API мессенджера, авторизация и сессии, ACL, идемпотентность сообщений, миграции, приватный S3, две API-реплики за HAProxy, Docker-конфигурация и собственные тесты.
|
||||
|
||||
- Загрузка возвращает `202 queued` после устойчивого сохранения оригинала в S3 staging и транзакции metadata + job + outbox. Статус доступен через GET; content до готовности возвращает 409.
|
||||
- Добавлены отдельные Valkey Streams broker с AOF и volume, outbox publisher и worker. Доставка at least once, ACK после устойчивого результата; дубли не создают второй набор объектов.
|
||||
- Worker сохраняет неизменённый оригинал, JPEG thumbnail, центрированный crop 128×128 и optimized WebP по контракту, с учётом EXIF orientation и прозрачности.
|
||||
- Реализованы пять автоматических попыток, backoff с jitter, lease/recovery, безопасные коды ошибок, реестр failed и явный replay с audit. Ready-задания повторно не запускаются.
|
||||
- Добавлены ограничения bytes/pixels/времени/памяти, ограниченная конкуренция и очистка staging после фиксации результата либо истечения retention failed.
|
||||
|
||||
### Запуск из чистого clone
|
||||
|
||||
Нужны Docker Engine с Compose v2, Python 3.11, Bash и make. Команды из корня репозитория:
|
||||
|
||||
```sh
|
||||
python3 -m venv .venv
|
||||
.venv/bin/python -m pip install -r requirements.txt -r requirements-dev.txt
|
||||
make check validate PYTHON=.venv/bin/python
|
||||
./scripts/start.sh
|
||||
./scripts/check.sh
|
||||
|
||||
# Полная проверка на отдельном временном Docker Compose project:
|
||||
./scripts/check-docker.sh
|
||||
|
||||
# Остановка обычного стенда с сохранением volumes:
|
||||
./scripts/stop.sh
|
||||
```
|
||||
|
||||
`start.sh` создаёт локальные `.env` и secrets, собирает приложение, выполняет миграции и bootstrap, ждёт готовности обеих API-реплик. API доступен на `http://localhost:8080`.
|
||||
|
||||
### Результаты проверок
|
||||
|
||||
Проверено 12.11.2026: `make check validate test`, 39 собственных API-тестов и все 17 групп полного сценария прошли. Воспроизведены остановка worker и рестарты API/broker, оба разрыва outbox, crash worker после PUT и перед ACK, дубли, исчерпание retry/replay, постоянная ошибка, timeout/MemoryError, очистка staging, недоступность broker, persistence и failover API. Отдельно подтверждён фактический restart всех 11 постоянных контейнеров с сохранением их ID и volumes.
|
||||
|
||||
- [Отчёт, архитектура, команды и ограничения](../REPORT.md).
|
||||
- [Полный вывод проверок](../evidence/lab5-checks.txt) и [JSON результатов](../evidence/lab5-checks.json).
|
||||
- [Доказательство рестартов](../evidence/lab5-restart.json) и [вывод persistence](../evidence/lab5-restart.txt).
|
||||
|
||||
### Ограничения / известные ошибки
|
||||
|
||||
Стенд учебный, на одном Docker-host; потеря host/disk/volumes и ручное удаление объектов не покрываются. Поддерживается pipeline_version=1; исследовательские треки на 5 не реализованы. При длительном простое worker outbox и staging требуют свободного диска; общей квоты принятой работы нет. Failed staging хранится 24 часа, после удаления replay запрещён. Превышение ресурсов обработки заканчивается failed после ограниченного числа попыток. Старые ready-вложения lab4 остаются доступны без автоматического создания вариантов.
|
||||
|
||||
### Источники и использование ИИ
|
||||
|
||||
Основа — ТЗ, OpenAPI, `attachment-states.md` и `jobs.schema.json` этого репозитория; официальная документация Redis Streams/XAUTOCLAIM, Valkey persistence, Pillow ImageOps и Python resource. Ссылки приведены в [REPORT.md](../REPORT.md#первичные-источники).
|
||||
|
||||
- [x] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
|
||||
- [x] Контракт и публичные тесты не ослаблены.
|
||||
- [x] make check и make test выполнены; приложен реальный вывод.
|
||||
- [x] Сценарии отказов/безопасности своего уровня воспроизведены.
|
||||
- [x] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
|
||||
- [x] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
name: Application
|
||||
on: [push, pull_request]
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
docker:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 20
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.11'
|
||||
- run: python -m venv .venv
|
||||
- run: .venv/bin/python -m pip install -r requirements.txt -r requirements-dev.txt
|
||||
- run: ./scripts/check-docker.sh
|
||||
- uses: actions/upload-artifact@v4
|
||||
if: always()
|
||||
with:
|
||||
name: lab5-evidence
|
||||
path: |
|
||||
evidence/lab5-checks.txt
|
||||
evidence/lab5-checks.json
|
||||
+28
@@ -0,0 +1,28 @@
|
||||
FROM python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
|
||||
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
APP_HOST=0.0.0.0 \
|
||||
APP_PORT=8080
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY requirements.txt .
|
||||
RUN python -m pip install --no-cache-dir -r requirements.txt \
|
||||
&& python -m pip check
|
||||
|
||||
RUN groupadd --gid 10001 app \
|
||||
&& useradd --uid 10001 --gid app --no-create-home app
|
||||
|
||||
COPY src/ ./src/
|
||||
COPY migrations/ ./migrations/
|
||||
COPY research/ ./research/
|
||||
|
||||
USER 10001:10001
|
||||
EXPOSE 8080
|
||||
STOPSIGNAL SIGTERM
|
||||
|
||||
HEALTHCHECK --interval=2s --timeout=3s --start-period=5s --retries=3 \
|
||||
CMD ["python", "-m", "src.healthcheck"]
|
||||
|
||||
CMD ["python", "-m", "src"]
|
||||
+154
@@ -0,0 +1,154 @@
|
||||
# Карта проекта для подготовки к защите
|
||||
|
||||
Составлена по исходному коду 13.11.2026. Ссылки вида `src/jobs.py:29` обозначают файл и начальную строку функции; при изменении исходников номера нужно перепроверять.
|
||||
|
||||
## Что это
|
||||
|
||||
Учебный HTTP-мессенджер на Python/FastAPI. Лабораторная 5 добавляет очередь и асинхронную обработку изображений; в REPORT.md заявлена оценка 4. Возможности предыдущих работ сохранены: пользователи, сессии, чаты, сообщения, вложения, пагинация и идемпотентность сообщений.
|
||||
|
||||
README.md описывает исходное задание и называет репозиторий skeleton. Фактическая реализация уже есть в src/ и compose.yaml. При ответах о поведении ориентироваться прежде всего на код. Ссылки в отчёте на REPORT.lab2/3/4.md не означают, что эти файлы присутствуют в текущем дереве.
|
||||
|
||||
```text
|
||||
Клиент → HAProxy → FastAPI ×2 → PostgreSQL (данные, jobs, outbox)
|
||||
→ Valkey (сессии)
|
||||
→ SeaweedFS S3 (staging)
|
||||
PostgreSQL outbox → publisher → отдельный Valkey Streams broker
|
||||
↓
|
||||
worker ×2 процесса
|
||||
↓
|
||||
subprocess Pillow → S3 результаты
|
||||
↓
|
||||
PostgreSQL ready → ACK
|
||||
cleanup → незавершённые загрузки и ненужный staging
|
||||
```
|
||||
|
||||
## Как запускается проект
|
||||
|
||||
Команда из корня в Bash/WSL/Git Bash при доступном Docker Compose: `./scripts/start.sh`.
|
||||
|
||||
1. `scripts/start.sh:5`: вызывает `python3 scripts/init_local.py`.
|
||||
2. `scripts/init_local.py:12`: создаёт отсутствующие .env и secrets, сохраняет существующие значения; генерирует конфигурацию доступа к S3.
|
||||
3. `scripts/start.sh:6`: `docker compose up -d --build lb cleanup publisher worker`. Остальные сервисы поднимаются через depends_on.
|
||||
4. `compose.yaml:90`: API собирается из Dockerfile, по умолчанию две реплики. PostgreSQL/сессии/миграции/S3/bootstrap перечислены в зависимостях API.
|
||||
5. `compose.yaml:107` → `src/migrate.py:20`: отдельный migrate применяет SQL по порядку, под advisory lock, запоминает SHA-256. `src/migrate.py:11`: API только проверяет применённую схему.
|
||||
6. `compose.yaml:144` → `src/bootstrap_s3.py:13`: подготовка приватного bucket.
|
||||
7. `Dockerfile:28`: команда контейнера API `python -m src`.
|
||||
8. `src/__main__.py:8`: Uvicorn загружает `src.app:app`, host/port из окружения, один процесс на контейнер.
|
||||
9. `src/app.py:24`: lifespan открывает пул БД и проверяет миграции; `src/app.py:33` создаёт FastAPI.
|
||||
10. `compose.yaml:169`: HAProxy публикуется на 127.0.0.1:8080; `infra/haproxy.cfg:34` — backend с roundrobin и проверкой readiness.
|
||||
11. `scripts/start.sh:7` → `scripts/wait-ready.sh:5`: ожидание стенда через `scripts/replicas.py wait`.
|
||||
|
||||
Остановка: `./scripts/stop.sh`; перезапуск: `./scripts/restart.sh`. Worker: `scripts/stop-worker.sh`, `scripts/start-worker.sh`. Остановка контейнеров с сохранением volumes отличается от удаления volumes.
|
||||
|
||||
## Общий путь HTTP-запроса
|
||||
|
||||
- `src/app.py:38`: middleware создаёт request_id, определяет пользователя (Basic для входа, Bearer для остальных защищённых API), проверяет JSON, добавляет X-Request-Id и X-Instance-Id, пишет время/статус в лог.
|
||||
- `src/models.py:8`: Pydantic-модели и проверки входных данных; модели пользователя/чата/сообщения начинаются со строк 21/27/40.
|
||||
- `src/app.py:77`: обработчики ошибок; неверные параметры дают контрактный 400, вместо стандартного FastAPI 422.
|
||||
- `src/errors.py:7`: ApiError; `src/errors.py:16`: общий формат HTTP-ошибки.
|
||||
- `src/db.py:28`: пул соединений; `src/db.py:42`: контекст отдельной транзакции, COMMIT при успешном выходе, rollback при ошибке.
|
||||
|
||||
## Регистрация, вход, сессия
|
||||
|
||||
- Регистрация: `src/app.py:112` → `src/passwords.py:12` (Argon2id) → `src/storage.py:35` (INSERT users; занятый username — 409).
|
||||
- Вход: middleware `src/app.py:38` → `src/auth.py:44` (Basic username/password) → `src/app.py:180` → `src/auth.py:61` (случайный Bearer token, запись в Valkey с TTL).
|
||||
- В Valkey ключ `session:SHA256(token)`, значение — user_id/expires_at: `src/auth.py:57`. Default TTL — 3600 с: `src/config.py:11`.
|
||||
- Защищённый запрос: `src/auth.py:70` читает сессию, проверяет срок и существование пользователя.
|
||||
- Выход: `src/app.py:185` → `src/auth.py:85`, удаление сессии.
|
||||
- Valkey сессий и Valkey broker — разные сервисы: `compose.yaml:38` и `compose.yaml:50`.
|
||||
|
||||
## Чаты, сообщения и доступ
|
||||
|
||||
- Создание чата: `src/app.py:122` → `src/storage.py:57`, чат и участники в одной транзакции.
|
||||
- Проверка членства: `src/storage.py:73`. Для недоступного чата возвращается 404.
|
||||
- Создание сообщения: `src/app.py:138` проверяет Idempotency-Key → `src/storage.py:106` проверяет доступ, вложения, сохраняет сообщение и результат повтора.
|
||||
- Идемпотентность сообщений: advisory lock по пользователю/пути чата/ключу, таблица message_idempotency. Тот же ключ и тело возвращают прежнее сообщение; другое тело — 409. TTL ≥24 ч: `src/config.py:18`.
|
||||
- queued/processing можно прикреплять к сообщению; failed — 409: `src/storage.py:106`.
|
||||
- История: `src/app.py:175` → `src/storage.py:144`. Чаты: `src/storage.py:87`.
|
||||
- Keyset-пагинация по (created_at,id), LIMIT size+1; cursor содержит scope/дату/UUID: `src/pagination.py:12`, `src/pagination.py:33`. Это не OFFSET и не snapshot между страницами.
|
||||
|
||||
## Загрузка изображения и ответ 202
|
||||
|
||||
`src/app.py:146` → `src/attachments.py:99` → `src/attachments.py:73`.
|
||||
|
||||
1. Проверка членства, имени, MIME; поток сырых PNG/JPEG, не multipart.
|
||||
2. Ограничения: до 10 MiB, до 40 млн пикселей; фактический формат, verify и полное декодирование: `src/attachments.py:19`, `src/attachments.py:47`. API проверяет изображение, но не генерирует варианты.
|
||||
3. Временный файл и SHA-256; семафор ограничивает приём/проверку одной загрузкой на API-реплику.
|
||||
4. Отдельно фиксируется attachment_uploads — журнал до S3 PUT.
|
||||
5. Под advisory lock выполняется PUT в `attachments/staging/<UUID>`.
|
||||
6. В одной транзакции: attachments(status=queued), attachment_jobs, attachment_outbox; удаление строки журнала.
|
||||
7. Ответ 202 после завершённого PUT и COMMIT. Возвращается снимок queued, даже если worker уже успел закончить.
|
||||
|
||||
Важная граница: 202 означает устойчивый приём и сохранённое намерение обработки; готовность проверяется отдельно. В PostgreSQL хранятся метаданные, байты — в S3. TemporaryFile не является durable staging.
|
||||
|
||||
## Outbox и брокер
|
||||
|
||||
- `src/publisher.py:8`: выбирает доступную неопубликованную запись outbox с FOR UPDATE SKIP LOCKED → XADD → published_at → COMMIT.
|
||||
- Сбой после COMMIT upload до публикации: запись остаётся в outbox. Сбой после XADD до COMMIT publisher: возможен повтор XADD.
|
||||
- `src/broker.py:26`: JSON содержит ID, версию, attempt, время; без байтов, паролей и клиентских путей.
|
||||
- `src/broker.py:18`: XGROUP CREATE MKSTREAM с началом 0, stream attachments:v1, группа image-workers.
|
||||
- `compose.yaml:50`: AOF appendfsync always, volume broker_data, noeviction.
|
||||
- `src/broker.py:33`: ACK и XDEL в одной транзакции брокера; здесь только одна consumer group.
|
||||
|
||||
## Worker, состояния, защита от дублей
|
||||
|
||||
- `src/worker.py:166`: supervisor запускает до двух consumer-процессов и заменяет умершие.
|
||||
- `src/worker.py:140`: XAUTOCLAIM старых pending, затем XREADGROUP новых, count=1 (prefetch=1).
|
||||
- `src/worker.py:89`: валидация сообщения → `src/jobs.py:29` (claim).
|
||||
- Claim блокирует job и attachment, проверяет версию/attempt/статус, выставляет lease_token/lease_until и processing.
|
||||
- `src/worker.py:40`: читает staging, сверяет размер/SHA-256, запускает отдельный subprocess Pillow с timeout.
|
||||
- Перед финальными PUT повторно блокирует строки, проверяет lease_token и срок аренды. Старый исполнитель не публикует результат после потери аренды.
|
||||
- Ключи детерминированы: `attachments/<UUID>/v1/{original,thumbnail,crop,optimized}`. Повтор PUT пишет в те же ключи.
|
||||
- После записи всех объектов одна транзакция фиксирует ready, variants и audit; затем ACK.
|
||||
- Дубль ready: `src/worker.py:28` проверяет состав вариантов и размеры объектов, после чего ACK; обработки заново нет.
|
||||
- Состояния: queued → processing → ready; временная ошибка → queued; постоянная ошибка/исчерпанный бюджет → failed.
|
||||
- Это at-least-once доставка и идемпотентный эффект. XADD/PUT могут повторяться; exactly-once доставки нет.
|
||||
|
||||
## Retry, зависшие работы, replay
|
||||
|
||||
- Исправление 13.11.2026: `src/jobs.py:51` продлевает действующий lease по токену. Consumer вызывает heartbeat каждые min(10, lease/3) секунд через `src/task_supervisor.py:49` и выполняет работу в `src/job_runner.py:9`.
|
||||
- Срок аренды не ограничивает общую длительность: разрешено TIMEOUT > LEASE. Независимый JOB_TASK_TIMEOUT_SECONDS (default 180 с) останавливает всё дерево задачи при зависании; ошибка продления/потеря аренды тоже останавливает обработку. Resize timeout остаётся отдельным.
|
||||
- Локальные сценарии supervisor: `student-tests/test_task_supervisor.py`. Полный Docker-прогон новой версии ещё требуется; старые evidence не подтверждают heartbeat.
|
||||
- Defaults: 5 попыток, timeout subprocess 60 с, lease 90 с, backoff base 2 с: `src/jobs.py:8`.
|
||||
- `src/jobs.py:72`: exponential backoff с jitter 0.5–1.5; базовая экспонента ограничена 60 с до jitter. Постоянная ошибка сразу failed, исчерпание попыток — retry_exhausted.
|
||||
- `src/jobs.py:91`: отказ записывается только при совпадении lease_token.
|
||||
- `src/jobs.py:98`: publisher вызывает recover для истёкших processing; причина worker_lost, ограниченный retry.
|
||||
- `src/jobs.py:111`: replay только failed с неудалённым staging; тот же job_id, attempt растёт, новый ограниченный бюджет и audit.
|
||||
- Реестр failed вместо отдельной DLQ: `src/job_admin.py:9`. Команды: `docker compose exec worker python -m src.job_admin failed` и `docker compose exec worker python -m src.job_admin replay <job_id>`.
|
||||
|
||||
## Варианты изображения
|
||||
|
||||
`src/image_pipeline.py:22`: EXIF orientation, белый фон прозрачности, RGB, удаление metadata.
|
||||
|
||||
| Объект | Правило |
|
||||
| --- | --- |
|
||||
| original | Исходные байты без перекодирования |
|
||||
| thumbnail | JPEG, вписать в 256×256, пропорции сохранены, без увеличения, quality=85 |
|
||||
| crop | JPEG 128×128, центрированный crop, увеличение возможно, quality=85 |
|
||||
| optimized | WebP, вписать в 1280×1280, без увеличения, quality=80, method=4 |
|
||||
|
||||
`src/image_pipeline.py:14`: размеры округляются вниз, минимум 1 px. `src/image_pipeline.py:58`: лимиты адресной памяти и CPU subprocess; wall timeout — `src/worker.py:40`.
|
||||
|
||||
## Чтение и очистка
|
||||
|
||||
- Metadata: `src/app.py:151` → `src/attachments.py:136`, доступ по членству в чате.
|
||||
- Content: `src/app.py:156`: не-ready — 409; отсутствующий вариант ready — 404; StreamingResponse через API без redirect. `src/attachments.py:145` закрывает S3 response.
|
||||
- `src/object_store.py:16`: приватный bucket через Minio SDK к SeaweedFS; `src/object_store.py:42` — PUT.
|
||||
- `src/cleanup.py:17`: orphan-журнал старше grace period; та же advisory lock, активные загрузки пропускаются.
|
||||
- `src/cleanup.py:48`: staging ready удаляется, failed хранится по retention (default сутки); queued/processing не удаляются. После удаления failed staging replay запрещён.
|
||||
|
||||
## Схема, диагностика, проверки
|
||||
|
||||
- `migrations/001_initial.sql`: пользователи/чаты/участники/сообщения; 002 — идемпотентность; 003 — вложения/журнал/связи; 004 — статусы/jobs/outbox/events.
|
||||
- `docs/schema.md`: описание схемы и транзакционных границ.
|
||||
- `src/app.py:99`: live без зависимостей; `src/app.py:104`: ready проверяет PostgreSQL, сессии и bucket. Broker/worker не входят в readiness API: намерение принимается через outbox при их простое.
|
||||
- `src/config.py:7`: секреты читаются из файлов, заданных переменными *_FILE; значения секретов не выводить при подготовке.
|
||||
- `make check`: структура/контракт без запуска приложения; `make validate`: OpenAPI; `make test`: smoke работающего API.
|
||||
- `scripts/check-docker.sh` → `student-tests/lab5_pipeline.py:22`: полный изолированный Docker-сценарий, включая падения и восстановление.
|
||||
- `student-tests/lab5_pipeline.py`: offline worker/рестарты, долгий runner с heartbeat, зависший runner с общим timeout, падение publisher/worker, retry/replay, лимиты subprocess, cleanup и offline broker. После изменений искать сценарии по комментариям и record(...).
|
||||
- `src/faults.py:9`: одноразовые failpoints, включаются только ENABLE_FAULT_INJECTION=1; crash до/после публикации/эффекта/ACK, timeout/memory/transient.
|
||||
- `evidence/lab5-checks.json`, `evidence/lab5-checks.txt`, `evidence/lab5-restart.json`: сохранённые результаты прошлых запусков, не подтверждение текущего живого стенда.
|
||||
|
||||
## Формат быстрых ответов
|
||||
|
||||
Сначала прямой ответ на вопрос в 1–3 предложениях. Ниже короткое пояснение и 1–3 ссылки на соответствующие строки кода. Термины объяснять по необходимости. По запросу о запуске показывать цепочку start.sh → Compose → Dockerfile → __main__.py → app/lifespan. При изменениях проверять номера строк заново.
|
||||
@@ -1,37 +1,191 @@
|
||||
# Отчёт — лабораторная 5
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
- Автор, группа: Савельев Иван, БВТ2304.
|
||||
- Целевая оценка: **4** — все требования уровней 3 и 4. Исследовательский трек оценки 5 не заявлен.
|
||||
- Стенд: Docker Compose; HAProxy, две реплики FastAPI, PostgreSQL, Valkey для сессий, SeaweedFS S3; отдельные Valkey Streams broker, outbox publisher и worker.
|
||||
- Commit/PR: после публикации. Evidence содержит source_commit и working_tree=true: проверялось рабочее дерево поверх указанного commit.
|
||||
- Использование ИИ: перенос базы, код, проверки и отчёт подготовлены с помощью Codex. Результаты проверок получены реальным запуском Docker, а не предполагаются по коду.
|
||||
|
||||
## Запуск из чистого clone
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
Нужны Docker Engine с Compose v2, Python 3.11, Bash и make. Все команды выполняются из корня репозитория. Внешние заранее установленные БД/брокер/S3 не нужны.
|
||||
|
||||
## Архитектура и решения
|
||||
```sh
|
||||
python3 -m venv .venv
|
||||
.venv/bin/python -m pip install -r requirements.txt -r requirements-dev.txt
|
||||
make check validate PYTHON=.venv/bin/python
|
||||
./scripts/start.sh
|
||||
./scripts/check.sh
|
||||
```
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab4, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
`start.sh` создаёт локальные `.env` и `secrets/`, собирает образ, выполняет миграции и bootstrap приватного bucket, поднимает сервисы и ждёт обе API-реплики через LB. Существующие настройки и пароли сохраняются. `.env`, secrets, виртуальное окружение и runtime data исключены из Git; Docker context содержит только исходники, зависимости и миграции.
|
||||
|
||||
API: http://localhost:8080. Если порт занят, задайте `HOST_PORT=8081` в `.env`; HTTP-проверки запускайте с `BASE_URL=http://localhost:8081`. Compose project `mtusi-lab5`, образы `mtusi-messenger:lab5` и volumes отделены от предыдущих лабораторных. Локальные credentials генерируются заново, файлы из lab4 не копируются.
|
||||
|
||||
```sh
|
||||
# Полный автоматический сценарий на отдельном временном project и свободном порту:
|
||||
./scripts/check-docker.sh
|
||||
|
||||
# Обычный стенд, остановка без потери данных:
|
||||
./scripts/stop.sh
|
||||
./scripts/start.sh
|
||||
|
||||
# Перезапуск всех постоянных процессов и хранилищ с сохранением volumes:
|
||||
./scripts/restart.sh
|
||||
|
||||
# Только для намеренного удаления учебных данных ТЕКУЩЕГО Compose project:
|
||||
docker compose down --volumes
|
||||
```
|
||||
|
||||
Полный сценарий создаёт синтетические данные в отдельном `lab5-check-<random>` и удаляет только его контейнеры/volumes. Пишет `evidence/lab5-checks.txt` и `evidence/lab5-checks.json`. Для диагностики сбоя можно явно задать `KEEP_TEST_STACK=1`; это оставит только неуспешный временный стенд. Основной стенд не затрагивается.
|
||||
|
||||
## Что перенесено и изменено
|
||||
|
||||
Из lab4 сохранены регистрация/Argon2id, Basic-вход и Bearer-сессии в Valkey, ACL чатов и вложений, сообщения и keyset-пагинация, X-Request-Id/X-Instance-Id, общая идемпотентность сообщений, две API-реплики за LB, приватный S3, лимиты загрузки, миграции и эксплуатационные скрипты. Предыдущие отчёты — `REPORT.lab2.md`, `REPORT.lab3.md`, `REPORT.lab4.md`; старые evidence относятся к соответствующим работам.
|
||||
|
||||
Объявленный переход API: upload возвращает **202 со снимком queued**, а клиент опрашивает GET metadata. Queued/processing уже можно прикрепить к сообщению в том же чате. Для failed новое прикрепление даёт 409. Content любого не-ready вложения даёт 409, отсутствующий вариант ready — 404. Оригинал скачивается побайтно без перекодирования и redirect; API не раскрывает storage keys/credentials. Контракт, README задания и публичные tests не изменялись.
|
||||
|
||||
Миграции 001–003 сохранены. Миграция 004 добавляет состояния/metadata вариантов и таблицы jobs, outbox, audit событий. Старые ready-вложения сохраняют свои ключи и читаются после обновления; варианты у старых записей не создаются автоматически. Схема описана в `docs/schema.md`.
|
||||
|
||||
## Архитектура и границы устойчивости
|
||||
|
||||
```text
|
||||
Клиент → HAProxy → API ×2 → PostgreSQL: metadata + job + outbox
|
||||
→ приватный S3 staging
|
||||
|
||||
PostgreSQL outbox → publisher → Valkey Streams (AOF, volume)
|
||||
↓ XREADGROUP, prefetch=1
|
||||
worker (2 процесса)
|
||||
↓
|
||||
отдельный resize subprocess
|
||||
↓
|
||||
S3 original + thumbnail + crop + optimized
|
||||
↓
|
||||
PostgreSQL COMMIT ready → broker ACK
|
||||
|
||||
cleanup → orphan PUT journal + staging только ready/просроченных failed
|
||||
```
|
||||
|
||||
PostgreSQL — источник истины для намерения обработки и состояния. Valkey Streams обеспечивает at-least-once доставку. Это отдельный service `broker`, не Pub/Sub и не тот же экземпляр, что сессии. Stream `attachments:v1`, группа `image-workers`, начало чтения `0`; topology создаётся идемпотентным XGROUP CREATE MKSTREAM. Сообщения не trim-ятся до ACK. После XACK запись удаляется XDEL в одной broker transaction; используется одна группа.
|
||||
|
||||
1. API принимает все bytes с ограничением размера, проверяет имя, MIME и настоящий PNG/JPEG. Resize/генерация вариантов в API не выполняются.
|
||||
2. Перед PUT фиксируется журнал `attachment_uploads`. Полный проверенный оригинал записывается в S3 `attachments/staging/<UUID>`. Это тот же приватный S3, но отдельный staging prefix; на этом этапе конечного original/вариантов ещё нет.
|
||||
3. Одна транзакция PostgreSQL вставляет queued metadata, job и outbox и удаляет запись журнала. **202 отправляется после PUT и успешного COMMIT.** Локальный TemporaryFile нужен только во время HTTP-приёма, устойчивым staging не является.
|
||||
4. Отдельный publisher читает due outbox с блокировкой SKIP LOCKED и делает XADD. Broker настроен AOF `appendfsync always`, volume, `noeviction`. Успешный XADD подтверждает приём брокером с этой настройкой persistence; это не consumer ACK. После ответа dispatcher отмечает published_at.
|
||||
5. Worker захватывает job транзакционно: status=processing, случайный lease_token и deadline. Данные для скачивания берёт из БД, а не из сообщения/filename. Проверяет размер и SHA-256 staging. Записывает original и все три варианта, затем в одной транзакции ставит ready и metadata/checksums. Только после COMMIT подтверждает доставку.
|
||||
|
||||
В сообщении только поля `jobs.schema.json`: schema_version, job_id, type, attachment_id, pipeline_version=1, attempt, created_at. Bytes, URL, credentials, filename и storage key отсутствуют. job_id постоянен при redelivery и managed retry; attempt начинается с 1 и меняется при новой управляемой попытке. Native stream ID — отдельный идентификатор доставки.
|
||||
|
||||
## Идемпотентность, retry и replay
|
||||
|
||||
Ключ эффекта `(attachment_id, pipeline_version, variant)` реализован детерминированными объектами:
|
||||
|
||||
```text
|
||||
attachments/<UUID>/v1/original
|
||||
attachments/<UUID>/v1/thumbnail
|
||||
attachments/<UUID>/v1/crop
|
||||
attachments/<UUID>/v1/optimized
|
||||
```
|
||||
|
||||
Число объектов конечного результата — четыре. Дубль queued/processing не захватывает уже арендованную job. Устаревшая попытка не меняет актуальное состояние. Перед PUT и COMMIT worker блокирует строки job/attachment и сверяет lease_token и deadline. При повторе после незавершённого PUT те же детерминированные объекты могут записаться ещё раз, второй набор не создаётся. Дубль ready проверяет полную metadata и размеры объектов через S3 HEAD, затем ACK, не меняя status/updated_at и не выполняя resize. В этой работе поддерживается только pipeline_version=1; трек versioned pipeline на 5 не заявлен.
|
||||
|
||||
Временные ошибки S3, resource limit и timeout дают queued с новым due outbox. Backoff: `uniform(0.5,1.5) × min(60, JOB_BACKOFF_SECONDS × 2^(attempt-1))`; интервал сохраняется в PostgreSQL. Default base=2 s, jitter присутствует. Автоматический бюджет — `JOB_MAX_ATTEMPTS=5`. При исчерпании — failed/error_code=retry_exhausted; последняя безопасная причина сохраняется в job. Нарушение checksum/размера staging или невалидное изображение — постоянный invalid_image, сразу failed без повторов. Тексты исключений и stack traces не передаются в API/registry.
|
||||
|
||||
Publisher также восстанавливает истёкший processing lease: записывает worker_lost и создаёт следующую попытку либо failed при исчерпании бюджета. Это не бесконечный retry. Pending-доставки умершего consumer забираются XAUTOCLAIM после lease timeout. Если recovery уже создал новую попытку, старое сообщение становится obsolete и получает ACK; намерение следующей попытки уже устойчиво сохранено.
|
||||
|
||||
Реестр failed и явный replay:
|
||||
|
||||
```sh
|
||||
docker compose exec -T worker python -m src.job_admin failed
|
||||
docker compose exec -T worker python -m src.job_admin replay <job_id>
|
||||
```
|
||||
|
||||
Replay разрешён только для failed с ещё сохранённым staging. Он сохраняет job_id/ключ эффекта, записывает operator_replay в audit, увеличивает глобальный attempt и выделяет новый ограниченный бюджет из пяти попыток. Ready/queued/processing повторять этой командой нельзя; повтор ready завершается ошибкой без изменения БД/объектов. Глобальный attempt монотонен, поэтому старые сообщения не запускают replay как новую работу. После удаления failed staging replay запрещён: исходник надо загрузить заново. HTTP endpoint управления заданиями не добавлен.
|
||||
|
||||
## Правила изображений и ограничения ресурсов
|
||||
|
||||
- Оригинал остаётся исходными bytes, MIME, size_bytes и SHA-256 относятся к нему.
|
||||
- EXIF orientation применяется к производным изображениям. Прозрачность компонуется на белый; производные RGB, без EXIF/ICC/comment metadata.
|
||||
- Thumbnail: JPEG quality=85, вписывание в 256×256 без увеличения, целочисленное округление вниз с минимумом 1 px. Fixture 320×200 → 256×160.
|
||||
- Crop: JPEG quality=85, центрированный квадрат 128×128, LANCZOS; маленький исходник может увеличиваться.
|
||||
- Optimized: WebP quality=80, method=4; вписывание в 1280×1280 без увеличения. Уменьшение байтов каждого маленького файла не обещается.
|
||||
- Загрузка ≤10 MiB и ≤40 млн pixels. Limit проверяется до полной распаковки; Content-Length и фактический поток проверяются независимо. Corrupt/MIME spoof — 415, bytes/pixels overflow — 413. API допускает одну проверяемую загрузку на реплику, таймаут тела 30 s и ожидания слота 20 s.
|
||||
- Worker concurrency=2, каждый consumer получает максимум одну доставку за раз. Для заданного Compose memory budget разрешено concurrency 1 или 2.
|
||||
- Каждый resize выполняется отдельным subprocess: wall timeout 60 s, CPU limit 60/61 s, address-space limit 640 MiB (`RLIMIT_AS`). При timeout subprocess убивается; MemoryError/resource termination не убивает основной consumer и проходит bounded retry. Worker container ограничен 1536 MiB, 2 CPU и 96 PID.
|
||||
- После исправления 13.11.2026 lease (default 90 s, минимум 15 s) продлевается consumer каждые `min(10, lease/3)` секунд. Продление проверяет статус, lease_token и срок аренды; истёкшую или чужую аренду не восстанавливает. Исправная задача может выполняться дольше первоначального lease; resize timeout и lease больше не связаны условием `lease > timeout`.
|
||||
- Вся попытка, включая скачивание, resize и публикацию, выполняется отдельным job_runner. Consumer контролирует независимый `JOB_TASK_TIMEOUT_SECONDS` (default 180 s в Compose, не меньше resize timeout). При превышении убивает дерево runner + resize и фиксирует ограниченный retry. Heartbeat не работает бесконечно для зависшего runner. При потере аренды/ошибке продления дерево также останавливается; ACK без устойчивого результата не отправляется. В Linux runner получает сигнал при смерти consumer и завершает свою группу процессов.
|
||||
- Во время финальной транзакции PUT/COMMIT строки удерживаются под fence; heartbeat пропускает занятую блокировку, как и recovery. Общий timeout при этом продолжает действовать.
|
||||
|
||||
Если корректное преобразование может занимать 95 s, можно задать `JOB_TIMEOUT_SECONDS=120`, `JOB_LEASE_SECONDS=90`, `JOB_TASK_TIMEOUT_SECONDS=180`: lease будет продлеваться, а отдельные таймауты останутся конечными. Defaults по-прежнему разрешают resize только до 60 s; heartbeat не отменяет этот лимит.
|
||||
- S3 операции имеют connect timeout 1 s/read timeout 3 s, SDK retries отключены. PostgreSQL I/O, pool wait и блокировки ограничены. CPU/resize выполняется вне DB-транзакции; транзакция публикации результата включает только fencing, четыре ограниченных PUT и COMMIT.
|
||||
|
||||
`WORKER_TASK_MEMORY_MB` ограничивает адресное пространство процесса, а Compose mem_limit — весь контейнер. Это разные ограничения; поддержка 40 млн pixels не обещает, что любой формат/ориентация уложится в память: превышение ресурсов заканчивается наблюдаемым failed после ограниченного бюджета, а не бесконечной обработкой.
|
||||
|
||||
## Staging retention и очистка
|
||||
|
||||
`cleanup` запускается раз в 30 s. Журнал orphan PUT старше `UPLOAD_CLEANUP_GRACE_SECONDS=3600` очищается под advisory lock: свежая/активная загрузка не удаляется. Crash после staging, но до metadata/job не дал 202; его журнал позволяет удалить orphan без обхода всего bucket.
|
||||
|
||||
Staging queued/processing не удаляется независимо от возраста. После безопасного COMMIT ready staging удаляется при ближайшем проходе; конечные четыре объекта сохраняются. Failed staging и частичные конечные объекты хранятся `STAGING_RETENTION_SECONDS=86400` после последнего перехода в failed, затем удаляются; metadata, failed registry и audit остаются. Операции DELETE идемпотентны, отметка staging_deleted_at фиксируется после удаления. Cleaner и replay блокируют те же строки, поэтому не удаляют staging текущей replay-работы.
|
||||
|
||||
## Сервисы, настройки и bootstrap
|
||||
|
||||
| Роль / service | Версия | Readiness / persistence / credentials |
|
||||
| --- | --- | --- |
|
||||
| API / api ×2 | Python 3.11.15; FastAPI 0.142.4 | non-root UID 10001, health live/ready; 384 MiB/1 CPU на реплику; Docker secrets |
|
||||
| БД / postgres | PostgreSQL 17.6-bookworm | pg_isready, postgres_data; отдельный migrate с checksum/advisory lock |
|
||||
| Сессии / valkey | Valkey 8.1.3 | PING, AOF always, valkey_data, внутренний порт |
|
||||
| Брокер / broker | Valkey 8.1.3 | авторизованный PING, AOF always, broker_data, отдельный пароль через secret |
|
||||
| Outbox / publisher | тот же собранный Python image | restart unless-stopped; DB+broker secrets; topology bootstrap идемпотентен |
|
||||
| Worker / worker | тот же Python image; Pillow 13.0.0 | restart supervisor, 2 consumer-процесса, leases; DB+broker+S3 secrets |
|
||||
| S3 / s3 + s3-storage | SeaweedFS 3.85 с digest | gateway probe + filer HTTP; s3_data; storage только в s3-internal |
|
||||
| S3 bootstrap / s3-bootstrap | MinIO SDK 7.2.20 | root secrets только для создания приватного bucket; повтор безопасен |
|
||||
| LB / lb | HAProxy 3.2.6-alpine | readiness backend; единственный порт 127.0.0.1:8080 |
|
||||
| Cleanup / cleanup | тот же Python image | restart unless-stopped, DB+S3 secrets; без broker credentials |
|
||||
|
||||
Python base image закреплён digest в Dockerfile; S3 image — digest в Compose. Все Python-зависимости закреплены в requirements.txt. Административные интерфейсы S3 и внутренние API/БД/broker ports наружу не опубликованы. S3 application credentials разрешают работу только своего bucket/prefix attachments/*; root используется только bootstrap. API/cleaner не получают broker password. `.env.example` содержит пути secret-файлов, а не их значения.
|
||||
|
||||
Соответствие настроек: APP_HOST/APP_PORT — адрес API внутри контейнера; HOST_PORT — порт LB; APP_ENV — учебная метка. SESSION_STORE=valkey и SESSION_TTL_SECONDS=3600 — общее session storage; CACHE_URL_FILE/DATABASE_URL_FILE — Docker secrets. API_REPLICAS=2; IDEMPOTENCY_TTL_SECONDS≥86400. S3_ENDPOINT/REGION/BUCKET и S3_*_FILE — приватный bucket. MAX_UPLOAD_BYTES/MAX_IMAGE_PIXELS и UPLOAD_CLEANUP_GRACE_SECONDS — приём и orphan cleanup. BROKER_URL_FILE — авторизованный Redis-compatible URL отдельного broker. WORKER_CONCURRENCY, WORKER_TASK_MEMORY_MB, JOB_TIMEOUT_SECONDS, JOB_LEASE_SECONDS, JOB_MAX_ATTEMPTS, JOB_BACKOFF_SECONDS — ограничения/повторы. STAGING_RETENTION_SECONDS — срок хранения failed исходников.
|
||||
|
||||
/health/ready API зависит от PostgreSQL, сессий и S3: они нужны синхронному пути. Broker/worker/publisher не являются зависимостями readiness API: при их остановке original+outbox надёжно принимаются, текстовые сообщения работают, resize синхронно не запускается. У фоновых процессов нет HTTP health endpoint; состояние jobs/pending/failed наблюдается через БД, broker и операторскую CLI.
|
||||
|
||||
## Доказательства критериев
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
Проверено 12.11.2026 до добавления heartbeat: все **17 групп проверок** и **39 API regression tests** прошли. Docker 29.0.1, aarch64, 10 CPU, 8 217 448 448 bytes RAM Docker VM. Эти evidence относятся к предыдущей реализации.
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
13.11.2026 после исправления локально прошли 6 проверок `python -m unittest discover -s student-tests -p test_task_supervisor.py -v` с реальными subprocess: работа дольше первой аренды, общий timeout при продолжающемся heartbeat, потеря аренды, ошибка продления, crash и убийство дочернего процесса. Полный Docker-прогон новой версии не выполнен: Docker Engine на текущей машине недоступен. В `lab5_pipeline.py` добавлены сценарии 20-секундной работы при lease=15 s без retry и зависания runner с общим timeout=25 s; они требуют повторного `./scripts/check-docker.sh`.
|
||||
|
||||
## Эксперименты и отказы
|
||||
Полный воспроизводимый запуск: `./scripts/check-docker.sh`. Исходник сценария — `student-tests/lab5_pipeline.py`. Результаты и реальные job/attachment IDs — `evidence/lab5-checks.json`, консольный вывод — `evidence/lab5-checks.txt`. Дополнительное доказательство реального restart обычного стенда с прежними container IDs и volumes — `evidence/lab5-restart.json`; воспроизводится `.venv/bin/python student-tests/verify_restart.py`.
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
| Критерий README | Команда / ключ проверки evidence | Наблюдаемый результат |
|
||||
| --- | --- | --- |
|
||||
| Сохранён lab4, переход 201→202, smoke | make check validate test; public_smoke_and_schema, student_api_regressions | Публичный lab5 сценарий и собственные API regression tests |
|
||||
| Отдельные broker/worker, persistence, лимиты | topology_and_limits | Реальные контейнеры/volumes, две API, broker AOF always, worker concurrency=2/prefetch=1 |
|
||||
| Worker остановлен → 202 → сообщение → рестарты → ready | worker_offline_api_broker_restart | Тот же attachment/job после рестарта API и broker; 409 до ready; original/три варианта доступны |
|
||||
| Original/thumbnail/crop/WebP по контракту | image_semantics | Bytes/checksums/MIME и реальные размеры; small/alpha/EXIF/floor cases |
|
||||
| Outbox, crash COMMIT→publish | outbox_crash_before_publish | Устойчивое намерение доставляется после автоматического restart publisher |
|
||||
| Неопределённый XADD, дубли | outbox_crash_after_publish | Повтор publish не создаёт второго результата |
|
||||
| Crash после side effect до COMMIT/ACK | worker_crash_after_objects, worker_crash_before_ack | Lease/reclaim восстанавливают работу; четыре canonical объекта; ready дубль — no-op |
|
||||
| Retry/backoff/jitter, failed registry, replay | bounded_retry_failed_registry_replay | 5 автоматических неудач → failed; audit содержит время/попытки; явный replay → ready; ready replay отклонён |
|
||||
| Постоянная ошибка | permanent_error | invalid_image с первой попытки, безопасный код |
|
||||
| Ограничение времени и памяти обработки | image_subprocess_timeout, image_subprocess_memory | Реальное убийство зависшего subprocess и MemoryError при выделении 1 GiB под лимитом 640 MiB; следующий retry завершает тот же job |
|
||||
| Безопасная очистка staging | staging_cleanup | Active staging сохранён; ready/expired failed удалены; replay без исходника запрещён |
|
||||
| Broker недоступен | broker_offline_outbox | API принимает 202 и текст; outbox доставляет после восстановления |
|
||||
| Сохранность после всех рестартов | make test-persistence; public_persistence | Сессия, история и original переживают рестарты с volumes |
|
||||
| Отказ API | make test-failover; public_api_failover | Другая реплика продолжает обслуживать подтверждённые сообщения без дублей |
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
Failpoints включаются только явным `ENABLE_FAULT_INJECTION=1` во временном тестовом project; по умолчанию отключены. Файлы управления доступны только через docker exec внутри контейнера, HTTP управления ими нет. `before_publish` завершает publisher после DB COMMIT намерения до XADD; `after_publish` — после XADD до фиксации outbox. `after_objects` завершает job_runner после PUT четырёх объектов до COMMIT ready; `before_ack` — после COMMIT ready до XACK. Маркеры срабатывания проверяются тестом, поэтому простой быстрый успех без внедрённого сбоя не засчитывается. Отдельный timeout failpoint усыпляет именно ограниченный resize subprocess, не подменяет результат mock-объектом. `before_download` с `delay:20` задерживает корректную попытку, с `timeout` — имитирует зависание runner до скачивания.
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
## Эксперименты и ограничения
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
Это проверка критериев на 4; исследование максимального throughput, сравнение concurrency/prefetch и versioned pipeline не выполнялись. Полный сценарий использует concurrency=2/prefetch=1, max_attempts=5; для быстрых fault checks timeout=5 s, lease=15 s, base backoff=0.15 s. Production-подобные default настройки остаются 60/90/2 s в .env.example. Дата, Docker/hardware и лимиты конкретного запуска записаны в evidence. Измерения latency/RSS этой проверки не трактуются как нагрузочный benchmark.
|
||||
|
||||
Обработка at-least-once, не exactly-once: XADD и PUT могут повториться, идемпотентен конечный эффект и его публикация. Сбой host/disk с потерей volumes, потеря единственной PostgreSQL/S3 или ручное удаление объектов не покрываются. Backup/replication этих хранилищ в lab5 не добавлялись. Очередь ограничена maxmemory/noeviction broker; outbox и staging во время длительного простоя требуют диска, общей квоты принятой работы эта оценка не требует. Повторы самого HTTP upload могут создать отдельные вложения по контракту.
|
||||
|
||||
## Первичные источники
|
||||
|
||||
- [Задание и объявленные переходы](README.md), [состояния](contracts/attachment-states.md), [схема job](contracts/jobs.schema.json).
|
||||
- [Redis Streams: consumer groups, pending, acknowledgements](https://redis.io/docs/latest/develop/data-types/streams/).
|
||||
- [XAUTOCLAIM: восстановление pending](https://redis.io/docs/latest/commands/xautoclaim/).
|
||||
- [Valkey persistence: AOF и fsync](https://valkey.io/topics/persistence/).
|
||||
- [Pillow ImageOps: EXIF transpose и центрированный fit](https://pillow.readthedocs.io/en/stable/reference/ImageOps.html).
|
||||
- [Python resource limits](https://docs.python.org/3/library/resource.html).
|
||||
|
||||
+220
@@ -0,0 +1,220 @@
|
||||
name: mtusi-lab5
|
||||
|
||||
x-application: &application
|
||||
image: mtusi-messenger:lab5
|
||||
env_file: .env
|
||||
environment: &pipeline-environment
|
||||
JOB_MAX_ATTEMPTS: ${JOB_MAX_ATTEMPTS:-5}
|
||||
JOB_TIMEOUT_SECONDS: ${JOB_TIMEOUT_SECONDS:-60}
|
||||
JOB_LEASE_SECONDS: ${JOB_LEASE_SECONDS:-90}
|
||||
JOB_TASK_TIMEOUT_SECONDS: ${JOB_TASK_TIMEOUT_SECONDS:-180}
|
||||
JOB_BACKOFF_SECONDS: ${JOB_BACKOFF_SECONDS:-2}
|
||||
WORKER_CONCURRENCY: ${WORKER_CONCURRENCY:-2}
|
||||
WORKER_TASK_MEMORY_MB: ${WORKER_TASK_MEMORY_MB:-640}
|
||||
ENABLE_FAULT_INJECTION: ${ENABLE_FAULT_INJECTION:-0}
|
||||
cpus: 1
|
||||
mem_limit: 384m
|
||||
|
||||
x-postgres: &postgres
|
||||
image: postgres:17.6-bookworm
|
||||
environment:
|
||||
POSTGRES_USER: messenger
|
||||
POSTGRES_DB: messenger
|
||||
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
|
||||
secrets: [postgres_password]
|
||||
cpus: 1
|
||||
mem_limit: 512m
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U messenger -d messenger"]
|
||||
interval: 2s
|
||||
timeout: 3s
|
||||
retries: 30
|
||||
|
||||
services:
|
||||
postgres:
|
||||
<<: *postgres
|
||||
volumes: [postgres_data:/var/lib/postgresql/data]
|
||||
|
||||
valkey:
|
||||
image: valkey/valkey:8.1.3
|
||||
command: ["valkey-server", "--appendonly", "yes", "--appendfsync", "always", "--save", "", "--maxmemory", "64mb", "--maxmemory-policy", "noeviction"]
|
||||
volumes: [valkey_data:/data]
|
||||
cpus: 0.5
|
||||
mem_limit: 128m
|
||||
healthcheck:
|
||||
test: ["CMD", "valkey-cli", "ping"]
|
||||
interval: 2s
|
||||
timeout: 3s
|
||||
retries: 30
|
||||
|
||||
broker:
|
||||
image: valkey/valkey:8.1.3
|
||||
command: ["sh", "-c", 'exec valkey-server --appendonly yes --appendfsync always --save "" --maxmemory 128mb --maxmemory-policy noeviction --requirepass "$$(cat /run/secrets/broker_password)"']
|
||||
secrets: [broker_password]
|
||||
volumes: [broker_data:/data]
|
||||
cpus: 0.5
|
||||
mem_limit: 256m
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", 'REDISCLI_AUTH="$$(cat /run/secrets/broker_password)" valkey-cli ping | grep -q PONG']
|
||||
interval: 2s
|
||||
timeout: 3s
|
||||
retries: 30
|
||||
|
||||
publisher:
|
||||
<<: *application
|
||||
command: ["python", "-m", "src.publisher"]
|
||||
secrets: [database_url, broker_url]
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
disable: true
|
||||
depends_on:
|
||||
migrate: {condition: service_completed_successfully}
|
||||
broker: {condition: service_healthy}
|
||||
|
||||
worker:
|
||||
<<: *application
|
||||
command: ["python", "-m", "src.worker"]
|
||||
secrets: [database_url, broker_url, s3_access_key_id, s3_secret_access_key]
|
||||
restart: unless-stopped
|
||||
mem_limit: 1536m
|
||||
cpus: 2
|
||||
pids_limit: 96
|
||||
healthcheck:
|
||||
disable: true
|
||||
depends_on:
|
||||
migrate: {condition: service_completed_successfully}
|
||||
broker: {condition: service_healthy}
|
||||
s3-bootstrap: {condition: service_completed_successfully}
|
||||
stop_grace_period: 8s
|
||||
|
||||
api:
|
||||
<<: *application
|
||||
build: .
|
||||
deploy:
|
||||
replicas: ${API_REPLICAS:-2}
|
||||
environment:
|
||||
<<: *pipeline-environment
|
||||
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
|
||||
secrets: [database_url, cache_url, s3_access_key_id, s3_secret_access_key]
|
||||
depends_on:
|
||||
postgres: {condition: service_healthy}
|
||||
valkey: {condition: service_healthy}
|
||||
migrate: {condition: service_completed_successfully}
|
||||
s3: {condition: service_healthy}
|
||||
s3-bootstrap: {condition: service_completed_successfully}
|
||||
stop_grace_period: 8s
|
||||
|
||||
migrate:
|
||||
<<: *application
|
||||
command: ["python", "-m", "src.migrate"]
|
||||
secrets: [database_url]
|
||||
healthcheck:
|
||||
disable: true
|
||||
depends_on:
|
||||
postgres: {condition: service_healthy}
|
||||
|
||||
s3:
|
||||
image: chrislusf/seaweedfs:3.85@sha256:49312939c00c01e5ee6afbd7d728b18027821d3764c35a797a72acd4fdf3296a
|
||||
command: ["s3", "-filer=s3-storage:8888", "-ip.bind=0.0.0.0", "-port=9000", "-config=/run/secrets/s3_config"]
|
||||
secrets: [s3_config]
|
||||
networks: [default, s3-internal]
|
||||
depends_on:
|
||||
s3-storage: {condition: service_healthy}
|
||||
cpus: 0.5
|
||||
mem_limit: 256m
|
||||
healthcheck:
|
||||
test: ["CMD", "nc", "-z", "127.0.0.1", "9000"]
|
||||
interval: 2s
|
||||
timeout: 2s
|
||||
retries: 30
|
||||
|
||||
s3-storage:
|
||||
image: chrislusf/seaweedfs:3.85@sha256:49312939c00c01e5ee6afbd7d728b18027821d3764c35a797a72acd4fdf3296a
|
||||
command: ["server", "-dir=/data", "-ip=s3-storage", "-ip.bind=0.0.0.0", "-filer", "-master.volumeSizeLimitMB=64", "-volume.max=64", "-volume.preStopSeconds=0"]
|
||||
volumes: [s3_data:/data]
|
||||
networks: [s3-internal]
|
||||
cpus: 1
|
||||
mem_limit: 512m
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:8888/"]
|
||||
interval: 2s
|
||||
timeout: 2s
|
||||
retries: 30
|
||||
|
||||
s3-bootstrap:
|
||||
<<: *application
|
||||
command: ["python", "-m", "src.bootstrap_s3"]
|
||||
environment:
|
||||
S3_ROOT_USER_FILE: /run/secrets/s3_root_user
|
||||
S3_ROOT_PASSWORD_FILE: /run/secrets/s3_root_password
|
||||
secrets: [s3_root_user, s3_root_password]
|
||||
cpus: 0.5
|
||||
mem_limit: 128m
|
||||
healthcheck:
|
||||
disable: true
|
||||
depends_on:
|
||||
s3: {condition: service_healthy}
|
||||
|
||||
cleanup:
|
||||
<<: *application
|
||||
command: ["python", "-m", "src.cleanup"]
|
||||
secrets: [database_url, s3_access_key_id, s3_secret_access_key]
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
disable: true
|
||||
depends_on:
|
||||
migrate: {condition: service_completed_successfully}
|
||||
s3-bootstrap: {condition: service_completed_successfully}
|
||||
|
||||
lb:
|
||||
image: haproxy:3.2.6-alpine
|
||||
ports: ["127.0.0.1:${HOST_PORT:-8080}:8080"]
|
||||
volumes: [./infra/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro]
|
||||
cpus: 0.5
|
||||
mem_limit: 64m
|
||||
depends_on:
|
||||
api: {condition: service_healthy}
|
||||
stop_grace_period: 10s
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:8080/health/ready"]
|
||||
interval: 2s
|
||||
timeout: 2s
|
||||
retries: 3
|
||||
|
||||
postgres-restore:
|
||||
<<: *postgres
|
||||
profiles: [research]
|
||||
volumes: [restore_data:/var/lib/postgresql/data]
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
valkey_data:
|
||||
restore_data:
|
||||
s3_data:
|
||||
broker_data:
|
||||
|
||||
networks:
|
||||
s3-internal:
|
||||
internal: true
|
||||
|
||||
secrets:
|
||||
broker_password:
|
||||
file: ./secrets/broker_password
|
||||
broker_url:
|
||||
file: ./secrets/broker_url
|
||||
postgres_password:
|
||||
file: ./secrets/postgres_password
|
||||
database_url:
|
||||
file: ./secrets/database_url
|
||||
cache_url:
|
||||
file: ./secrets/cache_url
|
||||
s3_root_user:
|
||||
file: ./secrets/s3_root_user
|
||||
s3_root_password:
|
||||
file: ./secrets/s3_root_password
|
||||
s3_access_key_id:
|
||||
file: ./secrets/s3_access_key_id
|
||||
s3_secret_access_key:
|
||||
file: ./secrets/s3_secret_access_key
|
||||
s3_config:
|
||||
file: ./secrets/s3_config
|
||||
@@ -0,0 +1,40 @@
|
||||
# Схема и транзакции
|
||||
|
||||
Лабораторная 5, оценка 4: клиент → HAProxy → FastAPI × 2 → PostgreSQL + Valkey sessions + приватный S3; outbox publisher → Valkey Streams broker → worker.
|
||||
|
||||
| Таблица PostgreSQL | Назначение и ограничения |
|
||||
| --- | --- |
|
||||
| users | UUID PK; username UNIQUE; password_hash Argon2id; display_name; created_at |
|
||||
| chats | UUID PK; title; created_at |
|
||||
| chat_members | PK (chat_id, user_id); FK на chats и users |
|
||||
| messages | UUID PK; FK на чат и членство отправителя; text до 4000 code points; created_at |
|
||||
| message_attachments | FK на messages и attachments; до 10 позиций; UNIQUE(message_id, position); PK(message_id, attachment_id) |
|
||||
| attachments | UUID PK; FK chat_id и (chat_id, owner_id); UNIQUE object_key; original metadata; status, variants, error_code, created_at/updated_at |
|
||||
| attachment_jobs | UUID job_id PK; UNIQUE attachment_id/staging_key; version, attempt/budget, lease, retention |
|
||||
| attachment_outbox | PK (job_id, attempt); available_at, published_at; индекс неопубликованных |
|
||||
| attachment_job_events | Audit переходов, safe error_code, attempt, время |
|
||||
| attachment_uploads | UUID PK, UNIQUE object_key, created_at; долговечный журнал незавершённых PUT |
|
||||
| message_idempotency | PK (user_id, chat_id, key); request_text, request_attachment_ids, message_id FK, expires_at |
|
||||
| schema_migrations | Имя и SHA-256 применённой миграции; applied_at |
|
||||
|
||||
В PostgreSQL нет bytes оригинала. `object_key=attachments/<UUID>/v1/original` назначает сервер; filename не участвует в ключе. До ответа 202 в таблице attachments есть status=queued, variants=[], error_code=null. Миграция 004 добавляет status, variants, error_code, updated_at; уже существующие ready-вложения lab4 не меняют ключи и остаются доступны.
|
||||
|
||||
`attachment_jobs` хранит один job_id на attachment_id, staging_key, pipeline_version=1, текущую попытку, бюджет, lease_token/lease_until и отметку удаления staging. `attachment_outbox(job_id,attempt)` создаётся в той же транзакции, что metadata/намерение. Dispatcher отмечает published_at после XADD; потеря ответа вызывает безопасный дубль. Отложенный retry получает отдельную строку outbox с available_at. `attachment_job_events` — audit переходов и явных replay без credentials и текста исключений.
|
||||
|
||||
Данные при 202: приватный S3 `attachments/staging/<UUID>` и COMMIT metadata+job+outbox. Готовность: worker сохраняет неизменённый original и три варианта по детерминированным ключам, затем атомарно фиксирует status=ready и JSON metadata. До ready content отвечает 409. Worker проверяет lease_token под блокировкой строк перед записью результатов. Consumer продлевает действующую аренду каждые min(10, lease/3) секунд, пока отдельный job_runner выполняет попытку. Просроченный/чужой lease продлевать нельзя. Общий JOB_TASK_TIMEOUT_SECONDS ограничивает работу runner независимо от heartbeat: по deadline убивается всё дерево обработки. Lease может быть меньше времени допустимой обработки. Cleaner не удаляет staging queued/processing; после ready удаляет staging, а failed хранит до STAGING_RETENTION_SECONDS.
|
||||
|
||||
Индексы: UNIQUE users(username); chat_members(user_id, chat_id); messages(chat_id, created_at, id); attachments(chat_id); attachment_uploads(created_at). Первичные/уникальные ключи индексируют поиск ID и object_key.
|
||||
|
||||
Создание чата и его участников атомарно. Неизвестный участник откатывает весь чат. Сообщение проверяет членство и все attachment_ids в том же чате, затем одной транзакцией записывает messages, message_attachments и результат идемпотентности. Составной FK дополнительно запрещает отправителя вне чата. Пустой/пробельный text разрешён только с вложением: это проверяет модель запроса; FK таблицы message_attachments запрещают несуществующие ссылки. Удаления вложений в HTTP API нет.
|
||||
|
||||
Общая advisory transaction lock сериализует область пользователь + POST + путь чата + Idempotency-Key. Сравниваются text и упорядоченный нормализованный список UUID; отсутствие attachment_ids эквивалентно []. Совпадение возвращает исходный 201 и Message, отличие — 409. TTL не меньше 24 часов. Чтение истории использует keyset (created_at,id), без кэша; на следующем запросе видны COMMIT любой реплики. При конкурентных вставках snapshot между страницами нет; запись с более ранними часами может оказаться до уже пройденного cursor.
|
||||
|
||||
Upload сначала полностью принимает ограниченный поток во временный файл и проверяет PNG/JPEG. Затем отдельной транзакцией фиксирует attachment_uploads. В следующей транзакции берётся advisory lock object_key, выполняется S3 PUT, вставляются queued metadata, job и outbox, удаляется журнал. 202 отправляется после COMMIT. S3 и PostgreSQL не образуют распределённую транзакцию.
|
||||
|
||||
Если PUT прошёл, а COMMIT нет, остаётся журнал. Если исход COMMIT неизвестен, немедленного удаления нет: metadata/job могли сохраниться, а удаление журнала атомарно с ними. Cleaner раз в 30 секунд берёт до 100 записей старше grace period (по умолчанию 3600 s, минимум 60 s), пробует ту же advisory lock и повторно проверяет запись. Активная загрузка пропускается; при наличии ready-метаданных удаляется только запись журнала; иначе удаляется S3 object и затем журнал. DELETE объекта идемпотентен. Сбой S3/БД откладывает обработку; следующий проход повторяет её. Это компенсация журналируемых загрузок уровня 4, без полного обхода bucket.
|
||||
|
||||
Миграции 001 и 002 сохранены. Миграция 003 добавляет вложения, связи, журнал и список вложений в ключе повтора; старые сообщения имеют пустой список. DDL выполняет один migrate-service под общей advisory lock; API проверяют версии и checksum.
|
||||
|
||||
Сервис valkey хранит только серверные сессии; отдельный сервис broker хранит Stream. Сессии: session:SHA256(token) → user_id, expires_at, TTL. Включены AOF appendfsync always и постоянный volume. Сырой токен не записывается в evidence. PostgreSQL, Valkey и S3 bucket проверяются /health/ready; /health/live не обращается к зависимостям.
|
||||
|
||||
Основа схемы перенесена из lab4; команды, измерения и ограничения прежней версии — в [REPORT.lab4.md](../REPORT.lab4.md). Исходные отчёты и результаты lab2/lab3 сохранены отдельно. Текущая лабораторная описывается в [REPORT.md](../REPORT.md).
|
||||
@@ -0,0 +1,66 @@
|
||||
UTC: 2026-10-22T11:17:22.562634+00:00
|
||||
Host: Darwin arm64 Python 3.11.9
|
||||
Docker: 29.0.1 aarch64 CPUs=10 RAM=8217448448
|
||||
Commit: 9961362 + working tree
|
||||
Command: ./scripts/check-docker.sh; project: lab2-check-ae280d00db
|
||||
PASS: start.sh built and started the stack, readiness reached
|
||||
api image=sha256:0b3dc5056cced642eea5ade11d235dce9f69f0d9e6d63956ce9479f2deaa10a0 started=2026-10-22T11:17:27.057932632Z memory=268435456 nano_cpus=1000000000
|
||||
postgres image=sha256:f3bd19c606e442c3d7bdfa8002e03fe260a1023351e0ea4598032022b68dd6e3 started=2026-10-22T11:17:24.411773881Z memory=536870912 nano_cpus=1000000000
|
||||
valkey image=sha256:fea8b3e67b15729d4bb70589eb03367bab9ad1ee89c876f54327fc7c6e618571 started=2026-10-22T11:17:24.406509756Z memory=134217728 nano_cpus=500000000
|
||||
PASS: non-root UID 10001; Python PID 1; healthcheck; named volumes; no API data volume
|
||||
Проверяем сервер: http://127.0.0.1:61900
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab2/mtusi2026lab2/.venv/bin/python scripts/check_contract.py
|
||||
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab2/mtusi2026lab2/.venv/bin/python tests/smoke.py --base-url "http://127.0.0.1:61900"
|
||||
PASS: lab2 public API scenario; infrastructure and grading evidence remain separate.
|
||||
test_chat_access (test_api.ApiTests.test_chat_access) ... ok
|
||||
test_chat_pagination_and_cursor_owner (test_api.ApiTests.test_chat_pagination_and_cursor_owner) ... ok
|
||||
test_cursor_cannot_be_used_for_another_chat_or_changed (test_api.ApiTests.test_cursor_cannot_be_used_for_another_chat_or_changed) ... ok
|
||||
test_default_page_size (test_api.ApiTests.test_default_page_size) ... ok
|
||||
test_duplicate_username (test_api.ApiTests.test_duplicate_username) ... ok
|
||||
test_framework_errors_and_request_ids (test_api.ApiTests.test_framework_errors_and_request_ids) ... ok
|
||||
test_identification_and_authentication_order (test_api.ApiTests.test_identification_and_authentication_order) ... ok
|
||||
test_invalid_chat_members (test_api.ApiTests.test_invalid_chat_members) ... ok
|
||||
test_invalid_chat_title (test_api.ApiTests.test_invalid_chat_title) ... ok
|
||||
test_invalid_limits_and_cursors (test_api.ApiTests.test_invalid_limits_and_cursors) ... ok
|
||||
test_invalid_message_fields (test_api.ApiTests.test_invalid_message_fields) ... ok
|
||||
test_invalid_user_fields (test_api.ApiTests.test_invalid_user_fields) ... ok
|
||||
test_malformed_json_and_encoding (test_api.ApiTests.test_malformed_json_and_encoding) ... ok
|
||||
test_message_pagination_and_changing_limit (test_api.ApiTests.test_message_pagination_and_changing_limit) ... ok
|
||||
test_unicode_and_text_preservation (test_api.ApiTests.test_unicode_and_text_preservation) ... ok
|
||||
test_wrong_content_type (test_api.ApiTests.test_wrong_content_type) ... ok
|
||||
test_simultaneous_registration_of_one_username (test_concurrency.ConcurrencyTests.test_simultaneous_registration_of_one_username) ... ok
|
||||
test_ten_clients_send_one_hundred_messages (test_concurrency.ConcurrencyTests.test_ten_clients_send_one_hundred_messages) ... ok
|
||||
test_logout_revokes_only_one_session (test_sessions.SessionTests.test_logout_revokes_only_one_session) ... ok
|
||||
test_password_validation (test_sessions.SessionTests.test_password_validation) ... ok
|
||||
test_sql_like_message_is_plain_text (test_sessions.SessionTests.test_sql_like_message_is_plain_text) ... ok
|
||||
test_unicode_password_with_colon_and_private_response (test_sessions.SessionTests.test_unicode_password_with_colon_and_private_response) ... ok
|
||||
test_wrong_password_and_basic_only_for_login (test_sessions.SessionTests.test_wrong_password_and_basic_only_for_login) ... ok
|
||||
|
||||
----------------------------------------------------------------------
|
||||
Ran 23 tests in 5.743s
|
||||
|
||||
OK
|
||||
PASS: 10 clients sent 100 messages; history contains exactly 100 unique messages.
|
||||
PASS: demo.sh registered users, logged in and read two messages
|
||||
PASS: Argon2id in PostgreSQL; logs do not contain message text, passwords or session tokens
|
||||
PASS: failed membership insert rolled back chat; migrations run twice without duplicate data
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab2/mtusi2026lab2/.venv/bin/python tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://127.0.0.1:61900"
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
Restart api 2026-10-22T11:17:27.057932632Z -> 2026-10-22T11:17:45.460071626Z
|
||||
Restart postgres 2026-10-22T11:17:24.411773881Z -> 2026-10-22T11:17:44.637353001Z
|
||||
Restart valkey 2026-10-22T11:17:24.406509756Z -> 2026-10-22T11:17:44.670095293Z
|
||||
PASS: API, PostgreSQL and Valkey really restarted; previous token still works
|
||||
PASS: valkey unavailable -> 503 in 0.11s; anonymous -> 401; readiness 503, liveness 200
|
||||
API ready
|
||||
PASS: valkey recovered without a new login
|
||||
PASS: postgres unavailable -> 503 in 2.01s; anonymous -> 401; readiness 503, liveness 200
|
||||
API ready
|
||||
PASS: postgres recovered without a new login
|
||||
API ready
|
||||
PASS: SESSION_TTL_SECONDS=2; valid token expires without logout
|
||||
PASS: SIGTERM with unfinished request: 5.54s; new requests refused after 0.16s; exit=0; OOM=false
|
||||
PASS: all lab2 Docker checks completed
|
||||
PASS: stop.sh removed containers and preserved named volumes
|
||||
Removed only temporary Compose project and its test volumes: lab2-check-ae280d00db
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,403 @@
|
||||
Command: ./scripts/check-docker.sh; project: lab3-check-a31f544685
|
||||
Docker: 29.0.1 aarch64 CPUs=10 RAM=8217448448
|
||||
Local .env and secrets are ready; values are not printed
|
||||
#1 [internal] load local bake definitions
|
||||
#1 reading from stdin 1.05kB done
|
||||
#1 DONE 0.0s
|
||||
|
||||
#2 [migrate internal] load build definition from Dockerfile
|
||||
#2 transferring dockerfile: 733B done
|
||||
#2 DONE 0.0s
|
||||
|
||||
#3 [api internal] load metadata for docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
|
||||
#3 DONE 0.0s
|
||||
|
||||
#4 [api internal] load .dockerignore
|
||||
#4 transferring context: 167B done
|
||||
#4 DONE 0.0s
|
||||
|
||||
#5 [migrate internal] load build context
|
||||
#5 transferring context: 809B done
|
||||
#5 DONE 0.0s
|
||||
|
||||
#6 [migrate 1/8] FROM docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
|
||||
#6 resolve docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3 done
|
||||
#6 DONE 0.0s
|
||||
|
||||
#7 [migrate 2/8] WORKDIR /app
|
||||
#7 CACHED
|
||||
|
||||
#8 [migrate 3/8] COPY requirements.txt .
|
||||
#8 CACHED
|
||||
|
||||
#9 [migrate 4/8] RUN python -m pip install --no-cache-dir -r requirements.txt && python -m pip check
|
||||
#9 CACHED
|
||||
|
||||
#10 [migrate 6/8] COPY src/ ./src/
|
||||
#10 CACHED
|
||||
|
||||
#11 [migrate 7/8] COPY migrations/ ./migrations/
|
||||
#11 CACHED
|
||||
|
||||
#12 [migrate 5/8] RUN groupadd --gid 10001 app && useradd --uid 10001 --gid app --no-create-home app
|
||||
#12 CACHED
|
||||
|
||||
#13 [migrate 8/8] COPY research/ ./research/
|
||||
#13 CACHED
|
||||
|
||||
#14 [api] exporting to image
|
||||
#14 exporting layers done
|
||||
#14 exporting manifest sha256:06163e45d3676c7cf4f7a1c7bfe108ed3b98559ebb485c7a09d1e87e6f418d17 done
|
||||
#14 exporting config sha256:3fed7460e9510eef7b2928f8d600d06d5d65329f74d820beac13809476b11398 done
|
||||
#14 exporting attestation manifest sha256:352319cde788733f2f237a2a78204630c542e7ca906cff14c902046e8c1d2e5b 0.0s done
|
||||
#14 exporting manifest list sha256:848e0c76f1678f35dd1b483e7c54552466999551ba7e77c0062bf346671e2902 done
|
||||
#14 naming to docker.io/library/mtusi-messenger:lab3 done
|
||||
#14 unpacking to docker.io/library/mtusi-messenger:lab3 done
|
||||
#14 DONE 0.1s
|
||||
|
||||
#15 [migrate] exporting to image
|
||||
#15 exporting layers done
|
||||
#15 exporting manifest sha256:a59568d34e2c5e8b4d23308342601fdaf8c07e78ec3065878679a067c082e952 done
|
||||
#15 exporting config sha256:b67b805ba1b7dd90f0a3e8444455488fe8c7f4118f14d671501dd63e85b59c97 done
|
||||
#15 exporting attestation manifest sha256:b1853c07ddac46c4b398b475052f334aa2753c2fe971e0e4fb2517b0e3e13656 0.0s done
|
||||
#15 exporting manifest list sha256:eff1abdd662eb60e2741e6ce6b8f364467d2817dafeef9754518247124a780eb done
|
||||
#15 naming to docker.io/library/mtusi-messenger:lab3 done
|
||||
#15 unpacking to docker.io/library/mtusi-messenger:lab3 done
|
||||
#15 DONE 0.1s
|
||||
|
||||
#16 [api] resolving provenance for metadata file
|
||||
#16 DONE 0.0s
|
||||
|
||||
#17 [migrate] resolving provenance for metadata file
|
||||
#17 DONE 0.0s
|
||||
mtusi-messenger:lab3 Built
|
||||
mtusi-messenger:lab3 Built
|
||||
Network lab3-check-a31f544685_default Creating
|
||||
Network lab3-check-a31f544685_default Created
|
||||
Volume lab3-check-a31f544685_postgres_data Creating
|
||||
Volume lab3-check-a31f544685_postgres_data Created
|
||||
Volume lab3-check-a31f544685_valkey_data Creating
|
||||
Volume lab3-check-a31f544685_valkey_data Created
|
||||
Container lab3-check-a31f544685-valkey-1 Creating
|
||||
Container lab3-check-a31f544685-postgres-1 Creating
|
||||
Container lab3-check-a31f544685-valkey-1 Created
|
||||
Container lab3-check-a31f544685-postgres-1 Created
|
||||
Container lab3-check-a31f544685-migrate-1 Creating
|
||||
Container lab3-check-a31f544685-migrate-1 Created
|
||||
Container lab3-check-a31f544685-api-2 Creating
|
||||
Container lab3-check-a31f544685-api-1 Creating
|
||||
Container lab3-check-a31f544685-api-1 Created
|
||||
Container lab3-check-a31f544685-api-2 Created
|
||||
Container lab3-check-a31f544685-lb-1 Creating
|
||||
Container lab3-check-a31f544685-lb-1 Created
|
||||
Container lab3-check-a31f544685-valkey-1 Starting
|
||||
Container lab3-check-a31f544685-postgres-1 Starting
|
||||
Container lab3-check-a31f544685-valkey-1 Started
|
||||
Container lab3-check-a31f544685-postgres-1 Started
|
||||
Container lab3-check-a31f544685-postgres-1 Waiting
|
||||
Container lab3-check-a31f544685-postgres-1 Healthy
|
||||
Container lab3-check-a31f544685-migrate-1 Starting
|
||||
Container lab3-check-a31f544685-migrate-1 Started
|
||||
Container lab3-check-a31f544685-migrate-1 Waiting
|
||||
Container lab3-check-a31f544685-postgres-1 Waiting
|
||||
Container lab3-check-a31f544685-valkey-1 Waiting
|
||||
Container lab3-check-a31f544685-valkey-1 Healthy
|
||||
Container lab3-check-a31f544685-postgres-1 Healthy
|
||||
Container lab3-check-a31f544685-migrate-1 Exited
|
||||
Container lab3-check-a31f544685-api-2 Starting
|
||||
Container lab3-check-a31f544685-api-2 Started
|
||||
Container lab3-check-a31f544685-api-1 Starting
|
||||
Container lab3-check-a31f544685-api-1 Started
|
||||
Container lab3-check-a31f544685-api-2 Waiting
|
||||
Container lab3-check-a31f544685-api-1 Waiting
|
||||
Container lab3-check-a31f544685-api-2 Healthy
|
||||
Container lab3-check-a31f544685-api-1 Healthy
|
||||
Container lab3-check-a31f544685-lb-1 Starting
|
||||
Container lab3-check-a31f544685-lb-1 Started
|
||||
API ready through LB: 7f6a80b69e32, f973eaec4d52
|
||||
PASS: 2 real API containers, CPU/RAM limits, API non-root and no public ports; only LB published
|
||||
Проверяем сервер: http://127.0.0.1:58474
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab3/mtusi2026lab3/.venv/bin/python scripts/check_contract.py
|
||||
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab3/mtusi2026lab3/.venv/bin/python tests/smoke.py --base-url "http://127.0.0.1:58474"
|
||||
PASS: lab3 public API scenario; infrastructure and grading evidence remain separate.
|
||||
test_chat_access (test_api.ApiTests.test_chat_access) ... ok
|
||||
test_chat_pagination_and_cursor_owner (test_api.ApiTests.test_chat_pagination_and_cursor_owner) ... ok
|
||||
test_cursor_cannot_be_used_for_another_chat_or_changed (test_api.ApiTests.test_cursor_cannot_be_used_for_another_chat_or_changed) ... ok
|
||||
test_default_page_size (test_api.ApiTests.test_default_page_size) ... ok
|
||||
test_duplicate_username (test_api.ApiTests.test_duplicate_username) ... ok
|
||||
test_framework_errors_and_request_ids (test_api.ApiTests.test_framework_errors_and_request_ids) ... ok
|
||||
test_identification_and_authentication_order (test_api.ApiTests.test_identification_and_authentication_order) ... ok
|
||||
test_invalid_chat_members (test_api.ApiTests.test_invalid_chat_members) ... ok
|
||||
test_invalid_chat_title (test_api.ApiTests.test_invalid_chat_title) ... ok
|
||||
test_invalid_limits_and_cursors (test_api.ApiTests.test_invalid_limits_and_cursors) ... ok
|
||||
test_invalid_message_fields (test_api.ApiTests.test_invalid_message_fields) ... ok
|
||||
test_invalid_user_fields (test_api.ApiTests.test_invalid_user_fields) ... ok
|
||||
test_malformed_json_and_encoding (test_api.ApiTests.test_malformed_json_and_encoding) ... ok
|
||||
test_message_pagination_and_changing_limit (test_api.ApiTests.test_message_pagination_and_changing_limit) ... ok
|
||||
test_unicode_and_text_preservation (test_api.ApiTests.test_unicode_and_text_preservation) ... ok
|
||||
test_wrong_content_type (test_api.ApiTests.test_wrong_content_type) ... ok
|
||||
test_simultaneous_registration_of_one_username (test_concurrency.ConcurrencyTests.test_simultaneous_registration_of_one_username) ... ok
|
||||
test_ten_clients_send_one_hundred_messages (test_concurrency.ConcurrencyTests.test_ten_clients_send_one_hundred_messages) ... ok
|
||||
test_conflict_and_semantic_json_replay (test_idempotency.IdempotencyTests.test_conflict_and_semantic_json_replay) ... ok
|
||||
test_key_format_and_missing_key (test_idempotency.IdempotencyTests.test_key_format_and_missing_key) ... ok
|
||||
test_parallel_replay_across_replicas (test_idempotency.IdempotencyTests.test_parallel_replay_across_replicas) ... ok
|
||||
test_replay_rechecks_session_and_access (test_idempotency.IdempotencyTests.test_replay_rechecks_session_and_access) ... ok
|
||||
test_scope_is_user_and_chat (test_idempotency.IdempotencyTests.test_scope_is_user_and_chat) ... ok
|
||||
test_logout_revokes_only_one_session (test_sessions.SessionTests.test_logout_revokes_only_one_session) ... ok
|
||||
test_password_validation (test_sessions.SessionTests.test_password_validation) ... ok
|
||||
test_sql_like_message_is_plain_text (test_sessions.SessionTests.test_sql_like_message_is_plain_text) ... ok
|
||||
test_unicode_password_with_colon_and_private_response (test_sessions.SessionTests.test_unicode_password_with_colon_and_private_response) ... ok
|
||||
test_wrong_password_and_basic_only_for_login (test_sessions.SessionTests.test_wrong_password_and_basic_only_for_login) ... ok
|
||||
|
||||
----------------------------------------------------------------------
|
||||
Ran 28 tests in 6.783s
|
||||
|
||||
OK
|
||||
PASS: 10 clients sent 100 messages; history contains exactly 100 unique messages.
|
||||
Container lab3-check-a31f544685-postgres-1 Running
|
||||
Container lab3-check-a31f544685-postgres-1 Running
|
||||
PASS: migrations only in one-shot service; repeated execution preserves data
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab3/mtusi2026lab3/.venv/bin/python tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://127.0.0.1:58474"
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
PASS: both API, LB, PostgreSQL, Valkey actually restarted; session and confirmed data preserved
|
||||
PASS: SIGTERM with unfinished request: {"instance": "7f6a80b69e32", "unfinished_request": true, "shutdown_seconds": 5.393, "new_connections_refused_seconds": 0.167, "exit_code": 0, "oom_killed": false}
|
||||
Container lab3-check-a31f544685-postgres-1 Waiting
|
||||
Container lab3-check-a31f544685-postgres-1 Healthy
|
||||
Container lab3-check-a31f544685-migrate-1 Starting
|
||||
Container lab3-check-a31f544685-migrate-1 Started
|
||||
Container lab3-check-a31f544685-valkey-1 Waiting
|
||||
Container lab3-check-a31f544685-migrate-1 Waiting
|
||||
Container lab3-check-a31f544685-postgres-1 Waiting
|
||||
Container lab3-check-a31f544685-valkey-1 Healthy
|
||||
Container lab3-check-a31f544685-postgres-1 Healthy
|
||||
Container lab3-check-a31f544685-migrate-1 Exited
|
||||
Container lab3-check-a31f544685-api-1 Starting
|
||||
Container lab3-check-a31f544685-api-1 Started
|
||||
API ready through LB: 7f6a80b69e32, f973eaec4d52
|
||||
Container lab3-check-a31f544685-valkey-1 Stopping
|
||||
Container lab3-check-a31f544685-valkey-1 Stopped
|
||||
PASS: valkey outage: API live=200, ready=503, LB ready=503
|
||||
Container lab3-check-a31f544685-valkey-1 Starting
|
||||
Container lab3-check-a31f544685-valkey-1 Started
|
||||
API ready through LB: 7f6a80b69e32, f973eaec4d52
|
||||
Container lab3-check-a31f544685-postgres-1 Stopping
|
||||
Container lab3-check-a31f544685-postgres-1 Stopped
|
||||
PASS: postgres outage: API live=200, ready=503, LB ready=503
|
||||
Container lab3-check-a31f544685-postgres-1 Starting
|
||||
Container lab3-check-a31f544685-postgres-1 Started
|
||||
API ready through LB: 7f6a80b69e32, f973eaec4d52
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab3/mtusi2026lab3/.venv/bin/python tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:58474"
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||
{"observed_instances_before": 2, "successful_writes": 113, "failed_attempts": 1, "first_recovery_seconds": 2.452}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
Container lab3-check-a31f544685-postgres-1 Waiting
|
||||
Container lab3-check-a31f544685-postgres-1 Healthy
|
||||
Container lab3-check-a31f544685-migrate-1 Starting
|
||||
Container lab3-check-a31f544685-migrate-1 Started
|
||||
Container lab3-check-a31f544685-postgres-1 Waiting
|
||||
Container lab3-check-a31f544685-valkey-1 Waiting
|
||||
Container lab3-check-a31f544685-migrate-1 Waiting
|
||||
Container lab3-check-a31f544685-postgres-1 Healthy
|
||||
Container lab3-check-a31f544685-valkey-1 Healthy
|
||||
Container lab3-check-a31f544685-migrate-1 Exited
|
||||
Container lab3-check-a31f544685-api-1 Starting
|
||||
Container lab3-check-a31f544685-api-1 Started
|
||||
API ready through LB: 7f6a80b69e32, f973eaec4d52
|
||||
Stopped API: 7f6a80b69e32 container=7f6a80b69e3297a4de72e756a6c9d867a108930c1ca0e57922b3821c26fd03e6
|
||||
{"label": "stop 7f6a80b69e32", "duration_seconds": 16.009, "successful_requests": 1576, "successful_writes": 788, "errors": 4, "successful_rps": 98.44, "first_success_after_action_seconds": {"POST": 0.014, "GET": 0.007}, "maximum_success_gap_seconds": {"POST": 1.957, "GET": 2.004}, "action_start_seconds": 1.003, "action_end_seconds": 1.666, "p50_ms": 7.813, "p95_ms": 16.393, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["7f6a80b69e32", "f973eaec4d52"]}
|
||||
Container lab3-check-a31f544685-postgres-1 Waiting
|
||||
Container lab3-check-a31f544685-postgres-1 Healthy
|
||||
Container lab3-check-a31f544685-migrate-1 Starting
|
||||
Container lab3-check-a31f544685-migrate-1 Started
|
||||
Container lab3-check-a31f544685-postgres-1 Waiting
|
||||
Container lab3-check-a31f544685-valkey-1 Waiting
|
||||
Container lab3-check-a31f544685-migrate-1 Waiting
|
||||
Container lab3-check-a31f544685-valkey-1 Healthy
|
||||
Container lab3-check-a31f544685-postgres-1 Healthy
|
||||
Container lab3-check-a31f544685-migrate-1 Exited
|
||||
Container lab3-check-a31f544685-api-1 Starting
|
||||
Container lab3-check-a31f544685-api-1 Started
|
||||
API ready through LB: 7f6a80b69e32, f973eaec4d52
|
||||
Stopped API: f973eaec4d52 container=f973eaec4d5290eb73cd96949877ccb8df1213523fce0ec05c6b40c1136bb31b
|
||||
{"label": "stop f973eaec4d52", "duration_seconds": 16.015, "successful_requests": 1574, "successful_writes": 786, "errors": 4, "successful_rps": 98.28, "first_success_after_action_seconds": {"POST": 0.002, "GET": 0.008}, "maximum_success_gap_seconds": {"POST": 2.006, "GET": 2.006}, "action_start_seconds": 1.005, "action_end_seconds": 1.713, "p50_ms": 7.98, "p95_ms": 17.217, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["7f6a80b69e32", "f973eaec4d52"]}
|
||||
Container lab3-check-a31f544685-postgres-1 Waiting
|
||||
Container lab3-check-a31f544685-postgres-1 Healthy
|
||||
Container lab3-check-a31f544685-migrate-1 Starting
|
||||
Container lab3-check-a31f544685-migrate-1 Started
|
||||
Container lab3-check-a31f544685-postgres-1 Waiting
|
||||
Container lab3-check-a31f544685-valkey-1 Waiting
|
||||
Container lab3-check-a31f544685-migrate-1 Waiting
|
||||
Container lab3-check-a31f544685-valkey-1 Healthy
|
||||
Container lab3-check-a31f544685-postgres-1 Healthy
|
||||
Container lab3-check-a31f544685-migrate-1 Exited
|
||||
Container lab3-check-a31f544685-api-2 Starting
|
||||
Container lab3-check-a31f544685-api-2 Started
|
||||
API ready through LB: 7f6a80b69e32, f973eaec4d52
|
||||
Local .env and secrets are ready; values are not printed
|
||||
#1 [internal] load local bake definitions
|
||||
#1 reading from stdin 1.03kB done
|
||||
#1 DONE 0.0s
|
||||
|
||||
#2 [internal] load build definition from Dockerfile
|
||||
#2 transferring dockerfile: 733B done
|
||||
#2 DONE 0.0s
|
||||
|
||||
#3 [internal] load metadata for docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
|
||||
#3 DONE 0.0s
|
||||
|
||||
#4 [internal] load .dockerignore
|
||||
#4 transferring context: 167B done
|
||||
#4 DONE 0.0s
|
||||
|
||||
#5 [internal] load build context
|
||||
#5 transferring context: 809B done
|
||||
#5 DONE 0.0s
|
||||
|
||||
#6 [1/8] FROM docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
|
||||
#6 resolve docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3 0.0s done
|
||||
#6 DONE 0.0s
|
||||
|
||||
#7 [4/8] RUN python -m pip install --no-cache-dir -r requirements.txt && python -m pip check
|
||||
#7 CACHED
|
||||
|
||||
#8 [3/8] COPY requirements.txt .
|
||||
#8 CACHED
|
||||
|
||||
#9 [5/8] RUN groupadd --gid 10001 app && useradd --uid 10001 --gid app --no-create-home app
|
||||
#9 CACHED
|
||||
|
||||
#10 [6/8] COPY src/ ./src/
|
||||
#10 CACHED
|
||||
|
||||
#11 [2/8] WORKDIR /app
|
||||
#11 CACHED
|
||||
|
||||
#12 [7/8] COPY migrations/ ./migrations/
|
||||
#12 CACHED
|
||||
|
||||
#13 [8/8] COPY research/ ./research/
|
||||
#13 CACHED
|
||||
|
||||
#14 exporting to image
|
||||
#14 exporting layers done
|
||||
#14 exporting manifest sha256:06163e45d3676c7cf4f7a1c7bfe108ed3b98559ebb485c7a09d1e87e6f418d17 done
|
||||
#14 exporting config sha256:3fed7460e9510eef7b2928f8d600d06d5d65329f74d820beac13809476b11398 done
|
||||
#14 exporting attestation manifest sha256:3a88ce0975ddcbab53c3ecb185da08ef114e818e9db952983e997cd47add67bf done
|
||||
#14 exporting manifest list sha256:433a8a3293202a92f092574052b474e8373c0d58649cbcf1c7ebc1af17933d3c done
|
||||
#14 naming to docker.io/library/mtusi-messenger:lab3 done
|
||||
#14 unpacking to docker.io/library/mtusi-messenger:lab3 done
|
||||
#14 DONE 0.1s
|
||||
|
||||
#15 resolving provenance for metadata file
|
||||
#15 DONE 0.0s
|
||||
mtusi-messenger:lab3 Built
|
||||
Local .env and secrets are ready; values are not printed
|
||||
Container lab3-check-a31f544685-postgres-1 Running
|
||||
#1 [internal] load local bake definitions
|
||||
#1 reading from stdin 576B done
|
||||
#1 DONE 0.0s
|
||||
|
||||
#2 [internal] load build definition from Dockerfile
|
||||
#2 transferring dockerfile: 733B done
|
||||
#2 DONE 0.0s
|
||||
|
||||
#3 [internal] load metadata for docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
|
||||
#3 DONE 0.0s
|
||||
|
||||
#4 [internal] load .dockerignore
|
||||
#4 transferring context: 167B done
|
||||
#4 DONE 0.0s
|
||||
|
||||
#5 [internal] load build context
|
||||
#5 transferring context: 809B done
|
||||
#5 DONE 0.0s
|
||||
|
||||
#6 [1/8] FROM docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
|
||||
#6 resolve docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3 done
|
||||
#6 DONE 0.0s
|
||||
|
||||
#7 [3/8] COPY requirements.txt .
|
||||
#7 CACHED
|
||||
|
||||
#8 [4/8] RUN python -m pip install --no-cache-dir -r requirements.txt && python -m pip check
|
||||
#8 CACHED
|
||||
|
||||
#9 [5/8] RUN groupadd --gid 10001 app && useradd --uid 10001 --gid app --no-create-home app
|
||||
#9 CACHED
|
||||
|
||||
#10 [6/8] COPY src/ ./src/
|
||||
#10 CACHED
|
||||
|
||||
#11 [7/8] COPY migrations/ ./migrations/
|
||||
#11 CACHED
|
||||
|
||||
#12 [2/8] WORKDIR /app
|
||||
#12 CACHED
|
||||
|
||||
#13 [8/8] COPY research/ ./research/
|
||||
#13 CACHED
|
||||
|
||||
#14 exporting to image
|
||||
#14 exporting layers done
|
||||
#14 exporting manifest sha256:a59568d34e2c5e8b4d23308342601fdaf8c07e78ec3065878679a067c082e952 done
|
||||
#14 exporting config sha256:b67b805ba1b7dd90f0a3e8444455488fe8c7f4118f14d671501dd63e85b59c97 done
|
||||
#14 exporting attestation manifest sha256:25ff7934ca5161da9b6c345af61b6a10b452580e4d349b35939d32feb66664ea done
|
||||
#14 exporting manifest list sha256:3e102cd9a2c1e81e95aed4a0d4de1ac94352370595fced350257813f7a07bef8 done
|
||||
#14 naming to docker.io/library/mtusi-messenger:lab3 done
|
||||
#14 unpacking to docker.io/library/mtusi-messenger:lab3 done
|
||||
#14 DONE 0.0s
|
||||
|
||||
#15 resolving provenance for metadata file
|
||||
#15 DONE 0.0s
|
||||
Migrations: already applied
|
||||
API ready through LB: 7f6a80b69e32, f973eaec4d52
|
||||
Replacing API: 7f6a80b69e32
|
||||
Container lab3-check-a31f544685-api-2 Running
|
||||
Container lab3-check-a31f544685-api-3 Creating
|
||||
Container lab3-check-a31f544685-api-3 Created
|
||||
Container lab3-check-a31f544685-api-3 Starting
|
||||
Container lab3-check-a31f544685-api-3 Started
|
||||
API ready through LB: ca9c7a52da66, f973eaec4d52
|
||||
Replacing API: f973eaec4d52
|
||||
Container lab3-check-a31f544685-api-3 Running
|
||||
Container lab3-check-a31f544685-api-4 Creating
|
||||
Container lab3-check-a31f544685-api-4 Created
|
||||
Container lab3-check-a31f544685-api-4 Starting
|
||||
Container lab3-check-a31f544685-api-4 Started
|
||||
API ready through LB: 244bf0b910eb, ca9c7a52da66
|
||||
PASS: all API containers replaced sequentially; stores and LB kept running
|
||||
{"label": "sequential API replacement", "duration_seconds": 23.774, "successful_requests": 2131, "successful_writes": 1063, "errors": 11, "successful_rps": 89.64, "first_success_after_action_seconds": {"POST": 0.018, "GET": 0.004}, "maximum_success_gap_seconds": {"POST": 1.059, "GET": 2.034}, "action_start_seconds": 1.005, "action_end_seconds": 19.63, "p50_ms": 7.723, "p95_ms": 13.565, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["244bf0b910eb", "7f6a80b69e32", "ca9c7a52da66", "f973eaec4d52"]}
|
||||
PASS: all lab3 grade-4 checks completed
|
||||
Container lab3-check-a31f544685-lb-1 Stopping
|
||||
Container lab3-check-a31f544685-lb-1 Stopped
|
||||
Container lab3-check-a31f544685-lb-1 Removing
|
||||
Container lab3-check-a31f544685-lb-1 Removed
|
||||
Container lab3-check-a31f544685-api-3 Stopping
|
||||
Container lab3-check-a31f544685-api-4 Stopping
|
||||
Container lab3-check-a31f544685-api-4 Stopped
|
||||
Container lab3-check-a31f544685-api-4 Removing
|
||||
Container lab3-check-a31f544685-api-4 Removed
|
||||
Container lab3-check-a31f544685-api-3 Stopped
|
||||
Container lab3-check-a31f544685-api-3 Removing
|
||||
Container lab3-check-a31f544685-api-3 Removed
|
||||
Container lab3-check-a31f544685-migrate-1 Stopping
|
||||
Container lab3-check-a31f544685-valkey-1 Stopping
|
||||
Container lab3-check-a31f544685-migrate-1 Stopped
|
||||
Container lab3-check-a31f544685-migrate-1 Removing
|
||||
Container lab3-check-a31f544685-migrate-1 Removed
|
||||
Container lab3-check-a31f544685-postgres-1 Stopping
|
||||
Container lab3-check-a31f544685-valkey-1 Stopped
|
||||
Container lab3-check-a31f544685-valkey-1 Removing
|
||||
Container lab3-check-a31f544685-valkey-1 Removed
|
||||
Container lab3-check-a31f544685-postgres-1 Stopped
|
||||
Container lab3-check-a31f544685-postgres-1 Removing
|
||||
Container lab3-check-a31f544685-postgres-1 Removed
|
||||
Volume lab3-check-a31f544685_valkey_data Removing
|
||||
Network lab3-check-a31f544685_default Removing
|
||||
Volume lab3-check-a31f544685_postgres_data Removing
|
||||
Volume lab3-check-a31f544685_valkey_data Removed
|
||||
Volume lab3-check-a31f544685_postgres_data Removed
|
||||
Network lab3-check-a31f544685_default Removed
|
||||
Removed only temporary test containers and volumes: lab3-check-a31f544685
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,533 @@
|
||||
Command: ./scripts/check-docker.sh; project: lab4-check-5aad8e0065
|
||||
Docker: 29.0.1 aarch64 CPUs=10 RAM=8217448448
|
||||
Local .env and secrets are ready; values are not printed
|
||||
#1 [internal] load local bake definitions
|
||||
#1 reading from stdin 564B done
|
||||
#1 DONE 0.0s
|
||||
|
||||
#2 [internal] load build definition from Dockerfile
|
||||
#2 transferring dockerfile: 733B done
|
||||
#2 DONE 0.0s
|
||||
|
||||
#3 [internal] load metadata for docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
|
||||
#3 DONE 0.0s
|
||||
|
||||
#4 [internal] load .dockerignore
|
||||
#4 transferring context: 167B done
|
||||
#4 DONE 0.0s
|
||||
|
||||
#5 [internal] load build context
|
||||
#5 transferring context: 1.01kB done
|
||||
#5 DONE 0.0s
|
||||
|
||||
#6 [1/8] FROM docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
|
||||
#6 resolve docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3 done
|
||||
#6 DONE 0.0s
|
||||
|
||||
#7 [4/8] RUN python -m pip install --no-cache-dir -r requirements.txt && python -m pip check
|
||||
#7 CACHED
|
||||
|
||||
#8 [7/8] COPY migrations/ ./migrations/
|
||||
#8 CACHED
|
||||
|
||||
#9 [2/8] WORKDIR /app
|
||||
#9 CACHED
|
||||
|
||||
#10 [6/8] COPY src/ ./src/
|
||||
#10 CACHED
|
||||
|
||||
#11 [3/8] COPY requirements.txt .
|
||||
#11 CACHED
|
||||
|
||||
#12 [5/8] RUN groupadd --gid 10001 app && useradd --uid 10001 --gid app --no-create-home app
|
||||
#12 CACHED
|
||||
|
||||
#13 [8/8] COPY research/ ./research/
|
||||
#13 CACHED
|
||||
|
||||
#14 exporting to image
|
||||
#14 exporting layers done
|
||||
#14 exporting manifest sha256:644f5d408505f5982b75f18404860de692ee837b2b2e18f5c92b0b057081b7ec done
|
||||
#14 exporting config sha256:726011343b32f5b6277348f2f8b38f95b8c31754ae0d21336375124a18aa2e0e done
|
||||
#14 exporting attestation manifest sha256:1146b210a20471d58663fb48a3a24ba22986971d0cce8777f912886768f43306 done
|
||||
#14 exporting manifest list sha256:a04b1bfe2aea8391c8f92dd952499559d766cc4b34091963fe57bcbaf6ec015d done
|
||||
#14 naming to docker.io/library/mtusi-messenger:lab4 done
|
||||
#14 unpacking to docker.io/library/mtusi-messenger:lab4 done
|
||||
#14 DONE 0.1s
|
||||
|
||||
#15 resolving provenance for metadata file
|
||||
#15 DONE 0.0s
|
||||
mtusi-messenger:lab4 Built
|
||||
Network lab4-check-5aad8e0065_default Creating
|
||||
Network lab4-check-5aad8e0065_default Created
|
||||
Network lab4-check-5aad8e0065_s3-internal Creating
|
||||
Network lab4-check-5aad8e0065_s3-internal Created
|
||||
Volume lab4-check-5aad8e0065_valkey_data Creating
|
||||
Volume lab4-check-5aad8e0065_valkey_data Created
|
||||
Volume lab4-check-5aad8e0065_postgres_data Creating
|
||||
Volume lab4-check-5aad8e0065_postgres_data Created
|
||||
Volume lab4-check-5aad8e0065_s3_data Creating
|
||||
Volume lab4-check-5aad8e0065_s3_data Created
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Creating
|
||||
Container lab4-check-5aad8e0065-valkey-1 Creating
|
||||
Container lab4-check-5aad8e0065-postgres-1 Creating
|
||||
Container lab4-check-5aad8e0065-valkey-1 Created
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Created
|
||||
Container lab4-check-5aad8e0065-s3-1 Creating
|
||||
Container lab4-check-5aad8e0065-postgres-1 Created
|
||||
Container lab4-check-5aad8e0065-migrate-1 Creating
|
||||
Container lab4-check-5aad8e0065-s3-1 Created
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Creating
|
||||
Container lab4-check-5aad8e0065-migrate-1 Created
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Created
|
||||
Container lab4-check-5aad8e0065-api-2 Creating
|
||||
Container lab4-check-5aad8e0065-api-1 Creating
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Creating
|
||||
Container lab4-check-5aad8e0065-api-1 Created
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Created
|
||||
Container lab4-check-5aad8e0065-api-2 Created
|
||||
Container lab4-check-5aad8e0065-lb-1 Creating
|
||||
Container lab4-check-5aad8e0065-lb-1 Created
|
||||
Container lab4-check-5aad8e0065-postgres-1 Starting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Started
|
||||
Container lab4-check-5aad8e0065-postgres-1 Started
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Starting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-valkey-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
Container lab4-check-5aad8e0065-api-2 Starting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Starting
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Started
|
||||
Container lab4-check-5aad8e0065-api-2 Started
|
||||
Container lab4-check-5aad8e0065-api-1 Starting
|
||||
Container lab4-check-5aad8e0065-api-1 Started
|
||||
Container lab4-check-5aad8e0065-api-2 Waiting
|
||||
Container lab4-check-5aad8e0065-api-1 Waiting
|
||||
Container lab4-check-5aad8e0065-api-2 Healthy
|
||||
Container lab4-check-5aad8e0065-api-1 Healthy
|
||||
Container lab4-check-5aad8e0065-lb-1 Starting
|
||||
Container lab4-check-5aad8e0065-lb-1 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
PASS: 2 real API containers, CPU/RAM limits, API non-root and no public ports; only LB published
|
||||
Проверяем сервер: http://127.0.0.1:57722
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab4/mtusi2026lab4/.venv/bin/python scripts/check_contract.py
|
||||
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab4/mtusi2026lab4/.venv/bin/python tests/smoke.py --base-url "http://127.0.0.1:57722"
|
||||
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
|
||||
test_chat_access (test_api.ApiTests.test_chat_access) ... ok
|
||||
test_chat_pagination_and_cursor_owner (test_api.ApiTests.test_chat_pagination_and_cursor_owner) ... ok
|
||||
test_cursor_cannot_be_used_for_another_chat_or_changed (test_api.ApiTests.test_cursor_cannot_be_used_for_another_chat_or_changed) ... ok
|
||||
test_default_page_size (test_api.ApiTests.test_default_page_size) ... ok
|
||||
test_duplicate_username (test_api.ApiTests.test_duplicate_username) ... ok
|
||||
test_framework_errors_and_request_ids (test_api.ApiTests.test_framework_errors_and_request_ids) ... ok
|
||||
test_identification_and_authentication_order (test_api.ApiTests.test_identification_and_authentication_order) ... ok
|
||||
test_invalid_chat_members (test_api.ApiTests.test_invalid_chat_members) ... ok
|
||||
test_invalid_chat_title (test_api.ApiTests.test_invalid_chat_title) ... ok
|
||||
test_invalid_limits_and_cursors (test_api.ApiTests.test_invalid_limits_and_cursors) ... ok
|
||||
test_invalid_message_fields (test_api.ApiTests.test_invalid_message_fields) ... ok
|
||||
test_invalid_user_fields (test_api.ApiTests.test_invalid_user_fields) ... ok
|
||||
test_malformed_json_and_encoding (test_api.ApiTests.test_malformed_json_and_encoding) ... ok
|
||||
test_message_pagination_and_changing_limit (test_api.ApiTests.test_message_pagination_and_changing_limit) ... ok
|
||||
test_unicode_and_text_preservation (test_api.ApiTests.test_unicode_and_text_preservation) ... ok
|
||||
test_wrong_content_type (test_api.ApiTests.test_wrong_content_type) ... ok
|
||||
test_access_and_authentication (test_attachments.AttachmentTests.test_access_and_authentication) ... ok
|
||||
test_attachment_only_message_and_other_member (test_attachments.AttachmentTests.test_attachment_only_message_and_other_member) ... ok
|
||||
test_chunked_limit_without_content_length (test_attachments.AttachmentTests.test_chunked_limit_without_content_length) ... ok
|
||||
test_content_length_limit_before_body (test_attachments.AttachmentTests.test_content_length_limit_before_body) ... ok
|
||||
test_corruption_and_mime_spoof (test_attachments.AttachmentTests.test_corruption_and_mime_spoof) ... ok
|
||||
test_filename_rejected_before_storage (test_attachments.AttachmentTests.test_filename_rejected_before_storage) ... ok
|
||||
test_foreign_and_unknown_attachment_ids (test_attachments.AttachmentTests.test_foreign_and_unknown_attachment_ids) ... ok
|
||||
test_idempotency_includes_ordered_attachment_ids (test_attachments.AttachmentTests.test_idempotency_includes_ordered_attachment_ids) ... ok
|
||||
test_invalid_attachment_lists (test_attachments.AttachmentTests.test_invalid_attachment_lists) ... ok
|
||||
test_pixel_limit_before_decode (test_attachments.AttachmentTests.test_pixel_limit_before_decode) ... ok
|
||||
test_png_and_jpeg_originals (test_attachments.AttachmentTests.test_png_and_jpeg_originals) ... ok
|
||||
test_simultaneous_registration_of_one_username (test_concurrency.ConcurrencyTests.test_simultaneous_registration_of_one_username) ... ok
|
||||
test_ten_clients_send_one_hundred_messages (test_concurrency.ConcurrencyTests.test_ten_clients_send_one_hundred_messages) ... ok
|
||||
test_conflict_and_semantic_json_replay (test_idempotency.IdempotencyTests.test_conflict_and_semantic_json_replay) ... ok
|
||||
test_key_format_and_missing_key (test_idempotency.IdempotencyTests.test_key_format_and_missing_key) ... ok
|
||||
test_parallel_replay_across_replicas (test_idempotency.IdempotencyTests.test_parallel_replay_across_replicas) ... ok
|
||||
test_replay_rechecks_session_and_access (test_idempotency.IdempotencyTests.test_replay_rechecks_session_and_access) ... ok
|
||||
test_scope_is_user_and_chat (test_idempotency.IdempotencyTests.test_scope_is_user_and_chat) ... ok
|
||||
test_logout_revokes_only_one_session (test_sessions.SessionTests.test_logout_revokes_only_one_session) ... ok
|
||||
test_password_validation (test_sessions.SessionTests.test_password_validation) ... ok
|
||||
test_sql_like_message_is_plain_text (test_sessions.SessionTests.test_sql_like_message_is_plain_text) ... ok
|
||||
test_unicode_password_with_colon_and_private_response (test_sessions.SessionTests.test_unicode_password_with_colon_and_private_response) ... ok
|
||||
test_wrong_password_and_basic_only_for_login (test_sessions.SessionTests.test_wrong_password_and_basic_only_for_login) ... ok
|
||||
|
||||
----------------------------------------------------------------------
|
||||
Ran 39 tests in 9.593s
|
||||
|
||||
OK
|
||||
PASS: 10 clients sent 100 messages; history contains exactly 100 unique messages.
|
||||
Container lab4-check-5aad8e0065-postgres-1 Running
|
||||
Container lab4-check-5aad8e0065-postgres-1 Running
|
||||
PASS: migrations only in one-shot service; repeated execution preserves data
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Stopping
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Stopped
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Running
|
||||
Container lab4-check-5aad8e0065-s3-1 Running
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Running
|
||||
Container lab4-check-5aad8e0065-s3-1 Running
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
PASS: private S3, limited credentials, isolated administration, repeated bootstrap
|
||||
Stopped API: ca1f62dfc015 container=ca1f62dfc0156fc235dbcd2f52b58eec2282e6ba8d3e056de11ca92758c69df0
|
||||
API ready through LB: 19a474c69210
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-valkey-1 Healthy
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
Container lab4-check-5aad8e0065-api-2 Starting
|
||||
Container lab4-check-5aad8e0065-api-2 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
PASS: original downloaded through another replica after stopping uploader
|
||||
PASS: post-PUT DB failure recorded; cleanup protects fresh, active and ready uploads
|
||||
Container lab4-check-5aad8e0065-s3-1 Stopping
|
||||
Container lab4-check-5aad8e0065-s3-1 Stopped
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-1 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
PASS: S3 outage gives API 503; recovery preserves original and session
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Starting
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Started
|
||||
PASS: 10 concurrent real PNG uploads; all checksums match; memory peaks: [215191552, 103378944]
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
|
||||
Container lab4-check-5aad8e0065-valkey-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-valkey-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab4/mtusi2026lab4/.venv/bin/python tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://127.0.0.1:57722"
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
PASS: both API, LB, PostgreSQL, Valkey, S3 and cleaner actually restarted; data and session preserved
|
||||
PASS: SIGTERM with unfinished request: {"instance": "19a474c69210", "unfinished_request": true, "shutdown_seconds": 5.523, "new_connections_refused_seconds": 0.185, "exit_code": 0, "oom_killed": false}
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
Container lab4-check-5aad8e0065-api-1 Starting
|
||||
Container lab4-check-5aad8e0065-api-1 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
Container lab4-check-5aad8e0065-valkey-1 Stopping
|
||||
Container lab4-check-5aad8e0065-valkey-1 Stopped
|
||||
PASS: valkey outage: API live=200, ready=503, LB ready=503
|
||||
Container lab4-check-5aad8e0065-valkey-1 Starting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
Container lab4-check-5aad8e0065-postgres-1 Stopping
|
||||
Container lab4-check-5aad8e0065-postgres-1 Stopped
|
||||
PASS: postgres outage: API live=200, ready=503, LB ready=503
|
||||
Container lab4-check-5aad8e0065-postgres-1 Starting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
Container lab4-check-5aad8e0065-s3-1 Stopping
|
||||
Container lab4-check-5aad8e0065-s3-1 Stopped
|
||||
PASS: s3 outage: API live=200, ready=503, LB ready=503
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-1 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
Container lab4-check-5aad8e0065-postgres-1 Paused
|
||||
Container lab4-check-5aad8e0065-postgres-1 Unpaused
|
||||
PASS: paused PostgreSQL: SQL and direct readiness bounded; live works; unpause restores API without restart
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab4/mtusi2026lab4/.venv/bin/python tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:57722"
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||
{"observed_instances_before": 2, "successful_writes": 110, "failed_attempts": 1, "first_recovery_seconds": 2.467}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Starting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
Container lab4-check-5aad8e0065-api-1 Starting
|
||||
Container lab4-check-5aad8e0065-api-1 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
Stopped API: 19a474c69210 container=19a474c69210a7be1c6631830e8fa7f251cf6dd3890c7407843090383543ee75
|
||||
{"label": "stop 19a474c69210", "duration_seconds": 16.012, "successful_requests": 1420, "successful_writes": 709, "errors": 4, "successful_rps": 88.68, "first_success_after_action_seconds": {"POST": 0.051, "GET": 0.004}, "maximum_success_gap_seconds": {"POST": 2.002, "GET": 1.995}, "action_start_seconds": 1.006, "action_end_seconds": 1.77, "p50_ms": 10.62, "p95_ms": 24.67, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["19a474c69210", "ca1f62dfc015"]}
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Starting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Healthy
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
Container lab4-check-5aad8e0065-api-1 Starting
|
||||
Container lab4-check-5aad8e0065-api-1 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
Stopped API: ca1f62dfc015 container=ca1f62dfc0156fc235dbcd2f52b58eec2282e6ba8d3e056de11ca92758c69df0
|
||||
{"label": "stop ca1f62dfc015", "duration_seconds": 16.008, "successful_requests": 1466, "successful_writes": 732, "errors": 4, "successful_rps": 91.58, "first_success_after_action_seconds": {"POST": 0.018, "GET": 0.028}, "maximum_success_gap_seconds": {"POST": 2.002, "GET": 1.999}, "action_start_seconds": 1.002, "action_end_seconds": 1.832, "p50_ms": 9.397, "p95_ms": 18.616, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["19a474c69210", "ca1f62dfc015"]}
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Healthy
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
Container lab4-check-5aad8e0065-api-2 Starting
|
||||
Container lab4-check-5aad8e0065-api-2 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
Local .env and secrets are ready; values are not printed
|
||||
#1 [internal] load local bake definitions
|
||||
#1 reading from stdin 564B done
|
||||
#1 DONE 0.0s
|
||||
|
||||
#2 [internal] load build definition from Dockerfile
|
||||
#2 transferring dockerfile: 733B done
|
||||
#2 DONE 0.0s
|
||||
|
||||
#3 [internal] load metadata for docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
|
||||
#3 DONE 0.0s
|
||||
|
||||
#4 [internal] load .dockerignore
|
||||
#4 transferring context: 167B done
|
||||
#4 DONE 0.0s
|
||||
|
||||
#5 [internal] load build context
|
||||
#5 transferring context: 1.01kB done
|
||||
#5 DONE 0.0s
|
||||
|
||||
#6 [1/8] FROM docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
|
||||
#6 resolve docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3 0.0s done
|
||||
#6 DONE 0.0s
|
||||
|
||||
#7 [3/8] COPY requirements.txt .
|
||||
#7 CACHED
|
||||
|
||||
#8 [6/8] COPY src/ ./src/
|
||||
#8 CACHED
|
||||
|
||||
#9 [2/8] WORKDIR /app
|
||||
#9 CACHED
|
||||
|
||||
#10 [7/8] COPY migrations/ ./migrations/
|
||||
#10 CACHED
|
||||
|
||||
#11 [5/8] RUN groupadd --gid 10001 app && useradd --uid 10001 --gid app --no-create-home app
|
||||
#11 CACHED
|
||||
|
||||
#12 [4/8] RUN python -m pip install --no-cache-dir -r requirements.txt && python -m pip check
|
||||
#12 CACHED
|
||||
|
||||
#13 [8/8] COPY research/ ./research/
|
||||
#13 CACHED
|
||||
|
||||
#14 exporting to image
|
||||
#14 exporting layers done
|
||||
#14 exporting manifest sha256:644f5d408505f5982b75f18404860de692ee837b2b2e18f5c92b0b057081b7ec done
|
||||
#14 exporting config sha256:726011343b32f5b6277348f2f8b38f95b8c31754ae0d21336375124a18aa2e0e done
|
||||
#14 exporting attestation manifest sha256:f4a5559518ea1ef0e3747d6084c8ad8dbc8b16dc41cb00062303d0f895b9a235 0.0s done
|
||||
#14 exporting manifest list sha256:93c838d55d79a0001346ecab28f72621d9540ba5c52eaeb1ca6020286dd05fd6 done
|
||||
#14 naming to docker.io/library/mtusi-messenger:lab4 done
|
||||
#14 unpacking to docker.io/library/mtusi-messenger:lab4 done
|
||||
#14 DONE 0.1s
|
||||
|
||||
#15 resolving provenance for metadata file
|
||||
#15 DONE 0.0s
|
||||
mtusi-messenger:lab4 Built
|
||||
Container lab4-check-5aad8e0065-postgres-1 Running
|
||||
Migrations: already applied
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
Replacing API: 19a474c69210
|
||||
Container lab4-check-5aad8e0065-api-2 Running
|
||||
Container lab4-check-5aad8e0065-api-3 Creating
|
||||
Container lab4-check-5aad8e0065-api-3 Created
|
||||
Container lab4-check-5aad8e0065-api-3 Starting
|
||||
Container lab4-check-5aad8e0065-api-3 Started
|
||||
API ready through LB: ca1f62dfc015, ed4f3c64a9f6
|
||||
Replacing API: ca1f62dfc015
|
||||
Container lab4-check-5aad8e0065-api-3 Running
|
||||
Container lab4-check-5aad8e0065-api-4 Creating
|
||||
Container lab4-check-5aad8e0065-api-4 Created
|
||||
Container lab4-check-5aad8e0065-api-4 Starting
|
||||
Container lab4-check-5aad8e0065-api-4 Started
|
||||
API ready through LB: 63549c0d63ba, ed4f3c64a9f6
|
||||
PASS: all API containers replaced sequentially; stores and LB kept running
|
||||
{"label": "sequential API replacement", "duration_seconds": 24.869, "successful_requests": 2254, "successful_writes": 1128, "errors": 8, "successful_rps": 90.63, "first_success_after_action_seconds": {"POST": 0.01, "GET": 0.021}, "maximum_success_gap_seconds": {"POST": 1.04, "GET": 1.04}, "action_start_seconds": 1.009, "action_end_seconds": 21.862, "p50_ms": 8.907, "p95_ms": 16.321, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["19a474c69210", "63549c0d63ba", "ca1f62dfc015", "ed4f3c64a9f6"]}
|
||||
PASS: all lab4 grade-4 checks completed
|
||||
Container lab4-check-5aad8e0065-lb-1 Stopping
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Stopping
|
||||
Container lab4-check-5aad8e0065-lb-1 Stopped
|
||||
Container lab4-check-5aad8e0065-lb-1 Removing
|
||||
Container lab4-check-5aad8e0065-lb-1 Removed
|
||||
Container lab4-check-5aad8e0065-api-3 Stopping
|
||||
Container lab4-check-5aad8e0065-api-4 Stopping
|
||||
Container lab4-check-5aad8e0065-api-3 Stopped
|
||||
Container lab4-check-5aad8e0065-api-3 Removing
|
||||
Container lab4-check-5aad8e0065-api-3 Removed
|
||||
Container lab4-check-5aad8e0065-api-4 Stopped
|
||||
Container lab4-check-5aad8e0065-api-4 Removing
|
||||
Container lab4-check-5aad8e0065-api-4 Removed
|
||||
Container lab4-check-5aad8e0065-valkey-1 Stopping
|
||||
Container lab4-check-5aad8e0065-valkey-1 Stopped
|
||||
Container lab4-check-5aad8e0065-valkey-1 Removing
|
||||
Container lab4-check-5aad8e0065-valkey-1 Removed
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Stopped
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Removing
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Removed
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Stopping
|
||||
Container lab4-check-5aad8e0065-migrate-1 Stopping
|
||||
Container lab4-check-5aad8e0065-migrate-1 Stopped
|
||||
Container lab4-check-5aad8e0065-migrate-1 Removing
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Stopped
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Removing
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Removed
|
||||
Container lab4-check-5aad8e0065-s3-1 Stopping
|
||||
Container lab4-check-5aad8e0065-migrate-1 Removed
|
||||
Container lab4-check-5aad8e0065-postgres-1 Stopping
|
||||
Container lab4-check-5aad8e0065-postgres-1 Stopped
|
||||
Container lab4-check-5aad8e0065-postgres-1 Removing
|
||||
Container lab4-check-5aad8e0065-postgres-1 Removed
|
||||
Container lab4-check-5aad8e0065-s3-1 Stopped
|
||||
Container lab4-check-5aad8e0065-s3-1 Removing
|
||||
Container lab4-check-5aad8e0065-s3-1 Removed
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Stopping
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Stopped
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Removing
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Removed
|
||||
Volume lab4-check-5aad8e0065_valkey_data Removing
|
||||
Network lab4-check-5aad8e0065_s3-internal Removing
|
||||
Volume lab4-check-5aad8e0065_postgres_data Removing
|
||||
Volume lab4-check-5aad8e0065_s3_data Removing
|
||||
Network lab4-check-5aad8e0065_default Removing
|
||||
Volume lab4-check-5aad8e0065_valkey_data Removed
|
||||
Volume lab4-check-5aad8e0065_s3_data Removed
|
||||
Volume lab4-check-5aad8e0065_postgres_data Removed
|
||||
Network lab4-check-5aad8e0065_s3-internal Removed
|
||||
Network lab4-check-5aad8e0065_default Removed
|
||||
Removed only temporary test containers and volumes: lab4-check-5aad8e0065
|
||||
@@ -0,0 +1,502 @@
|
||||
{
|
||||
"utc": "2026-11-12T15:23:27.573351+00:00",
|
||||
"command": "./scripts/check-docker.sh",
|
||||
"project": "lab5-check-d233766d34",
|
||||
"host": "Darwin arm64",
|
||||
"source_commit": "276dbdad8a34da4c5b57fbf1879ca169b581d75b",
|
||||
"working_tree": true,
|
||||
"configuration": {
|
||||
"concurrency": 2,
|
||||
"prefetch": 1,
|
||||
"attempts": 5,
|
||||
"timeout_seconds": 5,
|
||||
"lease_seconds": 15,
|
||||
"backoff_base_seconds": 0.15
|
||||
},
|
||||
"checks": {
|
||||
"topology_and_limits": {
|
||||
"passed": true,
|
||||
"api_replicas": 2,
|
||||
"broker": "Valkey Streams with AOF fsync always",
|
||||
"worker_processes": 2
|
||||
},
|
||||
"public_smoke_and_schema": {
|
||||
"passed": true
|
||||
},
|
||||
"student_api_regressions": {
|
||||
"passed": true
|
||||
},
|
||||
"worker_offline_api_broker_restart": {
|
||||
"passed": true,
|
||||
"attachment_id": "63406ab0-cc3c-4fa2-a0aa-e5e3353d5ae3",
|
||||
"job_id": "7f450ff7-cb2d-4189-a668-f1967dbd94e8",
|
||||
"message_id": "bc37bafb-efe1-4481-b304-c725c6e8368d",
|
||||
"variants": {
|
||||
"thumbnail": {
|
||||
"size": [
|
||||
256,
|
||||
160
|
||||
],
|
||||
"bytes": 5813
|
||||
},
|
||||
"crop": {
|
||||
"size": [
|
||||
128,
|
||||
128
|
||||
],
|
||||
"bytes": 2462
|
||||
},
|
||||
"optimized": {
|
||||
"size": [
|
||||
320,
|
||||
200
|
||||
],
|
||||
"bytes": 2462
|
||||
}
|
||||
}
|
||||
},
|
||||
"outbox_crash_before_publish": {
|
||||
"passed": true,
|
||||
"job_id": "a3973e2b-ff73-4bca-bc41-394f2b875d13",
|
||||
"automatic_recovery": true
|
||||
},
|
||||
"outbox_crash_after_publish": {
|
||||
"passed": true,
|
||||
"job_id": "007cb1d1-4b3b-4d12-9b2b-91a4682e8b08",
|
||||
"automatic_recovery": true
|
||||
},
|
||||
"worker_crash_after_objects": {
|
||||
"passed": true,
|
||||
"job_id": "21129707-7dd0-4a72-ac33-5691dae3194e",
|
||||
"attempt": 2,
|
||||
"canonical_objects": 4,
|
||||
"duplicate_noop": true
|
||||
},
|
||||
"worker_crash_before_ack": {
|
||||
"passed": true,
|
||||
"job_id": "2eeba68e-3fc7-42e0-adf9-13ac01c6c982",
|
||||
"attempt": 1,
|
||||
"canonical_objects": 4,
|
||||
"duplicate_noop": true
|
||||
},
|
||||
"bounded_retry_failed_registry_replay": {
|
||||
"passed": true,
|
||||
"job_id": "bf756bf3-c3ca-45d2-acdd-3e67eb882d4d",
|
||||
"max_automatic_attempts": 5,
|
||||
"audit": [
|
||||
{
|
||||
"event": "processing",
|
||||
"attempt": 1,
|
||||
"error_code": null,
|
||||
"at": "2026-11-12T15:25:18.431558+00:00"
|
||||
},
|
||||
{
|
||||
"event": "retry",
|
||||
"attempt": 1,
|
||||
"error_code": "storage_unavailable",
|
||||
"at": "2026-11-12T15:25:18.432821+00:00"
|
||||
},
|
||||
{
|
||||
"event": "processing",
|
||||
"attempt": 2,
|
||||
"error_code": null,
|
||||
"at": "2026-11-12T15:25:18.692895+00:00"
|
||||
},
|
||||
{
|
||||
"event": "retry",
|
||||
"attempt": 2,
|
||||
"error_code": "storage_unavailable",
|
||||
"at": "2026-11-12T15:25:18.69618+00:00"
|
||||
},
|
||||
{
|
||||
"event": "processing",
|
||||
"attempt": 3,
|
||||
"error_code": null,
|
||||
"at": "2026-11-12T15:25:19.206061+00:00"
|
||||
},
|
||||
{
|
||||
"event": "retry",
|
||||
"attempt": 3,
|
||||
"error_code": "storage_unavailable",
|
||||
"at": "2026-11-12T15:25:19.208065+00:00"
|
||||
},
|
||||
{
|
||||
"event": "processing",
|
||||
"attempt": 4,
|
||||
"error_code": null,
|
||||
"at": "2026-11-12T15:25:19.97778+00:00"
|
||||
},
|
||||
{
|
||||
"event": "retry",
|
||||
"attempt": 4,
|
||||
"error_code": "storage_unavailable",
|
||||
"at": "2026-11-12T15:25:19.978984+00:00"
|
||||
},
|
||||
{
|
||||
"event": "processing",
|
||||
"attempt": 5,
|
||||
"error_code": null,
|
||||
"at": "2026-11-12T15:25:21.777757+00:00"
|
||||
},
|
||||
{
|
||||
"event": "failed",
|
||||
"attempt": 5,
|
||||
"error_code": "retry_exhausted",
|
||||
"at": "2026-11-12T15:25:21.78129+00:00"
|
||||
},
|
||||
{
|
||||
"event": "operator_replay",
|
||||
"attempt": 5,
|
||||
"error_code": null,
|
||||
"at": "2026-11-12T15:25:22.680791+00:00"
|
||||
},
|
||||
{
|
||||
"event": "processing",
|
||||
"attempt": 6,
|
||||
"error_code": null,
|
||||
"at": "2026-11-12T15:25:22.802027+00:00"
|
||||
},
|
||||
{
|
||||
"event": "ready",
|
||||
"attempt": 6,
|
||||
"error_code": null,
|
||||
"at": "2026-11-12T15:25:23.151043+00:00"
|
||||
}
|
||||
],
|
||||
"ready_replay_rejected": true
|
||||
},
|
||||
"permanent_error": {
|
||||
"passed": true,
|
||||
"job_id": "c2056d32-08bf-4c8a-8980-2f8b6c2511c9",
|
||||
"attempt": 1,
|
||||
"error_code": "invalid_image"
|
||||
},
|
||||
"image_subprocess_timeout": {
|
||||
"passed": true,
|
||||
"job_id": "6e8efff5-d82d-45f4-af12-3b05dcdb0076",
|
||||
"timeout_seconds": 5,
|
||||
"task_memory_mb": 640,
|
||||
"recovered": true
|
||||
},
|
||||
"image_subprocess_memory": {
|
||||
"passed": true,
|
||||
"job_id": "ff1e7e76-b82d-4ba1-8ca9-79f3d78a4dfe",
|
||||
"timeout_seconds": 5,
|
||||
"task_memory_mb": 640,
|
||||
"recovered": true
|
||||
},
|
||||
"image_semantics": {
|
||||
"passed": true,
|
||||
"cases": [
|
||||
{
|
||||
"case": "small_alpha_on_white",
|
||||
"variants": {
|
||||
"thumbnail": {
|
||||
"size": [
|
||||
17,
|
||||
9
|
||||
],
|
||||
"bytes": 635
|
||||
},
|
||||
"crop": {
|
||||
"size": [
|
||||
128,
|
||||
128
|
||||
],
|
||||
"bytes": 883
|
||||
},
|
||||
"optimized": {
|
||||
"size": [
|
||||
17,
|
||||
9
|
||||
],
|
||||
"bytes": 56
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"case": "exif_orientation_6_and_floor",
|
||||
"variants": {
|
||||
"thumbnail": {
|
||||
"size": [
|
||||
160,
|
||||
256
|
||||
],
|
||||
"bytes": 1270
|
||||
},
|
||||
"crop": {
|
||||
"size": [
|
||||
128,
|
||||
128
|
||||
],
|
||||
"bytes": 886
|
||||
},
|
||||
"optimized": {
|
||||
"size": [
|
||||
201,
|
||||
321
|
||||
],
|
||||
"bytes": 210
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"staging_cleanup": {
|
||||
"passed": true,
|
||||
"active_preserved": true,
|
||||
"orphan_removed": true,
|
||||
"ready_removed": true,
|
||||
"expired_failed_removed": true,
|
||||
"expired_replay_rejected": true
|
||||
},
|
||||
"broker_offline_outbox": {
|
||||
"passed": true,
|
||||
"attachment_id": "3b3d3600-efcc-4840-ab76-07868501c21d",
|
||||
"recovered": true
|
||||
},
|
||||
"public_persistence": {
|
||||
"passed": true
|
||||
},
|
||||
"public_api_failover": {
|
||||
"passed": true
|
||||
}
|
||||
},
|
||||
"docker": "29.0.1 aarch64 CPUs=10 RAM=8217448448",
|
||||
"containers": [
|
||||
{
|
||||
"id": "9fbe552bf1bd8c2be76b3434cc5662479ca8d328ca698813ac9a785bcafc0cfd",
|
||||
"service": "api",
|
||||
"image": "mtusi-messenger:lab5",
|
||||
"memory_bytes": 402653184,
|
||||
"nano_cpus": 1000000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/run/secrets/s3_secret_access_key",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/database_url",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/cache_url",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/s3_access_key_id",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "ebc82ed28847cbaf6f474088c68005f05d49469010b45ff3767fb0a944ead6e0",
|
||||
"service": "api",
|
||||
"image": "mtusi-messenger:lab5",
|
||||
"memory_bytes": 402653184,
|
||||
"nano_cpus": 1000000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/run/secrets/s3_access_key_id",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/s3_secret_access_key",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/database_url",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/cache_url",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "26a48bd7b3390a4cd99667d4d99d6342edce8e26accaee3b181a4d842203261a",
|
||||
"service": "broker",
|
||||
"image": "valkey/valkey:8.1.3",
|
||||
"memory_bytes": 268435456,
|
||||
"nano_cpus": 500000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/data",
|
||||
"name": "lab5-check-d233766d34_broker_data"
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/broker_password",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "232fdc281b939bf905b4123102a6a6eb609a4ffcb730f6d03707beb07283d69c",
|
||||
"service": "cleanup",
|
||||
"image": "mtusi-messenger:lab5",
|
||||
"memory_bytes": 402653184,
|
||||
"nano_cpus": 1000000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/run/secrets/database_url",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/s3_access_key_id",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/s3_secret_access_key",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "e722801286bb112062f1da59af65dc21eddf5059ca086022e0d52f85a0f26154",
|
||||
"service": "lb",
|
||||
"image": "haproxy:3.2.6-alpine",
|
||||
"memory_bytes": 67108864,
|
||||
"nano_cpus": 500000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/usr/local/etc/haproxy/haproxy.cfg",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "ba806a4ce014a516ea867a516743de64bef0d449726bb544d265bf31b4055b1b",
|
||||
"service": "migrate",
|
||||
"image": "mtusi-messenger:lab5",
|
||||
"memory_bytes": 402653184,
|
||||
"nano_cpus": 1000000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/run/secrets/database_url",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "0fd62d960020290ba4089d05fdcc8da2bb0e2faf7528fc6ed3384e35236d4290",
|
||||
"service": "postgres",
|
||||
"image": "postgres:17.6-bookworm",
|
||||
"memory_bytes": 536870912,
|
||||
"nano_cpus": 1000000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/var/lib/postgresql/data",
|
||||
"name": "lab5-check-d233766d34_postgres_data"
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/postgres_password",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "c658ec3ef1e14ca2f1ca8f1530e6f13b37b99ce8389fafeae84e841e65daa2fc",
|
||||
"service": "publisher",
|
||||
"image": "mtusi-messenger:lab5",
|
||||
"memory_bytes": 402653184,
|
||||
"nano_cpus": 1000000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/run/secrets/database_url",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/broker_url",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "bc944a3a03374052aba43f42767c253755a921090fd15ead92fa2a924507c941",
|
||||
"service": "s3",
|
||||
"image": "chrislusf/seaweedfs:3.85@sha256:49312939c00c01e5ee6afbd7d728b18027821d3764c35a797a72acd4fdf3296a",
|
||||
"memory_bytes": 268435456,
|
||||
"nano_cpus": 500000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/run/secrets/s3_config",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/data",
|
||||
"name": "0496c4617fd5cac2843b4c391fc1eb42e15c90c4d9a35d092e71a6fd8815960f"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "9e7382c34331cb65974fa3feb248105c8055fa25c251151f0f004b65fc388923",
|
||||
"service": "s3-bootstrap",
|
||||
"image": "mtusi-messenger:lab5",
|
||||
"memory_bytes": 134217728,
|
||||
"nano_cpus": 500000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/run/secrets/s3_root_password",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/s3_root_user",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "b9821d3ec48ce18b8255a446cf99b02d6ea9658de406a15cc0ace7ccf7d7350e",
|
||||
"service": "s3-storage",
|
||||
"image": "chrislusf/seaweedfs:3.85@sha256:49312939c00c01e5ee6afbd7d728b18027821d3764c35a797a72acd4fdf3296a",
|
||||
"memory_bytes": 536870912,
|
||||
"nano_cpus": 1000000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/data",
|
||||
"name": "lab5-check-d233766d34_s3_data"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "c8ddc9fede2be42d05e79147dfc7dd87efede3b67f3226afeee1161fcaa49428",
|
||||
"service": "valkey",
|
||||
"image": "valkey/valkey:8.1.3",
|
||||
"memory_bytes": 134217728,
|
||||
"nano_cpus": 500000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/data",
|
||||
"name": "lab5-check-d233766d34_valkey_data"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "9fee2e7832c5c11e81bb56df9d435ad090fdb1fddb85efbf014ea6019559ed75",
|
||||
"service": "worker",
|
||||
"image": "mtusi-messenger:lab5",
|
||||
"memory_bytes": 1610612736,
|
||||
"nano_cpus": 2000000000,
|
||||
"mounts": [
|
||||
{
|
||||
"destination": "/run/secrets/broker_url",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/s3_access_key_id",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/s3_secret_access_key",
|
||||
"name": null
|
||||
},
|
||||
{
|
||||
"destination": "/run/secrets/database_url",
|
||||
"name": null
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"passed": true
|
||||
}
|
||||
@@ -0,0 +1,77 @@
|
||||
PASS: topology_and_limits {"api_replicas": 2, "broker": "Valkey Streams with AOF fsync always", "worker_processes": 2}
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab5/mtusi2026lab5/.venv/bin/python scripts/check_contract.py
|
||||
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab5/mtusi2026lab5/.venv/bin/python scripts/validate_openapi.py
|
||||
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab5/mtusi2026lab5/.venv/bin/python tests/smoke.py --base-url "http://127.0.0.1:57277"
|
||||
PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.
|
||||
PASS: public_smoke_and_schema {}
|
||||
PASS: 10 clients sent 100 messages; history contains exactly 100 unique messages.
|
||||
test_chat_access (test_api.ApiTests.test_chat_access) ... ok
|
||||
test_chat_pagination_and_cursor_owner (test_api.ApiTests.test_chat_pagination_and_cursor_owner) ... ok
|
||||
test_cursor_cannot_be_used_for_another_chat_or_changed (test_api.ApiTests.test_cursor_cannot_be_used_for_another_chat_or_changed) ... ok
|
||||
test_default_page_size (test_api.ApiTests.test_default_page_size) ... ok
|
||||
test_duplicate_username (test_api.ApiTests.test_duplicate_username) ... ok
|
||||
test_framework_errors_and_request_ids (test_api.ApiTests.test_framework_errors_and_request_ids) ... ok
|
||||
test_identification_and_authentication_order (test_api.ApiTests.test_identification_and_authentication_order) ... ok
|
||||
test_invalid_chat_members (test_api.ApiTests.test_invalid_chat_members) ... ok
|
||||
test_invalid_chat_title (test_api.ApiTests.test_invalid_chat_title) ... ok
|
||||
test_invalid_limits_and_cursors (test_api.ApiTests.test_invalid_limits_and_cursors) ... ok
|
||||
test_invalid_message_fields (test_api.ApiTests.test_invalid_message_fields) ... ok
|
||||
test_invalid_user_fields (test_api.ApiTests.test_invalid_user_fields) ... ok
|
||||
test_malformed_json_and_encoding (test_api.ApiTests.test_malformed_json_and_encoding) ... ok
|
||||
test_message_pagination_and_changing_limit (test_api.ApiTests.test_message_pagination_and_changing_limit) ... ok
|
||||
test_unicode_and_text_preservation (test_api.ApiTests.test_unicode_and_text_preservation) ... ok
|
||||
test_wrong_content_type (test_api.ApiTests.test_wrong_content_type) ... ok
|
||||
test_access_and_authentication (test_attachments.AttachmentTests.test_access_and_authentication) ... ok
|
||||
test_attachment_only_message_and_other_member (test_attachments.AttachmentTests.test_attachment_only_message_and_other_member) ... ok
|
||||
test_chunked_limit_without_content_length (test_attachments.AttachmentTests.test_chunked_limit_without_content_length) ... ok
|
||||
test_content_length_limit_before_body (test_attachments.AttachmentTests.test_content_length_limit_before_body) ... ok
|
||||
test_corruption_and_mime_spoof (test_attachments.AttachmentTests.test_corruption_and_mime_spoof) ... ok
|
||||
test_filename_rejected_before_storage (test_attachments.AttachmentTests.test_filename_rejected_before_storage) ... ok
|
||||
test_foreign_and_unknown_attachment_ids (test_attachments.AttachmentTests.test_foreign_and_unknown_attachment_ids) ... ok
|
||||
test_idempotency_includes_ordered_attachment_ids (test_attachments.AttachmentTests.test_idempotency_includes_ordered_attachment_ids) ... ok
|
||||
test_invalid_attachment_lists (test_attachments.AttachmentTests.test_invalid_attachment_lists) ... ok
|
||||
test_pixel_limit_before_decode (test_attachments.AttachmentTests.test_pixel_limit_before_decode) ... ok
|
||||
test_png_and_jpeg_originals (test_attachments.AttachmentTests.test_png_and_jpeg_originals) ... ok
|
||||
test_simultaneous_registration_of_one_username (test_concurrency.ConcurrencyTests.test_simultaneous_registration_of_one_username) ... ok
|
||||
test_ten_clients_send_one_hundred_messages (test_concurrency.ConcurrencyTests.test_ten_clients_send_one_hundred_messages) ... ok
|
||||
test_conflict_and_semantic_json_replay (test_idempotency.IdempotencyTests.test_conflict_and_semantic_json_replay) ... ok
|
||||
test_key_format_and_missing_key (test_idempotency.IdempotencyTests.test_key_format_and_missing_key) ... ok
|
||||
test_parallel_replay_across_replicas (test_idempotency.IdempotencyTests.test_parallel_replay_across_replicas) ... ok
|
||||
test_replay_rechecks_session_and_access (test_idempotency.IdempotencyTests.test_replay_rechecks_session_and_access) ... ok
|
||||
test_scope_is_user_and_chat (test_idempotency.IdempotencyTests.test_scope_is_user_and_chat) ... ok
|
||||
test_logout_revokes_only_one_session (test_sessions.SessionTests.test_logout_revokes_only_one_session) ... ok
|
||||
test_password_validation (test_sessions.SessionTests.test_password_validation) ... ok
|
||||
test_sql_like_message_is_plain_text (test_sessions.SessionTests.test_sql_like_message_is_plain_text) ... ok
|
||||
test_unicode_password_with_colon_and_private_response (test_sessions.SessionTests.test_unicode_password_with_colon_and_private_response) ... ok
|
||||
test_wrong_password_and_basic_only_for_login (test_sessions.SessionTests.test_wrong_password_and_basic_only_for_login) ... ok
|
||||
|
||||
----------------------------------------------------------------------
|
||||
Ran 39 tests in 6.981s
|
||||
|
||||
OK
|
||||
PASS: student_api_regressions {}
|
||||
PASS: worker_offline_api_broker_restart {"attachment_id": "63406ab0-cc3c-4fa2-a0aa-e5e3353d5ae3", "job_id": "7f450ff7-cb2d-4189-a668-f1967dbd94e8", "message_id": "bc37bafb-efe1-4481-b304-c725c6e8368d", "variants": {"thumbnail": {"size": [256, 160], "bytes": 5813}, "crop": {"size": [128, 128], "bytes": 2462}, "optimized": {"size": [320, 200], "bytes": 2462}}}
|
||||
PASS: outbox_crash_before_publish {"job_id": "a3973e2b-ff73-4bca-bc41-394f2b875d13", "automatic_recovery": true}
|
||||
PASS: outbox_crash_after_publish {"job_id": "007cb1d1-4b3b-4d12-9b2b-91a4682e8b08", "automatic_recovery": true}
|
||||
PASS: worker_crash_after_objects {"job_id": "21129707-7dd0-4a72-ac33-5691dae3194e", "attempt": 2, "canonical_objects": 4, "duplicate_noop": true}
|
||||
PASS: worker_crash_before_ack {"job_id": "2eeba68e-3fc7-42e0-adf9-13ac01c6c982", "attempt": 1, "canonical_objects": 4, "duplicate_noop": true}
|
||||
PASS: bounded_retry_failed_registry_replay {"job_id": "bf756bf3-c3ca-45d2-acdd-3e67eb882d4d", "max_automatic_attempts": 5, "audit": [{"event": "processing", "attempt": 1, "error_code": null, "at": "2026-11-12T15:25:18.431558+00:00"}, {"event": "retry", "attempt": 1, "error_code": "storage_unavailable", "at": "2026-11-12T15:25:18.432821+00:00"}, {"event": "processing", "attempt": 2, "error_code": null, "at": "2026-11-12T15:25:18.692895+00:00"}, {"event": "retry", "attempt": 2, "error_code": "storage_unavailable", "at": "2026-11-12T15:25:18.69618+00:00"}, {"event": "processing", "attempt": 3, "error_code": null, "at": "2026-11-12T15:25:19.206061+00:00"}, {"event": "retry", "attempt": 3, "error_code": "storage_unavailable", "at": "2026-11-12T15:25:19.208065+00:00"}, {"event": "processing", "attempt": 4, "error_code": null, "at": "2026-11-12T15:25:19.97778+00:00"}, {"event": "retry", "attempt": 4, "error_code": "storage_unavailable", "at": "2026-11-12T15:25:19.978984+00:00"}, {"event": "processing", "attempt": 5, "error_code": null, "at": "2026-11-12T15:25:21.777757+00:00"}, {"event": "failed", "attempt": 5, "error_code": "retry_exhausted", "at": "2026-11-12T15:25:21.78129+00:00"}, {"event": "operator_replay", "attempt": 5, "error_code": null, "at": "2026-11-12T15:25:22.680791+00:00"}, {"event": "processing", "attempt": 6, "error_code": null, "at": "2026-11-12T15:25:22.802027+00:00"}, {"event": "ready", "attempt": 6, "error_code": null, "at": "2026-11-12T15:25:23.151043+00:00"}], "ready_replay_rejected": true}
|
||||
PASS: permanent_error {"job_id": "c2056d32-08bf-4c8a-8980-2f8b6c2511c9", "attempt": 1, "error_code": "invalid_image"}
|
||||
PASS: image_subprocess_timeout {"job_id": "6e8efff5-d82d-45f4-af12-3b05dcdb0076", "timeout_seconds": 5, "task_memory_mb": 640, "recovered": true}
|
||||
PASS: image_subprocess_memory {"job_id": "ff1e7e76-b82d-4ba1-8ca9-79f3d78a4dfe", "timeout_seconds": 5, "task_memory_mb": 640, "recovered": true}
|
||||
PASS: image_semantics {"cases": [{"case": "small_alpha_on_white", "variants": {"thumbnail": {"size": [17, 9], "bytes": 635}, "crop": {"size": [128, 128], "bytes": 883}, "optimized": {"size": [17, 9], "bytes": 56}}}, {"case": "exif_orientation_6_and_floor", "variants": {"thumbnail": {"size": [160, 256], "bytes": 1270}, "crop": {"size": [128, 128], "bytes": 886}, "optimized": {"size": [201, 321], "bytes": 210}}}]}
|
||||
PASS: staging_cleanup {"active_preserved": true, "orphan_removed": true, "ready_removed": true, "expired_failed_removed": true, "expired_replay_rejected": true}
|
||||
PASS: broker_offline_outbox {"attachment_id": "3b3d3600-efcc-4840-ab76-07868501c21d", "recovered": true}
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab5/mtusi2026lab5/.venv/bin/python tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://127.0.0.1:57277"
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
PASS: public_persistence {}
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab5/mtusi2026lab5/.venv/bin/python tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:57277"
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||
{"observed_instances_before": 2, "successful_writes": 110, "failed_attempts": 1, "first_recovery_seconds": 2.797}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
PASS: public_api_failover {}
|
||||
PASS: all lab5 grade-4 criteria completed
|
||||
Removed isolated test stack: lab5-check-d233766d34
|
||||
@@ -0,0 +1,140 @@
|
||||
{
|
||||
"utc": "2026-11-12T15:28:31.543303+00:00",
|
||||
"command": ".venv/bin/python student-tests/verify_restart.py",
|
||||
"passed": true,
|
||||
"before": {
|
||||
"9e5a9d3d2bf0b2e47ee6d85b40b1899881db0d4d91d9ec09e50fb36aca67833c": {
|
||||
"service": "api",
|
||||
"started_at": "2026-11-12T15:28:13.153166256Z",
|
||||
"volumes": []
|
||||
},
|
||||
"83bab824b05accd2aa8829e83e272d72d4105644f8270cf3dc15f3412d569f5b": {
|
||||
"service": "api",
|
||||
"started_at": "2026-11-12T15:28:13.038450506Z",
|
||||
"volumes": []
|
||||
},
|
||||
"6f72735cbaed44f82555dcd550199764ab53a8c61a63c88f25c612fd39b6baba": {
|
||||
"service": "broker",
|
||||
"started_at": "2026-11-12T15:27:46.698266716Z",
|
||||
"volumes": [
|
||||
"mtusi-lab5_broker_data"
|
||||
]
|
||||
},
|
||||
"0f9060c7dc8473fa926c5affc560b3478b90c77771de9810ebba7858a30838d5": {
|
||||
"service": "cleanup",
|
||||
"started_at": "2026-11-12T15:28:13.036348173Z",
|
||||
"volumes": []
|
||||
},
|
||||
"d381adfb7273a5dd2f22d361a6e1ff490e5c2cf44ff147a8253f7f940cd01de4": {
|
||||
"service": "lb",
|
||||
"started_at": "2026-11-12T15:28:18.76757655Z",
|
||||
"volumes": []
|
||||
},
|
||||
"9906c3edca610723e6f09b64867fa71a543f18ef4b474804414e9f818a803115": {
|
||||
"service": "postgres",
|
||||
"started_at": "2026-11-12T15:27:46.700095216Z",
|
||||
"volumes": [
|
||||
"mtusi-lab5_postgres_data"
|
||||
]
|
||||
},
|
||||
"d705dceffd740900bbad04ee520bb96247f286c7c5381734a28bc1c9e30f27e4": {
|
||||
"service": "publisher",
|
||||
"started_at": "2026-11-12T15:27:49.924416218Z",
|
||||
"volumes": []
|
||||
},
|
||||
"cbe9063ff82960d9db58bc5713af90e4be3e72fdd1c08b0a92a93849dc6ad1aa": {
|
||||
"service": "s3",
|
||||
"started_at": "2026-11-12T15:28:09.808290754Z",
|
||||
"volumes": [
|
||||
"fe34bef7724beefe41335cd197f83aae91b16df70766cee6321bff12f765bc66"
|
||||
]
|
||||
},
|
||||
"6d1b6c402e19a068a0f2f4b9c06d5abbec7e41b5d778a883e6d045be409f6d4c": {
|
||||
"service": "s3-storage",
|
||||
"started_at": "2026-11-12T15:27:46.699445508Z",
|
||||
"volumes": [
|
||||
"mtusi-lab5_s3_data"
|
||||
]
|
||||
},
|
||||
"6d8c3e69c4b60acaf42e0259d367a0a3f2c06081a6349c284ddf9f2e9b5ba399": {
|
||||
"service": "valkey",
|
||||
"started_at": "2026-11-12T15:27:46.697619716Z",
|
||||
"volumes": [
|
||||
"mtusi-lab5_valkey_data"
|
||||
]
|
||||
},
|
||||
"28fabe35a9a5b57e0a3a96bb4ae5308fbd2fa825470799180a2471b55400206e": {
|
||||
"service": "worker",
|
||||
"started_at": "2026-11-12T15:28:13.037172673Z",
|
||||
"volumes": []
|
||||
}
|
||||
},
|
||||
"after": {
|
||||
"9e5a9d3d2bf0b2e47ee6d85b40b1899881db0d4d91d9ec09e50fb36aca67833c": {
|
||||
"service": "api",
|
||||
"started_at": "2026-11-12T15:28:33.373854585Z",
|
||||
"volumes": []
|
||||
},
|
||||
"83bab824b05accd2aa8829e83e272d72d4105644f8270cf3dc15f3412d569f5b": {
|
||||
"service": "api",
|
||||
"started_at": "2026-11-12T15:28:33.338416001Z",
|
||||
"volumes": []
|
||||
},
|
||||
"6f72735cbaed44f82555dcd550199764ab53a8c61a63c88f25c612fd39b6baba": {
|
||||
"service": "broker",
|
||||
"started_at": "2026-11-12T15:28:33.257433543Z",
|
||||
"volumes": [
|
||||
"mtusi-lab5_broker_data"
|
||||
]
|
||||
},
|
||||
"0f9060c7dc8473fa926c5affc560b3478b90c77771de9810ebba7858a30838d5": {
|
||||
"service": "cleanup",
|
||||
"started_at": "2026-11-12T15:28:40.990298547Z",
|
||||
"volumes": []
|
||||
},
|
||||
"d381adfb7273a5dd2f22d361a6e1ff490e5c2cf44ff147a8253f7f940cd01de4": {
|
||||
"service": "lb",
|
||||
"started_at": "2026-11-12T15:28:33.069501668Z",
|
||||
"volumes": []
|
||||
},
|
||||
"9906c3edca610723e6f09b64867fa71a543f18ef4b474804414e9f818a803115": {
|
||||
"service": "postgres",
|
||||
"started_at": "2026-11-12T15:28:33.134879876Z",
|
||||
"volumes": [
|
||||
"mtusi-lab5_postgres_data"
|
||||
]
|
||||
},
|
||||
"d705dceffd740900bbad04ee520bb96247f286c7c5381734a28bc1c9e30f27e4": {
|
||||
"service": "publisher",
|
||||
"started_at": "2026-11-12T15:28:40.90857463Z",
|
||||
"volumes": []
|
||||
},
|
||||
"cbe9063ff82960d9db58bc5713af90e4be3e72fdd1c08b0a92a93849dc6ad1aa": {
|
||||
"service": "s3",
|
||||
"started_at": "2026-11-12T15:28:33.167460626Z",
|
||||
"volumes": [
|
||||
"fe34bef7724beefe41335cd197f83aae91b16df70766cee6321bff12f765bc66"
|
||||
]
|
||||
},
|
||||
"6d1b6c402e19a068a0f2f4b9c06d5abbec7e41b5d778a883e6d045be409f6d4c": {
|
||||
"service": "s3-storage",
|
||||
"started_at": "2026-11-12T15:28:40.950084172Z",
|
||||
"volumes": [
|
||||
"mtusi-lab5_s3_data"
|
||||
]
|
||||
},
|
||||
"6d8c3e69c4b60acaf42e0259d367a0a3f2c06081a6349c284ddf9f2e9b5ba399": {
|
||||
"service": "valkey",
|
||||
"started_at": "2026-11-12T15:28:33.205284001Z",
|
||||
"volumes": [
|
||||
"mtusi-lab5_valkey_data"
|
||||
]
|
||||
},
|
||||
"28fabe35a9a5b57e0a3a96bb4ae5308fbd2fa825470799180a2471b55400206e": {
|
||||
"service": "worker",
|
||||
"started_at": "2026-11-12T15:28:33.299354043Z",
|
||||
"volumes": []
|
||||
}
|
||||
},
|
||||
"same_containers_and_volumes": true
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab5/mtusi2026lab5/.venv/bin/python tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:8080"
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
PASS: all persistent services actually restarted; container IDs and volumes preserved
|
||||
@@ -0,0 +1,76 @@
|
||||
{
|
||||
"utc": "2026-11-06T07:27:40.514878+00:00",
|
||||
"project": "lab4-pg-pause-30f1d978ce",
|
||||
"command": ".venv/bin/python student-tests/postgres_pause.py",
|
||||
"passed": true,
|
||||
"sql_without_server_timeout": {
|
||||
"elapsed_seconds": 2.001,
|
||||
"connection_closed": true
|
||||
},
|
||||
"direct_ready": {
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.012
|
||||
},
|
||||
"direct_live": {
|
||||
"status": 200,
|
||||
"elapsed_seconds": 0.006
|
||||
},
|
||||
"repeated_ready": [
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.11
|
||||
},
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.011
|
||||
},
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.014
|
||||
},
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.097
|
||||
},
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.194
|
||||
},
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.013
|
||||
},
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.299
|
||||
},
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.203
|
||||
},
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.013
|
||||
},
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.094
|
||||
}
|
||||
],
|
||||
"ready_after_12_seconds": {
|
||||
"status": 503,
|
||||
"elapsed_seconds": 0.009
|
||||
},
|
||||
"live_after_repeated_requests": {
|
||||
"status": 200,
|
||||
"elapsed_seconds": 0.006
|
||||
},
|
||||
"paused_seconds": 12.565,
|
||||
"recovered_ready": {
|
||||
"status": 200,
|
||||
"elapsed_seconds": 0.009
|
||||
},
|
||||
"recovery_seconds": 2.874,
|
||||
"api_not_restarted": true,
|
||||
"existing_session_and_message_preserved": true
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,32 @@
|
||||
Research project: lab2-research-85021a09d9
|
||||
Measuring two indexes on the same 100000 messages, three repeats...
|
||||
{
|
||||
"chat_only": {
|
||||
"insert_100000_median_ms": 1386.7587500000127,
|
||||
"index_bytes": 671744,
|
||||
"wal_bytes_median": 38822560,
|
||||
"queries": 180,
|
||||
"errors": 0,
|
||||
"concurrency": 1,
|
||||
"p50_ms": 1.6852079999978287,
|
||||
"p95_ms": 7.566457999985232,
|
||||
"p99_ms": 46.40233399999261,
|
||||
"successful_sql_rps": 282.01364182715855
|
||||
},
|
||||
"history": {
|
||||
"insert_100000_median_ms": 1404.4910420000178,
|
||||
"index_bytes": 7315456,
|
||||
"wal_bytes_median": 43108512,
|
||||
"queries": 180,
|
||||
"errors": 0,
|
||||
"concurrency": 1,
|
||||
"p50_ms": 0.22610399997802233,
|
||||
"p95_ms": 0.29704099995342403,
|
||||
"p99_ms": 0.3384169999662845,
|
||||
"successful_sql_rps": 4322.810483886352
|
||||
}
|
||||
}
|
||||
PASS: backup 0.54s; restore 0.68s; 100000 messages, all hashes and 12 samples match
|
||||
PASS: restored into a different container and fresh volume; one post-backup commit is absent as expected
|
||||
Results: evidence/research.json, evidence/query-plans.json; dump stays outside Git in backups/
|
||||
Removed only temporary research project: lab2-research-85021a09d9
|
||||
@@ -0,0 +1,40 @@
|
||||
global
|
||||
log stdout format raw local0
|
||||
maxconn 256
|
||||
|
||||
defaults
|
||||
mode http
|
||||
log global
|
||||
option dontlognull
|
||||
option http-server-close
|
||||
timeout connect 1s
|
||||
timeout client 40s
|
||||
timeout server 60s
|
||||
timeout http-request 5s
|
||||
timeout check 1s
|
||||
retries 1
|
||||
|
||||
resolvers docker
|
||||
nameserver dns 127.0.0.11:53
|
||||
accepted_payload_size 8192
|
||||
resolve_retries 3
|
||||
timeout resolve 1s
|
||||
timeout retry 1s
|
||||
hold valid 1s
|
||||
hold obsolete 1s
|
||||
hold nx 1s
|
||||
hold timeout 1s
|
||||
hold refused 1s
|
||||
hold other 1s
|
||||
|
||||
frontend public
|
||||
bind :8080
|
||||
default_backend api
|
||||
|
||||
backend api
|
||||
balance roundrobin
|
||||
option httpchk
|
||||
http-check send meth GET uri /health/ready ver HTTP/1.1 hdr Host localhost
|
||||
http-check expect status 200
|
||||
default-server inter 1s fastinter 1s downinter 1s fall 2 rise 2
|
||||
server-template replica 10 api:8080 check resolvers docker resolve-prefer ipv4 init-addr none
|
||||
@@ -0,0 +1,30 @@
|
||||
CREATE TABLE users (
|
||||
id uuid PRIMARY KEY,
|
||||
username varchar(32) NOT NULL UNIQUE CHECK (username ~ '^[a-z0-9_]{3,32}$'),
|
||||
display_name varchar(100) NOT NULL CHECK (char_length(display_name) >= 1),
|
||||
password_hash text NOT NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
|
||||
CREATE TABLE chats (
|
||||
id uuid PRIMARY KEY,
|
||||
title varchar(100) NOT NULL CHECK (char_length(title) >= 1),
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
|
||||
CREATE TABLE chat_members (
|
||||
chat_id uuid NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||
user_id uuid NOT NULL REFERENCES users(id),
|
||||
PRIMARY KEY (chat_id, user_id)
|
||||
);
|
||||
CREATE INDEX chat_members_user_idx ON chat_members (user_id, chat_id);
|
||||
|
||||
CREATE TABLE messages (
|
||||
id uuid PRIMARY KEY,
|
||||
chat_id uuid NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||
sender_id uuid NOT NULL,
|
||||
text text NOT NULL CHECK (char_length(text) BETWEEN 1 AND 4000 AND text !~ '^\s*$'),
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
FOREIGN KEY (chat_id, sender_id) REFERENCES chat_members(chat_id, user_id)
|
||||
);
|
||||
CREATE INDEX messages_history_idx ON messages (chat_id, created_at, id);
|
||||
@@ -0,0 +1,10 @@
|
||||
-- Добавочная миграция: существующие данные и таблицы lab2 сохраняются.
|
||||
CREATE TABLE message_idempotency (
|
||||
user_id uuid NOT NULL REFERENCES users(id),
|
||||
chat_id uuid NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||
key varchar(128) NOT NULL CHECK (key ~ '^[A-Za-z0-9._:-]{1,128}$'),
|
||||
request_text text NOT NULL,
|
||||
message_id uuid NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
|
||||
expires_at timestamptz NOT NULL,
|
||||
PRIMARY KEY (user_id, chat_id, key)
|
||||
);
|
||||
@@ -0,0 +1,34 @@
|
||||
-- Только добавления к lab3; старые сообщения и ключи сохраняются.
|
||||
CREATE TABLE attachments (
|
||||
id uuid PRIMARY KEY,
|
||||
chat_id uuid NOT NULL REFERENCES chats(id),
|
||||
owner_id uuid NOT NULL,
|
||||
object_key text NOT NULL UNIQUE,
|
||||
filename varchar(255) NOT NULL,
|
||||
content_type text NOT NULL CHECK (content_type IN ('image/png', 'image/jpeg')),
|
||||
size_bytes integer NOT NULL CHECK (size_bytes BETWEEN 1 AND 10485760),
|
||||
sha256 char(64) NOT NULL CHECK (sha256 ~ '^[a-f0-9]{64}$'),
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
FOREIGN KEY (chat_id, owner_id) REFERENCES chat_members(chat_id, user_id)
|
||||
);
|
||||
CREATE INDEX attachments_chat_idx ON attachments(chat_id);
|
||||
|
||||
ALTER TABLE messages DROP CONSTRAINT messages_text_check;
|
||||
ALTER TABLE messages ADD CONSTRAINT messages_text_check CHECK (char_length(text) <= 4000);
|
||||
CREATE TABLE message_attachments (
|
||||
message_id uuid NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
|
||||
attachment_id uuid NOT NULL REFERENCES attachments(id),
|
||||
position integer NOT NULL CHECK (position BETWEEN 0 AND 9),
|
||||
PRIMARY KEY (message_id, attachment_id),
|
||||
UNIQUE (message_id, position)
|
||||
);
|
||||
ALTER TABLE message_idempotency
|
||||
ADD COLUMN request_attachment_ids uuid[] NOT NULL DEFAULT '{}';
|
||||
|
||||
-- Фиксируется ДО S3 PUT. После COMMIT ready-метаданных удаляется атомарно с ними.
|
||||
CREATE TABLE attachment_uploads (
|
||||
id uuid PRIMARY KEY,
|
||||
object_key text NOT NULL UNIQUE,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
CREATE INDEX attachment_uploads_created_idx ON attachment_uploads(created_at);
|
||||
@@ -0,0 +1,42 @@
|
||||
-- Lab5: существующие ready-вложения lab4 остаются доступными.
|
||||
ALTER TABLE attachments
|
||||
ADD COLUMN status text NOT NULL DEFAULT 'ready' CHECK (status IN ('queued','processing','ready','failed')),
|
||||
ADD COLUMN variants jsonb NOT NULL DEFAULT '[]',
|
||||
ADD COLUMN error_code text,
|
||||
ADD COLUMN updated_at timestamptz NOT NULL DEFAULT clock_timestamp();
|
||||
UPDATE attachments SET updated_at=created_at;
|
||||
|
||||
CREATE TABLE attachment_jobs (
|
||||
id uuid PRIMARY KEY,
|
||||
attachment_id uuid NOT NULL UNIQUE REFERENCES attachments(id),
|
||||
pipeline_version integer NOT NULL DEFAULT 1 CHECK (pipeline_version=1),
|
||||
staging_key text NOT NULL UNIQUE,
|
||||
attempt integer NOT NULL DEFAULT 1 CHECK (attempt>=1),
|
||||
max_attempts integer NOT NULL CHECK (max_attempts>=1),
|
||||
replay_count integer NOT NULL DEFAULT 0,
|
||||
lease_token uuid,
|
||||
lease_until timestamptz,
|
||||
next_attempt_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
last_error text,
|
||||
staging_deleted_at timestamptz,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
CREATE INDEX attachment_jobs_lease_idx ON attachment_jobs(lease_until) WHERE lease_until IS NOT NULL;
|
||||
|
||||
CREATE TABLE attachment_outbox (
|
||||
job_id uuid NOT NULL REFERENCES attachment_jobs(id),
|
||||
attempt integer NOT NULL,
|
||||
available_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
published_at timestamptz,
|
||||
PRIMARY KEY(job_id, attempt)
|
||||
);
|
||||
CREATE INDEX attachment_outbox_due_idx ON attachment_outbox(available_at) WHERE published_at IS NULL;
|
||||
|
||||
CREATE TABLE attachment_job_events (
|
||||
id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
job_id uuid NOT NULL REFERENCES attachment_jobs(id),
|
||||
attempt integer NOT NULL,
|
||||
event text NOT NULL,
|
||||
error_code text,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
@@ -0,0 +1,36 @@
|
||||
# Основные библиотеки
|
||||
fastapi==0.142.4
|
||||
uvicorn==0.54.0
|
||||
pydantic==2.13.5
|
||||
|
||||
# Их зависимости
|
||||
annotated-doc==0.0.5
|
||||
annotated-types==0.8.0
|
||||
anyio==4.15.1
|
||||
click==8.5.0
|
||||
h11==0.16.0
|
||||
idna==3.20
|
||||
opentelemetry-api==1.45.1
|
||||
pydantic-core==2.46.5
|
||||
starlette==1.7.0
|
||||
typing-extensions==4.16.0
|
||||
typing-inspection==0.4.4
|
||||
|
||||
# PostgreSQL, сессии и Argon2id
|
||||
psycopg[binary,pool]==3.3.6
|
||||
redis==8.1.0
|
||||
argon2-cffi==25.1.0
|
||||
|
||||
# Транзитивные зависимости новых библиотек
|
||||
psycopg-binary==3.3.6
|
||||
psycopg-pool==3.3.3
|
||||
argon2-cffi-bindings==26.1.0
|
||||
cffi==2.1.1
|
||||
pycparser==3.11
|
||||
|
||||
# S3 и проверка настоящих PNG/JPEG (без изменения оригинала)
|
||||
minio==7.2.20
|
||||
Pillow==13.0.0
|
||||
certifi==2026.10.23
|
||||
pycryptodome==4.3.0
|
||||
urllib3==2.8.0
|
||||
@@ -0,0 +1,12 @@
|
||||
# Трек 2 на оценку 5
|
||||
|
||||
Запуск из корня: `./scripts/research.sh`. Нужны Docker Compose, Python 3 и интернет для первой сборки. Скрипт собирает API-образ и использует отдельный Compose project с новыми volumes. Текущий стенд не участвует.
|
||||
|
||||
- run.py управляет контейнерами, pg_dump, pg_restore, файлами evidence и cleanup.
|
||||
- benchmark.py выполняет SQL и сверку внутри API-образа. Перед заполнением требует пустую БД. Исследовательский флаг устанавливает только драйвер.
|
||||
|
||||
Датасет: 100000 сообщений в 4 чатах (70000/20000/9000/1000). Сравниваются индекс chat_id и индекс (chat_id,created_at,id), по 3 повтора с одинаковыми ограничениями и записями. На каждый чат — страницы около 10/50/90%, один прогрев и пять измерений, 51 строка. Измеряется INSERT SELECT с COMMIT, WAL и размер индекса. EXPLAIN и основные цифры сохраняются в evidence.
|
||||
|
||||
После dump создаётся ещё одно сообщение. Backup восстанавливается в другой контейнер и свежий volume. Сравниваются все счётчики, 12 полных записей и SHA-256 всех сообщений. Последующая запись ожидаемо отсутствует в восстановленном снимке.
|
||||
|
||||
Файлы research.json, query-plans.json и research.txt обновляются автоматически. Dump остаётся в игнорируемом backups/. Временные исследовательские volumes удаляются после проверки. Методика, фактические результаты и ограничения лабораторной 2 описаны в [REPORT.lab2.md](../REPORT.lab2.md).
|
||||
@@ -0,0 +1,168 @@
|
||||
"""SQL-эксперимент запускается только в отдельном research Compose project."""
|
||||
from datetime import datetime, timedelta, timezone
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import secrets
|
||||
import statistics
|
||||
import sys
|
||||
import time
|
||||
from uuid import UUID
|
||||
|
||||
import psycopg
|
||||
from psycopg.conninfo import make_conninfo
|
||||
from psycopg.rows import dict_row
|
||||
|
||||
from src import db, migrate, passwords
|
||||
from src.config import secret
|
||||
|
||||
|
||||
SIZES = [70000, 20000, 9000, 1000]
|
||||
START = datetime(2026, 1, 1, tzinfo=timezone.utc)
|
||||
QUERY = """SELECT * FROM messages WHERE chat_id=%s AND (created_at, id)>(%s, %s)
|
||||
ORDER BY created_at, id LIMIT %s"""
|
||||
|
||||
|
||||
def identifier(value):
|
||||
return UUID(hashlib.md5(value.encode(), usedforsecurity=False).hexdigest())
|
||||
|
||||
|
||||
def connect(restored=False):
|
||||
url = secret("DATABASE_URL")
|
||||
if restored:
|
||||
url = make_conninfo(url, host="postgres-restore")
|
||||
return psycopg.connect(url, autocommit=True, row_factory=dict_row,
|
||||
options="-c timezone=UTC -c statement_timeout=120000")
|
||||
|
||||
|
||||
def snapshot(conn):
|
||||
counts = {}
|
||||
for table in ("users", "chats", "chat_members", "messages", "schema_migrations"):
|
||||
# Имена таблиц — фиксированный список выше, не входные данные клиента.
|
||||
counts[table] = conn.execute("SELECT count(*) AS n FROM " + table).fetchone()["n"]
|
||||
digest = hashlib.sha256()
|
||||
for row in conn.execute("SELECT * FROM messages ORDER BY id"):
|
||||
digest.update((json.dumps(row, default=str, sort_keys=True, ensure_ascii=False) + "\n").encode())
|
||||
samples = []
|
||||
offset = 0
|
||||
for size in SIZES:
|
||||
for number in (offset + 1, offset + size // 2, offset + size):
|
||||
row = conn.execute("SELECT * FROM messages WHERE id=%s", (identifier(f"message:{number}"),)).fetchone()
|
||||
samples.append(json.loads(json.dumps(row, default=str, sort_keys=True, ensure_ascii=False)))
|
||||
offset += size
|
||||
return {"counts": counts, "messages_sha256": digest.hexdigest(), "samples": samples}
|
||||
|
||||
|
||||
def benchmark():
|
||||
db.pool.open(wait=True, timeout=15)
|
||||
try:
|
||||
migrate.apply()
|
||||
finally:
|
||||
db.pool.close()
|
||||
results, plans = [], []
|
||||
with connect() as conn:
|
||||
assert conn.execute("SELECT count(*) AS n FROM users").fetchone()["n"] == 0, "Research requires an empty database"
|
||||
encoded = passwords.hash_password(secrets.token_urlsafe(24))
|
||||
for n in range(1, 11):
|
||||
conn.execute("INSERT INTO users(id,username,display_name,password_hash) VALUES(%s,%s,%s,%s)",
|
||||
(identifier(f"user:{n}"), f"research_{n}", "Synthetic research user", encoded))
|
||||
for n in range(4):
|
||||
chat = identifier(f"chat:{n}")
|
||||
conn.execute("INSERT INTO chats(id,title) VALUES(%s,%s)", (chat, f"Research chat {n}"))
|
||||
for user in (1, 2):
|
||||
conn.execute("INSERT INTO chat_members(chat_id,user_id) VALUES(%s,%s)", (chat, identifier(f"user:{user}")))
|
||||
variants = [("chat_only", "chat_id"), ("history", "chat_id, created_at, id")]
|
||||
for repeat in range(3):
|
||||
order = variants if repeat % 2 == 0 else variants[::-1]
|
||||
for variant, columns in order:
|
||||
conn.execute("TRUNCATE messages")
|
||||
conn.execute("DROP INDEX IF EXISTS messages_history_idx")
|
||||
conn.execute("DROP INDEX IF EXISTS messages_chat_idx")
|
||||
index = "messages_history_idx" if variant == "history" else "messages_chat_idx"
|
||||
conn.execute("CREATE INDEX " + index + " ON messages (" + columns + ")")
|
||||
lsn = conn.execute("SELECT pg_current_wal_insert_lsn() AS lsn").fetchone()["lsn"]
|
||||
started = time.perf_counter()
|
||||
conn.execute("""INSERT INTO messages(id,chat_id,sender_id,text,created_at)
|
||||
SELECT md5('message:'||n)::uuid,
|
||||
md5('chat:'||(CASE WHEN n<=70000 THEN 0 WHEN n<=90000 THEN 1 WHEN n<=99000 THEN 2 ELSE 3 END))::uuid,
|
||||
md5('user:'||(1+n%2))::uuid,
|
||||
'Synthetic message '||n||' '||repeat('x',80),
|
||||
timestamptz '2026-01-01 00:00:00+00' + n*interval '1 microsecond'
|
||||
FROM generate_series(1,100000) AS n""")
|
||||
insert_ms = (time.perf_counter() - started) * 1000
|
||||
wal = conn.execute("SELECT pg_wal_lsn_diff(pg_current_wal_insert_lsn(), %s) AS bytes", (lsn,)).fetchone()["bytes"]
|
||||
conn.execute("VACUUM (ANALYZE) messages")
|
||||
index_bytes = conn.execute("SELECT pg_relation_size(%s::regclass) AS bytes", (index,)).fetchone()["bytes"]
|
||||
result = {"variant": variant, "repeat": repeat + 1, "insert_100000_ms": insert_ms,
|
||||
"wal_bytes": int(wal), "index_bytes": index_bytes, "queries": []}
|
||||
offset = 0
|
||||
for chat, size in enumerate(SIZES):
|
||||
for fraction in (0.1, 0.5, 0.9):
|
||||
number = offset + int(size * fraction)
|
||||
params = (identifier(f"chat:{chat}"), START + timedelta(microseconds=number), identifier(f"message:{number}"), 51)
|
||||
conn.execute(QUERY, params).fetchall() # один прогрев именно этой страницы
|
||||
times = []
|
||||
for _ in range(5):
|
||||
started = time.perf_counter()
|
||||
rows = conn.execute(QUERY, params).fetchall()
|
||||
times.append((time.perf_counter() - started) * 1000)
|
||||
assert len(rows) == 51
|
||||
plan = conn.execute("EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) " + QUERY, params).fetchone()["QUERY PLAN"][0]
|
||||
result["queries"].append({"chat_size": size, "position": fraction, "latency_ms": times,
|
||||
"explain_execution_ms": plan["Execution Time"]})
|
||||
if repeat == 0:
|
||||
plans.append({"variant": variant, "chat_size": size, "position": fraction, "plan": plan})
|
||||
offset += size
|
||||
results.append(result)
|
||||
# Последняя итерация заканчивается production-индексом history.
|
||||
assert conn.execute("SELECT to_regclass('messages_history_idx') AS name").fetchone()["name"]
|
||||
baseline = snapshot(conn)
|
||||
assert baseline["counts"]["messages"] == 100000
|
||||
summary = {}
|
||||
for variant, _ in variants:
|
||||
trials = [row for row in results if row["variant"] == variant]
|
||||
latencies = sorted(t for row in trials for q in row["queries"] for t in q["latency_ms"])
|
||||
summary[variant] = {
|
||||
"insert_100000_median_ms": statistics.median(row["insert_100000_ms"] for row in trials),
|
||||
"index_bytes": trials[-1]["index_bytes"],
|
||||
"wal_bytes_median": statistics.median(row["wal_bytes"] for row in trials),
|
||||
"queries": len(latencies), "errors": 0, "concurrency": 1,
|
||||
"p50_ms": statistics.median(latencies),
|
||||
"p95_ms": latencies[int(0.95 * (len(latencies) - 1))],
|
||||
"p99_ms": latencies[int(0.99 * (len(latencies) - 1))],
|
||||
"successful_sql_rps": len(latencies) / (sum(latencies) / 1000),
|
||||
}
|
||||
return {"postgres": conn.execute("SELECT version() AS version").fetchone()["version"],
|
||||
"messages": 100000, "chat_sizes": SIZES, "repeats": 3, "warmup_per_case": 1,
|
||||
"measurements_per_case": 5, "page_size": 51, "query": QUERY,
|
||||
"summary": summary, "trials": results, "plans": plans, "snapshot": baseline}
|
||||
|
||||
|
||||
def after_backup():
|
||||
with connect() as conn:
|
||||
message_id = identifier("after-backup")
|
||||
conn.execute("INSERT INTO messages(id,chat_id,sender_id,text) VALUES(%s,%s,%s,%s)",
|
||||
(message_id, identifier("chat:0"), identifier("user:1"), "Committed AFTER backup"))
|
||||
return {"id": str(message_id), "source_messages": conn.execute("SELECT count(*) AS n FROM messages").fetchone()["n"]}
|
||||
|
||||
|
||||
def verify():
|
||||
expected = json.load(sys.stdin)
|
||||
with connect(restored=True) as restored, connect() as source:
|
||||
actual = snapshot(restored)
|
||||
assert actual == expected, "Restored data differs from the snapshot"
|
||||
marker = (identifier("after-backup"),)
|
||||
assert restored.execute("SELECT 1 FROM messages WHERE id=%s", marker).fetchone() is None
|
||||
assert source.execute("SELECT 1 FROM messages WHERE id=%s", marker).fetchone() is not None
|
||||
return {"counts": actual["counts"], "messages_sha256": actual["messages_sha256"],
|
||||
"samples_verified": len(actual["samples"]), "all_messages_match": True,
|
||||
"after_backup_message_missing_in_restore": True,
|
||||
"source_messages": source.execute("SELECT count(*) AS n FROM messages").fetchone()["n"]}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if os.environ.get("LAB2_RESEARCH") != "1":
|
||||
raise SystemExit("Use ./scripts/research.sh: an isolated research database is required")
|
||||
operation = sys.argv[1] if len(sys.argv) > 1 else "benchmark"
|
||||
result = {"benchmark": benchmark, "after-backup": after_backup, "verify": verify}[operation]()
|
||||
print(json.dumps(result, ensure_ascii=False, default=str))
|
||||
@@ -0,0 +1,99 @@
|
||||
"""Создаёт изолированный стенд, измеряет SQL, делает dump и restore; пишет реальные результаты."""
|
||||
from datetime import datetime, timezone
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import platform
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from uuid import uuid4
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
project = "lab2-research-" + uuid4().hex[:10]
|
||||
env = dict(os.environ, COMPOSE_PROJECT_NAME=project)
|
||||
command = ["docker", "compose", "--profile", "research"]
|
||||
|
||||
|
||||
def compose(*args, stdin=None, stdout=subprocess.PIPE):
|
||||
return subprocess.run([*command, *args], cwd=ROOT, env=env, input=stdin,
|
||||
stdout=stdout, stderr=subprocess.PIPE, check=True, timeout=600)
|
||||
|
||||
|
||||
def run_module(operation, stdin=None):
|
||||
result = compose("run", "--rm", "--no-deps", "-T", "-e", "LAB2_RESEARCH=1", "api", "python", "-m", "research.benchmark", operation, stdin=stdin)
|
||||
return json.loads(result.stdout)
|
||||
|
||||
|
||||
def wait_database(service):
|
||||
for _ in range(60):
|
||||
cid = compose("ps", "-q", service).stdout.decode().strip()
|
||||
state = subprocess.run(["docker", "inspect", "--format", "{{.State.Health.Status}}", cid], capture_output=True, text=True, check=True)
|
||||
if state.stdout.strip() == "healthy":
|
||||
return
|
||||
time.sleep(1)
|
||||
raise RuntimeError(service + " did not become healthy")
|
||||
|
||||
|
||||
def main():
|
||||
evidence = ROOT / "evidence"
|
||||
evidence.mkdir(exist_ok=True)
|
||||
backups = ROOT / "backups"
|
||||
backups.mkdir(exist_ok=True)
|
||||
dump = backups / (project + ".dump")
|
||||
metadata = {"utc": datetime.now(timezone.utc).isoformat(), "project": project,
|
||||
"command": "./scripts/research.sh", "host": platform.system() + " " + platform.machine(),
|
||||
"commit": subprocess.run(["git", "rev-parse", "--short", "HEAD"], cwd=ROOT, capture_output=True, text=True).stdout.strip() + " + working tree",
|
||||
"postgres_limits": {"cpus": 1, "memory_mib": 512}, "api_runner_limits": {"cpus": 1, "memory_mib": 256}}
|
||||
started = time.perf_counter()
|
||||
try:
|
||||
print("Research project:", project, flush=True)
|
||||
compose("up", "-d", "postgres", "postgres-restore")
|
||||
wait_database("postgres")
|
||||
wait_database("postgres-restore")
|
||||
print("Measuring two indexes on the same 100000 messages, three repeats...", flush=True)
|
||||
result = run_module("benchmark")
|
||||
plans = result.pop("plans")
|
||||
(evidence / "query-plans.json").write_text(json.dumps(plans, ensure_ascii=False, indent=2))
|
||||
source_id = compose("ps", "-q", "postgres").stdout.decode().strip()
|
||||
restore_id = compose("ps", "-q", "postgres-restore").stdout.decode().strip()
|
||||
assert source_id != restore_id
|
||||
for label, cid in (("source", source_id), ("restored", restore_id)):
|
||||
inspected = json.loads(subprocess.run(["docker", "inspect", cid], capture_output=True, check=True).stdout)[0]
|
||||
metadata[label] = {"container": cid, "image": inspected["Image"],
|
||||
"volumes": [m["Name"] for m in inspected["Mounts"] if m["Type"] == "volume"]}
|
||||
assert metadata["source"]["volumes"] != metadata["restored"]["volumes"]
|
||||
metadata["docker"] = subprocess.run(["docker", "info", "--format", "{{.ServerVersion}} {{.Architecture}} CPUs={{.NCPU}} RAM={{.MemTotal}}"], capture_output=True, text=True, check=True).stdout.strip()
|
||||
metadata["stats_after_queries"] = subprocess.run(["docker", "stats", "--no-stream", "--format", "{{json .}}", source_id], capture_output=True, text=True, check=True).stdout.strip()
|
||||
backup_started = time.perf_counter()
|
||||
with dump.open("wb") as output:
|
||||
compose("exec", "-T", "postgres", "pg_dump", "-U", "messenger", "-d", "messenger", "--format=custom", stdout=output)
|
||||
backup_seconds = time.perf_counter() - backup_started
|
||||
marker = run_module("after-backup")
|
||||
restore_started = time.perf_counter()
|
||||
# Dump подаётся через stdin, без вывода данных или credentials в журнал.
|
||||
with dump.open("rb") as input_file:
|
||||
subprocess.run([*command, "exec", "-T", "postgres-restore", "pg_restore", "-U", "messenger", "-d", "messenger",
|
||||
"--exit-on-error", "--single-transaction", "--no-owner"], cwd=ROOT, env=env, stdin=input_file,
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, check=True, timeout=300)
|
||||
restore_seconds = time.perf_counter() - restore_started
|
||||
verification_started = time.perf_counter()
|
||||
verification = run_module("verify", stdin=json.dumps(result["snapshot"]).encode())
|
||||
result.update(metadata)
|
||||
result["backup"] = {"file": "backups/" + dump.name, "bytes": dump.stat().st_size, "seconds": backup_seconds}
|
||||
result["restore"] = {"seconds": restore_seconds, "verification_seconds": time.perf_counter() - verification_started,
|
||||
"verification": verification, "post_backup_insert": marker}
|
||||
result["total_seconds"] = time.perf_counter() - started
|
||||
(evidence / "research.json").write_text(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
print(json.dumps(result["summary"], ensure_ascii=False, indent=2), flush=True)
|
||||
print(f"PASS: backup {backup_seconds:.2f}s; restore {restore_seconds:.2f}s; 100000 messages, all hashes and 12 samples match", flush=True)
|
||||
print("PASS: restored into a different container and fresh volume; one post-backup commit is absent as expected", flush=True)
|
||||
print("Results: evidence/research.json, evidence/query-plans.json; dump stays outside Git in backups/", flush=True)
|
||||
finally:
|
||||
compose("down", "--volumes", "--timeout", "10")
|
||||
print("Removed only temporary research project:", project, flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+6
@@ -0,0 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$project_dir"
|
||||
python3 scripts/init_local.py
|
||||
docker compose build "$@" api
|
||||
Executable
+11
@@ -0,0 +1,11 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$project_dir"
|
||||
if [[ ! -x .venv/bin/python ]]; then
|
||||
echo "Создай .venv и установи requirements.txt по REPORT.md" >&2
|
||||
exit 1
|
||||
fi
|
||||
./scripts/build.sh
|
||||
mkdir -p evidence
|
||||
.venv/bin/python -u student-tests/lab5_pipeline.py 2>&1 | sed -E 's/[[:blank:]]+$//' | tee evidence/lab5-checks.txt
|
||||
Executable
+17
@@ -0,0 +1,17 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$project_dir"
|
||||
|
||||
if [[ ! -x .venv/bin/python ]]; then
|
||||
echo "Не хватает .venv" >&2
|
||||
echo "python3 -m venv .venv" >&2
|
||||
echo ".venv/bin/python -m pip install -r requirements.txt" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
export BASE_URL="${BASE_URL:-http://localhost:8080}"
|
||||
echo "Проверяем сервер: $BASE_URL"
|
||||
make check test PYTHON="$project_dir/.venv/bin/python" BASE_URL="$BASE_URL"
|
||||
.venv/bin/python -m unittest discover -s student-tests -v
|
||||
Executable
+28
@@ -0,0 +1,28 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
base_url="${BASE_URL:-http://localhost:8080}"
|
||||
suffix="$(date +%s)_$RANDOM"
|
||||
# Только синтетические пароли; токены остаются в памяти shell и не выводятся.
|
||||
password="Demo-password-$suffix"
|
||||
register() {
|
||||
local username="$1" display_name="$2"
|
||||
jq -nc --arg u "$username" --arg d "$display_name" --arg p "$password" \
|
||||
'{username:$u, display_name:$d, password:$p}' |
|
||||
curl -fsS -H 'Content-Type: application/json' --data-binary @- "$base_url/api/v1/users" | jq -er '.id'
|
||||
}
|
||||
login() {
|
||||
curl -fsS -X POST -u "$1:$password" "$base_url/api/v1/auth/sessions" | jq -er '.token'
|
||||
}
|
||||
alice_id="$(register "alice_$suffix" Алиса)"
|
||||
bob_id="$(register "bob_$suffix" Боб)"
|
||||
alice_token="$(login "alice_$suffix")"
|
||||
bob_token="$(login "bob_$suffix")"
|
||||
chat_id="$(jq -nc --arg a "$alice_id" --arg b "$bob_id" '{title:"Учебный чат",member_ids:[$a,$b]}' |
|
||||
curl -fsS -H "Authorization: Bearer $alice_token" -H 'Content-Type: application/json' \
|
||||
--data-binary @- "$base_url/api/v1/chats" | jq -er '.id')"
|
||||
for token in "$alice_token" "$bob_token"; do
|
||||
curl -fsS -H "Authorization: Bearer $token" -H 'Content-Type: application/json' \
|
||||
-H "Idempotency-Key: demo-$RANDOM-$RANDOM" \
|
||||
-d '{"text":"Привет из lab5! 👋"}' "$base_url/api/v1/chats/$chat_id/messages" | jq .
|
||||
done
|
||||
curl -fsS -H "Authorization: Bearer $bob_token" "$base_url/api/v1/chats/$chat_id/messages" | jq .
|
||||
@@ -0,0 +1,62 @@
|
||||
"""Создаёт локальное окружение и secrets; существующие значения сохраняет."""
|
||||
from pathlib import Path
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import secrets
|
||||
import shutil
|
||||
from urllib.parse import quote
|
||||
|
||||
|
||||
root = Path(__file__).resolve().parents[1]
|
||||
if not (root / ".env").exists():
|
||||
shutil.copyfile(root / ".env.example", root / ".env")
|
||||
directory = root / "secrets"
|
||||
directory.mkdir(exist_ok=True, mode=0o700)
|
||||
directory.chmod(0o700)
|
||||
password_file = directory / "postgres_password"
|
||||
if not password_file.exists():
|
||||
password_file.write_text(secrets.token_hex(24))
|
||||
password = password_file.read_text().strip()
|
||||
values = {
|
||||
"database_url": f"postgresql://messenger:{quote(password, safe='')}@postgres:5432/messenger",
|
||||
"cache_url": "redis://valkey:6379/0",
|
||||
"broker_password": secrets.token_hex(24),
|
||||
"s3_root_user": "root-" + secrets.token_hex(8),
|
||||
"s3_root_password": secrets.token_urlsafe(32),
|
||||
"s3_access_key_id": "api-" + secrets.token_hex(8),
|
||||
"s3_secret_access_key": secrets.token_urlsafe(32),
|
||||
}
|
||||
for name, value in values.items():
|
||||
path = directory / name
|
||||
if not path.exists():
|
||||
path.write_text(value)
|
||||
broker_url = directory / "broker_url"
|
||||
if not broker_url.exists():
|
||||
broker_url.write_text("redis://:" + (directory / "broker_password").read_text().strip() + "@broker:6379/0")
|
||||
broker_url.chmod(0o444)
|
||||
dotenv = dict(line.split("=", 1) for line in (root / ".env").read_text().splitlines()
|
||||
if line.strip() and not line.lstrip().startswith("#") and "=" in line)
|
||||
bucket = os.environ.get("S3_BUCKET", dotenv.get("S3_BUCKET", "messenger-attachments"))
|
||||
if re.fullmatch(r"[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]", bucket) is None:
|
||||
raise ValueError("Недопустимое имя S3 bucket")
|
||||
def local_secret(name):
|
||||
return (directory / name).read_text().strip()
|
||||
config = {"accounts": [{"id": "messenger-api", "displayName": "messenger-api"}], "identities": [
|
||||
{"name": "bootstrap", "credentials": [{"accessKey": local_secret("s3_root_user"),
|
||||
"secretKey": local_secret("s3_root_password")}], "actions": ["Admin"]},
|
||||
{"name": "messenger-api", "account": {"id": "messenger-api", "displayName": "messenger-api"},
|
||||
"credentials": [{"accessKey": local_secret("s3_access_key_id"), "secretKey": local_secret("s3_secret_access_key")}],
|
||||
"actions": ["Read:" + bucket, "List:" + bucket, "Write:" + bucket + "/attachments/*"]},
|
||||
]}
|
||||
config_file = directory / "s3_config"
|
||||
encoded_config = json.dumps(config)
|
||||
if not config_file.exists() or config_file.read_text() != encoded_config:
|
||||
temporary = directory / "s3_config.tmp"
|
||||
temporary.write_text(encoded_config)
|
||||
temporary.chmod(0o444)
|
||||
temporary.replace(config_file)
|
||||
# Каталог 0700 защищает файлы на хосте; 0444 позволяет USER 10001 читать bind-mounted secret.
|
||||
for name in ("postgres_password", "s3_config", *values):
|
||||
(directory / name).chmod(0o444)
|
||||
print("Local .env and secrets are ready; values are not printed")
|
||||
Executable
+6
@@ -0,0 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$project_dir"
|
||||
./scripts/build.sh
|
||||
docker compose run --rm migrate
|
||||
@@ -0,0 +1,90 @@
|
||||
"""Управление репликами только текущего Compose project через Docker CLI."""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import time
|
||||
from urllib.request import urlopen
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def command(*args):
|
||||
return subprocess.check_output(args, cwd=ROOT, text=True, timeout=120).strip()
|
||||
|
||||
|
||||
def replicas():
|
||||
ids = command("docker", "compose", "ps", "-q", "api").split()
|
||||
return json.loads(command("docker", "inspect", *ids)) if ids else []
|
||||
|
||||
|
||||
def base_url():
|
||||
config = json.loads(command("docker", "compose", "config", "--format", "json"))
|
||||
port = config["services"]["lb"]["ports"][0]["published"]
|
||||
return os.environ.get("BASE_URL", "http://127.0.0.1:" + str(port)).rstrip("/")
|
||||
|
||||
|
||||
def wait_ready():
|
||||
url = base_url() + "/health/ready"
|
||||
deadline = time.monotonic() + 90
|
||||
seen = set()
|
||||
while time.monotonic() < deadline:
|
||||
current = replicas()
|
||||
expected = {info["Config"]["Hostname"] for info in current}
|
||||
if expected and all(info["State"].get("Health", {}).get("Status") == "healthy" for info in current):
|
||||
try:
|
||||
with urlopen(url, timeout=2) as response:
|
||||
if response.status == 200:
|
||||
seen.add(response.headers.get("X-Instance-Id"))
|
||||
if expected <= seen:
|
||||
print("API ready through LB:", ", ".join(sorted(expected)), flush=True)
|
||||
return
|
||||
except OSError:
|
||||
pass
|
||||
time.sleep(0.2)
|
||||
raise RuntimeError("Не все работающие API-реплики готовы через LB за 90 секунд")
|
||||
|
||||
|
||||
def stop_one():
|
||||
target = os.environ.get("TARGET_INSTANCE_ID")
|
||||
if not target:
|
||||
raise RuntimeError("Задайте TARGET_INSTANCE_ID из X-Instance-Id")
|
||||
current = replicas()
|
||||
matches = [info for info in current if info["Config"]["Hostname"] == target]
|
||||
if len(current) < 2 or len(matches) != 1:
|
||||
raise RuntimeError("Нужны минимум две работающие реплики и ровно одна выбранная")
|
||||
selected = matches[0]
|
||||
command("docker", "stop", "--time", "8", selected["Id"])
|
||||
print("Stopped API:", target, "container=" + selected["Id"], flush=True)
|
||||
|
||||
|
||||
def rolling():
|
||||
wait_ready()
|
||||
originals = replicas()
|
||||
count = len(originals)
|
||||
if count < 2:
|
||||
raise RuntimeError("Для поочерёдной замены нужны минимум две API-реплики")
|
||||
for old in originals:
|
||||
active = replicas()
|
||||
healthy = [info for info in active if info["Id"] != old["Id"]
|
||||
and info["State"].get("Health", {}).get("Status") == "healthy"]
|
||||
if not healthy:
|
||||
raise RuntimeError("Нет готовой реплики, которая продолжит обслуживать запросы")
|
||||
print("Replacing API:", old["Config"]["Hostname"], flush=True)
|
||||
command("docker", "stop", "--time", "8", old["Id"])
|
||||
command("docker", "rm", old["Id"])
|
||||
# --no-recreate сохраняет остальные процессы; Compose создаёт только недостающий.
|
||||
command("docker", "compose", "up", "-d", "--no-deps", "--no-recreate", "--scale", f"api={count}", "api")
|
||||
wait_ready()
|
||||
assert len(replicas()) == count
|
||||
final = replicas()
|
||||
assert not ({info["Id"] for info in originals} & {info["Id"] for info in final})
|
||||
print("PASS: all API containers replaced sequentially; stores and LB kept running", flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("action", choices=["wait", "stop", "rolling"])
|
||||
action = parser.parse_args().action
|
||||
{"wait": wait_ready, "stop": stop_one, "rolling": rolling}[action]()
|
||||
Executable
+7
@@ -0,0 +1,7 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$project_dir"
|
||||
./scripts/build.sh
|
||||
mkdir -p evidence
|
||||
python3 -u research/run.py 2>&1 | tee evidence/research.txt
|
||||
Executable
+6
@@ -0,0 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$project_dir"
|
||||
docker compose restart --timeout 8 postgres valkey s3-storage s3 broker api lb cleanup publisher worker
|
||||
./scripts/wait-ready.sh
|
||||
Executable
+7
@@ -0,0 +1,7 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$project_dir"
|
||||
# Сначала собираем образ и применяем только обратно совместимые миграции.
|
||||
./scripts/migrate.sh
|
||||
exec python3 scripts/replicas.py rolling
|
||||
Executable
+5
@@ -0,0 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
# Локальный стенд: HAProxy, две API-реплики, PostgreSQL и Valkey.
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
exec "$project_dir/scripts/start.sh"
|
||||
Executable
+5
@@ -0,0 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$project_dir"
|
||||
docker compose start worker
|
||||
Executable
+7
@@ -0,0 +1,7 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$project_dir"
|
||||
python3 scripts/init_local.py
|
||||
docker compose up -d --build lb cleanup publisher worker
|
||||
./scripts/wait-ready.sh
|
||||
Executable
+5
@@ -0,0 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$project_dir"
|
||||
exec python3 scripts/replicas.py stop
|
||||
Executable
+5
@@ -0,0 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$project_dir"
|
||||
docker compose stop --timeout 8 worker
|
||||
Executable
+6
@@ -0,0 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$project_dir"
|
||||
docker compose down --timeout 10
|
||||
echo "Контейнеры удалены; именованные volumes сохранены"
|
||||
Executable
+5
@@ -0,0 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
project_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$project_dir"
|
||||
exec python3 scripts/replicas.py wait
|
||||
@@ -0,0 +1,16 @@
|
||||
"""Общий запуск"""
|
||||
|
||||
import os
|
||||
|
||||
import uvicorn
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
uvicorn.run(
|
||||
"src.app:app",
|
||||
host=os.environ.get("APP_HOST", "0.0.0.0"),
|
||||
port=int(os.environ.get("APP_PORT", "8080")),
|
||||
workers=1,
|
||||
access_log=False,
|
||||
timeout_graceful_shutdown=5,
|
||||
)
|
||||
+187
@@ -0,0 +1,187 @@
|
||||
"""HTTP-адреса, определение пользователя и обработка запросов"""
|
||||
|
||||
from contextlib import asynccontextmanager
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
from time import perf_counter
|
||||
from uuid import UUID, uuid4
|
||||
from urllib.parse import quote
|
||||
|
||||
from fastapi import FastAPI, Request, Response
|
||||
from fastapi.responses import StreamingResponse
|
||||
from starlette.concurrency import run_in_threadpool
|
||||
from fastapi.exceptions import RequestValidationError
|
||||
from starlette.exceptions import HTTPException
|
||||
|
||||
from src import attachments, auth, db, migrate, object_store, passwords, storage
|
||||
from src.errors import ApiError, error_response
|
||||
from src.models import CreateChat, CreateMessage, CreateUser
|
||||
from src.config import INSTANCE_ID
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app):
|
||||
await run_in_threadpool(db.pool.open, wait=True, timeout=15)
|
||||
try:
|
||||
await run_in_threadpool(migrate.check)
|
||||
yield
|
||||
finally:
|
||||
await run_in_threadpool(db.pool.close)
|
||||
|
||||
|
||||
app = FastAPI(title="Учебный мессенджер", redirect_slashes=False, lifespan=lifespan)
|
||||
logger = logging.getLogger("uvicorn.error")
|
||||
|
||||
|
||||
@app.middleware("http")
|
||||
async def handle_request(request: Request, call_next):
|
||||
request.state.request_id = str(uuid4())
|
||||
started = perf_counter()
|
||||
path = request.url.path
|
||||
try:
|
||||
is_registration = request.method == "POST" and path == "/api/v1/users"
|
||||
is_login = request.method == "POST" and path == "/api/v1/auth/sessions"
|
||||
is_upload = request.method == "POST" and re.fullmatch(r"/api/v1/chats/[^/]+/attachments", path)
|
||||
if is_login:
|
||||
request.state.user = await run_in_threadpool(auth.basic_user, request.headers.get("Authorization"))
|
||||
if await request.body():
|
||||
raise ApiError(400, "invalid_request", "Вход через Basic не принимает тело запроса")
|
||||
elif path.startswith("/api/v1/") and not is_registration:
|
||||
request.state.user = await run_in_threadpool(auth.current_user, request.headers.get("Authorization"))
|
||||
if request.method == "POST" and path.startswith("/api/v1/") and not is_login and not is_upload:
|
||||
content_type = request.headers.get("Content-Type", "").split(";", 1)[0].strip().lower()
|
||||
if content_type != "application/json":
|
||||
raise ApiError(415, "unsupported_media_type", "Ожидается application/json")
|
||||
try:
|
||||
json.loads((await request.body()).decode("utf-8"))
|
||||
except (ValueError, RecursionError):
|
||||
raise ApiError(400, "invalid_request", "Ожидается корректный JSON в UTF-8")
|
||||
response = await call_next(request)
|
||||
except ApiError as error:
|
||||
response = error_response(request, error)
|
||||
except Exception as error:
|
||||
# Не печатаем тело запроса или текст исключения: там могут быть сообщения.
|
||||
logger.error("%s request_id=%s", type(error).__name__, request.state.request_id)
|
||||
response = error_response(request, ApiError(500, "internal_error", "Внутренняя ошибка сервера"))
|
||||
response.headers["X-Request-Id"] = request.state.request_id
|
||||
response.headers["X-Instance-Id"] = INSTANCE_ID
|
||||
logger.info(
|
||||
"%s %s status=%s duration_ms=%.2f request_id=%s",
|
||||
request.method, path, response.status_code,
|
||||
(perf_counter() - started) * 1000, request.state.request_id,
|
||||
)
|
||||
return response
|
||||
|
||||
|
||||
@app.exception_handler(ApiError)
|
||||
def handle_api_error(request: Request, error: ApiError):
|
||||
return error_response(request, error)
|
||||
|
||||
|
||||
@app.exception_handler(RequestValidationError)
|
||||
def handle_validation_error(request: Request, error: RequestValidationError):
|
||||
# Контракт требует 400 и наш формат ошибки, а не стандартный ответ FastAPI.
|
||||
return error_response(request, ApiError(400, "invalid_request", "Неверные поля или параметры запроса"))
|
||||
|
||||
|
||||
@app.exception_handler(HTTPException)
|
||||
def handle_http_error(request: Request, error: HTTPException):
|
||||
code = "not_found" if error.status_code == 404 else "invalid_request"
|
||||
message = "Адрес не найден" if error.status_code == 404 else "Недопустимый HTTP-запрос"
|
||||
response = error_response(request, ApiError(error.status_code, code, message))
|
||||
if error.headers:
|
||||
response.headers.update(error.headers)
|
||||
return response
|
||||
|
||||
|
||||
@app.get("/health/live")
|
||||
def health_live():
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
@app.get("/health/ready")
|
||||
def health_ready():
|
||||
db.ready()
|
||||
auth.ready()
|
||||
object_store.ready()
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
@app.post("/api/v1/users", status_code=201)
|
||||
def create_user(data: CreateUser):
|
||||
return storage.create_user(data.username, data.display_name, passwords.hash_password(data.password))
|
||||
|
||||
|
||||
@app.get("/api/v1/users/me")
|
||||
def current_user(request: Request):
|
||||
return request.state.user
|
||||
|
||||
|
||||
@app.post("/api/v1/chats", status_code=201)
|
||||
def create_chat(request: Request, data: CreateChat):
|
||||
return storage.create_chat(request.state.user["id"], data.title, data.member_ids)
|
||||
|
||||
|
||||
@app.get("/api/v1/chats")
|
||||
def list_chats(request: Request, limit: str = "50", cursor: str | None = None):
|
||||
user_id = request.state.user["id"]
|
||||
return storage.list_chats(user_id, limit, cursor)
|
||||
|
||||
|
||||
@app.get("/api/v1/chats/{chat_id}")
|
||||
def get_chat(request: Request, chat_id: UUID):
|
||||
return storage.get_chat(request.state.user["id"], str(chat_id))
|
||||
|
||||
|
||||
@app.post("/api/v1/chats/{chat_id}/messages", status_code=201)
|
||||
def create_message(request: Request, chat_id: UUID, data: CreateMessage):
|
||||
keys = request.headers.getlist("Idempotency-Key")
|
||||
if len(keys) != 1 or re.fullmatch(r"[A-Za-z0-9._:-]{1,128}", keys[0]) is None:
|
||||
raise ApiError(400, "invalid_request", "Нужен Idempotency-Key: 1–128 символов ASCII [A-Za-z0-9._:-]")
|
||||
return storage.create_message(request.state.user["id"], str(chat_id), data.text, keys[0], data.attachment_ids)
|
||||
|
||||
|
||||
@app.post("/api/v1/chats/{chat_id}/attachments", status_code=202)
|
||||
async def upload_attachment(request: Request, chat_id: UUID, filename: str):
|
||||
return await attachments.upload(request, str(chat_id), filename)
|
||||
|
||||
|
||||
@app.get("/api/v1/attachments/{attachment_id}")
|
||||
def get_attachment(request: Request, attachment_id: UUID):
|
||||
return attachments.public_attachment(attachments.find(request.state.user["id"], str(attachment_id)))
|
||||
|
||||
|
||||
@app.get("/api/v1/attachments/{attachment_id}/content")
|
||||
def download_attachment(request: Request, attachment_id: UUID, variant: str = "original"):
|
||||
row = attachments.find(request.state.user["id"], str(attachment_id))
|
||||
if row["status"] != "ready":
|
||||
raise ApiError(409, "conflict", "Вложение не готово")
|
||||
info = row
|
||||
key = row["object_key"]
|
||||
if variant != "original":
|
||||
info = next((item for item in row["variants"] if item["name"] == variant), None)
|
||||
if info is None:
|
||||
raise ApiError(404, "not_found", "Вариант вложения не найден")
|
||||
key = row["object_key"].rsplit("/", 1)[0] + "/" + variant
|
||||
response = object_store.get(key)
|
||||
return StreamingResponse(attachments.chunks(response), media_type=info["content_type"], headers={
|
||||
"Content-Length": str(info["size_bytes"]), "X-Content-Type-Options": "nosniff",
|
||||
"Content-Disposition": "attachment; filename*=UTF-8''" + quote(row["filename"], safe=""),
|
||||
})
|
||||
|
||||
|
||||
@app.get("/api/v1/chats/{chat_id}/messages")
|
||||
def list_messages(request: Request, chat_id: UUID, limit: str = "50", cursor: str | None = None):
|
||||
return storage.list_messages(request.state.user["id"], str(chat_id), limit, cursor)
|
||||
|
||||
|
||||
@app.post("/api/v1/auth/sessions", status_code=201)
|
||||
def create_session(request: Request):
|
||||
return auth.create_session(request.state.user)
|
||||
|
||||
|
||||
@app.delete("/api/v1/auth/sessions/current", status_code=204)
|
||||
def delete_session(request: Request):
|
||||
auth.logout(request.headers.get("Authorization"))
|
||||
return Response(status_code=204)
|
||||
@@ -0,0 +1,151 @@
|
||||
"""Приём и проверка -> durable S3 staging + PostgreSQL outbox -> 202."""
|
||||
import asyncio
|
||||
import hashlib
|
||||
import os
|
||||
from tempfile import TemporaryFile
|
||||
import unicodedata
|
||||
from uuid import UUID, uuid4
|
||||
import warnings
|
||||
|
||||
from PIL import Image, UnidentifiedImageError
|
||||
from starlette.concurrency import run_in_threadpool
|
||||
from starlette.requests import ClientDisconnect
|
||||
|
||||
from src import db, jobs, object_store, storage
|
||||
from src.errors import ApiError
|
||||
|
||||
|
||||
MAX_BYTES = int(os.environ.get("MAX_UPLOAD_BYTES", "10485760"))
|
||||
MAX_PIXELS = int(os.environ.get("MAX_IMAGE_PIXELS", "40000000"))
|
||||
if not 1 <= MAX_BYTES <= 10485760 or not 1 <= MAX_PIXELS <= 40000000:
|
||||
raise ValueError("Лимиты загрузки должны быть положительными и не превышать контракт")
|
||||
Image.MAX_IMAGE_PIXELS = MAX_PIXELS
|
||||
warnings.filterwarnings("error", category=Image.DecompressionBombWarning)
|
||||
# Максимум одно полное декодирование на реплику, независимо от числа клиентов.
|
||||
upload_slot = asyncio.Semaphore(1)
|
||||
|
||||
|
||||
def public_attachment(row):
|
||||
return {"id": str(row["id"]), "chat_id": str(row["chat_id"]), "owner_id": str(row["owner_id"]),
|
||||
"filename": row["filename"], "content_type": row["content_type"],
|
||||
"size_bytes": row["size_bytes"], "sha256": row["sha256"], "status": row["status"],
|
||||
"variants": row["variants"], "error_code": row["error_code"], "created_at": storage.utc(row["created_at"]),
|
||||
"updated_at": storage.utc(row["updated_at"])}
|
||||
|
||||
|
||||
def validate_filename(filename):
|
||||
if not 1 <= len(filename) <= 255 or not filename.strip() or any(
|
||||
char in "/\\" or unicodedata.category(char) == "Cc" for char in filename
|
||||
):
|
||||
raise ApiError(400, "invalid_request", "Недопустимое имя файла")
|
||||
try:
|
||||
filename.encode("utf-8")
|
||||
except UnicodeError:
|
||||
raise ApiError(400, "invalid_request", "Недопустимое имя файла") from None
|
||||
|
||||
|
||||
def validate_image(file, declared_type):
|
||||
try:
|
||||
file.seek(0)
|
||||
with Image.open(file, formats=["PNG", "JPEG"]) as image:
|
||||
true_type = {"PNG": "image/png", "JPEG": "image/jpeg"}[image.format]
|
||||
if image.width * image.height > MAX_PIXELS:
|
||||
raise ApiError(413, "payload_too_large", "Слишком много пикселей")
|
||||
if true_type != declared_type:
|
||||
raise ApiError(415, "unsupported_media_type", "MIME не совпадает с содержимым")
|
||||
image.verify()
|
||||
# verify проверяет структуру/CRC, load дополнительно обнаруживает обрыв pixel data.
|
||||
file.seek(0)
|
||||
with Image.open(file, formats=["PNG", "JPEG"]) as image:
|
||||
image.load()
|
||||
return true_type
|
||||
except (Image.DecompressionBombError, Image.DecompressionBombWarning):
|
||||
raise ApiError(413, "payload_too_large", "Слишком много пикселей") from None
|
||||
except (UnidentifiedImageError, OSError, ValueError, SyntaxError):
|
||||
raise ApiError(415, "unsupported_media_type", "Повреждённый PNG/JPEG или другой формат") from None
|
||||
|
||||
|
||||
def lock_upload(conn, key, blocking=True):
|
||||
function = "pg_advisory_xact_lock" if blocking else "pg_try_advisory_xact_lock"
|
||||
return conn.execute(f"SELECT {function}(hashtextextended(%s, 4)) AS locked", (key,)).fetchone()["locked"]
|
||||
|
||||
|
||||
def save(user_id, chat_id, filename, content_type, file, size, sha256):
|
||||
attachment_id, job_id = uuid4(), uuid4()
|
||||
staging_key = "attachments/staging/" + attachment_id.hex
|
||||
final_key = "attachments/" + attachment_id.hex + "/v1/original"
|
||||
# Журнал фиксируется до PUT: crash до записи job оставит управляемый orphan.
|
||||
with db.connection() as conn:
|
||||
storage.require_member(conn, user_id, chat_id)
|
||||
conn.execute("INSERT INTO attachment_uploads(id, object_key) VALUES (%s, %s)", (attachment_id, staging_key))
|
||||
with db.connection() as conn:
|
||||
conn.execute("SET LOCAL idle_in_transaction_session_timeout='30s'")
|
||||
lock_upload(conn, staging_key)
|
||||
if not conn.execute("SELECT 1 FROM attachment_uploads WHERE id=%s", (attachment_id,)).fetchone():
|
||||
raise ApiError(503, "unavailable", "Загрузка больше не активна")
|
||||
object_store.put(staging_key, file, size, content_type)
|
||||
row = conn.execute("""INSERT INTO attachments(id,chat_id,owner_id,object_key,filename,
|
||||
content_type,size_bytes,sha256,status) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,'queued') RETURNING *""",
|
||||
(attachment_id, UUID(chat_id), UUID(user_id), final_key, filename, content_type, size, sha256)).fetchone()
|
||||
conn.execute("""INSERT INTO attachment_jobs(id,attachment_id,staging_key,max_attempts)
|
||||
VALUES (%s,%s,%s,%s)""", (job_id, attachment_id, staging_key, jobs.MAX_ATTEMPTS))
|
||||
conn.execute("INSERT INTO attachment_outbox(job_id,attempt) VALUES (%s,1)", (job_id,))
|
||||
conn.execute("DELETE FROM attachment_uploads WHERE id=%s", (attachment_id,))
|
||||
result = public_attachment(row)
|
||||
# Снимок queued даже если worker успеет закончить до отправки ответа.
|
||||
return result
|
||||
|
||||
|
||||
async def upload(request, chat_id, filename):
|
||||
user_id = request.state.user["id"]
|
||||
await run_in_threadpool(storage.get_chat, user_id, chat_id)
|
||||
validate_filename(filename)
|
||||
content_type = request.headers.get("Content-Type", "").split(";", 1)[0].strip().lower()
|
||||
if content_type not in {"image/png", "image/jpeg"}:
|
||||
raise ApiError(415, "unsupported_media_type", "Ожидаются сырые bytes PNG/JPEG")
|
||||
length = request.headers.get("Content-Length")
|
||||
if length is not None:
|
||||
if not length.isascii() or not length.isdecimal():
|
||||
raise ApiError(400, "invalid_request", "Неверный Content-Length")
|
||||
if len(length) > 10 or int(length) > MAX_BYTES:
|
||||
raise ApiError(413, "payload_too_large", "Файл больше допустимого размера")
|
||||
try:
|
||||
await asyncio.wait_for(upload_slot.acquire(), timeout=20)
|
||||
except TimeoutError:
|
||||
raise ApiError(503, "unavailable", "Загрузки заняты; повторите позднее") from None
|
||||
try:
|
||||
with TemporaryFile() as file:
|
||||
size = 0
|
||||
digest = hashlib.sha256()
|
||||
try:
|
||||
async with asyncio.timeout(30):
|
||||
async for chunk in request.stream():
|
||||
size += len(chunk)
|
||||
if size > MAX_BYTES:
|
||||
raise ApiError(413, "payload_too_large", "Файл больше допустимого размера")
|
||||
digest.update(chunk)
|
||||
await run_in_threadpool(file.write, chunk)
|
||||
except (TimeoutError, ClientDisconnect):
|
||||
raise ApiError(400, "invalid_request", "Загрузка прервана или истёк таймаут") from None
|
||||
true_type = await run_in_threadpool(validate_image, file, content_type)
|
||||
return await run_in_threadpool(save, user_id, chat_id, filename, true_type, file, size, digest.hexdigest())
|
||||
finally:
|
||||
upload_slot.release()
|
||||
|
||||
|
||||
def find(user_id, attachment_id):
|
||||
with db.connection() as conn:
|
||||
row = conn.execute("""SELECT a.* FROM attachments a JOIN chat_members m ON m.chat_id=a.chat_id
|
||||
WHERE a.id=%s AND m.user_id=%s""", (UUID(attachment_id), UUID(user_id))).fetchone()
|
||||
if row is None:
|
||||
raise ApiError(404, "not_found", "Вложение не найдено")
|
||||
return row
|
||||
|
||||
|
||||
def chunks(response):
|
||||
try:
|
||||
while chunk := response.read(65536):
|
||||
yield chunk
|
||||
finally:
|
||||
response.close()
|
||||
response.release_conn()
|
||||
+95
@@ -0,0 +1,95 @@
|
||||
"""Basic только для входа; Bearer-сессии с TTL и AOF находятся в Valkey."""
|
||||
import base64
|
||||
import binascii
|
||||
from contextlib import contextmanager
|
||||
from datetime import datetime, timedelta, timezone
|
||||
import hashlib
|
||||
import json
|
||||
import secrets
|
||||
|
||||
import redis
|
||||
|
||||
from src import passwords, storage
|
||||
from src.config import SESSION_TTL, secret
|
||||
from src.errors import ApiError
|
||||
|
||||
|
||||
sessions = redis.Redis.from_url(
|
||||
secret("CACHE_URL"), decode_responses=True,
|
||||
socket_connect_timeout=0.5, socket_timeout=0.5, max_connections=10,
|
||||
retry=redis.retry.Retry(redis.backoff.NoBackoff(), 0),
|
||||
)
|
||||
|
||||
|
||||
@contextmanager
|
||||
def available():
|
||||
try:
|
||||
yield
|
||||
except redis.RedisError:
|
||||
raise ApiError(503, "unavailable", "Valkey временно недоступен") from None
|
||||
|
||||
|
||||
def unauthorized(basic=False):
|
||||
challenge = 'Basic realm="messenger", charset="UTF-8"' if basic else 'Bearer realm="messenger"'
|
||||
return ApiError(401, "unauthorized", "Неверная или отсутствующая идентификация", {"WWW-Authenticate": challenge})
|
||||
|
||||
|
||||
def credential(header, scheme):
|
||||
parts = (header or "").split()
|
||||
if len(parts) != 2 or parts[0].lower() != scheme.lower():
|
||||
raise unauthorized(scheme == "Basic")
|
||||
return parts[1]
|
||||
|
||||
|
||||
def basic_user(header):
|
||||
value = credential(header, "Basic")
|
||||
try:
|
||||
decoded = base64.b64decode(value, validate=True).decode("utf-8")
|
||||
username, password = decoded.split(":", 1)
|
||||
except (ValueError, UnicodeError, binascii.Error):
|
||||
raise unauthorized(True) from None
|
||||
user = storage.find_login(username)
|
||||
if user is None or not passwords.matches(user["password_hash"], password):
|
||||
raise unauthorized(True)
|
||||
return storage.public_user(user)
|
||||
|
||||
|
||||
def session_key(token):
|
||||
return "session:" + hashlib.sha256(token.encode()).hexdigest()
|
||||
|
||||
|
||||
def create_session(user):
|
||||
token = secrets.token_urlsafe(32)
|
||||
expires = datetime.now(timezone.utc) + timedelta(seconds=SESSION_TTL)
|
||||
value = {"user_id": user["id"], "expires_at": storage.utc(expires)}
|
||||
with available():
|
||||
sessions.set(session_key(token), json.dumps(value), ex=SESSION_TTL)
|
||||
return {"token": token, "token_type": "Bearer", "expires_at": value["expires_at"], "user": user}
|
||||
|
||||
|
||||
def current_user(header):
|
||||
token = credential(header, "Bearer")
|
||||
with available():
|
||||
value = sessions.get(session_key(token))
|
||||
if value is None:
|
||||
raise unauthorized()
|
||||
data = json.loads(value)
|
||||
if datetime.fromisoformat(data["expires_at"].replace("Z", "+00:00")) <= datetime.now(timezone.utc):
|
||||
raise unauthorized()
|
||||
user = storage.find_user(data["user_id"])
|
||||
if user is None:
|
||||
raise unauthorized()
|
||||
return user
|
||||
|
||||
|
||||
def logout(header):
|
||||
token = credential(header, "Bearer")
|
||||
with available():
|
||||
removed = sessions.delete(session_key(token))
|
||||
if not removed:
|
||||
raise unauthorized()
|
||||
|
||||
|
||||
def ready():
|
||||
with available():
|
||||
sessions.ping()
|
||||
@@ -0,0 +1,38 @@
|
||||
"""Идемпотентное создание приватного bucket отдельной административной учёткой."""
|
||||
import os
|
||||
import time
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from minio import Minio
|
||||
from minio.error import MinioException, S3Error
|
||||
import urllib3
|
||||
|
||||
from src.config import secret
|
||||
|
||||
|
||||
def bootstrap():
|
||||
endpoint = urlsplit(os.environ.get("S3_ENDPOINT", "http://s3:9000"))
|
||||
client = Minio(endpoint.netloc, access_key=secret("S3_ROOT_USER"), secret_key=secret("S3_ROOT_PASSWORD"),
|
||||
secure=endpoint.scheme == "https", region=os.environ.get("S3_REGION", "us-east-1"),
|
||||
http_client=urllib3.PoolManager(timeout=urllib3.Timeout(connect=1, read=3), retries=False))
|
||||
bucket = os.environ.get("S3_BUCKET", "messenger-attachments")
|
||||
deadline = time.monotonic() + 60
|
||||
while True:
|
||||
try:
|
||||
if not client.bucket_exists(bucket):
|
||||
client.make_bucket(bucket)
|
||||
print("Private bucket is ready; static IAM permits only authenticated accounts", flush=True)
|
||||
return
|
||||
except S3Error as error:
|
||||
if error.code in {"BucketAlreadyOwnedByYou", "BucketAlreadyExists"}:
|
||||
return
|
||||
if time.monotonic() >= deadline:
|
||||
raise RuntimeError("S3 bootstrap failed") from None
|
||||
except (MinioException, urllib3.exceptions.HTTPError, OSError):
|
||||
if time.monotonic() >= deadline:
|
||||
raise RuntimeError("S3 bootstrap timed out") from None
|
||||
time.sleep(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
bootstrap()
|
||||
@@ -0,0 +1,38 @@
|
||||
"""Один durable Stream, consumer group; ACK только после эффекта/устойчивого retry."""
|
||||
import json
|
||||
|
||||
import redis
|
||||
|
||||
from src.config import secret
|
||||
|
||||
STREAM = 'attachments:v1'
|
||||
GROUP = 'image-workers'
|
||||
|
||||
|
||||
def connect():
|
||||
return redis.Redis.from_url(secret('BROKER_URL'), decode_responses=True,
|
||||
socket_connect_timeout=2, socket_timeout=3,
|
||||
retry_on_timeout=False)
|
||||
|
||||
|
||||
def bootstrap(client):
|
||||
try:
|
||||
client.xgroup_create(STREAM, GROUP, id='0', mkstream=True)
|
||||
except redis.ResponseError as error:
|
||||
if 'BUSYGROUP' not in str(error):
|
||||
raise
|
||||
|
||||
|
||||
def publish(client, job):
|
||||
payload = {'schema_version': 1, 'job_id': str(job['id']), 'type': 'attachment.process.v1',
|
||||
'attachment_id': str(job['attachment_id']), 'pipeline_version': job['pipeline_version'],
|
||||
'attempt': job['attempt'], 'created_at': job['created_at'].isoformat().replace('+00:00', 'Z')}
|
||||
return client.xadd(STREAM, {'job': json.dumps(payload)})
|
||||
|
||||
|
||||
def acknowledge(client, message_id):
|
||||
# Единственная группа: подтверждённые записи можно удалить. Pending не trim-им.
|
||||
with client.pipeline(transaction=True) as pipe:
|
||||
pipe.xack(STREAM, GROUP, message_id)
|
||||
pipe.xdel(STREAM, message_id)
|
||||
pipe.execute()
|
||||
@@ -0,0 +1,87 @@
|
||||
"""Очистка orphan PUT и staging только ready/просроченных failed; текущую работу не трогает."""
|
||||
import argparse
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
|
||||
from src import db, object_store
|
||||
from src.attachments import lock_upload
|
||||
|
||||
|
||||
GRACE_SECONDS = int(os.environ.get("UPLOAD_CLEANUP_GRACE_SECONDS", "3600"))
|
||||
if GRACE_SECONDS < 60:
|
||||
raise ValueError("Безопасный grace period должен быть не меньше 60 секунд")
|
||||
|
||||
|
||||
def cleanup_orphans():
|
||||
result = {"removed": 0, "kept_ready": 0, "skipped_active": 0}
|
||||
with db.connection() as conn:
|
||||
candidates = conn.execute("""SELECT id, object_key FROM attachment_uploads
|
||||
WHERE created_at < clock_timestamp() - %s * interval '1 second'
|
||||
ORDER BY created_at LIMIT 100""", (GRACE_SECONDS,)).fetchall()
|
||||
for row in candidates:
|
||||
with db.connection() as conn:
|
||||
if not lock_upload(conn, row["object_key"], blocking=False):
|
||||
result["skipped_active"] += 1
|
||||
continue
|
||||
pending = conn.execute("""SELECT 1 FROM attachment_uploads WHERE id=%s
|
||||
AND created_at < clock_timestamp() - %s * interval '1 second' FOR UPDATE""",
|
||||
(row["id"], GRACE_SECONDS)).fetchone()
|
||||
if not pending:
|
||||
continue
|
||||
ready = conn.execute("SELECT 1 FROM attachments WHERE object_key=%s", (row["object_key"],)).fetchone()
|
||||
if ready:
|
||||
result["kept_ready"] += 1
|
||||
else:
|
||||
object_store.remove(row["object_key"])
|
||||
result["removed"] += 1
|
||||
conn.execute("DELETE FROM attachment_uploads WHERE id=%s", (row["id"],))
|
||||
return result
|
||||
|
||||
|
||||
RETENTION = int(os.environ.get("STAGING_RETENTION_SECONDS", "86400"))
|
||||
if RETENTION < 60:
|
||||
raise ValueError("STAGING_RETENTION_SECONDS должен быть >=60")
|
||||
|
||||
|
||||
def cleanup_staging():
|
||||
removed = 0
|
||||
with db.connection() as conn:
|
||||
rows = conn.execute("""SELECT j.*,a.status,a.object_key FROM attachment_jobs j
|
||||
JOIN attachments a ON a.id=j.attachment_id WHERE j.staging_deleted_at IS NULL
|
||||
AND (a.status='ready' OR (a.status='failed' AND a.updated_at<clock_timestamp()-%s*interval '1 second'))
|
||||
LIMIT 20 FOR UPDATE OF j,a SKIP LOCKED""", (RETENTION,)).fetchall()
|
||||
for job in rows:
|
||||
object_store.remove(job['staging_key'])
|
||||
if job['status'] == 'failed':
|
||||
prefix = job['object_key'].rsplit('/', 1)[0] + '/'
|
||||
for variant in ('original', 'thumbnail', 'crop', 'optimized'):
|
||||
object_store.remove(prefix + variant)
|
||||
conn.execute('UPDATE attachment_jobs SET staging_deleted_at=clock_timestamp() WHERE id=%s', (job['id'],))
|
||||
removed += 1
|
||||
return removed
|
||||
|
||||
|
||||
def cleanup_once():
|
||||
return {**cleanup_orphans(), "staging_removed": cleanup_staging()}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--once", action="store_true")
|
||||
args = parser.parse_args()
|
||||
db.pool.open(wait=True, timeout=15)
|
||||
try:
|
||||
while True:
|
||||
try:
|
||||
print(json.dumps(cleanup_once()), flush=True)
|
||||
except Exception as error:
|
||||
if args.once:
|
||||
raise
|
||||
logging.error("Cleanup postponed: %s", type(error).__name__)
|
||||
if args.once:
|
||||
break
|
||||
time.sleep(30)
|
||||
finally:
|
||||
db.pool.close()
|
||||
@@ -0,0 +1,20 @@
|
||||
"""Настройки и чтение локальных Docker secrets."""
|
||||
import os
|
||||
import socket
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def secret(name):
|
||||
return Path(os.environ[name + "_FILE"]).read_text().strip()
|
||||
|
||||
|
||||
SESSION_TTL = int(os.environ.get("SESSION_TTL_SECONDS", "3600"))
|
||||
if SESSION_TTL < 1:
|
||||
raise ValueError("SESSION_TTL_SECONDS должен быть положительным")
|
||||
if os.environ.get("SESSION_STORE", "valkey") != "valkey":
|
||||
raise ValueError("В этой реализации SESSION_STORE=valkey")
|
||||
|
||||
INSTANCE_ID = socket.gethostname()
|
||||
IDEMPOTENCY_TTL = int(os.environ.get("IDEMPOTENCY_TTL_SECONDS", "86400"))
|
||||
if IDEMPOTENCY_TTL < 86400:
|
||||
raise ValueError("IDEMPOTENCY_TTL_SECONDS должен быть не меньше 86400")
|
||||
@@ -0,0 +1,52 @@
|
||||
"""Ограниченный пул; каждый with connection() — отдельная транзакция."""
|
||||
from contextlib import contextmanager
|
||||
|
||||
import psycopg
|
||||
from psycopg.rows import dict_row
|
||||
from psycopg_pool import ConnectionPool, PoolTimeout, TooManyRequests
|
||||
|
||||
from src.config import secret
|
||||
from src.errors import ApiError
|
||||
|
||||
|
||||
IO_TIMEOUT = 2.0
|
||||
|
||||
|
||||
class TimedConnection(psycopg.Connection):
|
||||
def wait(self, gen, interval=0.1, timeout=None):
|
||||
# statement_timeout работает на сервере и не помогает при docker pause.
|
||||
# Ограничиваем само ожидание I/O: проверку пула, SQL, COMMIT и ROLLBACK.
|
||||
timeout = IO_TIMEOUT if timeout is None else min(timeout, IO_TIMEOUT)
|
||||
try:
|
||||
return super().wait(gen, interval=interval, timeout=timeout)
|
||||
except psycopg.OperationalError:
|
||||
# Не ждём отмены/ROLLBACK от зависшего сервера. Пул заменит соединение.
|
||||
self.close()
|
||||
raise
|
||||
|
||||
|
||||
pool = ConnectionPool(
|
||||
conninfo=secret("DATABASE_URL"), open=False,
|
||||
connection_class=TimedConnection,
|
||||
min_size=1, max_size=5, max_waiting=20, timeout=2,
|
||||
check=ConnectionPool.check_connection,
|
||||
kwargs={
|
||||
"row_factory": dict_row,
|
||||
"connect_timeout": 2,
|
||||
"options": "-c timezone=UTC -c statement_timeout=3000 -c lock_timeout=2000 -c idle_in_transaction_session_timeout=5000",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@contextmanager
|
||||
def connection():
|
||||
try:
|
||||
with pool.connection() as conn:
|
||||
yield conn
|
||||
except (psycopg.OperationalError, PoolTimeout, TooManyRequests):
|
||||
raise ApiError(503, "unavailable", "PostgreSQL временно недоступен") from None
|
||||
|
||||
|
||||
def ready():
|
||||
with connection() as conn:
|
||||
conn.execute("SELECT 1")
|
||||
@@ -0,0 +1,27 @@
|
||||
"""общий формат ошибок"""
|
||||
|
||||
from fastapi import Request
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
|
||||
class ApiError(Exception):
|
||||
def __init__(self, status: int, code: str, message: str, headers=None):
|
||||
super().__init__(message)
|
||||
self.status = status
|
||||
self.code = code
|
||||
self.message = message
|
||||
self.headers = headers or {}
|
||||
|
||||
|
||||
def error_response(request: Request, error: ApiError):
|
||||
return JSONResponse(
|
||||
status_code=error.status,
|
||||
content={
|
||||
"error": {
|
||||
"code": error.code,
|
||||
"message": error.message,
|
||||
"request_id": request.state.request_id,
|
||||
}
|
||||
},
|
||||
headers={"X-Request-Id": request.state.request_id, **error.headers},
|
||||
)
|
||||
@@ -0,0 +1,33 @@
|
||||
"""Одноразовые failpoints для явных Docker-тестов; по умолчанию отключены."""
|
||||
import os
|
||||
from pathlib import Path
|
||||
import time
|
||||
|
||||
from src.errors import ApiError
|
||||
|
||||
|
||||
def hit(job_id, point):
|
||||
if os.environ.get('ENABLE_FAULT_INJECTION') != '1':
|
||||
return
|
||||
path = Path('/tmp/pipeline-faults') / (str(job_id) + '.' + point)
|
||||
try:
|
||||
action = path.read_text().strip()
|
||||
path.unlink()
|
||||
except FileNotFoundError:
|
||||
return
|
||||
path.with_suffix(path.suffix + '.hit').write_text(action)
|
||||
if action.startswith('transient:'):
|
||||
remaining = int(action.split(':')[1]) - 1
|
||||
if remaining:
|
||||
path.write_text('transient:' + str(remaining))
|
||||
action = 'transient'
|
||||
if action == 'crash':
|
||||
os._exit(86)
|
||||
if action == 'transient':
|
||||
raise ApiError(503, 'unavailable', 'Injected transient failure')
|
||||
if action == 'memory':
|
||||
bytearray(1024 * 1024 * 1024)
|
||||
if action == 'timeout':
|
||||
time.sleep(300)
|
||||
if action.startswith('delay:'):
|
||||
time.sleep(float(action.split(':', 1)[1]))
|
||||
@@ -0,0 +1,15 @@
|
||||
"""healthcheck докера"""
|
||||
|
||||
import json
|
||||
import os
|
||||
from urllib.request import urlopen
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
port = os.environ.get("APP_PORT", "8080")
|
||||
try:
|
||||
with urlopen(f"http://127.0.0.1:{port}/health/ready", timeout=2) as response:
|
||||
ready = response.status == 200 and json.load(response) == {"status": "ok"}
|
||||
raise SystemExit(0 if ready else 1)
|
||||
except (OSError, ValueError):
|
||||
raise SystemExit(1)
|
||||
@@ -0,0 +1,70 @@
|
||||
"""Одна ограниченная по памяти/времени обработка; оригинал не перекодируется."""
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import resource
|
||||
import sys
|
||||
import warnings
|
||||
|
||||
from PIL import Image, ImageOps
|
||||
|
||||
|
||||
def fit_size(size, limit):
|
||||
width, height = size
|
||||
if max(size) <= limit:
|
||||
return size
|
||||
# Целочисленное округление вниз, минимум 1 px, без float-погрешности.
|
||||
return max(1, width * limit // max(size)), max(1, height * limit // max(size))
|
||||
|
||||
|
||||
def transform(source, output):
|
||||
Image.MAX_IMAGE_PIXELS = int(os.environ.get('MAX_IMAGE_PIXELS', '40000000'))
|
||||
warnings.simplefilter('error', Image.DecompressionBombWarning)
|
||||
with Image.open(source, formats=['PNG', 'JPEG']) as image:
|
||||
if image.width * image.height > Image.MAX_IMAGE_PIXELS:
|
||||
raise ValueError('pixel_limit')
|
||||
image.load()
|
||||
ImageOps.exif_transpose(image, in_place=True)
|
||||
if image.mode in ('RGBA', 'LA') or 'transparency' in image.info:
|
||||
rgba = image.convert('RGBA')
|
||||
rgb = Image.new('RGB', image.size, 'white')
|
||||
rgb.paste(rgba, mask=rgba.getchannel('A'))
|
||||
rgba.close()
|
||||
else:
|
||||
rgb = image.convert('RGB')
|
||||
rgb.info.clear()
|
||||
variants = []
|
||||
for name, size, encoding, mime in [
|
||||
('thumbnail', fit_size(rgb.size, 256), 'JPEG', 'image/jpeg'),
|
||||
('crop', (128, 128), 'JPEG', 'image/jpeg'),
|
||||
('optimized', fit_size(rgb.size, 1280), 'WEBP', 'image/webp'),
|
||||
]:
|
||||
variant = ImageOps.fit(rgb, size, method=Image.Resampling.LANCZOS, centering=(0.5, 0.5)) if name == 'crop' else rgb.resize(size, Image.Resampling.LANCZOS)
|
||||
variant.info.clear()
|
||||
buffer = io.BytesIO()
|
||||
variant.save(buffer, format=encoding, quality=85 if encoding == 'JPEG' else 80,
|
||||
**({'method': 4} if encoding == 'WEBP' else {}))
|
||||
data = buffer.getvalue()
|
||||
(output / name).write_bytes(data)
|
||||
variants.append({'name': name, 'content_type': mime, 'size_bytes': len(data),
|
||||
'width': size[0], 'height': size[1], 'sha256': hashlib.sha256(data).hexdigest()})
|
||||
variant.close()
|
||||
rgb.close()
|
||||
(output / 'variants.json').write_text(json.dumps(variants))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
memory = int(os.environ.get('WORKER_TASK_MEMORY_MB', '640')) * 1024 * 1024
|
||||
resource.setrlimit(resource.RLIMIT_AS, (memory, memory))
|
||||
seconds = int(os.environ.get('JOB_TIMEOUT_SECONDS', '60'))
|
||||
resource.setrlimit(resource.RLIMIT_CPU, (seconds, seconds + 1))
|
||||
try:
|
||||
from src import faults
|
||||
faults.hit(sys.argv[3], 'transform')
|
||||
transform(Path(sys.argv[1]), Path(sys.argv[2]))
|
||||
except (ValueError, OSError, SyntaxError, Image.DecompressionBombWarning, Image.DecompressionBombError):
|
||||
sys.exit(65) # permanent invalid_image; stderr не содержит внутренние данные.
|
||||
except MemoryError:
|
||||
sys.exit(66)
|
||||
@@ -0,0 +1,29 @@
|
||||
"""Локальная операторская CLI: failed registry и безопасный replay по job_id."""
|
||||
import argparse
|
||||
import json
|
||||
|
||||
from src import db, jobs
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
sub = parser.add_subparsers(dest='command', required=True)
|
||||
sub.add_parser('failed')
|
||||
replay = sub.add_parser('replay')
|
||||
replay.add_argument('job_id')
|
||||
args = parser.parse_args()
|
||||
db.pool.open(wait=True, timeout=15)
|
||||
try:
|
||||
if args.command == 'replay':
|
||||
jobs.replay(args.job_id)
|
||||
print('Replay queued with audit; job_id=' + args.job_id)
|
||||
else:
|
||||
with db.connection() as conn:
|
||||
rows = conn.execute("""SELECT j.id AS job_id,j.attachment_id,j.attempt,j.max_attempts,
|
||||
a.error_code,j.last_error,j.staging_deleted_at FROM attachment_jobs j
|
||||
JOIN attachments a ON a.id=j.attachment_id WHERE a.status='failed' ORDER BY a.updated_at""").fetchall()
|
||||
print(json.dumps(rows, default=str, indent=2))
|
||||
except ValueError as error:
|
||||
parser.exit(2, str(error) + '\n')
|
||||
finally:
|
||||
db.pool.close()
|
||||
@@ -0,0 +1,31 @@
|
||||
"""Одна попытка в отдельном процессе под контролем consumer-supervisor."""
|
||||
import sys
|
||||
from uuid import UUID
|
||||
|
||||
from src import db, jobs, task_supervisor
|
||||
from src.worker import ProcessingError, process
|
||||
|
||||
|
||||
def main():
|
||||
task_supervisor.watch_parent(int(sys.argv[3]))
|
||||
db.pool.open(wait=True, timeout=15)
|
||||
try:
|
||||
with db.connection() as conn:
|
||||
job = jobs.locked(conn, UUID(sys.argv[1]))
|
||||
if not job or job['status'] != 'processing' or job['lease_token'] != UUID(sys.argv[2]):
|
||||
return 75
|
||||
try:
|
||||
if not process(job, sys.argv[4]):
|
||||
return 75
|
||||
except ProcessingError as error:
|
||||
jobs.fail(job, error.code, error.permanent)
|
||||
except Exception:
|
||||
jobs.fail(job, 'storage_unavailable')
|
||||
# Только после COMMIT ready/retry/failed родитель может отправить ACK.
|
||||
return 0
|
||||
finally:
|
||||
db.pool.close()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
+121
@@ -0,0 +1,121 @@
|
||||
"""Транзакционные переходы состояний, bounded retry и явный audit replay."""
|
||||
import os
|
||||
import random
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from src import db
|
||||
|
||||
MAX_ATTEMPTS = int(os.environ.get('JOB_MAX_ATTEMPTS', '5'))
|
||||
TIMEOUT = int(os.environ.get('JOB_TIMEOUT_SECONDS', '60'))
|
||||
LEASE = int(os.environ.get('JOB_LEASE_SECONDS', '90'))
|
||||
TASK_TIMEOUT = int(os.environ.get('JOB_TASK_TIMEOUT_SECONDS', str(max(180, TIMEOUT + 60))))
|
||||
HEARTBEAT = min(10, LEASE / 3)
|
||||
BACKOFF = float(os.environ.get('JOB_BACKOFF_SECONDS', '2'))
|
||||
if not 1 <= MAX_ATTEMPTS <= 20 or TIMEOUT < 1 or LEASE < 15 or TASK_TIMEOUT < TIMEOUT or BACKOFF <= 0:
|
||||
raise ValueError('Неверные retry/timeout/lease настройки')
|
||||
|
||||
|
||||
def event(conn, job, name, error=None):
|
||||
conn.execute('INSERT INTO attachment_job_events(job_id,attempt,event,error_code) VALUES (%s,%s,%s,%s)',
|
||||
(job['id'], job['attempt'], name, error))
|
||||
|
||||
|
||||
def locked(conn, job_id):
|
||||
return conn.execute('''SELECT j.*, a.status, a.object_key, a.variants, a.sha256, a.size_bytes,
|
||||
a.content_type FROM attachment_jobs j JOIN attachments a ON a.id=j.attachment_id
|
||||
WHERE j.id=%s FOR UPDATE OF j,a''', (job_id,)).fetchone()
|
||||
|
||||
|
||||
def claim(payload):
|
||||
with db.connection() as conn:
|
||||
job = locked(conn, UUID(payload['job_id']))
|
||||
if not job or str(job['attachment_id']) != payload['attachment_id'] or job['pipeline_version'] != payload['pipeline_version']:
|
||||
return 'obsolete', None
|
||||
if job['status'] in {'ready', 'failed'}:
|
||||
return job['status'], job
|
||||
if payload['attempt'] != job['attempt']:
|
||||
return 'obsolete', None
|
||||
if job['status'] != 'queued':
|
||||
return 'busy', None
|
||||
row = conn.execute('''UPDATE attachment_jobs SET lease_token=%s,
|
||||
lease_until=clock_timestamp()+%s*interval '1 second' WHERE id=%s
|
||||
AND next_attempt_at<=clock_timestamp() RETURNING *''', (uuid4(), LEASE, job['id'])).fetchone()
|
||||
if not row:
|
||||
return 'busy', None
|
||||
job.update(row)
|
||||
conn.execute("UPDATE attachments SET status='processing',updated_at=clock_timestamp() WHERE id=%s", (job['attachment_id'],))
|
||||
event(conn, job, 'processing')
|
||||
return 'claimed', job
|
||||
|
||||
|
||||
def renew(job):
|
||||
"""Продлить только действующую аренду; истёкшую/чужую не воскрешаем."""
|
||||
with db.connection() as conn:
|
||||
current = conn.execute('''SELECT j.lease_token,j.lease_until,a.status
|
||||
FROM attachment_jobs j JOIN attachments a ON a.id=j.attachment_id
|
||||
WHERE j.id=%s FOR UPDATE OF j,a SKIP LOCKED''', (job['id'],)).fetchone()
|
||||
if current is None:
|
||||
# Финальная запись объектов держит эти строки под fence. Не ждём
|
||||
# её блокировку: recover также пропустит её, общий timeout действует.
|
||||
current = conn.execute('''SELECT j.lease_token,a.status FROM attachment_jobs j
|
||||
JOIN attachments a ON a.id=j.attachment_id WHERE j.id=%s''', (job['id'],)).fetchone()
|
||||
return bool(current and current['status'] == 'processing'
|
||||
and current['lease_token'] == job['lease_token'])
|
||||
if current['status'] != 'processing' or current['lease_token'] != job['lease_token']:
|
||||
return False
|
||||
return conn.execute('''UPDATE attachment_jobs
|
||||
SET lease_until=clock_timestamp()+%s*interval '1 second'
|
||||
WHERE id=%s AND lease_until>clock_timestamp() RETURNING id''',
|
||||
(LEASE, job['id'])).fetchone() is not None
|
||||
|
||||
|
||||
def retry_or_fail(conn, job, code, permanent=False):
|
||||
# replay_count задаёт новый ограниченный бюджет, сохраняя глобальный номер attempt.
|
||||
exhausted = job['attempt'] >= job['max_attempts']
|
||||
if permanent or exhausted:
|
||||
final_code = code if permanent else 'retry_exhausted'
|
||||
conn.execute("UPDATE attachments SET status='failed',error_code=%s,updated_at=clock_timestamp() WHERE id=%s",
|
||||
(final_code, job['attachment_id']))
|
||||
conn.execute('UPDATE attachment_jobs SET lease_token=NULL,lease_until=NULL,last_error=%s WHERE id=%s', (code, job['id']))
|
||||
event(conn, job, 'failed', final_code)
|
||||
return
|
||||
delay = random.uniform(0.5, 1.5) * min(60, BACKOFF * 2 ** min(job['attempt'] - 1, 10))
|
||||
conn.execute('''UPDATE attachment_jobs SET attempt=attempt+1, lease_token=NULL,lease_until=NULL,
|
||||
last_error=%s,next_attempt_at=clock_timestamp()+%s*interval '1 second' WHERE id=%s''', (code, delay, job['id']))
|
||||
conn.execute("UPDATE attachments SET status='queued',error_code=NULL,updated_at=clock_timestamp() WHERE id=%s", (job['attachment_id'],))
|
||||
conn.execute('''INSERT INTO attachment_outbox(job_id,attempt,available_at)
|
||||
SELECT id,attempt,next_attempt_at FROM attachment_jobs WHERE id=%s''', (job['id'],))
|
||||
event(conn, job, 'retry', code)
|
||||
|
||||
|
||||
def fail(job, code, permanent=False):
|
||||
with db.connection() as conn:
|
||||
current = locked(conn, job['id'])
|
||||
if current['status'] == 'processing' and current['lease_token'] == job['lease_token']:
|
||||
retry_or_fail(conn, current, code, permanent)
|
||||
|
||||
|
||||
def recover():
|
||||
with db.connection() as conn:
|
||||
rows = conn.execute('''SELECT j.id FROM attachment_jobs j JOIN attachments a ON a.id=j.attachment_id
|
||||
WHERE a.status='processing' AND j.lease_until<clock_timestamp()
|
||||
ORDER BY j.lease_until LIMIT 20 FOR UPDATE OF j SKIP LOCKED''').fetchall()
|
||||
for row in rows:
|
||||
job = locked(conn, row['id'])
|
||||
expired = conn.execute('''SELECT lease_until<clock_timestamp() AS expired
|
||||
FROM attachment_jobs WHERE id=%s''', (job['id'],)).fetchone()['expired']
|
||||
if job['status'] == 'processing' and expired:
|
||||
retry_or_fail(conn, job, 'worker_lost')
|
||||
|
||||
|
||||
def replay(job_id):
|
||||
with db.connection() as conn:
|
||||
job = locked(conn, UUID(job_id))
|
||||
if not job or job['status'] != 'failed' or job['staging_deleted_at'] is not None:
|
||||
raise ValueError('Replay разрешён только для failed с сохранённым staging')
|
||||
conn.execute('''UPDATE attachment_jobs SET attempt=attempt+1,replay_count=replay_count+1,
|
||||
max_attempts=attempt+%s,next_attempt_at=clock_timestamp(),last_error=NULL,
|
||||
lease_token=NULL,lease_until=NULL WHERE id=%s''', (MAX_ATTEMPTS, job['id']))
|
||||
conn.execute("UPDATE attachments SET status='queued',error_code=NULL,updated_at=clock_timestamp() WHERE id=%s", (job['attachment_id'],))
|
||||
conn.execute('INSERT INTO attachment_outbox(job_id,attempt) SELECT id,attempt FROM attachment_jobs WHERE id=%s', (job['id'],))
|
||||
event(conn, job, 'operator_replay')
|
||||
@@ -0,0 +1,48 @@
|
||||
"""Версионируемые миграции с проверкой checksum и блокировкой."""
|
||||
import hashlib
|
||||
from pathlib import Path
|
||||
|
||||
from src import db
|
||||
|
||||
|
||||
DIRECTORY = Path(__file__).resolve().parents[1] / "migrations"
|
||||
|
||||
|
||||
def check():
|
||||
"""API только проверяет схему; DDL выполняет отдельный migrate-service."""
|
||||
with db.connection() as conn:
|
||||
for path in sorted(DIRECTORY.glob("*.sql")):
|
||||
row = conn.execute("SELECT checksum FROM schema_migrations WHERE version=%s", (path.name,)).fetchone()
|
||||
if row is None or row["checksum"] != hashlib.sha256(path.read_bytes()).hexdigest():
|
||||
raise RuntimeError("Сначала выполните миграции: " + path.name)
|
||||
|
||||
|
||||
def apply():
|
||||
applied = []
|
||||
with db.connection() as conn:
|
||||
conn.execute("SELECT pg_advisory_xact_lock(202602)")
|
||||
conn.execute("""
|
||||
CREATE TABLE IF NOT EXISTS schema_migrations (
|
||||
version text PRIMARY KEY, checksum text NOT NULL,
|
||||
applied_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
)
|
||||
""")
|
||||
for path in sorted(DIRECTORY.glob("*.sql")):
|
||||
checksum = hashlib.sha256(path.read_bytes()).hexdigest()
|
||||
existing = conn.execute("SELECT checksum FROM schema_migrations WHERE version=%s", (path.name,)).fetchone()
|
||||
if existing:
|
||||
if existing["checksum"] != checksum:
|
||||
raise RuntimeError("Изменена применённая миграция: " + path.name)
|
||||
continue
|
||||
conn.execute(path.read_text(), prepare=False)
|
||||
conn.execute("INSERT INTO schema_migrations(version, checksum) VALUES (%s, %s)", (path.name, checksum))
|
||||
applied.append(path.name)
|
||||
return applied
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
db.pool.open(wait=True, timeout=15)
|
||||
try:
|
||||
print("Migrations:", ", ".join(apply()) or "already applied")
|
||||
finally:
|
||||
db.pool.close()
|
||||
@@ -0,0 +1,56 @@
|
||||
"""допустимые поля и проверка данных"""
|
||||
|
||||
from uuid import UUID
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, ValidationInfo, field_validator, model_validator
|
||||
|
||||
|
||||
class RequestModel(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", strict=True)
|
||||
|
||||
@field_validator("*")
|
||||
@classmethod
|
||||
def check_strings(cls, value, info: ValidationInfo):
|
||||
if isinstance(value, str):
|
||||
if info.field_name != "text" and not value.strip():
|
||||
raise ValueError("Строка не должна состоять из пробелов")
|
||||
value.encode("utf-8")
|
||||
return value
|
||||
|
||||
|
||||
class CreateUser(RequestModel):
|
||||
username: str = Field(pattern=r"^[a-z0-9_]{3,32}$")
|
||||
display_name: str = Field(min_length=1, max_length=100)
|
||||
password: str = Field(min_length=12, max_length=128)
|
||||
|
||||
|
||||
class CreateChat(RequestModel):
|
||||
title: str = Field(min_length=1, max_length=100)
|
||||
member_ids: list[str] = Field(min_length=1, max_length=100)
|
||||
|
||||
@field_validator("member_ids")
|
||||
@classmethod
|
||||
def check_members(cls, values):
|
||||
member_ids = [str(UUID(value)) for value in values]
|
||||
if len(set(member_ids)) != len(member_ids):
|
||||
raise ValueError("Участники не должны повторяться")
|
||||
return member_ids
|
||||
|
||||
|
||||
class CreateMessage(RequestModel):
|
||||
text: str = Field(max_length=4000)
|
||||
attachment_ids: list[str] = Field(default_factory=list, max_length=10)
|
||||
|
||||
@field_validator("attachment_ids")
|
||||
@classmethod
|
||||
def check_attachments(cls, values):
|
||||
ids = [str(UUID(value)) for value in values]
|
||||
if len(ids) != len(set(ids)):
|
||||
raise ValueError("Вложения не должны повторяться")
|
||||
return ids
|
||||
|
||||
@model_validator(mode="after")
|
||||
def check_content(self):
|
||||
if not self.text.strip() and not self.attachment_ids:
|
||||
raise ValueError("Нужен текст или хотя бы одно вложение")
|
||||
return self
|
||||
@@ -0,0 +1,57 @@
|
||||
"""Один приватный bucket; короткие таймауты и отсутствие прозрачных повторов PUT."""
|
||||
from contextlib import contextmanager
|
||||
import os
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from minio import Minio
|
||||
from minio.error import MinioException
|
||||
import urllib3
|
||||
|
||||
from src.config import secret
|
||||
from src.errors import ApiError
|
||||
|
||||
|
||||
BUCKET = os.environ.get("S3_BUCKET", "messenger-attachments")
|
||||
endpoint = urlsplit(os.environ.get("S3_ENDPOINT", "http://s3:9000"))
|
||||
client = Minio(
|
||||
endpoint.netloc,
|
||||
access_key=secret("S3_ACCESS_KEY_ID"),
|
||||
secret_key=secret("S3_SECRET_ACCESS_KEY"),
|
||||
secure=endpoint.scheme == "https",
|
||||
region=os.environ.get("S3_REGION", "us-east-1"),
|
||||
http_client=urllib3.PoolManager(
|
||||
timeout=urllib3.Timeout(connect=1, read=3), retries=False, maxsize=4,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
@contextmanager
|
||||
def available():
|
||||
try:
|
||||
yield
|
||||
except (MinioException, urllib3.exceptions.HTTPError, OSError):
|
||||
raise ApiError(503, "unavailable", "Хранилище вложений временно недоступно") from None
|
||||
|
||||
|
||||
def ready():
|
||||
with available():
|
||||
if not client.bucket_exists(BUCKET):
|
||||
raise ApiError(503, "unavailable", "Хранилище вложений не готово")
|
||||
|
||||
|
||||
def put(key, file, size, content_type):
|
||||
with available():
|
||||
file.seek(0)
|
||||
# Весь обязательный диапазон <=10 MiB помещается в ОДИН PUT, без multipart.
|
||||
client.put_object(BUCKET, key, file, size, content_type=content_type,
|
||||
part_size=10 * 1024 * 1024, num_parallel_uploads=1)
|
||||
|
||||
|
||||
def get(key):
|
||||
with available():
|
||||
return client.get_object(BUCKET, key)
|
||||
|
||||
|
||||
def remove(key):
|
||||
with available():
|
||||
client.remove_object(BUCKET, key)
|
||||
@@ -0,0 +1,39 @@
|
||||
"""Курсор — scope, время и UUID; сами страницы выбирает PostgreSQL."""
|
||||
import base64
|
||||
import binascii
|
||||
from datetime import datetime
|
||||
import json
|
||||
import re
|
||||
from uuid import UUID
|
||||
|
||||
from src.errors import ApiError
|
||||
|
||||
|
||||
def parameters(limit, cursor, scope):
|
||||
try:
|
||||
if not re.fullmatch(r"[0-9]+", limit):
|
||||
raise ValueError
|
||||
size = int(limit)
|
||||
if not 1 <= size <= 100:
|
||||
raise ValueError
|
||||
anchor = None
|
||||
if cursor is not None:
|
||||
raw = base64.b64decode(cursor, altchars=b"-_", validate=True)
|
||||
data = json.loads(raw.decode("utf-8"))
|
||||
if not isinstance(data, dict) or set(data) != {"scope", "created_at", "id"}:
|
||||
raise ValueError
|
||||
if data["scope"] != scope or not isinstance(data["created_at"], str) or not data["created_at"].endswith("Z"):
|
||||
raise ValueError
|
||||
anchor = (datetime.fromisoformat(data["created_at"].replace("Z", "+00:00")), UUID(data["id"]))
|
||||
return size, anchor
|
||||
except (ValueError, TypeError, AttributeError, binascii.Error, RecursionError):
|
||||
raise ApiError(400, "invalid_request", "Неверный limit или cursor") from None
|
||||
|
||||
|
||||
def page(items, size, scope):
|
||||
cursor = None
|
||||
if len(items) > size:
|
||||
last = items[size - 1]
|
||||
value = {"scope": scope, "created_at": last["created_at"], "id": last["id"]}
|
||||
cursor = base64.urlsafe_b64encode(json.dumps(value).encode()).decode()
|
||||
return {"items": items[:size], "next_cursor": cursor}
|
||||
@@ -0,0 +1,22 @@
|
||||
"""Argon2id. Одновременно считаются не более двух хешей."""
|
||||
from threading import Semaphore
|
||||
|
||||
from argon2 import PasswordHasher
|
||||
from argon2.exceptions import VerificationError
|
||||
|
||||
|
||||
hasher = PasswordHasher(time_cost=2, memory_cost=19456, parallelism=1)
|
||||
slots = Semaphore(2)
|
||||
|
||||
|
||||
def hash_password(password):
|
||||
with slots:
|
||||
return hasher.hash(password)
|
||||
|
||||
|
||||
def matches(encoded, password):
|
||||
with slots:
|
||||
try:
|
||||
return hasher.verify(encoded, password)
|
||||
except VerificationError:
|
||||
return False
|
||||
@@ -0,0 +1,39 @@
|
||||
"""Outbox dispatcher: повтор XADD при неопределённом исходе безопасен."""
|
||||
import logging
|
||||
import time
|
||||
|
||||
from src import broker, db, faults, jobs
|
||||
|
||||
|
||||
def publish_one(client):
|
||||
with db.connection() as conn:
|
||||
job = conn.execute('''SELECT j.*,o.attempt AS delivery_attempt FROM attachment_outbox o
|
||||
JOIN attachment_jobs j ON j.id=o.job_id WHERE o.published_at IS NULL
|
||||
AND o.available_at<=clock_timestamp() ORDER BY o.available_at LIMIT 1 FOR UPDATE OF o SKIP LOCKED''').fetchone()
|
||||
if not job:
|
||||
return False
|
||||
job['attempt'] = job['delivery_attempt']
|
||||
faults.hit(job['id'], 'before_publish')
|
||||
broker.publish(client, job)
|
||||
faults.hit(job['id'], 'after_publish')
|
||||
conn.execute('UPDATE attachment_outbox SET published_at=clock_timestamp() WHERE job_id=%s AND attempt=%s',
|
||||
(job['id'], job['attempt']))
|
||||
return True
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
db.pool.open(wait=True, timeout=30)
|
||||
client = broker.connect()
|
||||
try:
|
||||
while True:
|
||||
try:
|
||||
jobs.recover()
|
||||
broker.bootstrap(client)
|
||||
if not publish_one(client):
|
||||
time.sleep(0.25)
|
||||
except Exception as error:
|
||||
logging.error('Outbox postponed: %s', type(error).__name__)
|
||||
time.sleep(1)
|
||||
finally:
|
||||
client.close()
|
||||
db.pool.close()
|
||||
+156
@@ -0,0 +1,156 @@
|
||||
"""Параметризованный SQL. Одна функция записи — одна транзакция."""
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from psycopg.errors import ForeignKeyViolation, UniqueViolation
|
||||
|
||||
from src import db, pagination
|
||||
from src.errors import ApiError
|
||||
from src.config import IDEMPOTENCY_TTL
|
||||
|
||||
|
||||
def utc(value):
|
||||
return value.isoformat(timespec="microseconds").replace("+00:00", "Z")
|
||||
|
||||
|
||||
def public_user(row):
|
||||
return {"id": str(row["id"]), "username": row["username"],
|
||||
"display_name": row["display_name"], "created_at": utc(row["created_at"])}
|
||||
|
||||
|
||||
def public_chat(row):
|
||||
return {"id": str(row["id"]), "title": row["title"],
|
||||
"member_ids": [str(value) for value in row["member_ids"]], "created_at": utc(row["created_at"])}
|
||||
|
||||
|
||||
def public_message(row):
|
||||
return {"id": str(row["id"]), "chat_id": str(row["chat_id"]), "sender_id": str(row["sender_id"]),
|
||||
"text": row["text"], "created_at": utc(row["created_at"]),
|
||||
"attachment_ids": [str(value) for value in row.get("attachment_ids", [])]}
|
||||
|
||||
|
||||
MESSAGE_ATTACHMENTS = """ARRAY(SELECT attachment_id FROM message_attachments
|
||||
WHERE message_id=m.id ORDER BY position) AS attachment_ids"""
|
||||
|
||||
|
||||
def create_user(username, display_name, password_hash):
|
||||
try:
|
||||
with db.connection() as conn:
|
||||
row = conn.execute("""INSERT INTO users(id, username, display_name, password_hash)
|
||||
VALUES (%s, %s, %s, %s) RETURNING *""", (uuid4(), username, display_name, password_hash)).fetchone()
|
||||
result = public_user(row)
|
||||
return result
|
||||
except UniqueViolation:
|
||||
raise ApiError(409, "conflict", "Имя пользователя уже занято") from None
|
||||
|
||||
|
||||
def find_login(username):
|
||||
with db.connection() as conn:
|
||||
return conn.execute("SELECT * FROM users WHERE username=%s", (username,)).fetchone()
|
||||
|
||||
|
||||
def find_user(user_id):
|
||||
with db.connection() as conn:
|
||||
row = conn.execute("SELECT * FROM users WHERE id=%s", (UUID(user_id),)).fetchone()
|
||||
return public_user(row) if row else None
|
||||
|
||||
|
||||
def create_chat(user_id, title, member_ids):
|
||||
if user_id not in member_ids:
|
||||
raise ApiError(400, "invalid_request", "Создатель должен входить в member_ids")
|
||||
try:
|
||||
with db.connection() as conn:
|
||||
row = conn.execute("INSERT INTO chats(id, title) VALUES (%s, %s) RETURNING *", (uuid4(), title)).fetchone()
|
||||
with conn.cursor() as cur:
|
||||
cur.executemany("INSERT INTO chat_members(chat_id, user_id) VALUES (%s, %s)",
|
||||
[(row["id"], UUID(member)) for member in member_ids])
|
||||
row["member_ids"] = member_ids
|
||||
result = public_chat(row)
|
||||
return result
|
||||
except ForeignKeyViolation:
|
||||
raise ApiError(400, "invalid_request", "Все участники должны существовать") from None
|
||||
|
||||
|
||||
def require_member(conn, user_id, chat_id):
|
||||
row = conn.execute("SELECT 1 FROM chat_members WHERE chat_id=%s AND user_id=%s", (UUID(chat_id), UUID(user_id))).fetchone()
|
||||
if row is None:
|
||||
raise ApiError(404, "not_found", "Чат не найден")
|
||||
|
||||
|
||||
def get_chat(user_id, chat_id):
|
||||
with db.connection() as conn:
|
||||
require_member(conn, user_id, chat_id)
|
||||
row = conn.execute("""SELECT c.*, ARRAY(SELECT user_id FROM chat_members WHERE chat_id=c.id ORDER BY user_id) AS member_ids
|
||||
FROM chats c WHERE c.id=%s""", (UUID(chat_id),)).fetchone()
|
||||
return public_chat(row)
|
||||
|
||||
|
||||
def list_chats(user_id, limit, cursor):
|
||||
scope = "chats:" + user_id
|
||||
size, anchor = pagination.parameters(limit, cursor, scope)
|
||||
with db.connection() as conn:
|
||||
if anchor:
|
||||
found = conn.execute("""SELECT 1 FROM chats c JOIN chat_members m ON m.chat_id=c.id
|
||||
WHERE m.user_id=%s AND (c.created_at, c.id)=(%s, %s)""", (UUID(user_id), *anchor)).fetchone()
|
||||
if not found:
|
||||
raise ApiError(400, "invalid_request", "Неверный cursor")
|
||||
condition = "AND (c.created_at, c.id) > (%s, %s)" if anchor else ""
|
||||
params = (UUID(user_id), *(anchor or ()), size + 1)
|
||||
rows = conn.execute("""WITH selected AS (
|
||||
SELECT c.* FROM chats c JOIN chat_members own ON own.chat_id=c.id
|
||||
WHERE own.user_id=%s """ + condition + """ ORDER BY c.created_at, c.id LIMIT %s
|
||||
) SELECT c.*, ARRAY(SELECT user_id FROM chat_members WHERE chat_id=c.id ORDER BY user_id) AS member_ids
|
||||
FROM selected c ORDER BY c.created_at, c.id""", params).fetchall()
|
||||
return pagination.page([public_chat(row) for row in rows], size, scope)
|
||||
|
||||
|
||||
def create_message(user_id, chat_id, text, key, attachment_ids=()):
|
||||
attachment_ids = [UUID(value) for value in attachment_ids]
|
||||
with db.connection() as conn:
|
||||
require_member(conn, user_id, chat_id)
|
||||
# Одна блокировка на область ключа во всех API-процессах.
|
||||
# Побочный эффект и результат повтора фиксируются одной транзакцией.
|
||||
scope = user_id + ":POST:/api/v1/chats/" + chat_id + "/messages:" + key
|
||||
conn.execute("SELECT pg_advisory_xact_lock(hashtextextended(%s, 0))", (scope,))
|
||||
previous = conn.execute("SELECT m.*, i.request_text, i.request_attachment_ids, " + MESSAGE_ATTACHMENTS + """ FROM message_idempotency i
|
||||
JOIN messages m ON m.id=i.message_id
|
||||
WHERE i.user_id=%s AND i.chat_id=%s AND i.key=%s AND i.expires_at>clock_timestamp()""",
|
||||
(UUID(user_id), UUID(chat_id), key)).fetchone()
|
||||
if previous:
|
||||
if previous["request_text"] != text or previous["request_attachment_ids"] != attachment_ids:
|
||||
raise ApiError(409, "conflict", "Этот ключ уже использован с другим сообщением")
|
||||
return public_message(previous)
|
||||
if attachment_ids:
|
||||
available = conn.execute("SELECT id,status FROM attachments WHERE chat_id=%s AND id=ANY(%s)",
|
||||
(UUID(chat_id), attachment_ids)).fetchall()
|
||||
if len(available) != len(attachment_ids):
|
||||
raise ApiError(404, "not_found", "Вложение не найдено в этом чате")
|
||||
if any(item["status"] == "failed" for item in available):
|
||||
raise ApiError(409, "conflict", "Вложение не обработано")
|
||||
conn.execute("DELETE FROM message_idempotency WHERE user_id=%s AND chat_id=%s AND key=%s",
|
||||
(UUID(user_id), UUID(chat_id), key))
|
||||
row = conn.execute("""INSERT INTO messages(id, chat_id, sender_id, text) VALUES (%s, %s, %s, %s)
|
||||
RETURNING *""", (uuid4(), UUID(chat_id), UUID(user_id), text)).fetchone()
|
||||
for position, attachment_id in enumerate(attachment_ids):
|
||||
conn.execute("INSERT INTO message_attachments(message_id, attachment_id, position) VALUES (%s, %s, %s)",
|
||||
(row["id"], attachment_id, position))
|
||||
conn.execute("""INSERT INTO message_idempotency(user_id, chat_id, key, request_text, request_attachment_ids, message_id, expires_at)
|
||||
VALUES (%s, %s, %s, %s, %s, %s, clock_timestamp() + %s * interval '1 second')""",
|
||||
(UUID(user_id), UUID(chat_id), key, text, attachment_ids, row["id"], IDEMPOTENCY_TTL))
|
||||
row["attachment_ids"] = attachment_ids
|
||||
result = public_message(row)
|
||||
return result
|
||||
|
||||
|
||||
def list_messages(user_id, chat_id, limit, cursor):
|
||||
scope = "messages:" + chat_id
|
||||
size, anchor = pagination.parameters(limit, cursor, scope)
|
||||
with db.connection() as conn:
|
||||
require_member(conn, user_id, chat_id)
|
||||
if anchor:
|
||||
found = conn.execute("SELECT 1 FROM messages WHERE chat_id=%s AND (created_at, id)=(%s, %s)", (UUID(chat_id), *anchor)).fetchone()
|
||||
if not found:
|
||||
raise ApiError(400, "invalid_request", "Неверный cursor")
|
||||
condition = "AND (m.created_at, m.id) > (%s, %s)" if anchor else ""
|
||||
rows = conn.execute("SELECT m.*, " + MESSAGE_ATTACHMENTS + " FROM messages m WHERE m.chat_id=%s " + condition +
|
||||
" ORDER BY m.created_at, m.id LIMIT %s", (UUID(chat_id), *(anchor or ()), size + 1)).fetchall()
|
||||
return pagination.page([public_message(row) for row in rows], size, scope)
|
||||
@@ -0,0 +1,69 @@
|
||||
"""Продление аренды и независимый wall timeout всего дерева обработки."""
|
||||
import os
|
||||
import ctypes
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
|
||||
class LeaseLost(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def watch_parent(expected_pid):
|
||||
"""В Linux смерть consumer также останавливает оставшееся дерево задачи."""
|
||||
if not sys.platform.startswith('linux'):
|
||||
return
|
||||
if os.getpgrp() != os.getpid():
|
||||
raise RuntimeError('Job runner must have its own process group')
|
||||
|
||||
def stop(signum, frame):
|
||||
os.killpg(os.getpgrp(), signal.SIGKILL)
|
||||
|
||||
signal.signal(signal.SIGTERM, stop)
|
||||
libc = ctypes.CDLL(None, use_errno=True)
|
||||
if libc.prctl(1, signal.SIGTERM, 0, 0, 0) != 0: # PR_SET_PDEATHSIG
|
||||
raise OSError(ctypes.get_errno(), 'Cannot watch consumer process')
|
||||
# Родитель мог погибнуть между Popen и установкой PR_SET_PDEATHSIG.
|
||||
if os.getppid() != expected_pid:
|
||||
stop(None, None)
|
||||
|
||||
|
||||
def stop_tree(process):
|
||||
if os.name == 'posix':
|
||||
try:
|
||||
os.killpg(process.pid, signal.SIGKILL)
|
||||
except ProcessLookupError:
|
||||
pass
|
||||
elif process.poll() is None:
|
||||
# Production работает в Linux; это также позволяет проверять на Windows.
|
||||
subprocess.run(['taskkill', '/PID', str(process.pid), '/T', '/F'],
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
|
||||
creationflags=subprocess.CREATE_NO_WINDOW, timeout=10, check=False)
|
||||
if process.poll() is None:
|
||||
process.kill()
|
||||
process.wait()
|
||||
|
||||
|
||||
def run(command, renew, timeout, interval):
|
||||
"""Heartbeat в управляющем цикле, без бесконечного фонового потока."""
|
||||
deadline = time.monotonic() + timeout
|
||||
options = {'start_new_session': True} if os.name == 'posix' else {
|
||||
'creationflags': subprocess.CREATE_NO_WINDOW}
|
||||
process = subprocess.Popen(command, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, **options)
|
||||
try:
|
||||
while True:
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
raise subprocess.TimeoutExpired(command, timeout)
|
||||
try:
|
||||
return process.wait(timeout=min(interval, remaining))
|
||||
except subprocess.TimeoutExpired:
|
||||
if time.monotonic() >= deadline:
|
||||
raise subprocess.TimeoutExpired(command, timeout) from None
|
||||
if not renew():
|
||||
raise LeaseLost()
|
||||
finally:
|
||||
# В том числе убиваем дочерний image_pipeline, если завис его родитель.
|
||||
stop_tree(process)
|
||||
+203
@@ -0,0 +1,203 @@
|
||||
"""Worker supervisors: concurrency процессов, prefetch=1, дочерний resize с timeout."""
|
||||
from datetime import datetime
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import multiprocessing
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
from tempfile import TemporaryDirectory
|
||||
import time
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from psycopg.types.json import Jsonb
|
||||
|
||||
from src import broker, db, faults, jobs, object_store, task_supervisor
|
||||
from src.errors import ApiError
|
||||
|
||||
|
||||
class ProcessingError(Exception):
|
||||
def __init__(self, code, permanent=False):
|
||||
self.code, self.permanent = code, permanent
|
||||
|
||||
|
||||
def verify_ready(job):
|
||||
# Старые ready из lab4 не имеют job; у новой job всегда все три варианта.
|
||||
if {v['name'] for v in job['variants']} != {'thumbnail', 'crop', 'optimized'}:
|
||||
raise ProcessingError('incomplete_result')
|
||||
for key, size in [(job['object_key'], job['size_bytes'])] + [
|
||||
(job['object_key'].rsplit('/', 1)[0] + '/' + v['name'], v['size_bytes']) for v in job['variants']
|
||||
]:
|
||||
with object_store.available():
|
||||
if object_store.client.stat_object(object_store.BUCKET, key).size != size:
|
||||
raise ProcessingError('incomplete_result')
|
||||
|
||||
|
||||
def process(job, workspace=None):
|
||||
faults.hit(job['id'], 'before_download')
|
||||
with TemporaryDirectory(prefix='image-job-', dir=workspace) as directory:
|
||||
output = Path(directory)
|
||||
source = output / 'original'
|
||||
response = object_store.get(job['staging_key'])
|
||||
size, digest = 0, hashlib.sha256()
|
||||
try:
|
||||
with source.open('wb') as file:
|
||||
while chunk := response.read(65536):
|
||||
size += len(chunk)
|
||||
if size > job['size_bytes'] or size > 10485760:
|
||||
raise ProcessingError('invalid_image', True)
|
||||
digest.update(chunk)
|
||||
file.write(chunk)
|
||||
finally:
|
||||
response.close()
|
||||
response.release_conn()
|
||||
if size != job['size_bytes'] or digest.hexdigest() != job['sha256']:
|
||||
raise ProcessingError('invalid_image', True)
|
||||
try:
|
||||
result = subprocess.run([sys.executable, '-m', 'src.image_pipeline', str(source), str(output), str(job['id'])],
|
||||
timeout=jobs.TIMEOUT, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
except subprocess.TimeoutExpired:
|
||||
raise ProcessingError('processing_timeout') from None
|
||||
if result.returncode:
|
||||
raise ProcessingError('invalid_image' if result.returncode == 65 else 'resource_limit', result.returncode == 65)
|
||||
variants = json.loads((output / 'variants.json').read_text())
|
||||
# Короткая транзакция fence перед side effects и публикацией результата.
|
||||
with db.connection() as conn:
|
||||
conn.execute("SET LOCAL idle_in_transaction_session_timeout='30s'")
|
||||
current = jobs.locked(conn, job['id'])
|
||||
active = conn.execute('SELECT lease_until>clock_timestamp() AS active FROM attachment_jobs WHERE id=%s', (job['id'],)).fetchone()['active']
|
||||
if current['status'] != 'processing' or current['lease_token'] != job['lease_token'] or not active:
|
||||
return False
|
||||
with source.open('rb') as file:
|
||||
object_store.put(job['object_key'], file, job['size_bytes'], job['content_type'])
|
||||
prefix = job['object_key'].rsplit('/', 1)[0] + '/'
|
||||
for variant in variants:
|
||||
with (output / variant['name']).open('rb') as file:
|
||||
object_store.put(prefix + variant['name'], file, variant['size_bytes'], variant['content_type'])
|
||||
faults.hit(job['id'], 'after_objects')
|
||||
conn.execute("UPDATE attachments SET status='ready',variants=%s,error_code=NULL,updated_at=clock_timestamp() WHERE id=%s", (Jsonb(variants), job['attachment_id']))
|
||||
conn.execute('UPDATE attachment_jobs SET lease_token=NULL,lease_until=NULL,last_error=NULL WHERE id=%s', (job['id'],))
|
||||
jobs.event(conn, job, 'ready')
|
||||
faults.hit(job['id'], 'before_ack')
|
||||
return True
|
||||
|
||||
|
||||
def handle(client, message_id, fields):
|
||||
try:
|
||||
payload = json.loads(fields['job'])
|
||||
required = {'schema_version', 'type', 'job_id', 'attachment_id', 'pipeline_version', 'attempt', 'created_at'}
|
||||
if not isinstance(payload, dict) or not required <= payload.keys() or payload.keys() - required - {'traceparent', 'tracestate'}:
|
||||
raise ValueError('invalid job')
|
||||
if type(payload['schema_version']) is not int or payload['schema_version'] != 1 or payload['type'] != 'attachment.process.v1':
|
||||
raise ValueError('invalid job')
|
||||
payload['job_id'] = str(UUID(payload['job_id']))
|
||||
payload['attachment_id'] = str(UUID(payload['attachment_id']))
|
||||
if type(payload['attempt']) is not int or payload['attempt'] < 1 or type(payload['pipeline_version']) is not int or payload['pipeline_version'] != 1:
|
||||
raise ValueError('invalid job')
|
||||
if not isinstance(payload['created_at'], str) or datetime.fromisoformat(payload['created_at'].replace('Z', '+00:00')).tzinfo is None:
|
||||
raise ValueError('invalid job')
|
||||
if 'traceparent' in payload and (not isinstance(payload['traceparent'], str) or re.fullmatch(r'00-[0-9a-f]{32}-[0-9a-f]{16}-[0-9a-f]{2}', payload['traceparent']) is None):
|
||||
raise ValueError('invalid trace context')
|
||||
if 'tracestate' in payload and (not isinstance(payload['tracestate'], str) or len(payload['tracestate']) > 512):
|
||||
raise ValueError('invalid trace context')
|
||||
except (ValueError, TypeError, KeyError):
|
||||
logging.error('Invalid broker message rejected')
|
||||
broker.acknowledge(client, message_id)
|
||||
return
|
||||
state, job = jobs.claim(payload)
|
||||
if state == 'busy':
|
||||
return
|
||||
if state == 'ready':
|
||||
verify_ready(job)
|
||||
if state == 'claimed':
|
||||
try:
|
||||
faults.hit(job['id'], 'before_process')
|
||||
# Временные файлы удаляет consumer даже после SIGKILL runner.
|
||||
with TemporaryDirectory(prefix='image-attempt-') as workspace:
|
||||
result = task_supervisor.run(
|
||||
[sys.executable, '-m', 'src.job_runner', str(job['id']), str(job['lease_token']),
|
||||
str(os.getpid()), workspace],
|
||||
renew=lambda: jobs.renew(job), timeout=jobs.TASK_TIMEOUT, interval=jobs.HEARTBEAT)
|
||||
if result != 0:
|
||||
# Crash/неизвестный COMMIT: не ACK, восстановление через lease.
|
||||
return
|
||||
except task_supervisor.LeaseLost:
|
||||
return
|
||||
except subprocess.TimeoutExpired:
|
||||
jobs.fail(job, 'processing_timeout')
|
||||
except ProcessingError as error:
|
||||
jobs.fail(job, error.code, error.permanent)
|
||||
except Exception:
|
||||
jobs.fail(job, 'storage_unavailable')
|
||||
# ready/failed/obsolete или записанный retry; при потере COMMIT исключение не даёт ACK.
|
||||
broker.acknowledge(client, message_id)
|
||||
|
||||
|
||||
def consume():
|
||||
db.pool.open(wait=True, timeout=30)
|
||||
client = broker.connect()
|
||||
consumer = 'worker-' + uuid4().hex
|
||||
cursor = '0-0'
|
||||
try:
|
||||
while True:
|
||||
try:
|
||||
broker.bootstrap(client)
|
||||
claimed = client.xautoclaim(broker.STREAM, broker.GROUP, consumer,
|
||||
min_idle_time=jobs.LEASE * 1000, start_id=cursor, count=1)
|
||||
cursor = claimed[0]
|
||||
entries = claimed[1]
|
||||
if not entries:
|
||||
batches = client.xreadgroup(broker.GROUP, consumer, {broker.STREAM: '>'}, count=1, block=1000)
|
||||
entries = batches[0][1] if batches else []
|
||||
for message_id, fields in entries:
|
||||
handle(client, message_id, fields)
|
||||
except Exception as error:
|
||||
logging.error('Worker postponed: %s', type(error).__name__)
|
||||
time.sleep(1)
|
||||
finally:
|
||||
client.close()
|
||||
db.pool.close()
|
||||
|
||||
|
||||
def main():
|
||||
concurrency = int(os.environ.get('WORKER_CONCURRENCY', '2'))
|
||||
if not 1 <= concurrency <= 2:
|
||||
raise ValueError('WORKER_CONCURRENCY: 1 или 2 для заданного memory limit')
|
||||
context = multiprocessing.get_context('spawn')
|
||||
processes = []
|
||||
stopping = False
|
||||
def stop(signum, frame):
|
||||
nonlocal stopping
|
||||
stopping = True
|
||||
signal.signal(signal.SIGTERM, stop)
|
||||
signal.signal(signal.SIGINT, stop)
|
||||
try:
|
||||
while not stopping:
|
||||
alive = []
|
||||
for child in processes:
|
||||
if child.is_alive():
|
||||
alive.append(child)
|
||||
else:
|
||||
child.join()
|
||||
processes = alive
|
||||
while len(processes) < concurrency:
|
||||
child = context.Process(target=consume)
|
||||
child.start()
|
||||
processes.append(child)
|
||||
time.sleep(0.25)
|
||||
finally:
|
||||
for child in processes:
|
||||
child.terminate()
|
||||
for child in processes:
|
||||
child.join(timeout=5)
|
||||
if child.is_alive():
|
||||
child.kill()
|
||||
child.join()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,275 @@
|
||||
"""Реальные отказы/очистка/память уровня 4; только временный тестовый стенд."""
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import socket
|
||||
import subprocess
|
||||
import threading
|
||||
import time
|
||||
from urllib.parse import urlsplit
|
||||
from uuid import uuid4
|
||||
|
||||
from PIL import Image
|
||||
|
||||
from helpers import Client, ROOT
|
||||
from client import Failure
|
||||
|
||||
|
||||
def run_checks(base, env, compose, apis, script, sql):
|
||||
result = {}
|
||||
client = Client(base, timeout=30)
|
||||
alice, bob, _, chat = client.bootstrap()
|
||||
png = (ROOT / "tests/fixtures/sample.png").read_bytes()
|
||||
upload_path = "/api/v1/chats/" + chat["id"] + "/attachments?filename=sample.png"
|
||||
|
||||
def upload():
|
||||
return client.call("POST", upload_path, 201, actor=alice, raw=png,
|
||||
headers={"Content-Type": "image/png"}, schema="Attachment")
|
||||
|
||||
def inside(code, data=None):
|
||||
completed = subprocess.run(["docker", "exec", "-i", apis()[0]["Id"], "python", "-c", code],
|
||||
input=json.dumps(data), capture_output=True, text=True, timeout=45, check=True)
|
||||
return json.loads(completed.stdout)
|
||||
|
||||
def cleanup():
|
||||
return inside("from src import db; from src.cleanup import cleanup_once; import json; "
|
||||
"db.pool.open(wait=True); print(json.dumps(cleanup_once())); db.pool.close()")
|
||||
|
||||
def stat(key):
|
||||
return inside("import json,sys; from src import object_store; from minio.error import S3Error\n"
|
||||
"key=json.load(sys.stdin)\n"
|
||||
"try:\n r=object_store.client.stat_object(object_store.BUCKET,key); print(json.dumps({'size':r.size}))\n"
|
||||
"except S3Error as e:\n print(json.dumps({'error':e.code}))", key)
|
||||
|
||||
def age(key):
|
||||
sql("UPDATE attachment_uploads SET created_at=clock_timestamp()-interval '2 hours' WHERE object_key='" + key + "'")
|
||||
|
||||
compose("stop", "cleanup")
|
||||
try:
|
||||
for _ in range(2):
|
||||
compose("run", "--rm", "s3-bootstrap")
|
||||
ready = upload()["json"]
|
||||
key = sql("SELECT object_key FROM attachments WHERE id='" + ready["id"] + "'")
|
||||
backend_id = compose("ps", "-q", "s3-storage")
|
||||
backend = json.loads(subprocess.check_output(["docker", "inspect", backend_id], text=True))[0]
|
||||
assert not set(backend["NetworkSettings"]["Networks"]) & set(apis()[0]["NetworkSettings"]["Networks"])
|
||||
backend_ip = next(iter(backend["NetworkSettings"]["Networks"].values()))["IPAddress"]
|
||||
# Вызов S3 напрямую из контейнера клиента, минуя HTTP API приложения.
|
||||
result["private_bucket"] = inside("""
|
||||
import io,json,socket,sys
|
||||
from urllib.error import HTTPError
|
||||
from urllib.request import urlopen
|
||||
from minio.error import S3Error
|
||||
from src import object_store
|
||||
data=json.load(sys.stdin); key=data['key']
|
||||
try:
|
||||
urlopen('http://s3:9000/'+object_store.BUCKET+'/'+key,timeout=3)
|
||||
raise AssertionError('public original')
|
||||
except HTTPError as e:
|
||||
assert e.code == 403
|
||||
for operation in (lambda: object_store.client.make_bucket('forbidden-bucket'),
|
||||
lambda: object_store.client.put_object(object_store.BUCKET,'outside-prefix',io.BytesIO(b'x'),1)):
|
||||
try:
|
||||
operation()
|
||||
raise AssertionError('excess application permissions')
|
||||
except S3Error as e:
|
||||
assert e.code == 'AccessDenied', e.code
|
||||
for port in (9333,8888,8080):
|
||||
with socket.socket() as connection:
|
||||
connection.settimeout(1)
|
||||
assert connection.connect_ex(('s3',port)) != 0, 'admin/backend port open'
|
||||
with socket.socket() as connection:
|
||||
connection.settimeout(.5)
|
||||
assert connection.connect_ex((data['backend_ip'],port)) != 0, 'storage network reachable from API'
|
||||
print(json.dumps({'anonymous_get':403,'create_bucket':'AccessDenied','write_outside_prefix':'AccessDenied',
|
||||
'admin_ports_unreachable':[9333,8888,8080],'storage_network_isolated':True,'bootstrap_runs':2}))
|
||||
""", {"key":key, "backend_ip":backend_ip})
|
||||
print("PASS: private S3, limited credentials, isolated administration, repeated bootstrap", flush=True)
|
||||
|
||||
# Останавливаем именно реплику, которая ответила на upload, а затем читаем другой.
|
||||
uploaded = upload()
|
||||
target = uploaded["headers"]["X-Instance-Id"]
|
||||
stopped_at = time.monotonic()
|
||||
subprocess.run([str(ROOT / "scripts/stop-one.sh")], cwd=ROOT,
|
||||
env=dict(env, TARGET_INSTANCE_ID=target), check=True, timeout=15)
|
||||
script("wait-ready.sh")
|
||||
while True:
|
||||
try:
|
||||
response = client.call("GET", "/api/v1/attachments/" + uploaded["json"]["id"] + "/content", actor=bob)
|
||||
break
|
||||
except Failure:
|
||||
assert time.monotonic() - stopped_at < 10, "download failover exceeded 10 seconds"
|
||||
time.sleep(.1)
|
||||
assert response["bytes"] == png and response["headers"]["X-Instance-Id"] != target
|
||||
stopped = next(info for info in apis() if info["Config"]["Hostname"] == target)
|
||||
assert not stopped["State"]["Running"]
|
||||
result["upload_replica_failover"] = {"uploaded_by": target, "stopped_container": stopped["Id"],
|
||||
"downloaded_by": response["headers"]["X-Instance-Id"], "sha256": hashlib.sha256(png).hexdigest(),
|
||||
"recovery_seconds":round(time.monotonic()-stopped_at,3)}
|
||||
compose("start", "api")
|
||||
script("wait-ready.sh")
|
||||
print("PASS: original downloaded through another replica after stopping uploader", flush=True)
|
||||
|
||||
# Незавершённое HTTP-тело не достигает S3 и не создаёт метаданных/журнала.
|
||||
before = sql("SELECT (SELECT count(*) FROM attachments)||':'||(SELECT count(*) FROM attachment_uploads)")
|
||||
address = urlsplit(base)
|
||||
with socket.create_connection((address.hostname, address.port), timeout=3) as connection:
|
||||
headers = ("POST " + upload_path + " HTTP/1.1\r\nHost: localhost\r\nAuthorization: Bearer " + alice["token"] +
|
||||
"\r\nContent-Type: image/png\r\nContent-Length: 1000000\r\n\r\n").encode()
|
||||
connection.sendall(headers + png[:30])
|
||||
time.sleep(0.5)
|
||||
assert sql("SELECT (SELECT count(*) FROM attachments)||':'||(SELECT count(*) FROM attachment_uploads)") == before
|
||||
result["interrupted_body"] = {"metadata_or_journal_created": False}
|
||||
|
||||
# Детерминированный реальный отказ PostgreSQL ПОСЛЕ успешного S3 PUT.
|
||||
sql("""CREATE FUNCTION lab4_reject_attachment() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||
BEGIN RAISE EXCEPTION 'injected storage failure' USING ERRCODE='08006'; END $$;
|
||||
CREATE TRIGGER lab4_reject BEFORE INSERT ON attachments FOR EACH ROW EXECUTE FUNCTION lab4_reject_attachment();""")
|
||||
try:
|
||||
failure = client.call("POST", upload_path, 503, actor=alice, raw=png, headers={"Content-Type": "image/png"})
|
||||
assert failure["json"]["error"]["code"] == "unavailable"
|
||||
finally:
|
||||
sql("DROP TRIGGER lab4_reject ON attachments; DROP FUNCTION lab4_reject_attachment()")
|
||||
pending_key = sql("SELECT object_key FROM attachment_uploads ORDER BY created_at DESC LIMIT 1")
|
||||
assert stat(pending_key)["size"] == len(png)
|
||||
assert sql("SELECT count(*) FROM attachments WHERE object_key='" + pending_key + "'") == "0"
|
||||
assert cleanup()["removed"] == 0
|
||||
age(pending_key)
|
||||
lock_code = ("from src import db; from src.attachments import lock_upload; import sys\n"
|
||||
"db.pool.open(wait=True)\nwith db.connection() as c:\n"
|
||||
" lock_upload(c,'" + pending_key + "'); print('locked',flush=True); sys.stdin.readline()\n"
|
||||
"db.pool.close()")
|
||||
holder = subprocess.Popen(["docker", "exec", "-i", apis()[0]["Id"], "python", "-u", "-c", lock_code],
|
||||
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
|
||||
try:
|
||||
assert holder.stdout.readline().strip() == "locked"
|
||||
assert cleanup()["skipped_active"] == 1
|
||||
assert stat(pending_key)["size"] == len(png)
|
||||
finally:
|
||||
holder.communicate("\n", timeout=10)
|
||||
assert cleanup()["removed"] == 1
|
||||
assert stat(pending_key).get("error") == "NoSuchKey"
|
||||
assert cleanup()["removed"] == 0
|
||||
# Даже устаревшая ошибочная запись журнала не позволяет удалить ready-файл.
|
||||
sql("INSERT INTO attachment_uploads(id,object_key,created_at) VALUES ('" + str(uuid4()) + "','" + key +
|
||||
"',clock_timestamp()-interval '2 hours')")
|
||||
assert cleanup()["kept_ready"] == 1
|
||||
assert client.call("GET", "/api/v1/attachments/" + ready["id"] + "/content", actor=bob)["bytes"] == png
|
||||
result["compensation"] = {"failure_after_put":503, "orphan_recorded":True, "fresh_kept":True,
|
||||
"active_locked_kept":True, "expired_removed":True, "ready_kept":True, "repeat_safe":True}
|
||||
print("PASS: post-PUT DB failure recorded; cleanup protects fresh, active and ready uploads", flush=True)
|
||||
|
||||
try:
|
||||
compose("stop", "--timeout", "8", "s3")
|
||||
result["s3_outage"] = inside("""
|
||||
import json,sys
|
||||
from urllib.request import Request,urlopen
|
||||
from urllib.error import HTTPError
|
||||
data=json.load(sys.stdin)
|
||||
statuses={}
|
||||
for name,method,path,body in [('live','GET','/health/live',None),('ready','GET','/health/ready',None),
|
||||
('download','GET','/api/v1/attachments/'+data['id']+'/content',None),
|
||||
('upload','POST',data['upload'],bytes.fromhex(data['png']))]:
|
||||
req=Request('http://127.0.0.1:8080'+path,data=body,method=method,
|
||||
headers={'Authorization':'Bearer '+data['token'],'Content-Type':'image/png'})
|
||||
try: response=urlopen(req,timeout=8)
|
||||
except HTTPError as error: response=error
|
||||
with response:
|
||||
statuses[name]=response.status
|
||||
payload=json.load(response)
|
||||
if name!='live':
|
||||
assert response.status==503 and payload['error']['code']=='unavailable'
|
||||
assert payload['error']['request_id']==response.headers['X-Request-Id']
|
||||
assert statuses['live']==200
|
||||
print(json.dumps(statuses))
|
||||
""", {"id":ready["id"], "token":alice["token"], "upload":upload_path, "png":png.hex()})
|
||||
finally:
|
||||
compose("start", "s3")
|
||||
script("wait-ready.sh")
|
||||
assert client.call("GET", "/api/v1/attachments/" + ready["id"] + "/content", actor=bob)["bytes"] == png
|
||||
sql("UPDATE attachment_uploads SET created_at=clock_timestamp()-interval '2 hours'")
|
||||
assert cleanup()["removed"] >= 1
|
||||
result["s3_outage"]["original_preserved_after_recovery"] = True
|
||||
print("PASS: S3 outage gives API 503; recovery preserves original and session", flush=True)
|
||||
|
||||
# Доказываем работу постоянного worker, а не только ручного вызова cleanup_once.
|
||||
inside("import io,json; from src import object_store; "
|
||||
"object_store.put('attachments/cleanup-worker-test',io.BytesIO(b'x'),1,'application/octet-stream'); print('{}')")
|
||||
sql("INSERT INTO attachment_uploads(id,object_key,created_at) VALUES ('" + str(uuid4()) +
|
||||
"','attachments/cleanup-worker-test',clock_timestamp()-interval '2 hours')")
|
||||
compose("start", "cleanup")
|
||||
deadline = time.monotonic() + 40
|
||||
while sql("SELECT count(*) FROM attachment_uploads WHERE object_key='attachments/cleanup-worker-test'") != "0":
|
||||
assert time.monotonic() < deadline, "automatic cleanup not running"
|
||||
time.sleep(0.2)
|
||||
assert stat('attachments/cleanup-worker-test').get("error") == "NoSuchKey"
|
||||
result["compensation"]["automatic_worker"] = True
|
||||
|
||||
# 10 одновременно отправляемых реальных PNG ~5.5 MiB; монитор cgroup каждые 20 ms.
|
||||
image = Image.frombytes("RGB", (1600, 1200), os.urandom(1600 * 1200 * 3))
|
||||
buffer = io.BytesIO()
|
||||
image.save(buffer, format="PNG")
|
||||
image.close()
|
||||
data = buffer.getvalue()
|
||||
expected_sha = hashlib.sha256(data).hexdigest()
|
||||
monitor_code = """
|
||||
from pathlib import Path
|
||||
import json,sys,threading,time
|
||||
root=Path('/sys/fs/cgroup'); stop=threading.Event(); samples=[]
|
||||
threading.Thread(target=lambda:(sys.stdin.readline(),stop.set()),daemon=True).start()
|
||||
start=time.monotonic(); before=int((root/'memory.current').read_text())
|
||||
print('ready',flush=True)
|
||||
while not stop.is_set():
|
||||
samples.append([round(time.monotonic()-start,3),int((root/'memory.current').read_text())])
|
||||
stop.wait(.02)
|
||||
print(json.dumps({'baseline_bytes':before,'sample_peak_bytes':max(s[1] for s in samples),
|
||||
'lifetime_peak_bytes':int((root/'memory.peak').read_text()),'samples':samples}),flush=True)
|
||||
"""
|
||||
monitors = []
|
||||
responses = []
|
||||
try:
|
||||
for info in apis():
|
||||
process = subprocess.Popen(["docker", "exec", "-i", info["Id"], "python", "-u", "-c", monitor_code],
|
||||
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
|
||||
assert process.stdout.readline().strip() == "ready"
|
||||
monitors.append((info, process))
|
||||
barrier = threading.Barrier(10)
|
||||
def send(_):
|
||||
concurrent_client = Client(base, timeout=60)
|
||||
barrier.wait(timeout=15)
|
||||
return concurrent_client.call("POST", upload_path, 201, actor=alice, raw=data,
|
||||
headers={"Content-Type":"image/png"}, schema="Attachment")
|
||||
started = time.monotonic()
|
||||
with ThreadPoolExecutor(max_workers=10) as pool:
|
||||
responses = list(pool.map(send, range(10)))
|
||||
elapsed = time.monotonic() - started
|
||||
finally:
|
||||
memory = []
|
||||
for info, process in monitors:
|
||||
output, error = process.communicate("\n", timeout=10)
|
||||
assert process.returncode == 0, error
|
||||
measurement = json.loads(output)
|
||||
measurement.update({"instance": info["Config"]["Hostname"], "limit_bytes": info["HostConfig"]["Memory"]})
|
||||
assert measurement["sample_peak_bytes"] < measurement["limit_bytes"]
|
||||
memory.append(measurement)
|
||||
assert len(responses) == 10
|
||||
assert len({r["headers"]["X-Instance-Id"] for r in responses}) >= 2
|
||||
for response in responses:
|
||||
assert response["json"]["sha256"] == expected_sha and response["json"]["size_bytes"] == len(data)
|
||||
original = client.call("GET", "/api/v1/attachments/" + response["json"]["id"] + "/content", actor=bob)
|
||||
assert hashlib.sha256(original["bytes"]).hexdigest() == expected_sha
|
||||
assert all(not i["State"]["OOMKilled"] for i in apis())
|
||||
timings = sorted(r["elapsed"] * 1000 for r in responses)
|
||||
result["parallel_uploads"] = {"concurrency":10, "successful":10, "errors":0,
|
||||
"dimensions":[1600,1200], "size_bytes":len(data), "sha256":expected_sha,
|
||||
"duration_seconds":round(elapsed,3), "successful_rps":round(10/elapsed,3),
|
||||
"p50_ms":round(timings[4],3), "p95_ms":round(timings[-1],3),
|
||||
"memory_sample_interval_seconds":0.02, "memory_includes_monitor_process":True,
|
||||
"memory":memory, "all_original_checksums_verified":True, "oom_killed":False}
|
||||
print("PASS: 10 concurrent real PNG uploads; all checksums match; memory peaks:",
|
||||
[m["sample_peak_bytes"] for m in memory], flush=True)
|
||||
return result
|
||||
finally:
|
||||
compose("start", "s3", "api", "cleanup")
|
||||
@@ -0,0 +1,319 @@
|
||||
"""Проверки уровня 4 в отдельном Compose project с синтетическими данными."""
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from datetime import datetime, timezone
|
||||
import json
|
||||
import os
|
||||
import platform
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from urllib.error import HTTPError
|
||||
from urllib.request import urlopen
|
||||
from uuid import uuid4
|
||||
|
||||
from helpers import Client, ROOT
|
||||
from client import Failure
|
||||
from resilience import history
|
||||
from attachment_checks import run_checks
|
||||
from postgres_pause import check_postgres_pause
|
||||
|
||||
|
||||
def docker(*args):
|
||||
return subprocess.check_output(["docker", *args], text=True, timeout=180).strip()
|
||||
|
||||
|
||||
def main():
|
||||
with socket.socket() as sock:
|
||||
sock.bind(("127.0.0.1", 0))
|
||||
port = sock.getsockname()[1]
|
||||
project = "lab4-check-" + uuid4().hex[:10]
|
||||
base = f"http://127.0.0.1:{port}"
|
||||
env = dict(os.environ, COMPOSE_PROJECT_NAME=project, HOST_PORT=str(port),
|
||||
API_REPLICAS="2", SESSION_TTL_SECONDS="3600", BASE_URL=base)
|
||||
result = {"utc": datetime.now(timezone.utc).isoformat(), "project": project,
|
||||
"base_url": base, "host": platform.system() + " " + platform.machine(),
|
||||
"docker": docker("info", "--format", "{{.ServerVersion}} {{.Architecture}} CPUs={{.NCPU}} RAM={{.MemTotal}}"),
|
||||
"commit": docker("compose", "version", "--short"),
|
||||
"profile": {"workers": 4, "pause_seconds": 0.05, "operation": "POST message then GET history, same-key retry after uncertain POST"}}
|
||||
result["compose_version"] = result.pop("commit")
|
||||
result["source_commit"] = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=ROOT, text=True).strip()
|
||||
result["working_tree"] = True
|
||||
|
||||
def compose(*args):
|
||||
return subprocess.check_output(["docker", "compose", *args], cwd=ROOT, env=env, text=True, timeout=180).strip()
|
||||
|
||||
def script(name):
|
||||
subprocess.run([str(ROOT / "scripts" / name)], cwd=ROOT, env=env, check=True, timeout=240)
|
||||
|
||||
def inspect_all():
|
||||
ids = compose("ps", "-a", "-q").split()
|
||||
return json.loads(docker("inspect", *ids))
|
||||
|
||||
def apis():
|
||||
return [i for i in inspect_all() if i["Config"]["Labels"]["com.docker.compose.service"] == "api"]
|
||||
|
||||
def metadata(info):
|
||||
return {"id": info["Id"], "instance": info["Config"]["Hostname"],
|
||||
"service": info["Config"]["Labels"]["com.docker.compose.service"],
|
||||
"image": info["Image"], "started_at": info["State"]["StartedAt"],
|
||||
"running": info["State"]["Running"], "exit_code": info["State"]["ExitCode"],
|
||||
"memory_bytes": info["HostConfig"]["Memory"], "nano_cpus": info["HostConfig"]["NanoCpus"],
|
||||
"published_ports": info["HostConfig"]["PortBindings"],
|
||||
"mounts": [{"type": m["Type"], "destination": m["Destination"], "name": m.get("Name")} for m in info["Mounts"]]}
|
||||
|
||||
def sql(query):
|
||||
return compose("exec", "-T", "postgres", "psql", "-U", "messenger", "-d", "messenger", "-At", "-c", query)
|
||||
|
||||
def check_shutdown():
|
||||
info = apis()[0]
|
||||
# Внутреннее подключение через docker exec не публикует порт API на хост.
|
||||
pending_code = """import socket,time
|
||||
s=socket.create_connection(('127.0.0.1',8080),timeout=2)
|
||||
s.sendall(b'POST /api/v1/users HTTP/1.1\\r\\nHost: localhost\\r\\nContent-Type: application/json\\r\\nContent-Length: 1000\\r\\nExpect: 100-continue\\r\\n\\r\\n')
|
||||
assert b'100 Continue' in s.recv(1024)
|
||||
s.sendall(b'{"username":')
|
||||
print('pending',flush=True)
|
||||
time.sleep(30)
|
||||
"""
|
||||
pending = subprocess.Popen(["docker", "exec", info["Id"], "python", "-u", "-c", pending_code],
|
||||
stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
|
||||
try:
|
||||
assert pending.stdout.readline().strip() == "pending"
|
||||
started = time.monotonic()
|
||||
docker("kill", "--signal=TERM", info["Id"])
|
||||
first_refusal = None
|
||||
while True:
|
||||
state = json.loads(docker("inspect", info["Id"]))[0]["State"]
|
||||
if not state["Running"]:
|
||||
break
|
||||
assert time.monotonic() - started < 8
|
||||
probe = subprocess.run(["docker", "exec", info["Id"], "python", "-c",
|
||||
"import socket; s=socket.socket(); s.settimeout(.2); raise SystemExit(s.connect_ex(('127.0.0.1',8080)))"],
|
||||
capture_output=True, timeout=3)
|
||||
if probe.returncode and first_refusal is None:
|
||||
first_refusal = time.monotonic() - started
|
||||
time.sleep(0.1)
|
||||
elapsed = time.monotonic() - started
|
||||
assert first_refusal is not None and first_refusal < elapsed < 8
|
||||
assert state["ExitCode"] == 0 and not state["OOMKilled"]
|
||||
logs = subprocess.run(["docker", "logs", info["Id"]], capture_output=True, text=True, check=True)
|
||||
assert "Application shutdown complete" in logs.stdout + logs.stderr
|
||||
result["sigterm"] = {"instance": info["Config"]["Hostname"], "unfinished_request": True,
|
||||
"shutdown_seconds": round(elapsed, 3), "new_connections_refused_seconds": round(first_refusal, 3),
|
||||
"exit_code": state["ExitCode"], "oom_killed": state["OOMKilled"]}
|
||||
print("PASS: SIGTERM with unfinished request:", json.dumps(result["sigterm"]), flush=True)
|
||||
finally:
|
||||
pending.communicate(timeout=10)
|
||||
compose("start", "api")
|
||||
script("wait-ready.sh")
|
||||
|
||||
def load_action(label, action, target=None):
|
||||
client = Client(base, timeout=2)
|
||||
alice, bob, _, chat = client.bootstrap()
|
||||
original = client.message(alice, chat, "До воздействия")["json"]
|
||||
stop = threading.Event()
|
||||
lock = threading.Lock()
|
||||
successes, errors, confirmed = [], [], {}
|
||||
started = time.monotonic()
|
||||
|
||||
def worker(index):
|
||||
c = Client(base, timeout=2)
|
||||
key = uuid4().hex
|
||||
while not stop.is_set():
|
||||
for method in ("POST", "GET"):
|
||||
try:
|
||||
if method == "POST":
|
||||
response = c.message(alice, chat, "load-" + key, key=key)
|
||||
with lock:
|
||||
confirmed[response["json"]["id"]] = response["json"]
|
||||
key = uuid4().hex
|
||||
else:
|
||||
response = c.call("GET", "/api/v1/chats/" + chat["id"] + "/messages?limit=100",
|
||||
actor=bob, schema="MessagePage")
|
||||
with lock:
|
||||
successes.append((time.monotonic() - started, method,
|
||||
response["headers"]["X-Instance-Id"], response["elapsed"]))
|
||||
except Failure as exc:
|
||||
with lock:
|
||||
errors.append((time.monotonic() - started, method, str(exc)))
|
||||
stop.wait(0.05)
|
||||
|
||||
with ThreadPoolExecutor(max_workers=4) as pool:
|
||||
futures = [pool.submit(worker, i) for i in range(4)]
|
||||
time.sleep(1)
|
||||
action_start = time.monotonic() - started
|
||||
try:
|
||||
action_info = action()
|
||||
action_end = time.monotonic() - started
|
||||
# Охватывает обнаружение отказа и устойчивую работу после него.
|
||||
time.sleep(max(3, 15 - (action_end - action_start)))
|
||||
finally:
|
||||
stop.set()
|
||||
for future in futures:
|
||||
future.result()
|
||||
elapsed = time.monotonic() - started
|
||||
items = history(client, bob, chat)
|
||||
by_id = {m["id"]: m for m in items}
|
||||
assert len(items) == len(by_id) == len({m["text"] for m in items}), "Повтор создал дубль"
|
||||
assert by_id.get(original["id"]) == original
|
||||
assert all(by_id.get(mid) == msg for mid, msg in confirmed.items()), "Потеря подтверждённого сообщения"
|
||||
after = [s for s in successes if s[0] >= action_end]
|
||||
assert {s[1] for s in after} == {"POST", "GET"}, "Нет чтений/записей после воздействия"
|
||||
recovery = {method: round(min(s[0] for s in successes if s[0] >= action_start and s[1] == method) - action_start, 3)
|
||||
for method in ("POST", "GET")}
|
||||
assert max(recovery.values()) <= 10, recovery
|
||||
max_gap = {}
|
||||
for method in ("POST", "GET"):
|
||||
moments = sorted(s[0] for s in successes if s[1] == method)
|
||||
max_gap[method] = round(max(b - a for a, b in zip(moments, moments[1:])), 3)
|
||||
assert max(max_gap.values()) <= 10, max_gap
|
||||
if target:
|
||||
assert target not in {s[2] for s in after[-20:]}, "Остановленная реплика ещё отвечает"
|
||||
buckets = []
|
||||
for second in range(int(elapsed) + 1):
|
||||
buckets.append({"second": second, "successes": sum(second <= s[0] < second + 1 for s in successes),
|
||||
"errors": sum(second <= e[0] < second + 1 for e in errors)})
|
||||
latencies = sorted(s[3] * 1000 for s in successes)
|
||||
summary = {"label": label, "duration_seconds": round(elapsed, 3), "successful_requests": len(successes),
|
||||
"successful_writes": len(confirmed), "errors": len(errors), "successful_rps": round(len(successes) / elapsed, 2),
|
||||
"first_success_after_action_seconds": recovery, "maximum_success_gap_seconds": max_gap,
|
||||
"action_start_seconds": round(action_start, 3), "action_end_seconds": round(action_end, 3),
|
||||
"p50_ms": round(latencies[len(latencies) // 2], 3),
|
||||
"p95_ms": round(latencies[int(len(latencies) * .95)], 3),
|
||||
"confirmed_data_preserved": True, "duplicate_messages": 0,
|
||||
"instances": sorted({s[2] for s in successes}), "per_second": buckets,
|
||||
"error_samples": errors[:5], "action": action_info}
|
||||
print(json.dumps({k: v for k, v in summary.items() if k not in ("per_second", "error_samples", "action")}, ensure_ascii=False), flush=True)
|
||||
return summary
|
||||
|
||||
print("Command: ./scripts/check-docker.sh; project:", project, flush=True)
|
||||
print("Docker:", result["docker"], flush=True)
|
||||
try:
|
||||
script("start.sh")
|
||||
initial = inspect_all()
|
||||
result["containers"] = [metadata(info) for info in initial]
|
||||
assert len(apis()) == 2
|
||||
for info in initial:
|
||||
assert info["HostConfig"]["Memory"] > 0 and info["HostConfig"]["NanoCpus"] > 0
|
||||
if info["Config"]["Labels"]["com.docker.compose.service"] != "lb":
|
||||
assert not info["HostConfig"]["PortBindings"]
|
||||
for info in apis():
|
||||
assert docker("exec", info["Id"], "id", "-u") == "10001"
|
||||
assert all(m["Destination"].startswith("/run/secrets/") for m in info["Mounts"])
|
||||
print("PASS: 2 real API containers, CPU/RAM limits, API non-root and no public ports; only LB published", flush=True)
|
||||
subprocess.run([str(ROOT / "scripts/check.sh")], cwd=ROOT, env=env, check=True, timeout=240)
|
||||
result["api_tests"] = "public smoke + student unittest suite passed"
|
||||
subprocess.run([str(ROOT / "scripts/demo.sh")], cwd=ROOT, env=env, check=True, stdout=subprocess.DEVNULL, timeout=30)
|
||||
assert sql("SELECT count(*) FROM schema_migrations") == "3"
|
||||
count = sql("SELECT count(*) FROM messages")
|
||||
for _ in range(2):
|
||||
compose("run", "--rm", "migrate")
|
||||
assert sql("SELECT count(*) FROM schema_migrations") == "3"
|
||||
assert sql("SELECT count(*) FROM messages") == count
|
||||
print("PASS: migrations only in one-shot service; repeated execution preserves data", flush=True)
|
||||
result["attachments"] = run_checks(base, env, compose, apis, script, sql)
|
||||
|
||||
replay_client = Client(base)
|
||||
replay_actor, _, _, replay_chat = replay_client.bootstrap()
|
||||
replay_key = uuid4().hex
|
||||
saved_message = replay_client.message(replay_actor, replay_chat, "Повтор после рестарта", key=replay_key)["json"]
|
||||
before = {i["Id"]: metadata(i) for i in inspect_all() if i["State"]["Running"]}
|
||||
subprocess.run(["make", "test-persistence", "PYTHON=" + sys.executable, "ACTION_SCRIPT=scripts/restart.sh"],
|
||||
cwd=ROOT, env=env, check=True, timeout=180)
|
||||
after = {i["Id"]: metadata(i) for i in inspect_all() if i["State"]["Running"]}
|
||||
assert before.keys() == after.keys()
|
||||
assert all(before[cid]["started_at"] != after[cid]["started_at"] for cid in before)
|
||||
assert replay_client.message(replay_actor, replay_chat, "Повтор после рестарта", key=replay_key)["json"] == saved_message
|
||||
result["persistence"] = {"before": list(before.values()), "after": list(after.values()), "passed": True,
|
||||
"idempotency_replay_after_restart": True}
|
||||
print("PASS: both API, LB, PostgreSQL, Valkey, S3 and cleaner actually restarted; data and session preserved", flush=True)
|
||||
|
||||
check_shutdown()
|
||||
|
||||
dependency_client = Client(base)
|
||||
actor = dependency_client.user("deps")
|
||||
result["dependency_readiness"] = []
|
||||
for dependency in ("valkey", "postgres", "s3"):
|
||||
try:
|
||||
compose("stop", "--timeout", "8", dependency)
|
||||
probe_id = apis()[0]["Id"]
|
||||
readiness = subprocess.run(["docker", "exec", probe_id, "python", "-m", "src.healthcheck"], capture_output=True, timeout=10)
|
||||
assert readiness.returncode == 1
|
||||
live = docker("exec", probe_id, "python", "-c",
|
||||
"from urllib.request import urlopen; print(urlopen('http://127.0.0.1:8080/health/live').status)")
|
||||
assert live == "200"
|
||||
deadline = time.monotonic() + 8
|
||||
while True:
|
||||
try:
|
||||
urlopen(base + "/health/ready", timeout=2).close()
|
||||
except HTTPError as error:
|
||||
if error.code == 503:
|
||||
break
|
||||
raise
|
||||
except OSError:
|
||||
# Уже отправленный запрос может истечь до исключения backend.
|
||||
pass
|
||||
assert time.monotonic() < deadline
|
||||
time.sleep(0.1)
|
||||
result["dependency_readiness"].append({"dependency": dependency, "api_live": 200, "api_ready": 503, "lb_ready": 503})
|
||||
print("PASS:", dependency, "outage: API live=200, ready=503, LB ready=503", flush=True)
|
||||
finally:
|
||||
compose("start", dependency)
|
||||
script("wait-ready.sh")
|
||||
dependency_client.call("GET", "/api/v1/users/me", actor=actor, schema="User")
|
||||
|
||||
result["postgres_pause"] = check_postgres_pause(compose, apis, base)
|
||||
|
||||
subprocess.run(["make", "test-failover", "PYTHON=" + sys.executable, "ACTION_SCRIPT=scripts/stop-one.sh"],
|
||||
cwd=ROOT, env=env, check=True, timeout=90)
|
||||
assert sum(i["State"]["Running"] for i in apis()) == 1
|
||||
result["public_failover"] = {"passed": True, "containers_after": [metadata(i) for i in apis()]}
|
||||
compose("start", "api")
|
||||
script("wait-ready.sh")
|
||||
|
||||
result["failovers"] = []
|
||||
for info in apis():
|
||||
target = info["Config"]["Hostname"]
|
||||
|
||||
def stop_target():
|
||||
subprocess.run([str(ROOT / "scripts/stop-one.sh")], cwd=ROOT,
|
||||
env=dict(env, TARGET_INSTANCE_ID=target), check=True, timeout=10)
|
||||
stopped = json.loads(docker("inspect", info["Id"]))[0]
|
||||
assert not stopped["State"]["Running"]
|
||||
assert not stopped["State"]["OOMKilled"] and stopped["State"]["ExitCode"] == 0
|
||||
# Uvicorn пишет в stderr; получаем оба потока без вывода запросов.
|
||||
logs_result = subprocess.run(["docker", "logs", info["Id"]], capture_output=True, text=True, check=True)
|
||||
assert "Application shutdown complete" in logs_result.stdout + logs_result.stderr
|
||||
return metadata(stopped)
|
||||
|
||||
result["failovers"].append(load_action("stop " + target, stop_target, target))
|
||||
compose("start", "api")
|
||||
script("wait-ready.sh")
|
||||
|
||||
before = {i["Id"]: metadata(i) for i in inspect_all() if i["State"]["Running"]}
|
||||
|
||||
def replace():
|
||||
script("rolling-update.sh")
|
||||
after = {i["Id"]: metadata(i) for i in inspect_all() if i["State"]["Running"]}
|
||||
old_api = {i["id"] for i in before.values() if i["service"] == "api"}
|
||||
new_api = {i["id"] for i in after.values() if i["service"] == "api"}
|
||||
assert len(new_api) == 2 and not old_api & new_api
|
||||
for cid, info in before.items():
|
||||
if info["service"] != "api":
|
||||
assert after[cid]["started_at"] == info["started_at"]
|
||||
return {"before": list(before.values()), "after": list(after.values())}
|
||||
|
||||
result["rolling_update"] = load_action("sequential API replacement", replace)
|
||||
result["passed"] = True
|
||||
print("PASS: all lab4 grade-4 checks completed", flush=True)
|
||||
finally:
|
||||
output = ROOT / "evidence" / "lab4-checks.json"
|
||||
output.write_text(json.dumps(result, ensure_ascii=False, indent=2) + "\n")
|
||||
compose("down", "--volumes", "--timeout", "8")
|
||||
print("Removed only temporary test containers and volumes:", project, flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,42 @@
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import unittest
|
||||
from urllib.parse import urlencode
|
||||
from uuid import uuid4
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
from client import Client
|
||||
|
||||
|
||||
class ApiTestCase(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.client = Client()
|
||||
self.alice, self.bob, self.eve, self.chat = self.client.bootstrap()
|
||||
self.path = "/api/v1/chats/" + self.chat["id"]
|
||||
|
||||
def assert_error(self, method, path, status, code, **kwargs):
|
||||
if method == "POST" and path.endswith("/messages"):
|
||||
kwargs["headers"] = {"Idempotency-Key": uuid4().hex, **kwargs.get("headers", {})}
|
||||
result = self.client.call(method, path, status, **kwargs)
|
||||
self.assertEqual(result["json"]["error"]["code"], code)
|
||||
return result["json"]["error"]
|
||||
|
||||
def read_all(self, path, actor, schema, limit=3):
|
||||
items = []
|
||||
cursor = None
|
||||
seen_cursors = set()
|
||||
while True:
|
||||
query = {"limit": limit}
|
||||
if cursor is not None:
|
||||
query["cursor"] = cursor
|
||||
page = self.client.call("GET", path + "?" + urlencode(query), actor=actor, schema=schema)["json"]
|
||||
self.assertLessEqual(len(page["items"]), limit)
|
||||
items.extend(page["items"])
|
||||
cursor = page["next_cursor"]
|
||||
if cursor is None:
|
||||
return items
|
||||
self.assertTrue(page["items"], "Пустая страница не должна вести к следующей")
|
||||
self.assertNotIn(cursor, seen_cursors, "Зацикливание пагинации")
|
||||
seen_cursors.add(cursor)
|
||||
@@ -0,0 +1,360 @@
|
||||
"""Реальный Compose-стенд: обязательный минимум и все критерии lab5 на 4."""
|
||||
from datetime import datetime, timezone
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import platform
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
from tempfile import TemporaryDirectory
|
||||
import time
|
||||
from uuid import uuid4
|
||||
|
||||
from PIL import Image
|
||||
|
||||
from helpers import Client, ROOT
|
||||
from verify_restart import verify as verify_restart
|
||||
|
||||
|
||||
def main():
|
||||
with socket.socket() as sock:
|
||||
sock.bind(('127.0.0.1', 0))
|
||||
port = sock.getsockname()[1]
|
||||
project = 'lab5-check-' + uuid4().hex[:10]
|
||||
base = 'http://127.0.0.1:' + str(port)
|
||||
env = dict(os.environ, COMPOSE_PROJECT_NAME=project, HOST_PORT=str(port), BASE_URL=base,
|
||||
API_REPLICAS='2', ENABLE_FAULT_INJECTION='1', JOB_TIMEOUT_SECONDS='5',
|
||||
JOB_LEASE_SECONDS='15', JOB_TASK_TIMEOUT_SECONDS='25',
|
||||
JOB_MAX_ATTEMPTS='5', JOB_BACKOFF_SECONDS='0.15')
|
||||
result = {'utc': datetime.now(timezone.utc).isoformat(), 'command': './scripts/check-docker.sh',
|
||||
'project': project, 'host': platform.system() + ' ' + platform.machine(),
|
||||
'source_commit': subprocess.check_output(['git', 'rev-parse', 'HEAD'], cwd=ROOT, text=True).strip(),
|
||||
'working_tree': True, 'configuration': {'concurrency': 2, 'prefetch': 1, 'attempts': 5,
|
||||
'timeout_seconds': 5, 'task_timeout_seconds': 25, 'lease_seconds': 15,
|
||||
'backoff_base_seconds': 0.15}, 'checks': {}}
|
||||
|
||||
def command(args, timeout=240, check=True):
|
||||
return subprocess.run(args, cwd=ROOT, env=env, text=True, capture_output=True, timeout=timeout, check=check)
|
||||
|
||||
def compose(*args, timeout=240, check=True):
|
||||
return command(['docker', 'compose', *args], timeout, check)
|
||||
|
||||
def sql(query):
|
||||
return compose('exec', '-T', 'postgres', 'psql', '-U', 'messenger', '-d', 'messenger', '-At', '-c', query).stdout.strip()
|
||||
|
||||
def worker_code(code, service='worker'):
|
||||
return compose('exec', '-T', service, 'python', '-c', code).stdout.strip()
|
||||
|
||||
def wait(predicate, seconds=60, description='condition'):
|
||||
deadline = time.monotonic() + seconds
|
||||
while time.monotonic() < deadline:
|
||||
value = predicate()
|
||||
if value:
|
||||
return value
|
||||
time.sleep(0.2)
|
||||
raise AssertionError('Timeout: ' + description)
|
||||
|
||||
def fault(job_id, point, action='crash', service='worker'):
|
||||
# docker cp позволяет установить failpoint также в остановленный publisher.
|
||||
container = compose('ps', '-aq', service).stdout.strip()
|
||||
with TemporaryDirectory(prefix='pipeline-fault-') as directory:
|
||||
source = Path(directory) / 'fault'
|
||||
source.write_text(action)
|
||||
source.chmod(0o644)
|
||||
command(['docker', 'cp', str(source), container + ':/tmp/pipeline-faults/' + job_id + '.' + point])
|
||||
|
||||
def hit(job_id, point, service='worker'):
|
||||
probe = compose('exec', '-T', service, 'python', '-c', 'from pathlib import Path; print(Path(' + repr('/tmp/pipeline-faults/' + job_id + '.' + point + '.hit') + ').exists())', check=False)
|
||||
return probe.returncode == 0 and probe.stdout.strip() == 'True'
|
||||
|
||||
def job_id(att):
|
||||
return sql("SELECT id FROM attachment_jobs WHERE attachment_id='" + att['id'] + "'")
|
||||
|
||||
def objects(att):
|
||||
prefix = 'attachments/' + att['id'].replace('-', '') + '/v1/'
|
||||
return json.loads(worker_code('import json; from src import object_store as s; print(json.dumps(sorted(o.object_name for o in s.client.list_objects(s.BUCKET,prefix=' + repr(prefix) + ',recursive=True))))', service='cleanup'))
|
||||
|
||||
def stage_exists(att):
|
||||
key = sql("SELECT staging_key FROM attachment_jobs WHERE attachment_id='" + att['id'] + "'")
|
||||
return int(worker_code('from src import object_store as s; print(sum(1 for o in s.client.list_objects(s.BUCKET,prefix=' + repr(key) + ',recursive=True)))', service='cleanup')) == 1
|
||||
|
||||
def record(name, **data):
|
||||
result['checks'][name] = {'passed': True, **data}
|
||||
print('PASS:', name, json.dumps(data, ensure_ascii=False), flush=True)
|
||||
|
||||
def metadata(att):
|
||||
return client.call('GET', '/api/v1/attachments/' + att['id'], actor=alice, schema='Attachment')['json']
|
||||
|
||||
def status(att, wanted, seconds=60):
|
||||
return wait(lambda: (value if (value := metadata(att))['status'] == wanted else None), seconds, wanted)
|
||||
|
||||
def upload(data=None, mime='image/png'):
|
||||
return client.call('POST', '/api/v1/chats/' + chat['id'] + '/attachments?filename=sample.png', 202,
|
||||
actor=alice, raw=data or png, headers={'Content-Type': mime}, schema='Attachment')['json']
|
||||
|
||||
def assert_variants(att, original):
|
||||
ready = status(att, 'ready')
|
||||
assert {v['name'] for v in ready['variants']} == {'thumbnail', 'crop', 'optimized'}
|
||||
original_response = client.call('GET', '/api/v1/attachments/' + att['id'] + '/content', actor=bob)
|
||||
assert original_response['bytes'] == original
|
||||
assert ready['sha256'] == hashlib.sha256(original).hexdigest()
|
||||
variants = {}
|
||||
for variant in ready['variants']:
|
||||
response = client.call('GET', '/api/v1/attachments/' + att['id'] + '/content?variant=' + variant['name'], actor=bob)
|
||||
data = response['bytes']
|
||||
assert len(data) == variant['size_bytes'] and hashlib.sha256(data).hexdigest() == variant['sha256']
|
||||
with Image.open(io.BytesIO(data)) as image:
|
||||
image.load()
|
||||
assert image.size == (variant['width'], variant['height'])
|
||||
assert image.format == ('WEBP' if variant['name'] == 'optimized' else 'JPEG')
|
||||
assert not image.getexif()
|
||||
variants[variant['name']] = {'size': list(image.size), 'bytes': len(data)}
|
||||
assert len(objects(att)) == 4
|
||||
return variants
|
||||
|
||||
def queue_payload(att):
|
||||
return json.loads(worker_code('''import json
|
||||
from src import db,broker
|
||||
from src.storage import utc
|
||||
db.pool.open(wait=True)
|
||||
with db.connection() as c:
|
||||
j=c.execute("SELECT * FROM attachment_jobs WHERE attachment_id=%s",(''' + repr(att['id']) + ''',)).fetchone()
|
||||
payload={'schema_version':1,'job_id':str(j['id']),'attachment_id':str(j['attachment_id']),'type':'attachment.process.v1','pipeline_version':1,'attempt':j['attempt'],'created_at':utc(j['created_at'])}
|
||||
print(json.dumps(payload))
|
||||
db.pool.close()
|
||||
'''))
|
||||
|
||||
def duplicate(att, count=2):
|
||||
payload = queue_payload(att)
|
||||
from jsonschema import Draft202012Validator, FormatChecker
|
||||
Draft202012Validator(json.loads((ROOT / 'contracts/jobs.schema.json').read_text()), format_checker=FormatChecker()).validate(payload)
|
||||
worker_code('import json; from src import broker; c=broker.connect(); p=' + repr(json.dumps(payload)) + '; [c.xadd(broker.STREAM,{"job":p}) for _ in range(' + str(count) + ')]')
|
||||
|
||||
try:
|
||||
result['docker'] = command(['docker', 'info', '--format', '{{.ServerVersion}} {{.Architecture}} CPUs={{.NCPU}} RAM={{.MemTotal}}']).stdout.strip()
|
||||
command([str(ROOT / 'scripts/start.sh')], timeout=300)
|
||||
for service in ('worker', 'publisher'):
|
||||
worker_code("from pathlib import Path; Path('/tmp/pipeline-faults').mkdir(exist_ok=True)", service)
|
||||
ids = compose('ps', '-aq').stdout.split()
|
||||
infos = json.loads(command(['docker', 'inspect', *ids]).stdout)
|
||||
result['containers'] = [{'id': info['Id'], 'service': info['Config']['Labels']['com.docker.compose.service'],
|
||||
'image': info['Config']['Image'], 'memory_bytes': info['HostConfig']['Memory'],
|
||||
'nano_cpus': info['HostConfig']['NanoCpus'],
|
||||
'mounts': [{'destination': m['Destination'], 'name': m.get('Name')} for m in info['Mounts']]} for info in infos]
|
||||
assert sum(i['Config']['Labels']['com.docker.compose.service'] == 'api' for i in infos) == 2
|
||||
for info in infos:
|
||||
assert info['HostConfig']['Memory'] and info['HostConfig']['NanoCpus']
|
||||
if info['Config']['Labels']['com.docker.compose.service'] != 'lb':
|
||||
assert not info['HostConfig']['PortBindings']
|
||||
record('topology_and_limits', api_replicas=2, broker='Valkey Streams with AOF fsync always', worker_processes=2)
|
||||
public = command(['make', 'check', 'validate', 'test', 'PYTHON=' + sys.executable])
|
||||
print(public.stdout, end='', flush=True)
|
||||
record('public_smoke_and_schema')
|
||||
unit = command([sys.executable, '-m', 'unittest', 'discover', '-s', 'student-tests', '-v'])
|
||||
print(unit.stdout + unit.stderr, end='', flush=True)
|
||||
record('student_api_regressions')
|
||||
client = Client(base)
|
||||
alice, bob, eve, chat = client.bootstrap()
|
||||
png = (ROOT / 'tests/fixtures/sample.png').read_bytes()
|
||||
|
||||
# Worker stopped: queued already linkable; API and broker restarts preserve accepted bytes/job.
|
||||
compose('stop', '--timeout', '3', 'worker')
|
||||
att = upload()
|
||||
assert att['status'] == 'queued' and stage_exists(att)
|
||||
client.call('GET', '/api/v1/attachments/' + att['id'] + '/content', 409, actor=alice)
|
||||
msg = client.message(bob, chat, '', attachments=[att['id']])['json']
|
||||
jid = job_id(att)
|
||||
wait(lambda: sql("SELECT published_at IS NOT NULL FROM attachment_outbox WHERE job_id='" + jid + "'") == 't')
|
||||
messages = json.loads(worker_code('from src import broker; import json; c=broker.connect(); print(json.dumps([json.loads(fields["job"]) for _,fields in c.xrange(broker.STREAM)]))', service='publisher'))
|
||||
from jsonschema import Draft202012Validator, FormatChecker
|
||||
validator = Draft202012Validator(json.loads((ROOT / 'contracts/jobs.schema.json').read_text()), format_checker=FormatChecker())
|
||||
assert messages
|
||||
for payload in messages:
|
||||
validator.validate(payload)
|
||||
compose('restart', '--timeout', '3', 'api', 'broker')
|
||||
command([str(ROOT / 'scripts/wait-ready.sh')])
|
||||
assert metadata(att)['status'] == 'queued'
|
||||
compose('start', 'worker')
|
||||
variants = assert_variants(att, png)
|
||||
assert variants['thumbnail']['size'] == [256, 160] and variants['crop']['size'] == [128, 128]
|
||||
record('worker_offline_api_broker_restart', attachment_id=att['id'], job_id=jid, message_id=msg['id'], variants=variants)
|
||||
|
||||
# Исправная попытка живёт дольше исходного lease без ложного retry.
|
||||
compose('stop', '--timeout', '3', 'publisher')
|
||||
att = upload(); jid = job_id(att)
|
||||
fault(jid, 'before_download', 'delay:20')
|
||||
compose('start', 'publisher')
|
||||
wait(lambda: hit(jid, 'before_download'), description='long task started')
|
||||
initial_lease = sql("SELECT lease_until FROM attachment_jobs WHERE id='" + jid + "'")
|
||||
assert worker_code('''from uuid import uuid4
|
||||
from src import db,jobs
|
||||
db.pool.open(wait=True)
|
||||
with db.connection() as c:
|
||||
j=jobs.locked(c,''' + repr(jid) + ''')
|
||||
j['lease_token']=uuid4()
|
||||
print(jobs.renew(j))
|
||||
db.pool.close()
|
||||
''') == 'False'
|
||||
wait(lambda: sql("SELECT lease_until FROM attachment_jobs WHERE id='" + jid + "'") != initial_lease,
|
||||
seconds=12, description='heartbeat extended lease')
|
||||
assert_variants(att, png)
|
||||
assert sql("SELECT attempt FROM attachment_jobs WHERE id='" + jid + "'") == '1'
|
||||
assert sql("SELECT count(*) FROM attachment_job_events WHERE job_id='" + jid + "' AND event='retry'") == '0'
|
||||
record('long_task_heartbeat', job_id=jid, delay_seconds=20, lease_seconds=15, attempt=1)
|
||||
|
||||
# Heartbeat не сохраняет зависшую работу бесконечно: общий deadline.
|
||||
compose('stop', '--timeout', '3', 'publisher')
|
||||
att = upload(); jid = job_id(att)
|
||||
fault(jid, 'before_download', 'timeout')
|
||||
compose('start', 'publisher')
|
||||
assert_variants(att, png)
|
||||
assert hit(jid, 'before_download')
|
||||
assert sql("SELECT count(*) FROM attachment_job_events WHERE job_id='" + jid + "' AND error_code='processing_timeout'") == '1'
|
||||
assert sql("SELECT attempt FROM attachment_jobs WHERE id='" + jid + "'") == '2'
|
||||
record('whole_task_timeout', job_id=jid, timeout_seconds=25, recovered=True)
|
||||
|
||||
# COMMIT -> publish gap and XADD -> outbox COMMIT gap, each crash is automatic.
|
||||
for point in ('before_publish', 'after_publish'):
|
||||
compose('stop', '--timeout', '3', 'publisher')
|
||||
att = upload(); jid = job_id(att)
|
||||
assert sql("SELECT published_at IS NULL FROM attachment_outbox WHERE job_id='" + jid + "'") == 't'
|
||||
fault(jid, point, service='publisher')
|
||||
compose('start', 'publisher')
|
||||
wait(lambda: hit(jid, point, 'publisher'), description=point)
|
||||
assert_variants(att, png)
|
||||
record('outbox_crash_' + point, job_id=jid, automatic_recovery=True)
|
||||
|
||||
# Worker crash after PUT before result COMMIT and after result COMMIT before ACK.
|
||||
for point in ('after_objects', 'before_ack'):
|
||||
compose('stop', '--timeout', '3', 'publisher')
|
||||
att = upload(); jid = job_id(att)
|
||||
fault(jid, point)
|
||||
compose('start', 'publisher')
|
||||
wait(lambda: hit(jid, point), description=point)
|
||||
assert_variants(att, png)
|
||||
if point == 'after_objects':
|
||||
assert int(sql("SELECT attempt FROM attachment_jobs WHERE id='" + jid + "'")) >= 2
|
||||
snapshot = metadata(att)
|
||||
duplicate(att, 4)
|
||||
wait(lambda: worker_code('from src import broker; c=broker.connect(); print(c.xpending(broker.STREAM,broker.GROUP)["pending"])') == '0', seconds=45, description='pending ACK')
|
||||
assert metadata(att) == snapshot and len(objects(att)) == 4
|
||||
record('worker_crash_' + point, job_id=jid, attempt=int(sql("SELECT attempt FROM attachment_jobs WHERE id='" + jid + "'")), canonical_objects=4, duplicate_noop=True)
|
||||
|
||||
# Transient S3 errors, jittered backoff, exhausted registry and explicit replay.
|
||||
compose('stop', '--timeout', '3', 'publisher')
|
||||
att = upload(); jid = job_id(att)
|
||||
fault(jid, 'before_process', 'transient:5')
|
||||
compose('start', 'publisher')
|
||||
status(att, 'failed')
|
||||
failed = metadata(att)
|
||||
assert failed['error_code'] == 'retry_exhausted'
|
||||
client.call('GET', '/api/v1/attachments/' + att['id'] + '/content', 409, actor=alice)
|
||||
client.call('POST', '/api/v1/chats/' + chat['id'] + '/messages', 409, actor=bob,
|
||||
body={'text': '', 'attachment_ids': [att['id']]}, headers={'Idempotency-Key': uuid4().hex})
|
||||
registry = json.loads(compose('exec', '-T', 'worker', 'python', '-m', 'src.job_admin', 'failed').stdout)
|
||||
assert any(row['job_id'] == jid for row in registry)
|
||||
compose('exec', '-T', 'worker', 'python', '-m', 'src.job_admin', 'replay', jid)
|
||||
assert_variants(att, png)
|
||||
attempt = sql("SELECT attempt FROM attachment_jobs WHERE id='" + jid + "'")
|
||||
rejected = compose('exec', '-T', 'worker', 'python', '-m', 'src.job_admin', 'replay', jid, check=False)
|
||||
assert rejected.returncode != 0
|
||||
assert sql("SELECT attempt FROM attachment_jobs WHERE id='" + jid + "'") == attempt
|
||||
audit = json.loads(sql("SELECT json_agg(json_build_object('event',event,'attempt',attempt,'error_code',error_code,'at',created_at) ORDER BY id) FROM attachment_job_events WHERE job_id='" + jid + "'"))
|
||||
record('bounded_retry_failed_registry_replay', job_id=jid, max_automatic_attempts=5, audit=audit, ready_replay_rejected=True)
|
||||
|
||||
# Permanent error from damaged staging: one attempt, safe code, no retry.
|
||||
compose('stop', '--timeout', '3', 'publisher')
|
||||
invalid = upload(); jid = job_id(invalid)
|
||||
key = sql("SELECT staging_key FROM attachment_jobs WHERE id='" + jid + "'")
|
||||
worker_code('import io; from src import object_store as s; s.put(' + repr(key) + ',io.BytesIO(b"bad"),3,"image/png")')
|
||||
compose('start', 'publisher')
|
||||
status(invalid, 'failed')
|
||||
assert metadata(invalid)['error_code'] == 'invalid_image'
|
||||
assert sql("SELECT attempt FROM attachment_jobs WHERE id='" + jid + "'") == '1'
|
||||
record('permanent_error', job_id=jid, attempt=1, error_code='invalid_image')
|
||||
|
||||
# Реальные timeout и MemoryError внутри ограниченного resize subprocess.
|
||||
for action, error_code in [('timeout', 'processing_timeout'), ('memory', 'resource_limit')]:
|
||||
compose('stop', '--timeout', '3', 'publisher')
|
||||
att = upload(); jid = job_id(att)
|
||||
fault(jid, 'transform', action)
|
||||
compose('start', 'publisher')
|
||||
assert_variants(att, png)
|
||||
assert sql("SELECT count(*) FROM attachment_job_events WHERE job_id='" + jid + "' AND error_code='" + error_code + "'") == '1'
|
||||
record('image_subprocess_' + action, job_id=jid, timeout_seconds=5, task_memory_mb=640, recovered=True)
|
||||
|
||||
# Alpha, small input, EXIF rotation and floor rounding in real HTTP downloads.
|
||||
cases = []
|
||||
alpha = io.BytesIO(); Image.new('RGBA', (17, 9), (255, 0, 0, 0)).save(alpha, format='PNG')
|
||||
att = upload(alpha.getvalue())
|
||||
data = assert_variants(att, alpha.getvalue())
|
||||
assert data['thumbnail']['size'] == [17, 9] and data['optimized']['size'] == [17, 9]
|
||||
thumb = client.call('GET', '/api/v1/attachments/' + att['id'] + '/content?variant=thumbnail', actor=bob)['bytes']
|
||||
with Image.open(io.BytesIO(thumb)) as image:
|
||||
assert all(channel >= 250 for channel in image.getpixel((8, 4)))
|
||||
cases.append({'case': 'small_alpha_on_white', 'variants': data})
|
||||
photo = io.BytesIO(); exif = Image.Exif(); exif[274] = 6
|
||||
Image.new('RGB', (321, 201), 'orange').save(photo, format='JPEG', exif=exif)
|
||||
att = upload(photo.getvalue(), 'image/jpeg')
|
||||
data = assert_variants(att, photo.getvalue())
|
||||
assert data['thumbnail']['size'] == [160, 256]
|
||||
cases.append({'case': 'exif_orientation_6_and_floor', 'variants': data})
|
||||
record('image_semantics', cases=cases)
|
||||
|
||||
# Cleanup preserves active staging even if old; removes ready and expired failed.
|
||||
compose('stop', '--timeout', '3', 'worker')
|
||||
active = upload()
|
||||
sql("UPDATE attachments SET created_at=clock_timestamp()-interval '2 days',updated_at=clock_timestamp()-interval '2 days' WHERE id='" + active['id'] + "'")
|
||||
sql("UPDATE attachments SET updated_at=clock_timestamp()-interval '2 days' WHERE id='" + invalid['id'] + "'")
|
||||
orphan_id = str(uuid4())
|
||||
orphan_key = 'attachments/staging/' + orphan_id.replace('-', '')
|
||||
sql("INSERT INTO attachment_uploads(id,object_key,created_at) VALUES ('" + orphan_id + "','" + orphan_key + "',clock_timestamp()-interval '2 days')")
|
||||
worker_code('import io; from src import object_store as s; s.put(' + repr(orphan_key) + ',io.BytesIO(b"orphan"),6,"image/png")', service='cleanup')
|
||||
compose('run', '--rm', '--no-deps', 'cleanup', 'python', '-m', 'src.cleanup', '--once')
|
||||
assert sql("SELECT count(*) FROM attachment_uploads WHERE id='" + orphan_id + "'") == '0'
|
||||
assert worker_code('from src import object_store as s; print(sum(1 for o in s.client.list_objects(s.BUCKET,prefix=' + repr(orphan_key) + ',recursive=True)))', service='cleanup') == '0'
|
||||
|
||||
assert stage_exists(active)
|
||||
assert not stage_exists(invalid)
|
||||
assert not stage_exists(att)
|
||||
rejected = compose('exec', '-T', 'cleanup', 'python', '-m', 'src.job_admin', 'replay', job_id(invalid), check=False)
|
||||
assert rejected.returncode != 0
|
||||
compose('start', 'worker'); assert_variants(active, png)
|
||||
record('staging_cleanup', active_preserved=True, orphan_removed=True, ready_removed=True, expired_failed_removed=True, expired_replay_rejected=True)
|
||||
|
||||
# Broker outage doesn't force synchronous resize: outbox recovers after restart.
|
||||
compose('stop', '--timeout', '3', 'broker')
|
||||
att = upload()
|
||||
assert att['status'] == 'queued'
|
||||
client.message(alice, chat, 'Broker offline; text API still works')
|
||||
compose('start', 'broker'); assert_variants(att, png)
|
||||
record('broker_offline_outbox', attachment_id=att['id'], recovered=True)
|
||||
|
||||
record('public_persistence', restart_proof=verify_restart(env))
|
||||
failover = command(['make', 'test-failover', 'PYTHON=' + sys.executable, 'ACTION_SCRIPT=scripts/stop-one.sh'])
|
||||
print(failover.stdout, end='', flush=True)
|
||||
compose('start', 'api')
|
||||
command([str(ROOT / 'scripts/wait-ready.sh')])
|
||||
record('public_api_failover')
|
||||
result['passed'] = True
|
||||
print('PASS: all lab5 grade-4 criteria completed', flush=True)
|
||||
except subprocess.CalledProcessError as error:
|
||||
# Команды не содержат credentials, но runtime logs не сохраняем автоматически.
|
||||
print('Command failed:', error.cmd[:4], error.returncode, flush=True)
|
||||
print(error.stdout, error.stderr, flush=True)
|
||||
raise
|
||||
finally:
|
||||
output = ROOT / 'evidence/lab5-checks.json'
|
||||
output.write_text(json.dumps(result, ensure_ascii=False, indent=2) + '\n')
|
||||
if os.environ.get('KEEP_TEST_STACK') == '1' and not result.get('passed'):
|
||||
print('Kept isolated failed test stack:', project, flush=True)
|
||||
else:
|
||||
compose('down', '--volumes', '--timeout', '5')
|
||||
print('Removed isolated test stack:', project, flush=True)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,145 @@
|
||||
"""Пауза живой PostgreSQL: прямой readiness, ограниченное SQL I/O и восстановление."""
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from datetime import datetime, timezone
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from uuid import uuid4
|
||||
|
||||
from helpers import Client, ROOT
|
||||
from client import Failure
|
||||
|
||||
|
||||
def check_postgres_pause(compose, apis, base):
|
||||
client = Client(base)
|
||||
alice, bob, _, chat = client.bootstrap()
|
||||
saved_message = client.message(alice, chat, "До паузы PostgreSQL")["json"]
|
||||
before = {info["Id"]:info["State"]["StartedAt"] for info in apis() if info["State"]["Running"]}
|
||||
probe_id = sorted(before)[0]
|
||||
|
||||
def direct_probe(path, expected):
|
||||
# HTTP напрямую в API, без HAProxy и его собственных таймаутов/503.
|
||||
code = """
|
||||
import json,time
|
||||
from urllib.error import HTTPError
|
||||
from urllib.request import urlopen
|
||||
started=time.monotonic()
|
||||
try: response=urlopen('http://127.0.0.1:8080/PATH',timeout=5)
|
||||
except HTTPError as error: response=error
|
||||
with response:
|
||||
body=json.load(response)
|
||||
elapsed=time.monotonic()-started
|
||||
assert response.status==EXPECTED, response.status
|
||||
assert response.headers['X-Request-Id'] and response.headers['X-Instance-Id']
|
||||
if response.status==503:
|
||||
assert body['error']['code']=='unavailable'
|
||||
assert body['error']['request_id']==response.headers['X-Request-Id']
|
||||
assert elapsed<4.5, elapsed
|
||||
print(json.dumps({'status':response.status,'elapsed_seconds':round(elapsed,3)}))
|
||||
""".replace("PATH", path).replace("EXPECTED", str(expected))
|
||||
completed = subprocess.run(["docker", "exec", probe_id, "python", "-c", code],
|
||||
capture_output=True, text=True, timeout=8, check=True)
|
||||
return json.loads(completed.stdout)
|
||||
|
||||
# Соединение получено ДО pause: проверяем именно выполняющийся SQL, а не
|
||||
# только timeout получения нового соединения или check callback пула.
|
||||
query_code = """
|
||||
import json,sys,time
|
||||
import psycopg
|
||||
from src import db
|
||||
db.pool.open(wait=True,timeout=10)
|
||||
conn=db.pool.getconn()
|
||||
conn.execute('SET statement_timeout=0')
|
||||
print('connected',flush=True)
|
||||
sys.stdin.readline()
|
||||
started=time.monotonic()
|
||||
try:
|
||||
try: conn.execute('SELECT 1')
|
||||
except psycopg.OperationalError:
|
||||
elapsed=time.monotonic()-started
|
||||
assert elapsed<4.5, elapsed
|
||||
assert conn.closed, 'timed out connection was not discarded'
|
||||
print(json.dumps({'elapsed_seconds':round(elapsed,3),'connection_closed':conn.closed}),flush=True)
|
||||
else: raise AssertionError('paused database answered a query')
|
||||
finally:
|
||||
db.pool.putconn(conn)
|
||||
db.pool.close()
|
||||
"""
|
||||
query = subprocess.Popen(["docker", "exec", "-i", probe_id, "python", "-u", "-c", query_code],
|
||||
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
|
||||
result = {}
|
||||
try:
|
||||
assert query.stdout.readline().strip() == "connected"
|
||||
compose("pause", "postgres")
|
||||
paused_at = time.monotonic()
|
||||
output, error = query.communicate("\n", timeout=10)
|
||||
assert query.returncode == 0, "SQL I/O deadline failed: " + error
|
||||
result["sql_without_server_timeout"] = json.loads(output)
|
||||
result["direct_ready"] = direct_probe("health/ready", 503)
|
||||
result["direct_live"] = direct_probe("health/live", 200)
|
||||
# Повторные запросы при длительной паузе не должны оставлять занятые потоки.
|
||||
with ThreadPoolExecutor(max_workers=10) as executor:
|
||||
result["repeated_ready"] = list(executor.map(lambda _: direct_probe("health/ready", 503), range(10)))
|
||||
# Сервер остаётся на pause не менее 12 секунд; ограничиваем именно API.
|
||||
remaining = 12 - (time.monotonic() - paused_at)
|
||||
if remaining > 0:
|
||||
time.sleep(remaining)
|
||||
result["ready_after_12_seconds"] = direct_probe("health/ready", 503)
|
||||
result["live_after_repeated_requests"] = direct_probe("health/live", 200)
|
||||
result["paused_seconds"] = round(time.monotonic() - paused_at, 3)
|
||||
finally:
|
||||
compose("unpause", "postgres")
|
||||
if query.poll() is None:
|
||||
query.communicate(timeout=10)
|
||||
|
||||
resumed_at = time.monotonic()
|
||||
while True:
|
||||
try:
|
||||
result["recovered_ready"] = direct_probe("health/ready", 200)
|
||||
client.call("GET", "/api/v1/users/me", actor=alice, schema="User")
|
||||
messages = client.call("GET", "/api/v1/chats/" + chat["id"] + "/messages", actor=bob, schema="MessagePage")["json"]["items"]
|
||||
assert saved_message in messages
|
||||
break
|
||||
except (subprocess.CalledProcessError, Failure):
|
||||
assert time.monotonic() - resumed_at < 30, "readiness did not recover after unpause"
|
||||
time.sleep(.2)
|
||||
assert before == {info["Id"]:info["State"]["StartedAt"] for info in apis() if info["State"]["Running"]}
|
||||
result.update({"recovery_seconds":round(time.monotonic()-resumed_at,3), "api_not_restarted":True,
|
||||
"existing_session_and_message_preserved":True, "passed":True})
|
||||
print("PASS: paused PostgreSQL: SQL and direct readiness bounded; live works; unpause restores API without restart", flush=True)
|
||||
return result
|
||||
|
||||
|
||||
def main():
|
||||
with socket.socket() as listener:
|
||||
listener.bind(("127.0.0.1", 0))
|
||||
port = listener.getsockname()[1]
|
||||
project = "lab4-pg-pause-" + uuid4().hex[:10]
|
||||
base = f"http://127.0.0.1:{port}"
|
||||
env = dict(os.environ, COMPOSE_PROJECT_NAME=project, HOST_PORT=str(port), API_REPLICAS="2", BASE_URL=base)
|
||||
|
||||
def compose(*args):
|
||||
return subprocess.check_output(["docker", "compose", *args], cwd=ROOT, env=env, text=True, timeout=180).strip()
|
||||
|
||||
def apis():
|
||||
ids = compose("ps", "-a", "-q", "api").split()
|
||||
return json.loads(subprocess.check_output(["docker", "inspect", *ids], text=True))
|
||||
|
||||
result = {"utc":datetime.now(timezone.utc).isoformat(), "project":project,
|
||||
"command":".venv/bin/python student-tests/postgres_pause.py", "passed":False}
|
||||
try:
|
||||
compose("up", "-d", "--no-build", "lb")
|
||||
subprocess.run([str(ROOT / "scripts/wait-ready.sh")], cwd=ROOT, env=env, check=True, timeout=120)
|
||||
result.update(check_postgres_pause(compose, apis, base))
|
||||
print(json.dumps(result, ensure_ascii=False), flush=True)
|
||||
finally:
|
||||
Path(ROOT / "evidence/postgres-pause.json").write_text(json.dumps(result, ensure_ascii=False, indent=2) + "\n")
|
||||
compose("down", "--volumes", "--timeout", "8")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,164 @@
|
||||
import base64
|
||||
import json
|
||||
from urllib.parse import urlencode
|
||||
from uuid import uuid4
|
||||
|
||||
from helpers import ApiTestCase
|
||||
|
||||
|
||||
class ApiTests(ApiTestCase):
|
||||
def test_duplicate_username(self):
|
||||
self.assert_error("POST", "/api/v1/users", 409, "conflict", body=self.alice["registration"])
|
||||
|
||||
def test_invalid_user_fields(self):
|
||||
invalid_bodies = [
|
||||
{}, [], None,
|
||||
{"username": "ab", "display_name": "Имя"},
|
||||
{"username": "UPPER", "display_name": "Имя"},
|
||||
{"username": "name\n", "display_name": "Имя"},
|
||||
{"username": "язык", "display_name": "Имя"},
|
||||
{"username": "a" * 33, "display_name": "Имя"},
|
||||
{"username": 123, "display_name": "Имя"},
|
||||
{"username": "valid_name", "display_name": " \t\n"},
|
||||
{"username": "valid_name", "display_name": "👋" * 101},
|
||||
{"username": "valid_name", "display_name": False},
|
||||
{"username": "valid_name", "display_name": "Имя", "extra": 1},
|
||||
]
|
||||
for body in invalid_bodies:
|
||||
if isinstance(body, dict):
|
||||
body = {**body, "password": "Synthetic-password-123"}
|
||||
with self.subTest(body=body):
|
||||
self.assert_error("POST", "/api/v1/users", 400, "invalid_request",
|
||||
raw=json.dumps(body).encode(), headers={"Content-Type": "application/json"})
|
||||
|
||||
def test_malformed_json_and_encoding(self):
|
||||
invalid_bodies = [
|
||||
b'{"username":', b'', b'not JSON', b'\xff',
|
||||
'{"username":"valid_name","display_name":"Имя"}'.encode("utf-16"),
|
||||
b'{"username":"valid_name","display_name":"\\ud800"}',
|
||||
]
|
||||
for body in invalid_bodies:
|
||||
with self.subTest(body=body):
|
||||
self.assert_error("POST", "/api/v1/users", 400, "invalid_request",
|
||||
raw=body, headers={"Content-Type": "application/json"})
|
||||
|
||||
def test_wrong_content_type(self):
|
||||
self.assert_error("POST", "/api/v1/users", 415, "unsupported_media_type",
|
||||
raw=b'{}', headers={"Content-Type": "text/plain"})
|
||||
|
||||
def test_identification_and_authentication_order(self):
|
||||
self.assert_error("GET", "/api/v1/users/me", 401, "unauthorized")
|
||||
self.assert_error("GET", "/api/v1/users/me", 401, "unauthorized",
|
||||
headers={"X-User-Id": self.alice["id"]})
|
||||
self.assert_error("GET", "/api/v1/users/me", 401, "unauthorized",
|
||||
headers={"Authorization": "Bearer unknown-session"})
|
||||
self.assert_error("POST", self.path + "/messages", 401, "unauthorized",
|
||||
raw=b'{', headers={"Content-Type": "application/json"})
|
||||
me = self.client.call("GET", "/api/v1/users/me", schema="User", actor=self.alice,
|
||||
headers={"X-User-Id": self.bob["id"]})["json"]
|
||||
self.assertEqual(me, self.alice["user"])
|
||||
|
||||
def test_invalid_chat_members(self):
|
||||
invalid_members = [
|
||||
[], [self.bob["id"]], [self.alice["id"], str(uuid4())],
|
||||
[self.alice["id"], self.alice["id"]],
|
||||
[self.alice["id"], self.alice["id"].upper()],
|
||||
[self.alice["id"], "not-a-uuid"], [True], self.alice["id"],
|
||||
[self.alice["id"]] + [str(uuid4()) for _ in range(100)],
|
||||
]
|
||||
for members in invalid_members:
|
||||
with self.subTest(members=members):
|
||||
self.assert_error("POST", "/api/v1/chats", 400, "invalid_request", actor=self.alice,
|
||||
body={"title": "Чат", "member_ids": members})
|
||||
chats = self.read_all("/api/v1/chats", self.alice, "ChatPage")
|
||||
self.assertEqual([chat["id"] for chat in chats], [self.chat["id"]])
|
||||
|
||||
def test_invalid_chat_title(self):
|
||||
for title in ("", " \n", "a" * 101, 123, None):
|
||||
with self.subTest(title=title):
|
||||
self.assert_error("POST", "/api/v1/chats", 400, "invalid_request", actor=self.alice,
|
||||
body={"title": title, "member_ids": [self.alice["id"]]})
|
||||
|
||||
def test_chat_access(self):
|
||||
outsider = self.assert_error("GET", self.path, 404, "not_found", actor=self.eve)
|
||||
missing = self.assert_error("GET", "/api/v1/chats/" + str(uuid4()), 404, "not_found", actor=self.eve)
|
||||
self.assertEqual(outsider["message"], missing["message"])
|
||||
self.assert_error("GET", self.path + "/messages", 404, "not_found", actor=self.eve)
|
||||
self.assert_error("POST", self.path + "/messages", 404, "not_found", actor=self.eve, body={"text": "Чужое"})
|
||||
self.assertEqual(self.read_all("/api/v1/chats", self.eve, "ChatPage"), [])
|
||||
self.assert_error("GET", "/api/v1/chats/not-a-uuid", 400, "invalid_request", actor=self.alice)
|
||||
|
||||
def test_invalid_message_fields(self):
|
||||
for text in ("", " \t\n", "👋" * 4001, 123, True, None, []):
|
||||
with self.subTest(text_type=type(text).__name__):
|
||||
self.assert_error("POST", self.path + "/messages", 400, "invalid_request",
|
||||
actor=self.alice, body={"text": text})
|
||||
self.assert_error("POST", self.path + "/messages", 400, "invalid_request", actor=self.alice,
|
||||
body={"text": "Подмена автора", "sender_id": self.bob["id"]})
|
||||
self.assertEqual(self.read_all(self.path + "/messages", self.alice, "MessagePage"), [])
|
||||
|
||||
def test_unicode_and_text_preservation(self):
|
||||
for text in (" Привет, мир 👋\n", "👋" * 4000):
|
||||
message = self.client.message(self.bob, self.chat, text)["json"]
|
||||
self.assertEqual(message["text"], text)
|
||||
self.assertEqual(message["sender_id"], self.bob["id"])
|
||||
items = self.read_all(self.path + "/messages", self.alice, "MessagePage")
|
||||
self.assertEqual([item["text"] for item in items], [" Привет, мир 👋\n", "👋" * 4000])
|
||||
|
||||
def test_message_pagination_and_changing_limit(self):
|
||||
sent = [self.client.message(self.alice, self.chat, f"Сообщение {i}")["json"] for i in range(8)]
|
||||
items = self.read_all(self.path + "/messages", self.bob, "MessagePage", limit=3)
|
||||
self.assertEqual(items, sorted(sent, key=lambda item: (item["created_at"], item["id"])))
|
||||
first = self.client.call("GET", self.path + "/messages?limit=1", actor=self.alice, schema="MessagePage")["json"]
|
||||
query = urlencode({"limit": 100, "cursor": first["next_cursor"]})
|
||||
rest = self.client.call("GET", self.path + "/messages?" + query, actor=self.alice, schema="MessagePage")["json"]
|
||||
self.assertEqual(first["items"] + rest["items"], items)
|
||||
self.assertIsNone(rest["next_cursor"])
|
||||
|
||||
def test_default_page_size(self):
|
||||
for i in range(51):
|
||||
self.client.message(self.alice, self.chat, f"Сообщение {i}")
|
||||
page = self.client.call("GET", self.path + "/messages", actor=self.alice, schema="MessagePage")["json"]
|
||||
self.assertEqual(len(page["items"]), 50)
|
||||
self.assertIsNotNone(page["next_cursor"])
|
||||
|
||||
def test_chat_pagination_and_cursor_owner(self):
|
||||
for i in range(4):
|
||||
self.client.call("POST", "/api/v1/chats", 201, actor=self.alice,
|
||||
body={"title": f"Чат {i}", "member_ids": [self.alice["id"], self.bob["id"]]})
|
||||
chats = self.read_all("/api/v1/chats", self.alice, "ChatPage", limit=2)
|
||||
self.assertEqual(len(chats), 5)
|
||||
self.assertEqual(len({chat["id"] for chat in chats}), 5)
|
||||
self.assertEqual(chats, sorted(chats, key=lambda item: (item["created_at"], item["id"])))
|
||||
first = self.client.call("GET", "/api/v1/chats?limit=1", actor=self.alice, schema="ChatPage")["json"]
|
||||
query = urlencode({"cursor": first["next_cursor"]})
|
||||
self.assert_error("GET", "/api/v1/chats?" + query, 400, "invalid_request", actor=self.bob)
|
||||
|
||||
def test_invalid_limits_and_cursors(self):
|
||||
bad_queries = [{"limit": value} for value in ("0", "101", "-1", "1.5", "1.0", "abc", "")]
|
||||
bad_queries += [{"cursor": value} for value in ("", "not-base64!", "e30=", "W10=")]
|
||||
for path in ("/api/v1/chats", self.path + "/messages"):
|
||||
for query in bad_queries:
|
||||
with self.subTest(path=path, query=query):
|
||||
self.assert_error("GET", path + "?" + urlencode(query), 400, "invalid_request", actor=self.alice)
|
||||
|
||||
def test_cursor_cannot_be_used_for_another_chat_or_changed(self):
|
||||
for i in range(2):
|
||||
self.client.message(self.alice, self.chat, f"Сообщение {i}")
|
||||
first = self.client.call("GET", self.path + "/messages?limit=1", actor=self.alice, schema="MessagePage")["json"]
|
||||
cursor = first["next_cursor"]
|
||||
other = self.client.call("POST", "/api/v1/chats", 201, actor=self.alice, schema="Chat",
|
||||
body={"title": "Другой чат", "member_ids": [self.alice["id"]]})["json"]
|
||||
path = "/api/v1/chats/" + other["id"] + "/messages?" + urlencode({"cursor": cursor})
|
||||
self.assert_error("GET", path, 400, "invalid_request", actor=self.alice)
|
||||
data = json.loads(base64.urlsafe_b64decode(cursor))
|
||||
data["id"] = str(uuid4())
|
||||
changed = base64.urlsafe_b64encode(json.dumps(data).encode()).decode()
|
||||
self.assert_error("GET", self.path + "/messages?" + urlencode({"cursor": changed}),
|
||||
400, "invalid_request", actor=self.alice)
|
||||
|
||||
def test_framework_errors_and_request_ids(self):
|
||||
first = self.assert_error("GET", "/unknown", 404, "not_found")
|
||||
second = self.assert_error("GET", "/unknown", 404, "not_found")
|
||||
self.assertNotEqual(first["request_id"], second["request_id"])
|
||||
self.assert_error("POST", "/health/live", 405, "invalid_request")
|
||||
@@ -0,0 +1,135 @@
|
||||
import hashlib
|
||||
import http.client
|
||||
import io
|
||||
import json
|
||||
import struct
|
||||
from urllib.parse import urlencode, urlsplit
|
||||
from uuid import uuid4
|
||||
import zlib
|
||||
|
||||
from PIL import Image
|
||||
|
||||
from helpers import ApiTestCase, ROOT
|
||||
|
||||
|
||||
class AttachmentTests(ApiTestCase):
|
||||
def upload(self, data=None, filename="photo.png", content_type="image/png", expected=202, actor=None):
|
||||
if data is None:
|
||||
data = (ROOT / "tests/fixtures/sample.png").read_bytes()
|
||||
return self.client.call("POST", self.path + "/attachments?" + urlencode({"filename": filename}),
|
||||
expected, actor=actor or self.alice, raw=data,
|
||||
headers={"Content-Type": content_type}, schema="Attachment" if expected == 202 else None)
|
||||
|
||||
def test_png_and_jpeg_originals(self):
|
||||
jpeg = io.BytesIO()
|
||||
Image.new("RGB", (100, 60), "orange").save(jpeg, format="JPEG")
|
||||
for data, mime, filename in (((ROOT / "tests/fixtures/sample.png").read_bytes(), "image/png", "Фото 👋.png"),
|
||||
(jpeg.getvalue(), "image/jpeg", "photo.jpeg")):
|
||||
with self.subTest(mime=mime):
|
||||
attachment = self.upload(data, filename, mime)["json"]
|
||||
self.assertEqual(attachment["filename"], filename)
|
||||
self.assertEqual(attachment["status"], "queued")
|
||||
attachment = self.client.wait_ready(self.bob, attachment)
|
||||
self.assertEqual({v["name"] for v in attachment["variants"]}, {"thumbnail", "crop", "optimized"})
|
||||
self.assertIsNone(attachment["error_code"])
|
||||
self.assertEqual(attachment["sha256"], hashlib.sha256(data).hexdigest())
|
||||
self.assertEqual(attachment["size_bytes"], len(data))
|
||||
self.assertEqual(self.client.call("GET", "/api/v1/attachments/" + attachment["id"],
|
||||
actor=self.bob, schema="Attachment")["json"], attachment)
|
||||
result = self.client.call("GET", "/api/v1/attachments/" + attachment["id"] + "/content", actor=self.bob)
|
||||
self.assertEqual(result["bytes"], data)
|
||||
self.assertEqual(result["headers"].get_content_type(), mime)
|
||||
self.assertEqual(result["headers"]["X-Content-Type-Options"], "nosniff")
|
||||
|
||||
def test_corruption_and_mime_spoof(self):
|
||||
png = (ROOT / "tests/fixtures/sample.png").read_bytes()
|
||||
for data, mime in ((png, "image/jpeg"), (png[:50], "image/png"),
|
||||
(b"not an image", "image/png"), (b"", "image/png"),
|
||||
(b"GIF89a", "image/png"), (png, "application/octet-stream")):
|
||||
with self.subTest(size=len(data), mime=mime):
|
||||
result = self.upload(data, content_type=mime, expected=415)
|
||||
self.assertEqual(result["json"]["error"]["code"], "unsupported_media_type")
|
||||
|
||||
def test_filename_rejected_before_storage(self):
|
||||
for filename in ("", " ", "../a.png", "a/b.png", "a\\b.png", "a\x00.png", "a\r\n.png", "a\x7f.png", "a" * 256):
|
||||
with self.subTest(filename=repr(filename)):
|
||||
result = self.upload(filename=filename, expected=400)
|
||||
self.assertEqual(result["json"]["error"]["code"], "invalid_request")
|
||||
self.client.call("POST", self.path + "/attachments", 400, actor=self.alice, raw=b"x",
|
||||
headers={"Content-Type": "image/png"})
|
||||
|
||||
def test_content_length_limit_before_body(self):
|
||||
self.client.call("POST", self.path + "/attachments?filename=a.png", 413, actor=self.alice,
|
||||
raw=b"x", headers={"Content-Type": "image/png", "Content-Length": str(10485761)})
|
||||
|
||||
def test_chunked_limit_without_content_length(self):
|
||||
target = urlsplit(self.client.base)
|
||||
connection = http.client.HTTPConnection(target.hostname, target.port, timeout=15)
|
||||
try:
|
||||
connection.request("POST", self.path + "/attachments?filename=large.png",
|
||||
body=iter([b"x" * 65536] * 160 + [b"x"]),
|
||||
headers={"Authorization": "Bearer " + self.alice["token"], "Content-Type": "image/png"},
|
||||
encode_chunked=True)
|
||||
response = connection.getresponse()
|
||||
self.assertEqual(response.status, 413)
|
||||
self.assertEqual(json.loads(response.read())["error"]["code"], "payload_too_large")
|
||||
finally:
|
||||
connection.close()
|
||||
|
||||
def test_pixel_limit_before_decode(self):
|
||||
def chunk(kind, data):
|
||||
return struct.pack(">I", len(data)) + kind + data + struct.pack(">I", zlib.crc32(kind + data))
|
||||
# Размеры в корректном IHDR превышают лимит; pixel data вообще не выделяется.
|
||||
data = b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", struct.pack(">IIBBBBB", 10000, 4001, 8, 2, 0, 0, 0))
|
||||
data += chunk(b"IDAT", b"") + chunk(b"IEND", b"")
|
||||
self.upload(data, expected=413)
|
||||
|
||||
def test_access_and_authentication(self):
|
||||
attachment = self.upload()["json"]
|
||||
for suffix in ("", "/content"):
|
||||
path = "/api/v1/attachments/" + attachment["id"] + suffix
|
||||
self.assert_error("GET", path, 401, "unauthorized")
|
||||
self.assert_error("GET", path, 404, "not_found", actor=self.eve)
|
||||
self.assert_error("GET", "/api/v1/attachments/" + str(uuid4()) + suffix,
|
||||
404, "not_found", actor=self.eve)
|
||||
self.upload(expected=404, actor=self.eve)
|
||||
self.client.call("POST", self.path + "/attachments?filename=../bad", 401,
|
||||
raw=b"x", headers={"Content-Type": "image/png"})
|
||||
self.client.wait_ready(self.alice, attachment)
|
||||
self.assert_error("GET", "/api/v1/attachments/" + attachment["id"] + "/content?variant=missing",
|
||||
404, "not_found", actor=self.alice)
|
||||
|
||||
def test_attachment_only_message_and_other_member(self):
|
||||
attachment = self.upload()["json"]
|
||||
message = self.client.message(self.bob, self.chat, "", attachments=[attachment["id"]])["json"]
|
||||
self.assertEqual(message["attachment_ids"], [attachment["id"]])
|
||||
self.assertEqual(message["sender_id"], self.bob["id"])
|
||||
self.assertEqual(self.read_all(self.path + "/messages", self.alice, "MessagePage"), [message])
|
||||
|
||||
def test_foreign_and_unknown_attachment_ids(self):
|
||||
attachment = self.upload()["json"]
|
||||
other = self.client.call("POST", "/api/v1/chats", 201, actor=self.alice, schema="Chat",
|
||||
body={"title": "Другой чат", "member_ids": [self.alice["id"]]})["json"]
|
||||
for chat_id, ids in ((self.chat["id"], [str(uuid4())]), (other["id"], [attachment["id"]])):
|
||||
self.assert_error("POST", "/api/v1/chats/" + chat_id + "/messages", 404, "not_found", actor=self.alice,
|
||||
body={"text": "", "attachment_ids": ids})
|
||||
|
||||
def test_invalid_attachment_lists(self):
|
||||
attachment = self.upload()["json"]
|
||||
for ids in ([attachment["id"], attachment["id"].upper()], [str(uuid4()) for _ in range(11)],
|
||||
["not-a-uuid"], [True], None, "uuid"):
|
||||
self.assert_error("POST", self.path + "/messages", 400, "invalid_request", actor=self.alice,
|
||||
body={"text": "Текст", "attachment_ids": ids})
|
||||
|
||||
def test_idempotency_includes_ordered_attachment_ids(self):
|
||||
first, second = self.upload()["json"], self.upload()["json"]
|
||||
key = uuid4().hex
|
||||
ids = [first["id"], second["id"]]
|
||||
saved = self.client.message(self.alice, self.chat, "", key=key, attachments=ids)["json"]
|
||||
self.assertEqual(self.client.message(self.alice, self.chat, "", key=key, attachments=ids)["json"], saved)
|
||||
for changed in ([second["id"], first["id"]], [first["id"]]):
|
||||
self.assert_error("POST", self.path + "/messages", 409, "conflict", actor=self.alice,
|
||||
body={"text": "", "attachment_ids": changed}, headers={"Idempotency-Key": key})
|
||||
key = uuid4().hex
|
||||
saved = self.client.message(self.alice, self.chat, "Без файла", key=key)["json"]
|
||||
self.assertEqual(self.client.message(self.alice, self.chat, "Без файла", key=key, attachments=[])["json"], saved)
|
||||
@@ -0,0 +1,47 @@
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
import threading
|
||||
from uuid import uuid4
|
||||
|
||||
from helpers import ApiTestCase, Client
|
||||
|
||||
|
||||
class ConcurrencyTests(ApiTestCase):
|
||||
def test_ten_clients_send_one_hundred_messages(self):
|
||||
barrier = threading.Barrier(10)
|
||||
|
||||
def send_messages(worker):
|
||||
client = Client()
|
||||
actor = self.alice if worker % 2 == 0 else self.bob
|
||||
barrier.wait(timeout=10)
|
||||
sent = []
|
||||
for index in range(10):
|
||||
message = client.message(actor, self.chat, f"parallel-{worker}-{index}")["json"]
|
||||
self.assertEqual(message["sender_id"], actor["id"])
|
||||
sent.append(message)
|
||||
return sent
|
||||
|
||||
with ThreadPoolExecutor(max_workers=10) as pool:
|
||||
batches = list(pool.map(send_messages, range(10)))
|
||||
sent = [message for batch in batches for message in batch]
|
||||
history = self.read_all(self.path + "/messages", self.alice, "MessagePage", limit=17)
|
||||
self.assertEqual(len(history), 100)
|
||||
self.assertEqual(len({message["id"] for message in history}), 100)
|
||||
self.assertEqual(history, sorted(sent, key=lambda item: (item["created_at"], item["id"])))
|
||||
self.assertEqual({message["text"] for message in history},
|
||||
{f"parallel-{worker}-{index}" for worker in range(10) for index in range(10)})
|
||||
print("PASS: 10 clients sent 100 messages; history contains exactly 100 unique messages.")
|
||||
|
||||
def test_simultaneous_registration_of_one_username(self):
|
||||
username = "race_" + uuid4().hex[:20]
|
||||
barrier = threading.Barrier(10)
|
||||
|
||||
def register(worker):
|
||||
client = Client()
|
||||
barrier.wait(timeout=10)
|
||||
return client.call("POST", "/api/v1/users", [201, 409],
|
||||
body={"username": username, "display_name": f"Клиент {worker}", "password": "Synthetic-password-123"})["status"]
|
||||
|
||||
with ThreadPoolExecutor(max_workers=10) as pool:
|
||||
statuses = list(pool.map(register, range(10)))
|
||||
self.assertEqual(statuses.count(201), 1)
|
||||
self.assertEqual(statuses.count(409), 9)
|
||||
@@ -0,0 +1,62 @@
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
import json
|
||||
import threading
|
||||
from uuid import uuid4
|
||||
|
||||
from helpers import ApiTestCase, Client
|
||||
|
||||
|
||||
class IdempotencyTests(ApiTestCase):
|
||||
def test_parallel_replay_across_replicas(self):
|
||||
key = uuid4().hex
|
||||
barrier = threading.Barrier(20)
|
||||
|
||||
def send(_):
|
||||
client = Client()
|
||||
barrier.wait(timeout=10)
|
||||
return client.message(self.alice, self.chat, "Одно сообщение 👋", key=key)
|
||||
|
||||
with ThreadPoolExecutor(max_workers=20) as pool:
|
||||
responses = list(pool.map(send, range(20)))
|
||||
self.assertGreaterEqual(len({r["headers"]["X-Instance-Id"] for r in responses}), 2)
|
||||
self.assertTrue(all(r["json"] == responses[0]["json"] for r in responses))
|
||||
self.assertEqual(self.read_all(self.path + "/messages", self.bob, "MessagePage"), [responses[0]["json"]])
|
||||
|
||||
def test_key_format_and_missing_key(self):
|
||||
for key in (None, "", " ", "a b", "a/b", "a" * 129):
|
||||
with self.subTest(key=key):
|
||||
headers = {} if key is None else {"Idempotency-Key": key}
|
||||
self.client.call("POST", self.path + "/messages", 400, actor=self.alice,
|
||||
body={"text": "Текст"}, headers=headers)
|
||||
for key in ("A", "a" * 128, "ABCdef012._:-"):
|
||||
self.client.message(self.alice, self.chat, key, key=key)
|
||||
|
||||
def test_conflict_and_semantic_json_replay(self):
|
||||
key = uuid4().hex
|
||||
first = self.client.message(self.alice, self.chat, "Текст 👋", key=key)["json"]
|
||||
replay = self.client.call("POST", self.path + "/messages", 201, actor=self.alice,
|
||||
raw=json.dumps({"text": "Текст 👋"}, ensure_ascii=True, indent=2).encode(),
|
||||
headers={"Content-Type": "application/json", "Idempotency-Key": key},
|
||||
schema="Message")["json"]
|
||||
self.assertEqual(first, replay)
|
||||
self.assert_error("POST", self.path + "/messages", 409, "conflict", actor=self.alice,
|
||||
body={"text": "Другой"}, headers={"Idempotency-Key": key})
|
||||
self.assertEqual(self.read_all(self.path + "/messages", self.bob, "MessagePage"), [first])
|
||||
|
||||
def test_scope_is_user_and_chat(self):
|
||||
key = uuid4().hex
|
||||
first = self.client.message(self.alice, self.chat, "Алиса", key=key)["json"]
|
||||
second = self.client.message(self.bob, self.chat, "Боб", key=key)["json"]
|
||||
other = self.client.call("POST", "/api/v1/chats", 201, actor=self.alice, schema="Chat",
|
||||
body={"title": "Другой чат", "member_ids": [self.alice["id"]]})["json"]
|
||||
third = self.client.message(self.alice, other, "Другой чат", key=key)["json"]
|
||||
self.assertEqual(len({first["id"], second["id"], third["id"]}), 3)
|
||||
|
||||
def test_replay_rechecks_session_and_access(self):
|
||||
key = uuid4().hex
|
||||
self.client.message(self.alice, self.chat, "Текст", key=key)
|
||||
self.client.call("POST", self.path + "/messages", 404, actor=self.eve,
|
||||
body={"text": "Текст"}, headers={"Idempotency-Key": key})
|
||||
self.client.call("DELETE", "/api/v1/auth/sessions/current", 204, actor=self.alice)
|
||||
self.client.call("POST", self.path + "/messages", 401, actor=self.alice,
|
||||
body={"text": "Текст"}, headers={"Idempotency-Key": key})
|
||||
@@ -0,0 +1,51 @@
|
||||
import base64
|
||||
from uuid import uuid4
|
||||
|
||||
from helpers import ApiTestCase
|
||||
|
||||
|
||||
class SessionTests(ApiTestCase):
|
||||
def basic(self, username, password):
|
||||
value = base64.b64encode((username + ":" + password).encode()).decode()
|
||||
return {"Authorization": "Basic " + value}
|
||||
|
||||
def test_wrong_password_and_basic_only_for_login(self):
|
||||
registration = self.alice["registration"]
|
||||
self.assert_error("POST", "/api/v1/auth/sessions", 401, "unauthorized",
|
||||
headers=self.basic(registration["username"], "wrong-password"))
|
||||
self.assert_error("POST", "/api/v1/auth/sessions", 401, "unauthorized")
|
||||
self.assert_error("POST", "/api/v1/auth/sessions", 401, "unauthorized",
|
||||
headers={"Authorization": "Basic not-base64!"})
|
||||
self.assert_error("GET", "/api/v1/users/me", 401, "unauthorized",
|
||||
headers=self.basic(registration["username"], registration["password"]))
|
||||
|
||||
def test_logout_revokes_only_one_session(self):
|
||||
registration = self.alice["registration"]
|
||||
second = self.client.call("POST", "/api/v1/auth/sessions", 201, schema="Session",
|
||||
headers=self.basic(registration["username"], registration["password"]))["json"]
|
||||
self.client.call("DELETE", "/api/v1/auth/sessions/current", 204, actor=self.alice)
|
||||
self.assert_error("GET", "/api/v1/users/me", 401, "unauthorized", actor=self.alice)
|
||||
self.assert_error("DELETE", "/api/v1/auth/sessions/current", 401, "unauthorized", actor=self.alice)
|
||||
self.client.call("GET", "/api/v1/users/me", schema="User", headers={"Authorization": "Bearer " + second["token"]})
|
||||
|
||||
def test_password_validation(self):
|
||||
for password in ("short", "x" * 129, 123, None, " " * 12):
|
||||
body = {"username": "pw_" + uuid4().hex[:20], "display_name": "Пароль", "password": password}
|
||||
self.assert_error("POST", "/api/v1/users", 400, "invalid_request", body=body)
|
||||
|
||||
def test_unicode_password_with_colon_and_private_response(self):
|
||||
username = "pw_" + uuid4().hex[:20]
|
||||
password = "Пароль:с:двоеточием-123"
|
||||
user = self.client.call("POST", "/api/v1/users", 201, schema="User",
|
||||
body={"username": username, "display_name": "Пароль", "password": password})["json"]
|
||||
session = self.client.call("POST", "/api/v1/auth/sessions", 201, schema="Session",
|
||||
headers=self.basic(username, password))["json"]
|
||||
self.assertEqual(session["user"], user)
|
||||
self.assertEqual(set(user), {"id", "username", "display_name", "created_at"})
|
||||
self.assertGreaterEqual(len(session["token"]), 32)
|
||||
|
||||
def test_sql_like_message_is_plain_text(self):
|
||||
text = "'; DROP TABLE messages; -- Привет"
|
||||
message = self.client.message(self.alice, self.chat, text)["json"]
|
||||
self.assertEqual(message["text"], text)
|
||||
self.assertEqual(self.read_all(self.path + "/messages", self.bob, "MessagePage"), [message])
|
||||
@@ -0,0 +1,88 @@
|
||||
"""Локальные проверки реальных subprocess без Docker и внешних библиотек.
|
||||
|
||||
python -m unittest discover -s student-tests -p test_task_supervisor.py -v
|
||||
"""
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
from tempfile import TemporaryDirectory
|
||||
import time
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
from src import task_supervisor
|
||||
|
||||
|
||||
class TaskSupervisorTests(unittest.TestCase):
|
||||
def test_successful_work_outlives_initial_lease(self):
|
||||
lease = 0.5
|
||||
expires = time.monotonic() + lease
|
||||
renewals = []
|
||||
|
||||
def renew():
|
||||
nonlocal expires
|
||||
now = time.monotonic()
|
||||
self.assertLess(now, expires, 'renewal must happen before lease expires')
|
||||
expires = now + lease
|
||||
renewals.append(now)
|
||||
return True
|
||||
|
||||
started = time.monotonic()
|
||||
code = task_supervisor.run([sys.executable, '-c', 'import time; time.sleep(1.1)'],
|
||||
renew, timeout=5, interval=0.1)
|
||||
self.assertEqual(code, 0)
|
||||
self.assertGreater(time.monotonic() - started, lease)
|
||||
self.assertGreaterEqual(len(renewals), 3)
|
||||
|
||||
def test_heartbeat_does_not_bypass_total_timeout(self):
|
||||
renewals = []
|
||||
|
||||
def renew():
|
||||
renewals.append(True)
|
||||
return True
|
||||
|
||||
started = time.monotonic()
|
||||
with self.assertRaises(subprocess.TimeoutExpired):
|
||||
task_supervisor.run([sys.executable, '-c', 'import time; time.sleep(60)'],
|
||||
renew, timeout=0.5, interval=0.1)
|
||||
self.assertTrue(renewals)
|
||||
self.assertLess(time.monotonic() - started, 5)
|
||||
|
||||
def test_lost_lease_stops_work(self):
|
||||
with self.assertRaises(task_supervisor.LeaseLost):
|
||||
task_supervisor.run([sys.executable, '-c', 'import time; time.sleep(60)'],
|
||||
lambda: False, timeout=10, interval=0.1)
|
||||
|
||||
def test_renewal_error_stops_work(self):
|
||||
def renew():
|
||||
raise ConnectionError('database unavailable')
|
||||
|
||||
with self.assertRaises(ConnectionError):
|
||||
task_supervisor.run([sys.executable, '-c', 'import time; time.sleep(60)'],
|
||||
renew, timeout=10, interval=0.1)
|
||||
|
||||
def test_child_failure_is_not_success(self):
|
||||
self.assertEqual(task_supervisor.run([sys.executable, '-c', 'raise SystemExit(86)'],
|
||||
lambda: True, timeout=5, interval=0.1), 86)
|
||||
|
||||
def test_timeout_kills_descendant_too(self):
|
||||
with TemporaryDirectory() as directory:
|
||||
marker = Path(directory) / 'survived'
|
||||
started = Path(directory) / 'started'
|
||||
descendant = ('import time; from pathlib import Path; '
|
||||
f'Path({str(started)!r}).write_text("started"); '
|
||||
'time.sleep(1.5); '
|
||||
f'Path({str(marker)!r}).write_text("orphan")')
|
||||
parent = ('import subprocess,sys,time; '
|
||||
f'subprocess.Popen([sys.executable,"-c",{descendant!r}]); '
|
||||
'time.sleep(60)')
|
||||
with self.assertRaises(subprocess.TimeoutExpired):
|
||||
task_supervisor.run([sys.executable, '-c', parent], lambda: True,
|
||||
timeout=0.8, interval=0.1)
|
||||
self.assertTrue(started.exists(), 'descendant must start before timeout')
|
||||
time.sleep(1.6)
|
||||
self.assertFalse(marker.exists(), 'descendant survived parent timeout')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -0,0 +1,36 @@
|
||||
"""Публичный persistence + доказательство реального restart каждого постоянного сервиса."""
|
||||
from datetime import datetime, timezone
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def verify(env):
|
||||
def run(*args):
|
||||
return subprocess.check_output(args, cwd=ROOT, env=env, text=True, timeout=180).strip()
|
||||
def snapshot():
|
||||
ids = run('docker', 'compose', 'ps', '-q').split()
|
||||
infos = json.loads(run('docker', 'inspect', *ids))
|
||||
return {info['Id']: {'service': info['Config']['Labels']['com.docker.compose.service'],
|
||||
'started_at': info['State']['StartedAt'],
|
||||
'volumes': sorted(m['Name'] for m in info['Mounts'] if m['Type'] == 'volume')}
|
||||
for info in infos}
|
||||
before = snapshot()
|
||||
print(run('make', 'test-persistence', 'PYTHON=' + sys.executable, 'ACTION_SCRIPT=scripts/restart.sh'), flush=True)
|
||||
after = snapshot()
|
||||
assert before.keys() == after.keys()
|
||||
assert all(before[key]['started_at'] != after[key]['started_at'] for key in before)
|
||||
assert all(before[key]['volumes'] == after[key]['volumes'] for key in before)
|
||||
required = {'api', 'lb', 'postgres', 'valkey', 'broker', 's3', 's3-storage', 'cleanup', 'publisher', 'worker'}
|
||||
assert required == {value['service'] for value in after.values()}
|
||||
return {'passed': True, 'before': before, 'after': after, 'same_containers_and_volumes': True}
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
result = {'utc': datetime.now(timezone.utc).isoformat(), 'command': '.venv/bin/python student-tests/verify_restart.py', **verify(dict(os.environ))}
|
||||
(ROOT / 'evidence/lab5-restart.json').write_text(json.dumps(result, indent=2) + '\n')
|
||||
print('PASS: all persistent services actually restarted; container IDs and volumes preserved')
|
||||
Reference in New Issue
Block a user