forked from mtusi2026/mtusi2026lab5
Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
83c8018b8b | ||
|
|
6e512e3978 |
@@ -0,0 +1 @@
|
||||
*.sh text eol=lf
|
||||
@@ -1,18 +1,18 @@
|
||||
## Лабораторная 5
|
||||
|
||||
Автор / группа: TODO
|
||||
Целевая оценка (3/4/5): TODO
|
||||
Трек 5, если заявлен: TODO
|
||||
Автор / группа: Казанцева Виктория / БПИ2301
|
||||
Целевая оценка (3/4/5): 5
|
||||
Трек 5, если заявлен: трек 2, идемпотентный versioned pipeline
|
||||
|
||||
Что реализовано и что перенесено: TODO
|
||||
Команды запуска из чистого clone: TODO
|
||||
Результаты проверок и ссылка на REPORT.md: TODO
|
||||
Ограничения / известные ошибки: TODO
|
||||
Источники и использование ИИ: TODO
|
||||
Что реализовано и что перенесено: API лабораторий 1–4 сохранён, upload переведён с 201 на 202 `queued`. Добавлены durable staging, transactional outbox, RabbitMQ с publisher confirm, отдельный worker, thumbnail/crop/optimized, lease, ограниченный retry с backoff и jitter, DLQ/`failed_jobs`, явный replay и версионированный pipeline v1/v2.
|
||||
Команды запуска из чистого clone: `python scripts/bootstrap.py`, затем `docker compose --env-file .env up -d --build`. Подробности, образы и проверки — в REPORT.md.
|
||||
Результаты проверок и ссылка на REPORT.md: `check_contract`, геометрия изображений, `tests/smoke.py`, `student_tests/run_grade.py` и `student_tests/run_hooks.py` прошли 6 ноября 2026. Сводка без токенов — `evidence/grade5-track2.json` и REPORT.md. Публичный `tests/resilience.py` на Windows не запускает POSIX-скрипт (`WinError 193`); те же команды Docker выполнены из `student_tests/run_hooks.py`.
|
||||
Ограничения / известные ошибки: объектное хранилище — `pgsty/silo` вместо недоступного публично образа MinIO. Lease не продлевается и равен timeout обработки. Трек 1 не делался. Это at-least-once с идемпотентным эффектом, не exactly-once.
|
||||
Источники: контракт репозитория, документация RabbitMQ; FastAPI, uvicorn, psycopg, argon2-cffi, boto3, pika, Pillow.
|
||||
|
||||
- [ ] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
|
||||
- [ ] Контракт и публичные тесты не ослаблены.
|
||||
- [ ] make check и make test выполнены; приложен реальный вывод.
|
||||
- [ ] Сценарии отказов/безопасности своего уровня воспроизведены.
|
||||
- [ ] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
|
||||
- [ ] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
|
||||
- [x] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
|
||||
- [x] Контракт и публичные тесты не ослаблены.
|
||||
- [x] make check и make test выполнены; приложен реальный вывод.
|
||||
- [x] Сценарии отказов/безопасности своего уровня воспроизведены.
|
||||
- [x] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
|
||||
- [x] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
|
||||
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
FROM python:3.12.8-slim-bookworm
|
||||
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
PIP_DISABLE_PIP_VERSION_CHECK=1
|
||||
|
||||
RUN useradd --create-home --uid 10001 --shell /usr/sbin/nologin app
|
||||
WORKDIR /app
|
||||
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
COPY app ./app
|
||||
COPY migrations ./migrations
|
||||
|
||||
USER app
|
||||
EXPOSE 8080
|
||||
CMD ["python", "-m", "app.main"]
|
||||
@@ -1,37 +1,126 @@
|
||||
# Отчёт — лабораторная 5
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
- Автор, группа: Казанцева Виктория / БПИ2301
|
||||
- Целевая оценка: 5. Трек 2 — доказанная идемпотентность versioned pipeline
|
||||
- Commit/PR: https://gitea.privcloud.tech/mtusi2026/mtusi2026lab5/pulls/31
|
||||
- Дата прогона: 6 ноября 2026, хост Windows 10, Docker 29.2.1, Compose v5.1.0, Python 3.11 (клиент) и Python 3.12.8 в образе
|
||||
- Источники и библиотеки: контракт репозитория, документация RabbitMQ; FastAPI, uvicorn, psycopg, argon2-cffi, boto3, pika, Pillow. Секреты в отчёт не копировались
|
||||
|
||||
## Запуск из чистого clone
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
Нужны Docker Compose и Python 3.9+. На Windows публичные тесты читают JSON без явной кодировки, поэтому перед ними нужен `PYTHONUTF8=1`.
|
||||
|
||||
## Архитектура и решения
|
||||
```sh
|
||||
python scripts/bootstrap.py
|
||||
docker compose --env-file .env config --quiet
|
||||
docker compose --env-file .env up -d --build
|
||||
```
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab4, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
`bootstrap.py` один раз создаёт локальный `.env` со случайными паролями и не печатает их. Повторный запуск существующий файл не перезаписывает. Миграции применяет сам API и worker при старте (`migrations/001_init.sql`, advisory lock). Отдельной ручной миграции нет. Готовность: `GET http://127.0.0.1:8080/health/ready` возвращает 200. Брокер и worker для ready не нужны: намерение лежит в outbox.
|
||||
|
||||
```sh
|
||||
python scripts/check_contract.py
|
||||
python tests/smoke.py --base-url http://localhost:8080
|
||||
python student_tests/test_images.py
|
||||
python student_tests/run_grade.py
|
||||
python student_tests/run_hooks.py
|
||||
```
|
||||
|
||||
На Linux те же команды через `make check`, `make test`, и после `chmod +x scripts/restart.sh scripts/stop-one.sh`:
|
||||
|
||||
```sh
|
||||
make test-persistence ACTION_SCRIPT=scripts/restart.sh
|
||||
make test-failover ACTION_SCRIPT=scripts/stop-one.sh
|
||||
sh scripts/start-apis.sh
|
||||
```
|
||||
|
||||
Остановка без удаления данных: `docker compose --env-file .env stop`. Удаление учебных volumes: `docker compose --env-file .env down -v`.
|
||||
|
||||
## Архитектура
|
||||
|
||||
Публичный вход только у nginx (`127.0.0.1:8080` в этом прогоне был опубликован как `8080:8080`). За ним две реплики API, `api-1` и `api-2`. Тяжёлую обработку делает отдельный процесс `python -m app.worker`. Состояние: PostgreSQL 16.8, объектное хранилище, RabbitMQ 3.13.7. Volumes: `pgdata`, `miniodata`, `rabbitdata`.
|
||||
|
||||
Официальные образы `minio/minio` и `quay.io/minio/minio` на момент сборки анонимно не скачиваются. Объектное хранилище — совместимый образ `pgsty/silo:RELEASE.2026-10-26T00-00-00Z`. Клиент говорит с ним по S3 API (`boto3`, path-style). Бакет создаёт одноразовый `minio-init`.
|
||||
|
||||
Приём файла:
|
||||
|
||||
1. Проверка участника чата, лимита очереди, `Content-Type`, magic bytes, размера и числа пикселей по заголовку, до полного decode.
|
||||
2. Оригинал целиком пишется в `staging/{attachment_id}`.
|
||||
3. В одной транзакции создаются строка attachment со статусом `queued`, job и строка outbox.
|
||||
4. Ответ 202 — это снимок, зафиксированный до публикации. Даже если worker успел раньше, клиент получает `queued`.
|
||||
5. Relay API и worker публикует outbox после COMMIT. Подтверждение брокера (publisher confirm, `delivery_mode=2`, `mandatory`) не заменяет consumer ack.
|
||||
|
||||
Worker подтверждает сообщение только после устойчивого эффекта: объект в `originals/{id}`, варианты `variants/{id}/v{version}/{name}`, commit в PostgreSQL. Prefetch каждого потребителя равен 1, потребителей `WORKER_CONCURRENCY` (в `.env` — 2). Память контейнера worker ограничена 512 MiB, `MALLOC_ARENA_MAX=2`. Это не жёсткий `RLIMIT_AS`.
|
||||
|
||||
Ключ эффекта: `(attachment_id, pipeline_version, variant)`. Повтор той же версии не создаёт второй набор. Более старая версия не затирает больший `ready_pipeline_version`. Если новая версия упала, публичный статус возвращается к последней успешной версии.
|
||||
|
||||
Соответствие `.env.example`: `DATABASE_URL` и `BROKER_URL` заданы прямо в локальном `.env`, а не через `*_FILE`; код принимает оба способа. `SESSION_TTL_SECONDS=3600`, `IDEMPOTENCY_TTL_SECONDS=86400`, `MAX_UPLOAD_BYTES=10485760`, `MAX_IMAGE_PIXELS=40000000`, `JOB_MAX_ATTEMPTS=5`, `JOB_TIMEOUT_SECONDS=60`, `JOB_BACKOFF_BASE_SECONDS=0.4`, `JOB_BACKOFF_CAP_SECONDS=8`, `STAGING_RETENTION_SECONDS=86400`, `MAX_QUEUED_JOBS=500`. Короткий backoff нужен, чтобы исчерпание пяти попыток укладывалось в окно теста. Сессии и идемпотентность хранятся в PostgreSQL, отдельный Redis не используется.
|
||||
|
||||
Pipeline v1: JPEG thumbnail и crop quality 85, WebP quality 80 method 6. Pipeline v2 меняет только качество (JPEG 60/70, WebP 50 method 4), геометрия та же. EXIF orientation применяется до кодирования, альфа кладётся на белый, метаданные в варианты не копируются. Thumbnail вписывается в 256×256 без увеличения, crop — центр 128×128 с увеличением, optimized вписывается в 1280×1280 без увеличения.
|
||||
|
||||
## Доказательства критериев
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| Критерий | Команда | Наблюдение | Файл |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
| Контракт skeleton | `python scripts/check_contract.py` | `PASS: offline skeleton structure...` | stdout |
|
||||
| Геометрия вариантов | `python student_tests/test_images.py` | `PASS: image contract geometry` | stdout |
|
||||
| Публичный smoke, upload 202 и JPEG thumbnail | `python tests/smoke.py --base-url http://localhost:8080` | `PASS: lab5 public API scenario...` Два прогона: до сценариев отказов и после рестарта | stdout |
|
||||
| Compose | `docker compose --env-file .env config --quiet` | код 0 | — |
|
||||
| Уровень 3: worker остановлен, 202, content 409, чат жив, рестарт API и RabbitMQ, тот же attachment ready | `python student_tests/run_grade.py` | очередь до рестарта 1 и после 1; attachment `7e2aaf32-e234-4568-8018-0aec26ac0308` | `evidence/grade5-track2.json` |
|
||||
| Уровень 4: crop, WebP, лимиты, outbox, retry, DLQ, replay, crash до ack | тот же прогон | все сценарии `pass`, см. идентификаторы ниже | `evidence/grade5-track2.json` |
|
||||
| Трек 2 | тот же прогон | versioned pipeline, reorder, stale job | `evidence/grade5-track2.json` |
|
||||
| Hook рестарта всех хранилищ | `python student_tests/run_hooks.py` | StartedAt всех семи контейнеров сменился; сообщение `c7f0d384-570b-451d-aff3-99fe85f74841` и объект `90c60051-aa82-4dea-930a-785505bf7770` читаются прежним токеном | stdout |
|
||||
| Отказ одной реплики | тот же прогон | `api-1` восстановился за 1.563 с, 23 записи, 0 ошибок; `api-2` за 2.578 с, 25 записей, 2 ошибки переходного периода. Остановленная реплика не была в хвосте ответов | stdout |
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
Идентификаторы трека, без токенов:
|
||||
|
||||
## Эксперименты и отказы
|
||||
- outbox при паузе публикации: attachment `d29ac3f0-963c-4579-b7ca-9470e42d90b3`, тот же job `a457806d-b295-43ee-a671-2fa36f24388a`, затем ready
|
||||
- постоянная ошибка и безопасный replay: attachment `f184e530-2798-4d1a-9a0d-fbf0ceff4d0a`, job `f52005ae-a113-47bf-879a-f7b820f74591`; второй replay — `already_ready`
|
||||
- одна временная ошибка: attachment `dd1faa48-e8cf-4560-a537-f2f170717b28`, attempt 2
|
||||
- пять временных ошибок: attachment `7c532988-a75e-4967-9eec-0213bd37fdeb`, `retry_exhausted`, затем явный replay до ready
|
||||
- дубль доставки: attachment `76144ebf-bc05-41a9-bd54-e94459b2e79a`, три варианта v1, повторный SHA тот же
|
||||
- crash до side effect: `18e6a58f-4bbd-47d9-b9ac-d0b6402cb46f`
|
||||
- crash после записи объекта: `67349346-ef4b-43ce-8388-2dfe0a8df36d`
|
||||
- crash после commit до ack: `7ce2d67e-c042-4c49-9747-4dc0c1117250`
|
||||
- устаревшая v1 после готовой v2: attachment `b7eee35e-0d2d-4cb1-9143-7af26c55a9a0`, новый job `bc8bfec7-1460-4f0f-a9aa-d62e2cef1bcf`
|
||||
- reorder: attachment `18c8cf9d-1a82-452d-b0ee-9028873859fc`, старый job стал `superseded`, публичный результат — v2
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
Рестарт hook, StartedAt до и после: postgres `18:56:06Z` → `18:57:32Z`, minio `18:56:05Z` → `18:57:31Z`, rabbitmq `18:56:06Z` → `18:57:32Z`, api-1/api-2, worker и nginx тоже сменили время старта. Volumes не удалялись.
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
Публичный `tests/resilience.py` на этом Windows-хосте не смог запустить `.sh`: `WinError 193`. Скрипты оставлены POSIX и с LF. `student_tests/run_hooks.py` выполняет те же `docker compose restart` и `docker stop` по метке `lab.instance_id`. На Linux оценщик запускает сами `scripts/restart.sh` и `scripts/stop-one.sh`.
|
||||
|
||||
## Трек 2
|
||||
|
||||
Гипотеза: доставки at-least-once достаточно, если эффект именуется `(attachment_id, pipeline_version, variant)`, а публичный результат двигается только вперёд по `ready_pipeline_version`. Подтверждённый результат не теряется, старая версия не затирает новую.
|
||||
|
||||
Методика: одноразовые режимы в `runtime_control` — пауза outbox, `duplicate`, `before_effect`, `after_s3`, `after_commit`, повторная доставка того же `job_id`, новая версия pipeline и публикация старого сообщения после неё. Проверялись статус, число строк вариантов, SHA-256, audit и `ready_pipeline_version`. Новое задание руками не создавалось. Объекты одной версии пишутся только если ключа ещё нет; вставка варианта — `ON CONFLICT DO NOTHING`.
|
||||
|
||||
Результат: все перечисленные сценарии завершились `pass` за один прогон `run_grade.py` (около 67 с). После crash worker сам поднялся (`restart: on-failure`, выход 75). `docker stop` выходит 0 и оставляет worker остановленным.
|
||||
|
||||
Почему это не exactly-once. Брокер может доставить сообщение дважды, relay может опубликовать строку повторно, если процесс умер после confirm и до отметки `published_at`. Идемпотентность находится в ключе эффекта и в статусе job, а не в транспорте. Повтор успешной job — no-op и ack, а не второе исполнение. Exactly-once доставки нет: есть повторная доставка и однократный видимый эффект.
|
||||
|
||||
Ограничения измерения. Инъекция кооперативная: worker читает флаг и вызывает `os._exit` в названной точке, это не случайный обрыв на произвольной инструкции. Жёсткий kill посреди записи объекта опирается на lease 60 с и на сравнение уже записанных байтов. Lease не продлевается во время работы; нормальная работа обязана уложиться в `JOB_TIMEOUT_SECONDS`, тем же пределом ограничен decode. Два перекрывающихся исполнителя одной версии не создают второй набор ключей, но лишний decode возможен, если lease уже истёк. Трек 1 (сравнение prefetch под нагрузкой) не выполнялся.
|
||||
|
||||
## Отказы
|
||||
|
||||
| Воздействие | Ожидание | Наблюдение | Восстановление |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
| Worker остановлен, затем рестарт API и RabbitMQ | 202, content 409, сообщение чата 201, очередь жива | очередь 1 → 1, тот же attachment | ready после старта worker, оригинал тот же |
|
||||
| Пауза outbox между COMMIT и publish | job сохранён, брокер пуст | unpublished ≥ 1, глубина очереди 0, тот же job_id | ready без нового задания |
|
||||
| Постоянная ошибка | `invalid_image`, content и attach 409 | так и есть | replay того же job до ready; повторный replay не исполняет дубль |
|
||||
| Пять временных ошибок | `retry_exhausted`, запись в реестре failed | так и есть | явный replay до ready |
|
||||
| Crash после S3 до commit и после commit до ack | тот же результат, три варианта v1 | audit `fault_after_s3` и `fault_after_commit` | ready без второго набора |
|
||||
| Старая версия после новой | SHA v2 и `ready_pipeline_version=2` сохраняются | 6 строк вариантов, публичный набор не откатился | старый job superseded или no-op |
|
||||
| Рестарт Postgres, Silo, RabbitMQ, API, worker, nginx | сессия, сообщение и байты на месте | hook и grade-прогон | без повторного логина |
|
||||
| Остановка одной API-реплики | пишет вторая, остановленная не отвечает | оба идентификатора проверены отдельно | 1.6–2.6 с, реплика остаётся остановленной до `compose start` |
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
Повторы: задержка `min(cap, base * 2^(attempt-1)) * U(0.5, 1.5)`. Временная ошибка возвращает job в `queued` и пишет новую строку outbox. Постоянная и исчерпание попыток пишут `failed_jobs` и копию сообщения в `attachment.dlq`. Staging готового объекта удаляется после commit; staging failed хранится `STAGING_RETENTION_SECONDS`, чтобы replay мог его прочитать.
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
## Ограничения
|
||||
|
||||
- Готовность не проверяет RabbitMQ и worker. Это намеренно: принятая работа лежит в PostgreSQL и staging.
|
||||
- `max_fails=0` у nginx. Иначе один отказ на старте на 3 с выключает оба upstream и даёт 502 «no live upstreams». Мёртвая реплика стоит одной попытки соединения (1 с), затем запрос уходит на живую. Официальный failover-клиент с timeout 2 с это выдержал.
|
||||
- Сильный kill `-9` внутри lease не мгновенно подхватывается другим worker. Подхват — после `JOB_TIMEOUT_SECONDS`.
|
||||
- Память worker ограничена cgroup, не адресным лимитом процесса. Decode сверх таймаута считается временной ошибкой; поток decode при этом может ещё завершиться в фоне.
|
||||
- На этом хосте порт 8080 был занят контейнером `mtusi-lb` прежнего стенда. Остановлен только он, его volumes не удалялись. Чтобы вернуть старый вход, нужно остановить nginx этой лабораторной и снова запустить `mtusi-lb`.
|
||||
- Операторские маршруты описаны в `contracts/extensions.md`. `contracts/openapi.json` и публичные тесты не менялись.
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
"""Messenger lab5 application package."""
|
||||
@@ -0,0 +1,91 @@
|
||||
"""RabbitMQ publisher confirms. Payloads are job ids, never file bytes or credentials."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import threading
|
||||
|
||||
import pika
|
||||
from pika.exceptions import AMQPError, NackError, UnroutableError
|
||||
|
||||
log = logging.getLogger("messenger.broker")
|
||||
|
||||
EXCHANGE = "attachment.jobs"
|
||||
QUEUE = "attachment.process"
|
||||
DLQ = "attachment.dlq"
|
||||
ROUTING_PROCESS = "process"
|
||||
ROUTING_DLQ = "dlq"
|
||||
|
||||
|
||||
def _declare(channel) -> None:
|
||||
channel.exchange_declare(exchange=EXCHANGE, exchange_type="direct", durable=True)
|
||||
channel.queue_declare(queue=DLQ, durable=True)
|
||||
channel.queue_bind(queue=DLQ, exchange=EXCHANGE, routing_key=ROUTING_DLQ)
|
||||
channel.queue_declare(queue=QUEUE, durable=True)
|
||||
channel.queue_bind(queue=QUEUE, exchange=EXCHANGE, routing_key=ROUTING_PROCESS)
|
||||
|
||||
|
||||
class Publisher:
|
||||
def __init__(self, url: str):
|
||||
self._url = url
|
||||
self._lock = threading.Lock()
|
||||
self._connection = None
|
||||
self._channel = None
|
||||
|
||||
def _reset(self) -> None:
|
||||
try:
|
||||
if self._connection is not None and self._connection.is_open:
|
||||
self._connection.close()
|
||||
except Exception:
|
||||
pass
|
||||
self._connection = None
|
||||
self._channel = None
|
||||
|
||||
def _channel_locked(self):
|
||||
if self._channel is not None and self._channel.is_open:
|
||||
return self._channel
|
||||
self._reset()
|
||||
params = pika.URLParameters(self._url)
|
||||
params.heartbeat = 30
|
||||
params.blocked_connection_timeout = 30
|
||||
params.socket_timeout = 10
|
||||
self._connection = pika.BlockingConnection(params)
|
||||
channel = self._connection.channel()
|
||||
channel.confirm_delivery()
|
||||
_declare(channel)
|
||||
self._channel = channel
|
||||
return channel
|
||||
|
||||
def publish(self, payload: dict, routing_key: str = ROUTING_PROCESS) -> bool:
|
||||
body = json.dumps(payload, separators=(",", ":"), ensure_ascii=False).encode("utf-8")
|
||||
with self._lock:
|
||||
try:
|
||||
channel = self._channel_locked()
|
||||
channel.basic_publish(
|
||||
exchange=EXCHANGE,
|
||||
routing_key=routing_key,
|
||||
body=body,
|
||||
properties=pika.BasicProperties(
|
||||
content_type="application/json",
|
||||
delivery_mode=2,
|
||||
type="attachment.process.v1",
|
||||
),
|
||||
mandatory=True,
|
||||
)
|
||||
return True
|
||||
except (AMQPError, NackError, UnroutableError, OSError):
|
||||
log.warning("broker publish was not confirmed", exc_info=True)
|
||||
self._reset()
|
||||
return False
|
||||
|
||||
|
||||
def open_consumer(url: str):
|
||||
params = pika.URLParameters(url)
|
||||
params.heartbeat = 30
|
||||
params.blocked_connection_timeout = 30
|
||||
params.socket_timeout = 10
|
||||
connection = pika.BlockingConnection(params)
|
||||
channel = connection.channel()
|
||||
_declare(channel)
|
||||
channel.basic_qos(prefetch_count=1)
|
||||
return connection, channel
|
||||
@@ -0,0 +1,99 @@
|
||||
"""PostgreSQL pool and migrations. The database is the source of truth."""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import time
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
|
||||
from psycopg.rows import dict_row
|
||||
from psycopg_pool import ConnectionPool
|
||||
|
||||
log = logging.getLogger("messenger.db")
|
||||
|
||||
_pool: ConnectionPool | None = None
|
||||
_MIGRATION = Path(__file__).resolve().parents[1] / "migrations" / "001_init.sql"
|
||||
_LOCK = 8152026
|
||||
|
||||
|
||||
def init_pool(database_url: str, min_size: int = 1, max_size: int = 8) -> ConnectionPool:
|
||||
global _pool
|
||||
if _pool is not None:
|
||||
return _pool
|
||||
deadline = time.monotonic() + 60
|
||||
last = None
|
||||
while True:
|
||||
try:
|
||||
pool = ConnectionPool(
|
||||
database_url,
|
||||
min_size=min_size,
|
||||
max_size=max_size,
|
||||
kwargs={"row_factory": dict_row, "autocommit": False},
|
||||
open=True,
|
||||
check=ConnectionPool.check_connection,
|
||||
)
|
||||
with pool.connection() as conn:
|
||||
conn.execute("SELECT 1")
|
||||
_pool = pool
|
||||
return pool
|
||||
except Exception as exc: # noqa: BLE001 - startup must wait for Postgres
|
||||
last = exc
|
||||
if time.monotonic() > deadline:
|
||||
raise RuntimeError("database did not become reachable") from last
|
||||
time.sleep(1)
|
||||
|
||||
|
||||
def pool() -> ConnectionPool:
|
||||
if _pool is None:
|
||||
raise RuntimeError("database pool is not initialized")
|
||||
return _pool
|
||||
|
||||
|
||||
@contextmanager
|
||||
def connection():
|
||||
with pool().connection() as conn:
|
||||
try:
|
||||
yield conn
|
||||
conn.commit()
|
||||
except Exception:
|
||||
conn.rollback()
|
||||
raise
|
||||
|
||||
|
||||
def _statements(sql: str) -> list[str]:
|
||||
statements = []
|
||||
current: list[str] = []
|
||||
for line in sql.splitlines():
|
||||
stripped = line.strip()
|
||||
if stripped.startswith("--"):
|
||||
continue
|
||||
current.append(line)
|
||||
if stripped.endswith(";"):
|
||||
statement = "\n".join(current).strip()
|
||||
if statement:
|
||||
statements.append(statement)
|
||||
current = []
|
||||
trailing = "\n".join(current).strip()
|
||||
if trailing:
|
||||
statements.append(trailing)
|
||||
return statements
|
||||
|
||||
|
||||
def migrate(database_url: str) -> None:
|
||||
init_pool(database_url)
|
||||
statements = _statements(_MIGRATION.read_text(encoding="utf-8"))
|
||||
with connection() as conn:
|
||||
conn.execute("SELECT pg_advisory_xact_lock(%s)", (_LOCK,))
|
||||
for statement in statements:
|
||||
conn.execute(statement)
|
||||
log.info("schema ready")
|
||||
|
||||
|
||||
def ping() -> bool:
|
||||
try:
|
||||
with connection() as conn:
|
||||
conn.execute("SELECT 1")
|
||||
return True
|
||||
except Exception:
|
||||
log.warning("database ping failed", exc_info=True)
|
||||
return False
|
||||
@@ -0,0 +1,19 @@
|
||||
"""API errors that map onto the contract body. Details never include SQL or secrets."""
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
class ApiError(Exception):
|
||||
def __init__(self, status: int, code: str, message: str, headers: dict | None = None):
|
||||
super().__init__(message)
|
||||
self.status = status
|
||||
self.code = code
|
||||
self.message = message
|
||||
self.headers = headers or {}
|
||||
|
||||
|
||||
def unauthorized(kind: str) -> ApiError:
|
||||
if kind == "basic":
|
||||
header = 'Basic realm="messenger", charset="UTF-8"'
|
||||
else:
|
||||
header = 'Bearer realm="messenger"'
|
||||
return ApiError(401, "unauthorized", "Требуется действительная аутентификация", {"WWW-Authenticate": header})
|
||||
+1070
File diff suppressed because it is too large
Load Diff
+182
@@ -0,0 +1,182 @@
|
||||
"""Image inspection and variant rendering.
|
||||
|
||||
Thumbnail: JPEG, fit inside 256×256, no upscale, EXIF orientation applied, alpha on white.
|
||||
Crop: JPEG, centered square resized to 128×128; upscale is allowed.
|
||||
Optimized: WebP, fit inside 1280×1280, no upscale. Quality and metadata policy live in
|
||||
pipeline_params and are recorded in the report.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import math
|
||||
from dataclasses import dataclass
|
||||
|
||||
from PIL import Image, ImageOps, UnidentifiedImageError
|
||||
|
||||
PNG_MAGIC = b"\x89PNG\r\n\x1a\n"
|
||||
_SOF = {0xC0, 0xC1, 0xC2, 0xC3, 0xC5, 0xC6, 0xC7, 0xC9, 0xCA, 0xCB, 0xCD, 0xCE, 0xCF}
|
||||
|
||||
|
||||
class ImageRejected(Exception):
|
||||
def __init__(self, status: int, code: str, message: str):
|
||||
super().__init__(message)
|
||||
self.status = status
|
||||
self.code = code
|
||||
self.message = message
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PipelineParams:
|
||||
version: int
|
||||
jpeg_quality: int
|
||||
crop_quality: int
|
||||
webp_quality: int
|
||||
webp_method: int
|
||||
|
||||
|
||||
def pipeline_params(version: int) -> PipelineParams:
|
||||
"""Version 1 is the course pipeline. Version 2 changes only quality, so a stale
|
||||
writer can be detected by checksum without changing geometry."""
|
||||
if version == 1:
|
||||
return PipelineParams(1, jpeg_quality=85, crop_quality=85, webp_quality=80, webp_method=6)
|
||||
if version == 2:
|
||||
return PipelineParams(2, jpeg_quality=60, crop_quality=70, webp_quality=50, webp_method=4)
|
||||
raise ImageRejected(400, "invalid_request", "Неизвестная версия pipeline")
|
||||
|
||||
|
||||
def _png_size(data: bytes) -> tuple[int, int]:
|
||||
if len(data) < 24 or data[:8] != PNG_MAGIC or data[12:16] != b"IHDR":
|
||||
raise ImageRejected(415, "unsupported_media_type", "Файл не является корректным PNG")
|
||||
if b"IEND" not in data:
|
||||
raise ImageRejected(415, "unsupported_media_type", "PNG повреждён")
|
||||
width = int.from_bytes(data[16:20], "big")
|
||||
height = int.from_bytes(data[20:24], "big")
|
||||
return width, height
|
||||
|
||||
|
||||
def _jpeg_size(data: bytes) -> tuple[int, int]:
|
||||
if len(data) < 4 or data[:2] != b"\xff\xd8":
|
||||
raise ImageRejected(415, "unsupported_media_type", "Файл не является корректным JPEG")
|
||||
index = 2
|
||||
while index < len(data):
|
||||
if data[index] != 0xFF:
|
||||
raise ImageRejected(415, "unsupported_media_type", "JPEG повреждён")
|
||||
while index < len(data) and data[index] == 0xFF:
|
||||
index += 1
|
||||
if index >= len(data):
|
||||
break
|
||||
marker = data[index]
|
||||
index += 1
|
||||
if marker in {0xD8, 0xD9, 0xDA} or marker == 0x01 or 0xD0 <= marker <= 0xD7:
|
||||
if marker in {0xD9, 0xDA}:
|
||||
break
|
||||
continue
|
||||
if index + 2 > len(data):
|
||||
break
|
||||
length = int.from_bytes(data[index:index + 2], "big")
|
||||
if length < 2 or index + length > len(data):
|
||||
raise ImageRejected(415, "unsupported_media_type", "JPEG повреждён")
|
||||
if marker in _SOF:
|
||||
if length < 8:
|
||||
raise ImageRejected(415, "unsupported_media_type", "JPEG повреждён")
|
||||
height = int.from_bytes(data[index + 3:index + 5], "big")
|
||||
width = int.from_bytes(data[index + 5:index + 7], "big")
|
||||
return width, height
|
||||
index += length
|
||||
raise ImageRejected(415, "unsupported_media_type", "Не удалось прочитать размеры JPEG")
|
||||
|
||||
|
||||
def inspect_upload(data: bytes, declared: str, max_bytes: int, max_pixels: int) -> tuple[str, int, int]:
|
||||
"""Validate a client upload from headers and the container, before a full decode."""
|
||||
if len(data) > max_bytes:
|
||||
raise ImageRejected(413, "payload_too_large", "Превышен размер файла")
|
||||
if not data:
|
||||
raise ImageRejected(415, "unsupported_media_type", "Пустое изображение")
|
||||
if declared == "image/png":
|
||||
if not data.startswith(PNG_MAGIC):
|
||||
raise ImageRejected(415, "unsupported_media_type", "Содержимое не соответствует PNG")
|
||||
width, height = _png_size(data)
|
||||
content_type = "image/png"
|
||||
elif declared == "image/jpeg":
|
||||
if not data.startswith(b"\xff\xd8\xff"):
|
||||
raise ImageRejected(415, "unsupported_media_type", "Содержимое не соответствует JPEG")
|
||||
width, height = _jpeg_size(data)
|
||||
content_type = "image/jpeg"
|
||||
else:
|
||||
raise ImageRejected(415, "unsupported_media_type", "Допустимы только PNG и JPEG")
|
||||
if width < 1 or height < 1:
|
||||
raise ImageRejected(415, "unsupported_media_type", "Некорректные размеры изображения")
|
||||
if width > max_pixels or height > max_pixels or width * height > max_pixels:
|
||||
raise ImageRejected(413, "payload_too_large", "Превышено число пикселей")
|
||||
return content_type, width, height
|
||||
|
||||
|
||||
def fit_inside(width: int, height: int, box: int, allow_upscale: bool) -> tuple[int, int]:
|
||||
scale = min(box / width, box / height)
|
||||
if not allow_upscale:
|
||||
scale = min(scale, 1.0)
|
||||
return max(1, math.floor(width * scale)), max(1, math.floor(height * scale))
|
||||
|
||||
|
||||
def _to_rgb_white(image: Image.Image) -> Image.Image:
|
||||
if image.mode == "RGB":
|
||||
return image
|
||||
if image.mode == "P" and "transparency" in image.info:
|
||||
image = image.convert("RGBA")
|
||||
if image.mode in {"RGBA", "LA"}:
|
||||
rgba = image.convert("RGBA")
|
||||
background = Image.new("RGB", rgba.size, (255, 255, 255))
|
||||
background.paste(rgba, mask=rgba.getchannel("A"))
|
||||
return background
|
||||
return image.convert("RGB")
|
||||
|
||||
|
||||
def decode_image(data: bytes, max_pixels: int) -> Image.Image:
|
||||
Image.MAX_IMAGE_PIXELS = max_pixels
|
||||
try:
|
||||
image = Image.open(io.BytesIO(data))
|
||||
width, height = image.size
|
||||
if width < 1 or height < 1 or width * height > max_pixels:
|
||||
raise ImageRejected(413, "payload_too_large", "Превышено число пикселей")
|
||||
image = ImageOps.exif_transpose(image)
|
||||
width, height = image.size
|
||||
if width * height > max_pixels:
|
||||
raise ImageRejected(413, "payload_too_large", "Превышено число пикселей")
|
||||
image.load()
|
||||
except ImageRejected:
|
||||
raise
|
||||
except (UnidentifiedImageError, Image.DecompressionBombError, OSError, ValueError) as exc:
|
||||
raise ImageRejected(415, "unsupported_media_type", "Изображение повреждено") from exc
|
||||
return _to_rgb_white(image)
|
||||
|
||||
|
||||
def _jpeg_bytes(image: Image.Image, quality: int) -> bytes:
|
||||
buffer = io.BytesIO()
|
||||
image.save(buffer, format="JPEG", quality=quality, optimize=True, progressive=False)
|
||||
return buffer.getvalue()
|
||||
|
||||
|
||||
def _webp_bytes(image: Image.Image, quality: int, method: int) -> bytes:
|
||||
buffer = io.BytesIO()
|
||||
image.save(buffer, format="WEBP", quality=quality, method=method, lossless=False, exact=False)
|
||||
return buffer.getvalue()
|
||||
|
||||
|
||||
def render_variants(source: Image.Image, params: PipelineParams) -> dict[str, dict]:
|
||||
width, height = source.size
|
||||
thumb_w, thumb_h = fit_inside(width, height, 256, allow_upscale=False)
|
||||
thumbnail = source if (thumb_w, thumb_h) == (width, height) else source.resize((thumb_w, thumb_h), Image.Resampling.LANCZOS)
|
||||
side = min(width, height)
|
||||
left = (width - side) // 2
|
||||
top = (height - side) // 2
|
||||
crop = source.crop((left, top, left + side, top + side)).resize((128, 128), Image.Resampling.LANCZOS)
|
||||
opt_w, opt_h = fit_inside(width, height, 1280, allow_upscale=False)
|
||||
optimized = source if (opt_w, opt_h) == (width, height) else source.resize((opt_w, opt_h), Image.Resampling.LANCZOS)
|
||||
thumb_bytes = _jpeg_bytes(thumbnail, params.jpeg_quality)
|
||||
crop_bytes = _jpeg_bytes(crop, params.crop_quality)
|
||||
webp_bytes = _webp_bytes(optimized, params.webp_quality, params.webp_method)
|
||||
return {
|
||||
"thumbnail": {"bytes": thumb_bytes, "content_type": "image/jpeg", "width": thumb_w, "height": thumb_h},
|
||||
"crop": {"bytes": crop_bytes, "content_type": "image/jpeg", "width": 128, "height": 128},
|
||||
"optimized": {"bytes": webp_bytes, "content_type": "image/webp", "width": opt_w, "height": opt_h},
|
||||
}
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
"""API process entrypoint."""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
import uvicorn
|
||||
|
||||
from app.broker import Publisher
|
||||
from app.db import migrate
|
||||
from app.httpapi import create_app
|
||||
from app.outbox import start_relay
|
||||
from app.runtime import init
|
||||
from app.s3store import ObjectStore
|
||||
from app.settings import load
|
||||
|
||||
|
||||
def main() -> None:
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s %(message)s")
|
||||
settings = load()
|
||||
migrate(settings.database_url)
|
||||
store = ObjectStore(
|
||||
settings.s3_endpoint,
|
||||
settings.s3_region,
|
||||
settings.s3_bucket,
|
||||
settings.s3_access_key_id,
|
||||
settings.s3_secret_access_key,
|
||||
)
|
||||
store.ensure_bucket()
|
||||
init(settings, store, Publisher(settings.broker_url))
|
||||
start_relay()
|
||||
uvicorn.run(create_app(), host=settings.host, port=settings.port, log_level="info", access_log=False)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+114
@@ -0,0 +1,114 @@
|
||||
"""Transactional outbox and operator controls shared by the API and the worker."""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import uuid
|
||||
|
||||
from psycopg.types.json import Jsonb
|
||||
|
||||
from app import runtime
|
||||
from app.broker import ROUTING_DLQ
|
||||
from app.db import connection
|
||||
from app.timeutil import iso
|
||||
|
||||
log = logging.getLogger("messenger.outbox")
|
||||
|
||||
|
||||
def payload_for(job_id, attachment_id, pipeline_version: int, attempt: int, created_at) -> dict:
|
||||
created = created_at if isinstance(created_at, str) else iso(created_at)
|
||||
return {
|
||||
"schema_version": 1,
|
||||
"job_id": str(job_id),
|
||||
"type": "attachment.process.v1",
|
||||
"attachment_id": str(attachment_id),
|
||||
"pipeline_version": int(pipeline_version),
|
||||
"attempt": int(attempt),
|
||||
"created_at": created,
|
||||
}
|
||||
|
||||
|
||||
def add_outbox(conn, job_id, attachment_id, pipeline_version: int, attempt: int, created_at) -> dict:
|
||||
body = payload_for(job_id, attachment_id, pipeline_version, attempt, created_at)
|
||||
conn.execute(
|
||||
"INSERT INTO outbox (job_id, payload) VALUES (%s, %s)",
|
||||
(job_id, Jsonb(body)),
|
||||
)
|
||||
return body
|
||||
|
||||
|
||||
def audit(conn, job_id, attachment_id, action: str, detail: dict) -> None:
|
||||
conn.execute(
|
||||
"INSERT INTO job_audit (job_id, attachment_id, action, detail) VALUES (%s, %s, %s, %s)",
|
||||
(job_id, attachment_id, action, Jsonb(detail)),
|
||||
)
|
||||
|
||||
|
||||
def outbox_paused(conn) -> bool:
|
||||
row = conn.execute("SELECT value FROM runtime_control WHERE key='outbox_paused'").fetchone()
|
||||
return bool(row and row["value"] is True)
|
||||
|
||||
|
||||
def relay_tick(limit: int = 10) -> int:
|
||||
"""Publish committed outbox rows. A crash after COMMIT and before confirm leaves the row unpublished."""
|
||||
settings, _store, publisher = runtime.require()
|
||||
published = 0
|
||||
with connection() as conn:
|
||||
if outbox_paused(conn):
|
||||
return 0
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE outbox AS o
|
||||
SET published_at = clock_timestamp()
|
||||
FROM jobs AS j
|
||||
WHERE o.job_id = j.job_id
|
||||
AND o.published_at IS NULL
|
||||
AND j.status IN ('ready', 'failed', 'superseded')
|
||||
"""
|
||||
)
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT o.id, o.payload
|
||||
FROM outbox AS o
|
||||
JOIN jobs AS j ON j.job_id = o.job_id
|
||||
WHERE o.published_at IS NULL
|
||||
AND j.status = 'queued'
|
||||
AND (j.next_attempt_at IS NULL OR j.next_attempt_at <= clock_timestamp())
|
||||
ORDER BY o.id
|
||||
FOR UPDATE OF o SKIP LOCKED
|
||||
LIMIT %s
|
||||
""",
|
||||
(limit,),
|
||||
).fetchall()
|
||||
for row in rows:
|
||||
if publisher.publish(row["payload"]):
|
||||
conn.execute("UPDATE outbox SET published_at = clock_timestamp() WHERE id = %s", (row["id"],))
|
||||
published += 1
|
||||
else:
|
||||
break
|
||||
if published:
|
||||
log.info("published %s job message(s)", published)
|
||||
return published
|
||||
|
||||
|
||||
def publish_dlq(body: dict) -> None:
|
||||
_settings, _store, publisher = runtime.require()
|
||||
publisher.publish(body, routing_key=ROUTING_DLQ)
|
||||
|
||||
|
||||
def start_relay() -> None:
|
||||
import threading
|
||||
import time
|
||||
|
||||
def loop() -> None:
|
||||
while True:
|
||||
try:
|
||||
relay_tick()
|
||||
except Exception:
|
||||
log.warning("outbox relay iteration failed", exc_info=True)
|
||||
time.sleep(0.4)
|
||||
|
||||
threading.Thread(target=loop, name="outbox-relay", daemon=True).start()
|
||||
|
||||
|
||||
def new_job_id() -> uuid.UUID:
|
||||
return uuid.uuid4()
|
||||
+592
@@ -0,0 +1,592 @@
|
||||
"""Idempotent attachment pipeline.
|
||||
|
||||
The effect key is (attachment_id, pipeline_version, variant). Object keys include that
|
||||
identity, database rows use it as a primary key, and a completed job is a no-op.
|
||||
Delivery is at least once; this module does not claim exactly-once delivery.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
import os
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
|
||||
from psycopg.types.json import Jsonb
|
||||
|
||||
from app import runtime
|
||||
from app.db import connection
|
||||
from app.images import ImageRejected, decode_image, pipeline_params, render_variants
|
||||
from app.outbox import add_outbox, audit, publish_dlq
|
||||
|
||||
log = logging.getLogger("messenger.pipeline")
|
||||
|
||||
VARIANT_NAMES = ("thumbnail", "crop", "optimized")
|
||||
|
||||
|
||||
class TransientJobError(Exception):
|
||||
def __init__(self, message: str):
|
||||
super().__init__(message)
|
||||
self.message = message
|
||||
|
||||
|
||||
class PermanentJobError(Exception):
|
||||
def __init__(self, code: str, message: str):
|
||||
super().__init__(message)
|
||||
self.code = code
|
||||
self.message = message
|
||||
|
||||
|
||||
def _safe(text: str) -> str:
|
||||
cleaned = " ".join(text.split())
|
||||
return cleaned[:180] or "processing failed"
|
||||
|
||||
|
||||
def _parse_message(body: dict) -> dict:
|
||||
required = {"schema_version", "job_id", "type", "attachment_id", "pipeline_version", "attempt", "created_at"}
|
||||
if not isinstance(body, dict) or set(body) - required - {"traceparent", "tracestate"}:
|
||||
raise PermanentJobError("processing_failed", "invalid job message")
|
||||
if any(key not in body for key in required):
|
||||
raise PermanentJobError("processing_failed", "invalid job message")
|
||||
if body["schema_version"] != 1 or body["type"] != "attachment.process.v1":
|
||||
raise PermanentJobError("processing_failed", "invalid job message")
|
||||
try:
|
||||
return {
|
||||
"job_id": uuid.UUID(str(body["job_id"])),
|
||||
"attachment_id": uuid.UUID(str(body["attachment_id"])),
|
||||
"pipeline_version": int(body["pipeline_version"]),
|
||||
"attempt": int(body["attempt"]),
|
||||
"created_at": str(body["created_at"]),
|
||||
}
|
||||
except (ValueError, TypeError) as exc:
|
||||
raise PermanentJobError("processing_failed", "invalid job message") from exc
|
||||
|
||||
|
||||
def _read_fault(conn) -> dict:
|
||||
row = conn.execute("SELECT value FROM runtime_control WHERE key = 'worker_fault' FOR UPDATE").fetchone()
|
||||
value = row["value"] if row else {}
|
||||
return dict(value) if isinstance(value, dict) else {}
|
||||
|
||||
|
||||
def _write_fault(conn, value: dict) -> None:
|
||||
conn.execute("UPDATE runtime_control SET value = %s WHERE key = 'worker_fault'", (Jsonb(value),))
|
||||
|
||||
|
||||
def _consume_counted_fault() -> str | None:
|
||||
"""Disarm one-shot and counted faults. Crash faults stay armed until their injection point."""
|
||||
with connection() as conn:
|
||||
value = _read_fault(conn)
|
||||
if not value.get("armed"):
|
||||
return None
|
||||
mode = value.get("mode")
|
||||
if mode == "transient":
|
||||
remaining = int(value.get("remaining") or 1) - 1
|
||||
if remaining <= 0:
|
||||
_write_fault(conn, {"mode": None, "armed": False})
|
||||
else:
|
||||
_write_fault(conn, {"mode": "transient", "armed": True, "remaining": remaining})
|
||||
return "transient"
|
||||
if mode in {"permanent", "duplicate"}:
|
||||
_write_fault(conn, {"mode": None, "armed": False})
|
||||
return mode
|
||||
if mode in {"before_effect", "after_s3", "after_commit"}:
|
||||
return mode
|
||||
return None
|
||||
|
||||
|
||||
def _crash(mode: str, job_id, attachment_id) -> None:
|
||||
with connection() as conn:
|
||||
value = _read_fault(conn)
|
||||
if value.get("armed") and value.get("mode") == mode:
|
||||
_write_fault(conn, {"mode": None, "armed": False})
|
||||
audit(conn, job_id, attachment_id, "fault_" + mode, {"mode": mode})
|
||||
if mode in {"before_effect", "after_s3"}:
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE jobs
|
||||
SET status = 'queued', lease_until = NULL, lease_owner = NULL, updated_at = clock_timestamp()
|
||||
WHERE job_id = %s AND status = 'processing'
|
||||
""",
|
||||
(job_id,),
|
||||
)
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachments AS a
|
||||
SET status = 'queued', updated_at = clock_timestamp()
|
||||
FROM jobs AS j
|
||||
WHERE j.job_id = %s AND a.id = j.attachment_id
|
||||
AND a.pipeline_version = j.pipeline_version AND a.status = 'processing'
|
||||
""",
|
||||
(job_id,),
|
||||
)
|
||||
log.warning("fault injection exit: %s", mode)
|
||||
os._exit(75)
|
||||
|
||||
|
||||
def _load(conn, message: dict):
|
||||
job = conn.execute("SELECT * FROM jobs WHERE job_id = %s FOR UPDATE", (message["job_id"],)).fetchone()
|
||||
attachment = None
|
||||
if job is not None:
|
||||
attachment = conn.execute("SELECT * FROM attachments WHERE id = %s FOR UPDATE", (job["attachment_id"],)).fetchone()
|
||||
return job, attachment
|
||||
|
||||
|
||||
def _finish_noop(conn, job, attachment, action: str, redelivered: bool) -> None:
|
||||
audit(conn, job["job_id"] if job else None, attachment["id"] if attachment else None, action, {"redelivered": redelivered})
|
||||
|
||||
|
||||
def _supersede(conn, job, attachment, redelivered: bool) -> None:
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE jobs
|
||||
SET status = 'superseded', lease_until = NULL, lease_owner = NULL, updated_at = clock_timestamp()
|
||||
WHERE job_id = %s AND status IN ('queued', 'processing')
|
||||
""",
|
||||
(job["job_id"],),
|
||||
)
|
||||
_finish_noop(conn, job, attachment, "superseded", redelivered)
|
||||
|
||||
|
||||
def _acquire(conn, job, worker_id: str, timeout_seconds: int):
|
||||
return conn.execute(
|
||||
"""
|
||||
UPDATE jobs
|
||||
SET status = 'processing',
|
||||
lease_owner = %s,
|
||||
lease_until = clock_timestamp() + make_interval(secs => %s),
|
||||
updated_at = clock_timestamp()
|
||||
WHERE job_id = %s
|
||||
AND status IN ('queued', 'processing')
|
||||
AND (lease_until IS NULL OR lease_until < clock_timestamp() OR lease_owner = %s)
|
||||
RETURNING job_id
|
||||
""",
|
||||
(worker_id, timeout_seconds, job["job_id"], worker_id),
|
||||
).fetchone()
|
||||
|
||||
|
||||
def _schedule_retry(conn, job, attachment, message: str) -> None:
|
||||
settings, _store, _publisher = runtime.require()
|
||||
attempt = int(job["attempt"])
|
||||
if attempt >= settings.job_max_attempts:
|
||||
_fail(conn, job, attachment, "retry_exhausted", "transient", "retry budget exhausted")
|
||||
return
|
||||
import random
|
||||
|
||||
delay = min(settings.job_backoff_cap_seconds, settings.job_backoff_base_seconds * (2 ** (attempt - 1)))
|
||||
delay = delay * (0.5 + random.random())
|
||||
next_attempt = attempt + 1
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE jobs
|
||||
SET status = 'queued', attempt = %s, next_attempt_at = clock_timestamp() + make_interval(secs => %s),
|
||||
lease_until = NULL, lease_owner = NULL, error_class = 'transient', error_code = 'processing_failed',
|
||||
last_error = %s, updated_at = clock_timestamp()
|
||||
WHERE job_id = %s
|
||||
""",
|
||||
(next_attempt, delay, _safe(message), job["job_id"]),
|
||||
)
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachments
|
||||
SET status = 'queued', error_code = NULL, updated_at = clock_timestamp()
|
||||
WHERE id = %s AND pipeline_version = %s AND status <> 'ready'
|
||||
""",
|
||||
(attachment["id"], job["pipeline_version"]),
|
||||
)
|
||||
add_outbox(conn, job["job_id"], attachment["id"], job["pipeline_version"], next_attempt, job["created_at"])
|
||||
audit(conn, job["job_id"], attachment["id"], "retry", {"attempt": next_attempt, "delay_seconds": round(delay, 3)})
|
||||
|
||||
|
||||
def _fail(conn, job, attachment, code: str, error_class: str, message: str) -> None:
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE jobs
|
||||
SET status = 'failed', error_code = %s, error_class = %s, last_error = %s,
|
||||
lease_until = NULL, lease_owner = NULL, next_attempt_at = NULL, updated_at = clock_timestamp()
|
||||
WHERE job_id = %s
|
||||
""",
|
||||
(code, error_class, _safe(message), job["job_id"]),
|
||||
)
|
||||
if attachment["ready_pipeline_version"] is not None:
|
||||
# A failed newer attempt must not discard an already published result.
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachments
|
||||
SET status = 'ready', error_code = NULL, pipeline_version = ready_pipeline_version,
|
||||
updated_at = clock_timestamp()
|
||||
WHERE id = %s AND ready_pipeline_version IS NOT NULL
|
||||
""",
|
||||
(attachment["id"],),
|
||||
)
|
||||
else:
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachments
|
||||
SET status = 'failed', error_code = %s, updated_at = clock_timestamp()
|
||||
WHERE id = %s AND pipeline_version = %s AND status <> 'ready'
|
||||
""",
|
||||
(code, attachment["id"], job["pipeline_version"]),
|
||||
)
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO failed_jobs (job_id, attachment_id, pipeline_version, attempt, error_code, error_class, safe_message)
|
||||
VALUES (%s, %s, %s, %s, %s, %s, %s)
|
||||
""",
|
||||
(job["job_id"], attachment["id"], job["pipeline_version"], job["attempt"], code, error_class, _safe(message)),
|
||||
)
|
||||
audit(conn, job["job_id"], attachment["id"], "failed", {"error_code": code, "error_class": error_class})
|
||||
publish_dlq(
|
||||
{
|
||||
"schema_version": 1,
|
||||
"job_id": str(job["job_id"]),
|
||||
"type": "attachment.process.v1",
|
||||
"attachment_id": str(attachment["id"]),
|
||||
"pipeline_version": int(job["pipeline_version"]),
|
||||
"attempt": int(job["attempt"]),
|
||||
"created_at": message_created(job),
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def message_created(job) -> str:
|
||||
from app.timeutil import iso as format_iso
|
||||
|
||||
value = job["created_at"]
|
||||
return value if isinstance(value, str) else format_iso(value)
|
||||
|
||||
|
||||
def _put_original(store, attachment, data: bytes) -> None:
|
||||
existing = store.get(attachment["original_key"])
|
||||
digest = hashlib.sha256(data).hexdigest()
|
||||
if digest != attachment["sha256"]:
|
||||
raise PermanentJobError("processing_failed", "checksum mismatch")
|
||||
if existing is None:
|
||||
store.put(attachment["original_key"], data, attachment["content_type"])
|
||||
return
|
||||
if hashlib.sha256(existing).hexdigest() != attachment["sha256"]:
|
||||
raise TransientJobError("stored original does not match the accepted checksum")
|
||||
|
||||
|
||||
def _store_variants(store, attachment, version: int, rendered: dict, max_pixels: int) -> list[dict]:
|
||||
stored = []
|
||||
for name in VARIANT_NAMES:
|
||||
item = rendered[name]
|
||||
key = f"variants/{attachment['id']}/v{version}/{name}"
|
||||
existing = store.get(key)
|
||||
if existing is None:
|
||||
store.put(key, item["bytes"], item["content_type"])
|
||||
blob = item["bytes"]
|
||||
width, height = item["width"], item["height"]
|
||||
else:
|
||||
blob = existing
|
||||
if hashlib.sha256(blob).hexdigest() == hashlib.sha256(item["bytes"]).hexdigest():
|
||||
width, height = item["width"], item["height"]
|
||||
else:
|
||||
probed = decode_image(blob, max_pixels)
|
||||
width, height = probed.size
|
||||
stored.append(
|
||||
{
|
||||
"name": name,
|
||||
"content_type": item["content_type"] if existing is None else _content_type_for(name, blob, item["content_type"]),
|
||||
"size_bytes": len(blob),
|
||||
"width": width,
|
||||
"height": height,
|
||||
"sha256": hashlib.sha256(blob).hexdigest(),
|
||||
"object_key": key,
|
||||
}
|
||||
)
|
||||
return stored
|
||||
|
||||
|
||||
def _content_type_for(name: str, blob: bytes, encoded_type: str) -> str:
|
||||
if blob.startswith(b"\xff\xd8"):
|
||||
return "image/jpeg"
|
||||
if len(blob) >= 12 and blob[:4] == b"RIFF" and blob[8:12] == b"WEBP":
|
||||
return "image/webp"
|
||||
return encoded_type
|
||||
|
||||
|
||||
def _source_bytes(store, attachment) -> bytes:
|
||||
data = store.get(attachment["staging_key"])
|
||||
if data is None:
|
||||
data = store.get(attachment["original_key"])
|
||||
if data is None:
|
||||
raise TransientJobError("source object is not available yet")
|
||||
return data
|
||||
|
||||
|
||||
def _commit_success(job, attachment, variants: list[dict], worker_id: str, crash_after_commit: bool) -> bool:
|
||||
"""Returns True when this version became the published result."""
|
||||
settings, store, _publisher = runtime.require()
|
||||
with connection() as conn:
|
||||
current = conn.execute("SELECT pipeline_version, ready_pipeline_version FROM attachments WHERE id = %s FOR UPDATE", (attachment["id"],)).fetchone()
|
||||
for item in variants:
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO attachment_variants (
|
||||
attachment_id, pipeline_version, name, content_type, size_bytes, width, height, sha256, object_key
|
||||
) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)
|
||||
ON CONFLICT (attachment_id, pipeline_version, name) DO NOTHING
|
||||
""",
|
||||
(
|
||||
attachment["id"],
|
||||
job["pipeline_version"],
|
||||
item["name"],
|
||||
item["content_type"],
|
||||
item["size_bytes"],
|
||||
item["width"],
|
||||
item["height"],
|
||||
item["sha256"],
|
||||
item["object_key"],
|
||||
),
|
||||
)
|
||||
updated = conn.execute(
|
||||
"""
|
||||
UPDATE attachments
|
||||
SET status = 'ready', ready_pipeline_version = %s, error_code = NULL, updated_at = clock_timestamp()
|
||||
WHERE id = %s
|
||||
AND pipeline_version = %s
|
||||
AND (ready_pipeline_version IS NULL OR ready_pipeline_version <= %s)
|
||||
RETURNING id
|
||||
""",
|
||||
(job["pipeline_version"], attachment["id"], job["pipeline_version"], job["pipeline_version"]),
|
||||
).fetchone()
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE jobs
|
||||
SET status = 'ready', error_code = NULL, error_class = NULL, last_error = NULL,
|
||||
lease_until = NULL, lease_owner = NULL, next_attempt_at = NULL, updated_at = clock_timestamp()
|
||||
WHERE job_id = %s
|
||||
""",
|
||||
(job["job_id"],),
|
||||
)
|
||||
if crash_after_commit:
|
||||
_write_fault(conn, {"mode": None, "armed": False})
|
||||
audit(conn, job["job_id"], attachment["id"], "fault_after_commit", {"worker": worker_id})
|
||||
audit(
|
||||
conn,
|
||||
job["job_id"],
|
||||
attachment["id"],
|
||||
"committed",
|
||||
{"pipeline_version": job["pipeline_version"], "published": bool(updated), "variants": len(variants)},
|
||||
)
|
||||
published = updated is not None and current is not None
|
||||
if crash_after_commit:
|
||||
os._exit(75)
|
||||
if published:
|
||||
try:
|
||||
store.delete(attachment["staging_key"])
|
||||
with connection() as conn:
|
||||
conn.execute(
|
||||
"UPDATE attachments SET staging_deleted_at = clock_timestamp() WHERE id = %s AND status = 'ready'",
|
||||
(attachment["id"],),
|
||||
)
|
||||
except Exception:
|
||||
log.warning("staging cleanup deferred for %s", attachment["id"])
|
||||
return published
|
||||
|
||||
|
||||
def _execute(message: dict, attachment: dict, worker_id: str, fault: str | None) -> None:
|
||||
settings, store, _publisher = runtime.require()
|
||||
if fault == "before_effect":
|
||||
_crash("before_effect", message["job_id"], attachment["id"])
|
||||
deadline = time.monotonic() + settings.job_timeout_seconds
|
||||
data = _source_bytes(store, attachment)
|
||||
if len(data) > settings.max_upload_bytes:
|
||||
raise PermanentJobError("invalid_image", "object exceeds the byte limit")
|
||||
if time.monotonic() > deadline:
|
||||
raise TransientJobError("processing deadline exceeded")
|
||||
try:
|
||||
image = _decode_limited(data, settings.max_image_pixels, max(1, deadline - time.monotonic()))
|
||||
params = pipeline_params(int(attachment_version(message)))
|
||||
rendered = render_variants(image, params)
|
||||
except ImageRejected as exc:
|
||||
raise PermanentJobError("invalid_image", exc.message) from exc
|
||||
if time.monotonic() > deadline:
|
||||
raise TransientJobError("processing deadline exceeded")
|
||||
_put_original(store, attachment, data)
|
||||
variants = _store_variants(store, attachment, message["pipeline_version"], rendered, settings.max_image_pixels)
|
||||
if fault == "after_s3":
|
||||
_crash("after_s3", message["job_id"], attachment["id"])
|
||||
_commit_success(message_as_job(message, attachment), attachment, variants, worker_id, fault == "after_commit")
|
||||
|
||||
|
||||
def attachment_version(message: dict) -> int:
|
||||
return int(message["pipeline_version"])
|
||||
|
||||
|
||||
def message_as_job(message: dict, attachment: dict) -> dict:
|
||||
return {
|
||||
"job_id": message["job_id"],
|
||||
"pipeline_version": message["pipeline_version"],
|
||||
"attempt": message["attempt"],
|
||||
"created_at": message["created_at"],
|
||||
"attachment_id": attachment["id"],
|
||||
}
|
||||
|
||||
|
||||
def _decode_limited(data: bytes, max_pixels: int, timeout: float):
|
||||
box: dict = {}
|
||||
|
||||
def run() -> None:
|
||||
try:
|
||||
box["image"] = decode_image(data, max_pixels)
|
||||
except Exception as exc: # noqa: BLE001 - transferred to the caller
|
||||
box["error"] = exc
|
||||
|
||||
thread = threading.Thread(target=run, daemon=True)
|
||||
thread.start()
|
||||
thread.join(timeout)
|
||||
if thread.is_alive():
|
||||
raise TransientJobError("decode timed out")
|
||||
if "error" in box:
|
||||
raise box["error"]
|
||||
return box["image"]
|
||||
|
||||
|
||||
def _handle_known_error(message: dict, exc: Exception) -> None:
|
||||
with connection() as conn:
|
||||
job, attachment = _load(conn, message)
|
||||
if job is None or attachment is None or job["status"] in {"ready", "failed", "superseded"}:
|
||||
return
|
||||
if isinstance(exc, PermanentJobError):
|
||||
_fail(conn, job, attachment, exc.code, "permanent", exc.message)
|
||||
elif isinstance(exc, TransientJobError):
|
||||
_schedule_retry(conn, job, attachment, exc.message)
|
||||
else:
|
||||
_schedule_retry(conn, job, attachment, "temporary processing failure")
|
||||
|
||||
|
||||
def process_delivery(raw: bytes, redelivered: bool, worker_id: str) -> str:
|
||||
"""Returns 'ack' or 'nack'. Ack happens only after the durable decision is committed."""
|
||||
import json
|
||||
|
||||
try:
|
||||
body = json.loads(raw.decode("utf-8"))
|
||||
message = _parse_message(body)
|
||||
except (UnicodeError, json.JSONDecodeError, PermanentJobError):
|
||||
log.warning("discarding a job message that does not match the contract")
|
||||
return "ack"
|
||||
try:
|
||||
fault = _consume_counted_fault()
|
||||
except Exception:
|
||||
log.warning("fault control unreadable", exc_info=True)
|
||||
return "nack"
|
||||
if fault == "transient":
|
||||
_handle_known_error(message, TransientJobError("injected transient failure"))
|
||||
return "ack"
|
||||
if fault == "permanent":
|
||||
_handle_known_error(message, PermanentJobError("invalid_image", "injected permanent failure"))
|
||||
return "ack"
|
||||
runs = 2 if fault == "duplicate" else 1
|
||||
for _ in range(runs):
|
||||
outcome = _process_once(message, redelivered, worker_id, fault if runs == 1 else None)
|
||||
if outcome == "nack":
|
||||
return "nack"
|
||||
fault = None
|
||||
return "ack"
|
||||
|
||||
|
||||
def _process_once(message: dict, redelivered: bool, worker_id: str, fault: str | None) -> str:
|
||||
settings, _store, _publisher = runtime.require()
|
||||
with connection() as conn:
|
||||
job, attachment = _load(conn, message)
|
||||
if job is None or attachment is None:
|
||||
audit(conn, message["job_id"], message["attachment_id"], "missing", {"redelivered": redelivered})
|
||||
return "ack"
|
||||
if job["status"] in {"ready", "superseded", "failed"}:
|
||||
_finish_noop(conn, job, attachment, "noop_" + job["status"], redelivered)
|
||||
return "ack"
|
||||
if message["attempt"] != int(job["attempt"]):
|
||||
_finish_noop(conn, job, attachment, "noop_attempt", redelivered)
|
||||
return "ack"
|
||||
newer_ready = (
|
||||
attachment["ready_pipeline_version"] is not None
|
||||
and int(attachment["ready_pipeline_version"]) > int(job["pipeline_version"])
|
||||
)
|
||||
newer_desired = int(attachment["pipeline_version"]) > int(job["pipeline_version"])
|
||||
if newer_ready or newer_desired:
|
||||
_supersede(conn, job, attachment, redelivered)
|
||||
return "ack"
|
||||
if _acquire(conn, job, worker_id, settings.job_timeout_seconds) is None:
|
||||
return "nack"
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachments
|
||||
SET status = 'processing', updated_at = clock_timestamp()
|
||||
WHERE id = %s AND pipeline_version = %s AND status IN ('queued', 'processing')
|
||||
""",
|
||||
(attachment["id"], job["pipeline_version"]),
|
||||
)
|
||||
audit(conn, job["job_id"], attachment["id"], "leased", {"worker": worker_id, "redelivered": redelivered})
|
||||
try:
|
||||
_execute(message, attachment, worker_id, fault)
|
||||
except (PermanentJobError, TransientJobError, ImageRejected) as exc:
|
||||
if isinstance(exc, ImageRejected):
|
||||
exc = PermanentJobError("invalid_image", exc.message)
|
||||
_handle_known_error(message, exc)
|
||||
except Exception:
|
||||
log.warning("unexpected pipeline error", exc_info=True)
|
||||
_handle_known_error(message, TransientJobError("temporary processing failure"))
|
||||
return "ack"
|
||||
|
||||
|
||||
def recover_expired_leases() -> int:
|
||||
recovered = 0
|
||||
with connection() as conn:
|
||||
rows = conn.execute(
|
||||
"""
|
||||
UPDATE jobs
|
||||
SET status = 'queued', lease_until = NULL, lease_owner = NULL, updated_at = clock_timestamp()
|
||||
WHERE status = 'processing' AND lease_until IS NOT NULL AND lease_until < clock_timestamp()
|
||||
RETURNING job_id, attachment_id, pipeline_version, attempt, created_at
|
||||
"""
|
||||
).fetchall()
|
||||
for job in rows:
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachments
|
||||
SET status = 'queued', updated_at = clock_timestamp()
|
||||
WHERE id = %s AND pipeline_version = %s AND status = 'processing'
|
||||
""",
|
||||
(job["attachment_id"], job["pipeline_version"]),
|
||||
)
|
||||
add_outbox(conn, job["job_id"], job["attachment_id"], job["pipeline_version"], job["attempt"], job["created_at"])
|
||||
audit(conn, job["job_id"], job["attachment_id"], "lease_expired", {})
|
||||
recovered += 1
|
||||
if recovered:
|
||||
log.info("recovered %s expired lease(s)", recovered)
|
||||
return recovered
|
||||
|
||||
|
||||
def sweep_staging() -> None:
|
||||
settings, store, _publisher = runtime.require()
|
||||
with connection() as conn:
|
||||
ready = conn.execute(
|
||||
"""
|
||||
SELECT id, staging_key FROM attachments
|
||||
WHERE staging_deleted_at IS NULL AND status = 'ready'
|
||||
LIMIT 25
|
||||
"""
|
||||
).fetchall()
|
||||
failed = conn.execute(
|
||||
"""
|
||||
SELECT id, staging_key FROM attachments
|
||||
WHERE staging_deleted_at IS NULL AND status = 'failed'
|
||||
AND updated_at < clock_timestamp() - make_interval(secs => %s)
|
||||
LIMIT 25
|
||||
""",
|
||||
(settings.staging_retention_seconds,),
|
||||
).fetchall()
|
||||
for row in list(ready) + list(failed):
|
||||
try:
|
||||
store.delete(row["staging_key"])
|
||||
except Exception:
|
||||
log.warning("staging delete failed")
|
||||
continue
|
||||
with connection() as conn:
|
||||
conn.execute(
|
||||
"UPDATE attachments SET staging_deleted_at = clock_timestamp() WHERE id = %s",
|
||||
(row["id"],),
|
||||
)
|
||||
@@ -0,0 +1,37 @@
|
||||
"""Shared runtime objects created at process startup."""
|
||||
from __future__ import annotations
|
||||
|
||||
from argon2 import PasswordHasher
|
||||
from argon2.low_level import Type
|
||||
|
||||
from app.broker import Publisher
|
||||
from app.settings import Settings
|
||||
from app.s3store import ObjectStore
|
||||
|
||||
settings: Settings | None = None
|
||||
store: ObjectStore | None = None
|
||||
publisher: Publisher | None = None
|
||||
hasher = PasswordHasher(
|
||||
time_cost=2,
|
||||
memory_cost=19456,
|
||||
parallelism=1,
|
||||
hash_len=32,
|
||||
salt_len=16,
|
||||
type=Type.ID,
|
||||
)
|
||||
dummy_hash: str | None = None
|
||||
|
||||
|
||||
def init(app_settings: Settings, object_store: ObjectStore, message_publisher: Publisher) -> None:
|
||||
global settings, store, publisher, dummy_hash
|
||||
settings = app_settings
|
||||
store = object_store
|
||||
publisher = message_publisher
|
||||
if dummy_hash is None:
|
||||
dummy_hash = hasher.hash("dummy-password-for-equal-time")
|
||||
|
||||
|
||||
def require() -> tuple[Settings, ObjectStore, Publisher]:
|
||||
if settings is None or store is None or publisher is None:
|
||||
raise RuntimeError("runtime is not initialized")
|
||||
return settings, store, publisher
|
||||
@@ -0,0 +1,60 @@
|
||||
"""Private S3-compatible store. Object keys are server-generated."""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
import boto3
|
||||
from botocore.client import Config
|
||||
from botocore.exceptions import ClientError
|
||||
|
||||
log = logging.getLogger("messenger.s3")
|
||||
|
||||
|
||||
class ObjectStore:
|
||||
def __init__(self, endpoint: str, region: str, bucket: str, access_key: str, secret_key: str):
|
||||
self.bucket = bucket
|
||||
self.client = boto3.client(
|
||||
"s3",
|
||||
endpoint_url=endpoint,
|
||||
region_name=region,
|
||||
aws_access_key_id=access_key,
|
||||
aws_secret_access_key=secret_key,
|
||||
config=Config(
|
||||
s3={"addressing_style": "path"},
|
||||
signature_version="s3v4",
|
||||
retries={"max_attempts": 3, "mode": "standard"},
|
||||
connect_timeout=3,
|
||||
read_timeout=20,
|
||||
),
|
||||
)
|
||||
|
||||
def ensure_bucket(self) -> None:
|
||||
try:
|
||||
self.client.head_bucket(Bucket=self.bucket)
|
||||
except ClientError:
|
||||
self.client.create_bucket(Bucket=self.bucket)
|
||||
log.info("bucket ready")
|
||||
|
||||
def healthy(self) -> bool:
|
||||
try:
|
||||
self.client.head_bucket(Bucket=self.bucket)
|
||||
return True
|
||||
except Exception:
|
||||
log.warning("object store ping failed", exc_info=True)
|
||||
return False
|
||||
|
||||
def put(self, key: str, data: bytes, content_type: str) -> None:
|
||||
self.client.put_object(Bucket=self.bucket, Key=key, Body=data, ContentType=content_type)
|
||||
|
||||
def get(self, key: str) -> bytes | None:
|
||||
try:
|
||||
response = self.client.get_object(Bucket=self.bucket, Key=key)
|
||||
except ClientError as exc:
|
||||
code = exc.response.get("Error", {}).get("Code", "")
|
||||
if code in {"404", "NoSuchKey", "NotFound"}:
|
||||
return None
|
||||
raise
|
||||
return response["Body"].read()
|
||||
|
||||
def delete(self, key: str) -> None:
|
||||
self.client.delete_object(Bucket=self.bucket, Key=key)
|
||||
@@ -0,0 +1,74 @@
|
||||
"""Process configuration. Secrets come from the environment or *_FILE, never from the image."""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def _raw(name: str, default: str | None = None) -> str:
|
||||
file_name = os.environ.get(name + "_FILE")
|
||||
if file_name:
|
||||
return Path(file_name).read_text(encoding="utf-8").strip()
|
||||
if name in os.environ and os.environ[name] != "":
|
||||
return os.environ[name]
|
||||
if default is not None:
|
||||
return default
|
||||
raise RuntimeError("missing configuration " + name)
|
||||
|
||||
|
||||
def _int(name: str, default: int) -> int:
|
||||
return int(_raw(name, str(default)))
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Settings:
|
||||
instance_id: str
|
||||
database_url: str
|
||||
session_ttl_seconds: int
|
||||
idempotency_ttl_seconds: int
|
||||
s3_endpoint: str
|
||||
s3_region: str
|
||||
s3_bucket: str
|
||||
s3_access_key_id: str
|
||||
s3_secret_access_key: str
|
||||
max_upload_bytes: int
|
||||
max_image_pixels: int
|
||||
broker_url: str
|
||||
worker_concurrency: int
|
||||
job_max_attempts: int
|
||||
job_timeout_seconds: int
|
||||
job_backoff_base_seconds: float
|
||||
job_backoff_cap_seconds: float
|
||||
staging_retention_seconds: int
|
||||
operator_token: str
|
||||
max_queued_jobs: int
|
||||
host: str
|
||||
port: int
|
||||
|
||||
|
||||
def load() -> Settings:
|
||||
return Settings(
|
||||
instance_id=_raw("INSTANCE_ID", "api-local"),
|
||||
database_url=_raw("DATABASE_URL"),
|
||||
session_ttl_seconds=_int("SESSION_TTL_SECONDS", 3600),
|
||||
idempotency_ttl_seconds=_int("IDEMPOTENCY_TTL_SECONDS", 86400),
|
||||
s3_endpoint=_raw("S3_ENDPOINT", "http://minio:9000"),
|
||||
s3_region=_raw("S3_REGION", "us-east-1"),
|
||||
s3_bucket=_raw("S3_BUCKET", "messenger-attachments"),
|
||||
s3_access_key_id=_raw("S3_ACCESS_KEY_ID"),
|
||||
s3_secret_access_key=_raw("S3_SECRET_ACCESS_KEY"),
|
||||
max_upload_bytes=_int("MAX_UPLOAD_BYTES", 10485760),
|
||||
max_image_pixels=_int("MAX_IMAGE_PIXELS", 40000000),
|
||||
broker_url=_raw("BROKER_URL"),
|
||||
worker_concurrency=max(1, _int("WORKER_CONCURRENCY", 2)),
|
||||
job_max_attempts=max(1, _int("JOB_MAX_ATTEMPTS", 5)),
|
||||
job_timeout_seconds=max(1, _int("JOB_TIMEOUT_SECONDS", 60)),
|
||||
job_backoff_base_seconds=float(_raw("JOB_BACKOFF_BASE_SECONDS", "0.5")),
|
||||
job_backoff_cap_seconds=float(_raw("JOB_BACKOFF_CAP_SECONDS", "30")),
|
||||
staging_retention_seconds=_int("STAGING_RETENTION_SECONDS", 86400),
|
||||
operator_token=_raw("OPERATOR_TOKEN", ""),
|
||||
max_queued_jobs=_int("MAX_QUEUED_JOBS", 500),
|
||||
host=_raw("APP_HOST", "0.0.0.0"),
|
||||
port=_int("APP_PORT", 8080),
|
||||
)
|
||||
@@ -0,0 +1,15 @@
|
||||
"""UTC timestamps rendered as RFC 3339 with a Z suffix."""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
|
||||
|
||||
def utcnow() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def iso(value: datetime) -> str:
|
||||
if value.tzinfo is None:
|
||||
value = value.replace(tzinfo=timezone.utc)
|
||||
value = value.astimezone(timezone.utc)
|
||||
return value.strftime("%Y-%m-%dT%H:%M:%S.") + f"{value.microsecond:06d}Z"
|
||||
+105
@@ -0,0 +1,105 @@
|
||||
"""Separate worker process. It never serves the public HTTP API."""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import signal
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
|
||||
from app import runtime
|
||||
from app.broker import QUEUE, open_consumer
|
||||
from app.db import migrate
|
||||
from app.outbox import start_relay
|
||||
from app.pipeline import process_delivery, recover_expired_leases, sweep_staging
|
||||
from app.runtime import init
|
||||
from app.s3store import ObjectStore
|
||||
from app.settings import load
|
||||
|
||||
log = logging.getLogger("messenger.worker")
|
||||
|
||||
|
||||
def _on_term(_signum, _frame) -> None:
|
||||
os._exit(0)
|
||||
|
||||
|
||||
def _consume(index: int, broker_url: str) -> None:
|
||||
worker_id = f"worker-{index}-{uuid.uuid4()}"
|
||||
while True:
|
||||
connection = None
|
||||
try:
|
||||
connection, channel = open_consumer(broker_url)
|
||||
|
||||
def on_message(channel, method, _properties, body, connection=connection, worker_id=worker_id):
|
||||
def work() -> None:
|
||||
try:
|
||||
result = process_delivery(body, bool(method.redelivered), worker_id)
|
||||
except Exception:
|
||||
log.exception("delivery failed")
|
||||
result = "nack"
|
||||
if result != "ack":
|
||||
time.sleep(1)
|
||||
|
||||
def finish() -> None:
|
||||
if result == "ack":
|
||||
channel.basic_ack(method.delivery_tag)
|
||||
else:
|
||||
channel.basic_nack(method.delivery_tag, requeue=True)
|
||||
|
||||
connection.add_callback_threadsafe(finish)
|
||||
|
||||
threading.Thread(target=work, name=f"delivery-{index}", daemon=True).start()
|
||||
|
||||
channel.basic_consume(queue=QUEUE, on_message_callback=on_message, auto_ack=False)
|
||||
log.info("consumer %s waiting; prefetch=1", worker_id)
|
||||
channel.start_consuming()
|
||||
except Exception:
|
||||
log.warning("consumer connection dropped", exc_info=True)
|
||||
time.sleep(1)
|
||||
finally:
|
||||
try:
|
||||
if connection is not None and connection.is_open:
|
||||
connection.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def _maintain() -> None:
|
||||
while True:
|
||||
try:
|
||||
recover_expired_leases()
|
||||
sweep_staging()
|
||||
except Exception:
|
||||
log.warning("maintenance iteration failed", exc_info=True)
|
||||
time.sleep(2)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s %(message)s")
|
||||
signal.signal(signal.SIGTERM, _on_term)
|
||||
signal.signal(signal.SIGINT, _on_term)
|
||||
settings = load()
|
||||
migrate(settings.database_url)
|
||||
store = ObjectStore(
|
||||
settings.s3_endpoint,
|
||||
settings.s3_region,
|
||||
settings.s3_bucket,
|
||||
settings.s3_access_key_id,
|
||||
settings.s3_secret_access_key,
|
||||
)
|
||||
store.ensure_bucket()
|
||||
from app.broker import Publisher
|
||||
|
||||
init(settings, store, Publisher(settings.broker_url))
|
||||
start_relay()
|
||||
threading.Thread(target=_maintain, name="maintenance", daemon=True).start()
|
||||
for index in range(settings.worker_concurrency):
|
||||
threading.Thread(target=_consume, args=(index, settings.broker_url), name=f"consumer-{index}", daemon=True).start()
|
||||
log.info("worker started with concurrency %s", settings.worker_concurrency)
|
||||
while True:
|
||||
time.sleep(3600)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+155
@@ -0,0 +1,155 @@
|
||||
name: mtusi-lab5
|
||||
|
||||
x-app: &app
|
||||
image: mtusi-lab5-app:local
|
||||
build:
|
||||
context: .
|
||||
env_file:
|
||||
- .env
|
||||
init: true
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
minio:
|
||||
condition: service_healthy
|
||||
minio-init:
|
||||
condition: service_completed_successfully
|
||||
rabbitmq:
|
||||
condition: service_healthy
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16.8-alpine
|
||||
container_name: mtusi-lab5-postgres
|
||||
environment:
|
||||
POSTGRES_USER: ${POSTGRES_USER}
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
|
||||
POSTGRES_DB: ${POSTGRES_DB}
|
||||
volumes:
|
||||
- pgdata:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U $$POSTGRES_USER -d $$POSTGRES_DB"]
|
||||
interval: 2s
|
||||
timeout: 3s
|
||||
retries: 30
|
||||
restart: unless-stopped
|
||||
|
||||
minio:
|
||||
image: pgsty/silo:RELEASE.2026-10-26T00-00-00Z
|
||||
container_name: mtusi-lab5-minio
|
||||
command: ["server", "/data", "--console-address", ":9001"]
|
||||
environment:
|
||||
MINIO_ROOT_USER: ${MINIO_ROOT_USER}
|
||||
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}
|
||||
volumes:
|
||||
- miniodata:/data
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://127.0.0.1:9000/minio/health/live"]
|
||||
interval: 2s
|
||||
timeout: 3s
|
||||
retries: 30
|
||||
start_period: 5s
|
||||
restart: unless-stopped
|
||||
|
||||
minio-init:
|
||||
image: pgsty/silo:RELEASE.2026-10-26T00-00-00Z
|
||||
container_name: mtusi-lab5-minio-init
|
||||
depends_on:
|
||||
minio:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
MINIO_ROOT_USER: ${MINIO_ROOT_USER}
|
||||
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}
|
||||
S3_BUCKET: ${S3_BUCKET}
|
||||
entrypoint: ["/bin/sh", "-c"]
|
||||
command:
|
||||
- |
|
||||
set -eu
|
||||
mc alias set local http://minio:9000 "$$MINIO_ROOT_USER" "$$MINIO_ROOT_PASSWORD"
|
||||
mc mb -p "local/$$S3_BUCKET" || true
|
||||
mc anonymous set none "local/$$S3_BUCKET"
|
||||
restart: "no"
|
||||
|
||||
rabbitmq:
|
||||
image: rabbitmq:3.13.7-management-alpine
|
||||
container_name: mtusi-lab5-rabbitmq
|
||||
environment:
|
||||
RABBITMQ_DEFAULT_USER: ${RABBITMQ_DEFAULT_USER}
|
||||
RABBITMQ_DEFAULT_PASS: ${RABBITMQ_DEFAULT_PASS}
|
||||
volumes:
|
||||
- rabbitdata:/var/lib/rabbitmq
|
||||
- ./infra/rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf:ro
|
||||
healthcheck:
|
||||
test: ["CMD", "rabbitmq-diagnostics", "-q", "ping"]
|
||||
interval: 5s
|
||||
timeout: 10s
|
||||
retries: 20
|
||||
start_period: 20s
|
||||
restart: unless-stopped
|
||||
|
||||
api-1:
|
||||
<<: *app
|
||||
container_name: mtusi-lab5-api-1
|
||||
environment:
|
||||
INSTANCE_ID: api-1
|
||||
labels:
|
||||
lab.role: api
|
||||
lab.instance_id: api-1
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=3)"]
|
||||
interval: 3s
|
||||
timeout: 5s
|
||||
retries: 30
|
||||
start_period: 20s
|
||||
|
||||
api-2:
|
||||
<<: *app
|
||||
container_name: mtusi-lab5-api-2
|
||||
environment:
|
||||
INSTANCE_ID: api-2
|
||||
labels:
|
||||
lab.role: api
|
||||
lab.instance_id: api-2
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=3)"]
|
||||
interval: 3s
|
||||
timeout: 5s
|
||||
retries: 30
|
||||
start_period: 20s
|
||||
|
||||
worker:
|
||||
<<: *app
|
||||
container_name: mtusi-lab5-worker
|
||||
command: ["python", "-m", "app.worker"]
|
||||
environment:
|
||||
INSTANCE_ID: worker-1
|
||||
MALLOC_ARENA_MAX: "2"
|
||||
mem_limit: 512m
|
||||
restart: on-failure
|
||||
labels:
|
||||
lab.role: worker
|
||||
|
||||
nginx:
|
||||
image: nginx:1.27.5-alpine
|
||||
container_name: mtusi-lab5-nginx
|
||||
depends_on:
|
||||
api-1:
|
||||
condition: service_healthy
|
||||
api-2:
|
||||
condition: service_healthy
|
||||
ports:
|
||||
- "8080:8080"
|
||||
volumes:
|
||||
- ./infra/nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:8080/health/live"]
|
||||
interval: 3s
|
||||
timeout: 3s
|
||||
retries: 20
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
pgdata:
|
||||
miniodata:
|
||||
rabbitdata:
|
||||
@@ -0,0 +1,30 @@
|
||||
# Расширения лабораторной 5
|
||||
|
||||
Обязательный контракт `contracts/openapi.json` не изменён. Ниже — операторский интерфейс и параметры pipeline, нужные для уровня 4 и трека 5. Эти маршруты не входят в публичный API и требуют заголовок `X-Operator-Token`.
|
||||
|
||||
## Pipeline
|
||||
|
||||
Ключ эффекта: `(attachment_id, pipeline_version, variant)`.
|
||||
|
||||
| Версия | Назначение | JPEG thumbnail/crop | WebP optimized |
|
||||
| --- | --- | --- | --- |
|
||||
| 1 | Основной курс | quality 85, optimize, без progressive и без EXIF | quality 80, method 6, lossy |
|
||||
| 2 | Эксперимент идемпотентности | quality 60 / 70 | quality 50, method 4 |
|
||||
|
||||
Геометрия одинаковая: thumbnail вписывается в 256×256 без увеличения, crop — центр 128×128, optimized вписывается в 1280×1280 без увеличения. EXIF orientation применяется до кодирования, альфа-канал кладётся на белый. Метаданные исходника в варианты не копируются. Версия 2 нужна только чтобы отличить новый результат от старого по SHA-256.
|
||||
|
||||
Повторная доставка, смена порядка и устаревшая версия не создают второй набор объектов одной и той же версии и не заменяют уже опубликованный более новый `ready_pipeline_version`.
|
||||
|
||||
## Операторские маршруты
|
||||
|
||||
- `POST /internal/control/outbox` `{"paused": true}` — остановить публикацию outbox после COMMIT.
|
||||
- `POST /internal/control/fault` `{"mode": "before_effect"|"after_s3"|"after_commit"|"duplicate"|"transient"|"permanent", "remaining": 1}` — одноразовая инъекция. `transient` уменьшает `remaining`.
|
||||
- `POST /internal/control/backpressure` `{"max_queued_jobs": 0}` или `null` — вернуть лимит окружения.
|
||||
- `GET /internal/metrics`, `GET /internal/attachments/{id}`, `GET /internal/failed`.
|
||||
- `POST /internal/attachments/{id}/replay` — новая попытка того же `job_id`. Уже `ready` задание не исполняется повторно.
|
||||
- `POST /internal/attachments/{id}/reprocess` `{"pipeline_version": 2}` — новая версия pipeline.
|
||||
- `POST /internal/jobs/{id}/redeliver` — ещё одна доставка того же сообщения брокером.
|
||||
|
||||
## Повторы
|
||||
|
||||
`JOB_MAX_ATTEMPTS` по умолчанию 5. Временная ошибка возвращает задание в `queued` с задержкой `base * 2^(attempt-1)`, умноженной на случайный коэффициент 0.5–1.5, но не дольше `JOB_BACKOFF_CAP_SECONDS`. Постоянная ошибка и исчерпание попыток пишут `failed_jobs` и копию сообщения в DLQ. Коды: `invalid_image`, `processing_failed`, `retry_exhausted`.
|
||||
@@ -0,0 +1,144 @@
|
||||
{
|
||||
"track": 2,
|
||||
"results": [
|
||||
{
|
||||
"scenario": "grade4 variants and broker payload",
|
||||
"result": "pass",
|
||||
"detail": {
|
||||
"attachment_id": "9fa448b7-09b8-4913-8b7f-2c739b5755b3",
|
||||
"job_id": "590aa490-01f5-407d-91ea-495ebf067b4e"
|
||||
}
|
||||
},
|
||||
{
|
||||
"scenario": "idempotency key order",
|
||||
"result": "pass",
|
||||
"detail": {
|
||||
"message_id": "b0f70dd7-777f-40d4-82e5-9b0ed7a49f83"
|
||||
}
|
||||
},
|
||||
{
|
||||
"scenario": "validation",
|
||||
"result": "pass",
|
||||
"detail": {
|
||||
"rejected": [
|
||||
"extra_field",
|
||||
"invalid_image",
|
||||
"pixel_limit"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"scenario": "backpressure",
|
||||
"result": "pass",
|
||||
"detail": {
|
||||
"max_queued_jobs": 0,
|
||||
"status": 503
|
||||
}
|
||||
},
|
||||
{
|
||||
"scenario": "outbox commit/publish gap",
|
||||
"result": "pass",
|
||||
"detail": {
|
||||
"attachment_id": "2a18cbd8-25bf-4d7f-86df-df2eeba65217",
|
||||
"job_id": "4f6cb35f-26b0-4ac3-ba82-f4df1a531876"
|
||||
}
|
||||
},
|
||||
{
|
||||
"scenario": "worker stop, broker restart, resume",
|
||||
"result": "pass",
|
||||
"detail": {
|
||||
"attachment_id": "54a139ee-963c-48c5-8bae-13853c4630bb",
|
||||
"queue_before": 1,
|
||||
"queue_after_restart": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"scenario": "permanent failure, DLQ registry, safe replay",
|
||||
"result": "pass",
|
||||
"detail": {
|
||||
"attachment_id": "27931bef-3404-4b63-a7d7-1c757f43263f",
|
||||
"job_id": "b2c59c26-07ac-4419-9b85-a0fb6fab79e2"
|
||||
}
|
||||
},
|
||||
{
|
||||
"scenario": "transient retry with backoff",
|
||||
"result": "pass",
|
||||
"detail": {
|
||||
"attachment_id": "27158119-313a-4582-bd19-73a700ce5cbc",
|
||||
"attempt": 2
|
||||
}
|
||||
},
|
||||
{
|
||||
"scenario": "retry budget then explicit replay",
|
||||
"result": "pass",
|
||||
"detail": {
|
||||
"attachment_id": "ea24b78e-fd31-4a47-ab82-05b0583964b6"
|
||||
}
|
||||
},
|
||||
{
|
||||
"scenario": "duplicate delivery is a no-op",
|
||||
"result": "pass",
|
||||
"detail": {
|
||||
"attachment_id": "3d7f00ae-089b-474b-9f6c-e92becc45a3f",
|
||||
"job_id": "ffc5490a-e210-4652-a3a8-63210a1a6675"
|
||||
}
|
||||
},
|
||||
{
|
||||
"scenario": "crash before side effect",
|
||||
"result": "pass",
|
||||
"detail": {
|
||||
"attachment_id": "e7367911-2e2b-4a36-8001-b777fc1a0352",
|
||||
"audit": "fault_before_effect"
|
||||
}
|
||||
},
|
||||
{
|
||||
"scenario": "crash after object write",
|
||||
"result": "pass",
|
||||
"detail": {
|
||||
"attachment_id": "946b250d-8ead-4a2b-9dc4-797db8a27fed",
|
||||
"audit": "fault_after_s3"
|
||||
}
|
||||
},
|
||||
{
|
||||
"scenario": "crash after commit before ack",
|
||||
"result": "pass",
|
||||
"detail": {
|
||||
"attachment_id": "d3bd57dd-2ee9-48ba-8ded-3d94a798db6a",
|
||||
"audit": "fault_after_commit"
|
||||
}
|
||||
},
|
||||
{
|
||||
"scenario": "versioned pipeline rejects stale result",
|
||||
"result": "pass",
|
||||
"detail": {
|
||||
"attachment_id": "1dc4f3f5-74bd-4f6a-820a-3a2fa00c5f80",
|
||||
"new_job_id": "613fa584-6bbb-441c-b280-2354b9d4837c"
|
||||
}
|
||||
},
|
||||
{
|
||||
"scenario": "reordered older job cannot win",
|
||||
"result": "pass",
|
||||
"detail": {
|
||||
"attachment_id": "e83b88ea-54ea-4f20-bcb1-60bbe3ba41b7"
|
||||
}
|
||||
},
|
||||
{
|
||||
"scenario": "volume restart preserves session message and object",
|
||||
"result": "pass",
|
||||
"detail": {
|
||||
"message_id": "f7f9c162-7ac1-4f61-9bba-a6a02e0bf805",
|
||||
"attachment_id": "1dc4f3f5-74bd-4f6a-820a-3a2fa00c5f80"
|
||||
}
|
||||
},
|
||||
{
|
||||
"scenario": "two API replicas",
|
||||
"result": "pass",
|
||||
"detail": {
|
||||
"instances": [
|
||||
"api-1",
|
||||
"api-2"
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
worker_processes auto;
|
||||
error_log /dev/stderr warn;
|
||||
pid /tmp/nginx.pid;
|
||||
|
||||
events {
|
||||
worker_connections 1024;
|
||||
}
|
||||
|
||||
http {
|
||||
access_log off;
|
||||
client_max_body_size 12m;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Connection "";
|
||||
gzip off;
|
||||
|
||||
upstream messenger_api {
|
||||
server api-1:8080 max_fails=0;
|
||||
server api-2:8080 max_fails=0;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 8080;
|
||||
location / {
|
||||
proxy_pass http://messenger_api;
|
||||
proxy_connect_timeout 1s;
|
||||
proxy_send_timeout 60s;
|
||||
proxy_read_timeout 60s;
|
||||
proxy_next_upstream error timeout http_502 http_503 non_idempotent;
|
||||
proxy_next_upstream_tries 2;
|
||||
proxy_next_upstream_timeout 3s;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
disk_free_limit.absolute = 50MB
|
||||
consumer_timeout = 3600000
|
||||
@@ -0,0 +1,154 @@
|
||||
-- Idempotent schema for lab5. Applied under a session advisory lock at startup.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS users (
|
||||
id uuid PRIMARY KEY,
|
||||
username text NOT NULL UNIQUE,
|
||||
display_name text NOT NULL,
|
||||
password_hash text NOT NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS sessions (
|
||||
token_hash text PRIMARY KEY,
|
||||
user_id uuid NOT NULL REFERENCES users (id),
|
||||
expires_at timestamptz NOT NULL,
|
||||
revoked_at timestamptz
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS sessions_user_idx ON sessions (user_id);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chats (
|
||||
id uuid PRIMARY KEY,
|
||||
title text NOT NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chat_members (
|
||||
chat_id uuid NOT NULL REFERENCES chats (id),
|
||||
user_id uuid NOT NULL REFERENCES users (id),
|
||||
position integer NOT NULL,
|
||||
PRIMARY KEY (chat_id, user_id)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS chat_members_user_idx ON chat_members (user_id, chat_id);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS messages (
|
||||
id uuid PRIMARY KEY,
|
||||
chat_id uuid NOT NULL REFERENCES chats (id),
|
||||
sender_id uuid NOT NULL REFERENCES users (id),
|
||||
body text NOT NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS messages_chat_order_idx ON messages (chat_id, created_at, id);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS attachments (
|
||||
id uuid PRIMARY KEY,
|
||||
chat_id uuid NOT NULL REFERENCES chats (id),
|
||||
owner_id uuid NOT NULL REFERENCES users (id),
|
||||
filename text NOT NULL,
|
||||
content_type text NOT NULL,
|
||||
size_bytes integer NOT NULL,
|
||||
sha256 text NOT NULL,
|
||||
status text NOT NULL,
|
||||
error_code text,
|
||||
pipeline_version integer NOT NULL,
|
||||
ready_pipeline_version integer,
|
||||
staging_key text NOT NULL,
|
||||
original_key text NOT NULL,
|
||||
staging_deleted_at timestamptz,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
updated_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
CONSTRAINT attachments_status_chk CHECK (status IN ('queued', 'processing', 'ready', 'failed'))
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS message_attachments (
|
||||
message_id uuid NOT NULL REFERENCES messages (id),
|
||||
attachment_id uuid NOT NULL REFERENCES attachments (id),
|
||||
position integer NOT NULL,
|
||||
PRIMARY KEY (message_id, position)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS idempotency_keys (
|
||||
user_id uuid NOT NULL REFERENCES users (id),
|
||||
http_method text NOT NULL,
|
||||
path text NOT NULL,
|
||||
idem_key text NOT NULL,
|
||||
request_hash text NOT NULL,
|
||||
response_json jsonb,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
PRIMARY KEY (user_id, http_method, path, idem_key)
|
||||
);
|
||||
|
||||
-- One logical job per (attachment, pipeline version). job_id is stable across redelivery and replay.
|
||||
CREATE TABLE IF NOT EXISTS jobs (
|
||||
job_id uuid PRIMARY KEY,
|
||||
attachment_id uuid NOT NULL REFERENCES attachments (id),
|
||||
pipeline_version integer NOT NULL,
|
||||
attempt integer NOT NULL,
|
||||
status text NOT NULL,
|
||||
error_code text,
|
||||
error_class text,
|
||||
last_error text,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
updated_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
next_attempt_at timestamptz,
|
||||
lease_until timestamptz,
|
||||
lease_owner text,
|
||||
CONSTRAINT jobs_version_uniq UNIQUE (attachment_id, pipeline_version),
|
||||
CONSTRAINT jobs_status_chk CHECK (status IN ('queued', 'processing', 'ready', 'failed', 'superseded'))
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS jobs_status_idx ON jobs (status, next_attempt_at);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS attachment_variants (
|
||||
attachment_id uuid NOT NULL REFERENCES attachments (id),
|
||||
pipeline_version integer NOT NULL,
|
||||
name text NOT NULL,
|
||||
content_type text NOT NULL,
|
||||
size_bytes integer NOT NULL,
|
||||
width integer NOT NULL,
|
||||
height integer NOT NULL,
|
||||
sha256 text NOT NULL,
|
||||
object_key text NOT NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
PRIMARY KEY (attachment_id, pipeline_version, name)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS outbox (
|
||||
id bigserial PRIMARY KEY,
|
||||
job_id uuid NOT NULL REFERENCES jobs (job_id),
|
||||
payload jsonb NOT NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
published_at timestamptz
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS outbox_unpublished_idx ON outbox (id) WHERE published_at IS NULL;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS failed_jobs (
|
||||
id bigserial PRIMARY KEY,
|
||||
job_id uuid NOT NULL,
|
||||
attachment_id uuid NOT NULL,
|
||||
pipeline_version integer NOT NULL,
|
||||
attempt integer NOT NULL,
|
||||
error_code text NOT NULL,
|
||||
error_class text NOT NULL,
|
||||
safe_message text NOT NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
replayed_at timestamptz
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS job_audit (
|
||||
id bigserial PRIMARY KEY,
|
||||
job_id uuid,
|
||||
attachment_id uuid,
|
||||
action text NOT NULL,
|
||||
detail jsonb NOT NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS runtime_control (
|
||||
key text PRIMARY KEY,
|
||||
value jsonb NOT NULL
|
||||
);
|
||||
|
||||
INSERT INTO runtime_control (key, value) VALUES
|
||||
('outbox_paused', 'false'::jsonb),
|
||||
('worker_fault', '{"mode": null, "armed": false}'::jsonb),
|
||||
('max_queued_jobs', 'null'::jsonb)
|
||||
ON CONFLICT (key) DO NOTHING;
|
||||
@@ -0,0 +1,7 @@
|
||||
fastapi==0.115.6
|
||||
uvicorn==0.34.0
|
||||
psycopg[binary,pool]==3.2.4
|
||||
argon2-cffi==23.1.0
|
||||
boto3==1.35.99
|
||||
pika==1.3.2
|
||||
Pillow==11.1.0
|
||||
@@ -0,0 +1,54 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Create a local .env for the synthetic stand. Existing files are left untouched."""
|
||||
import secrets
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
TARGET = ROOT / ".env"
|
||||
|
||||
|
||||
def main() -> None:
|
||||
if TARGET.exists():
|
||||
print("local env already exists")
|
||||
return
|
||||
database_password = secrets.token_hex(16)
|
||||
object_password = secrets.token_hex(16)
|
||||
broker_password = secrets.token_hex(16)
|
||||
operator = secrets.token_hex(16)
|
||||
lines = [
|
||||
"POSTGRES_USER=messenger",
|
||||
"POSTGRES_PASSWORD=" + database_password,
|
||||
"POSTGRES_DB=messenger",
|
||||
"MINIO_ROOT_USER=labminio",
|
||||
"MINIO_ROOT_PASSWORD=" + object_password,
|
||||
"S3_ACCESS_KEY_ID=labminio",
|
||||
"S3_SECRET_ACCESS_KEY=" + object_password,
|
||||
"S3_ENDPOINT=http://minio:9000",
|
||||
"S3_REGION=us-east-1",
|
||||
"S3_BUCKET=messenger-attachments",
|
||||
"RABBITMQ_DEFAULT_USER=messenger",
|
||||
"RABBITMQ_DEFAULT_PASS=" + broker_password,
|
||||
"DATABASE_URL=postgresql://messenger:" + database_password + "@postgres:5432/messenger",
|
||||
"BROKER_URL=amqp://messenger:" + broker_password + "@rabbitmq:5672/",
|
||||
"SESSION_TTL_SECONDS=3600",
|
||||
"IDEMPOTENCY_TTL_SECONDS=86400",
|
||||
"MAX_UPLOAD_BYTES=10485760",
|
||||
"MAX_IMAGE_PIXELS=40000000",
|
||||
"WORKER_CONCURRENCY=2",
|
||||
"JOB_MAX_ATTEMPTS=5",
|
||||
"JOB_TIMEOUT_SECONDS=60",
|
||||
"JOB_BACKOFF_BASE_SECONDS=0.4",
|
||||
"JOB_BACKOFF_CAP_SECONDS=8",
|
||||
"STAGING_RETENTION_SECONDS=86400",
|
||||
"MAX_QUEUED_JOBS=500",
|
||||
"OPERATOR_TOKEN=" + operator,
|
||||
"APP_HOST=0.0.0.0",
|
||||
"APP_PORT=8080",
|
||||
"",
|
||||
]
|
||||
TARGET.write_text("\n".join(lines), encoding="utf-8", newline="\n")
|
||||
print("wrote local env")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+6
@@ -0,0 +1,6 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
# Restart API and every stateful dependency. Volumes are not removed.
|
||||
docker compose --env-file .env restart postgres minio rabbitmq api-1 api-2 worker nginx
|
||||
exit 0
|
||||
Executable
+5
@@ -0,0 +1,5 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
docker compose --env-file .env start api-1 api-2
|
||||
exit 0
|
||||
Executable
+11
@@ -0,0 +1,11 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
: "${TARGET_INSTANCE_ID:?test provides target API instance alias}"
|
||||
cid=$(docker ps -q --filter "label=lab.instance_id=${TARGET_INSTANCE_ID}")
|
||||
if [ -z "$cid" ]; then
|
||||
echo "no running API replica for the requested instance" >&2
|
||||
exit 1
|
||||
fi
|
||||
# Leave this replica stopped. Do not stop the database, broker, or load balancer.
|
||||
docker stop -t 5 $cid >/dev/null
|
||||
exit 0
|
||||
@@ -0,0 +1,359 @@
|
||||
"""Grade 4/5 scenarios for the local stand. Prints no session tokens."""
|
||||
import hashlib
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from urllib import error, request
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
from client import Client, Failure
|
||||
|
||||
BASE = "http://localhost:8080"
|
||||
EVIDENCE = ROOT / "evidence" / "grade5-track2.json"
|
||||
ALLOWED_MESSAGE_KEYS = {"schema_version", "job_id", "type", "attachment_id", "pipeline_version", "attempt", "created_at", "traceparent", "tracestate"}
|
||||
|
||||
|
||||
def env_value(name):
|
||||
for line in (ROOT / ".env").read_text(encoding="utf-8").splitlines():
|
||||
if line.startswith(name + "="):
|
||||
return line.split("=", 1)[1]
|
||||
raise SystemExit("missing local setting " + name)
|
||||
|
||||
|
||||
TOKEN = env_value("OPERATOR_TOKEN")
|
||||
RESULTS = []
|
||||
|
||||
|
||||
def record(name, detail):
|
||||
RESULTS.append({"scenario": name, "result": "pass", "detail": detail})
|
||||
print("PASS:", name)
|
||||
|
||||
|
||||
def fail(name, message):
|
||||
RESULTS.append({"scenario": name, "result": "fail", "detail": message})
|
||||
EVIDENCE.parent.mkdir(exist_ok=True)
|
||||
EVIDENCE.write_text(json.dumps(RESULTS, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
raise SystemExit(name + ": " + message)
|
||||
|
||||
|
||||
def compose(*args):
|
||||
completed = subprocess.run(["docker", "compose", "--env-file", ".env", *args], cwd=ROOT, text=True, capture_output=True)
|
||||
if completed.returncode != 0:
|
||||
raise SystemExit("compose " + " ".join(args) + " failed: " + completed.stderr[-400:])
|
||||
return completed.stdout
|
||||
|
||||
|
||||
def queue_depth():
|
||||
completed = subprocess.run(
|
||||
["docker", "exec", "mtusi-lab5-rabbitmq", "rabbitmqctl", "list_queues", "name", "messages"],
|
||||
text=True, capture_output=True,
|
||||
)
|
||||
if completed.returncode != 0:
|
||||
return None
|
||||
for line in completed.stdout.splitlines():
|
||||
parts = line.split()
|
||||
if len(parts) == 2 and parts[0] == "attachment.process" and parts[1].isdigit():
|
||||
return int(parts[1])
|
||||
return 0
|
||||
|
||||
|
||||
def operator(method, path, body=None, expected=200):
|
||||
data = None if body is None else json.dumps(body).encode("utf-8")
|
||||
headers = {"X-Operator-Token": TOKEN, "Accept": "application/json"}
|
||||
if body is not None:
|
||||
headers["Content-Type"] = "application/json"
|
||||
call = request.Request(BASE + path, data=data, headers=headers, method=method)
|
||||
try:
|
||||
response = request.urlopen(call, timeout=10)
|
||||
except error.HTTPError as exc:
|
||||
response = exc
|
||||
payload = json.loads(response.read().decode("utf-8"))
|
||||
if response.status != expected:
|
||||
raise Failure(method + " " + path + " -> " + str(response.status) + " " + payload.get("error", {}).get("code", ""))
|
||||
return payload
|
||||
|
||||
|
||||
def reset_controls():
|
||||
operator("POST", "/internal/control/fault", {"mode": None})
|
||||
operator("POST", "/internal/control/outbox", {"paused": False})
|
||||
operator("POST", "/internal/control/backpressure", {"max_queued_jobs": None})
|
||||
compose("start", "worker")
|
||||
|
||||
|
||||
def wait_health(timeout=90):
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
Client(BASE, timeout=3).call("GET", "/health/ready", schema="Health")
|
||||
return
|
||||
except Failure:
|
||||
time.sleep(0.5)
|
||||
raise SystemExit("API did not become ready")
|
||||
|
||||
|
||||
def wait_attachment(client, actor, attachment_id, wanted, timeout=90):
|
||||
deadline = time.monotonic() + timeout
|
||||
last = None
|
||||
while time.monotonic() < deadline:
|
||||
last = client.call("GET", "/api/v1/attachments/" + attachment_id, actor=actor, schema="Attachment")["json"]
|
||||
if last["status"] == wanted:
|
||||
return last
|
||||
if wanted != "failed" and last["status"] == "failed":
|
||||
raise Failure("attachment failed with " + str(last["error_code"]))
|
||||
time.sleep(0.4)
|
||||
raise Failure("timed out waiting for " + wanted + ", last status " + str(last and last["status"]))
|
||||
|
||||
|
||||
def upload_sample(client, actor, chat):
|
||||
content = (ROOT / "tests" / "fixtures" / "sample.png").read_bytes()
|
||||
attachment = client.call(
|
||||
"POST",
|
||||
"/api/v1/chats/" + chat["id"] + "/attachments?filename=sample.png",
|
||||
202,
|
||||
actor=actor,
|
||||
raw=content,
|
||||
headers={"Content-Type": "image/png"},
|
||||
schema="Attachment",
|
||||
)["json"]
|
||||
if attachment["status"] != "queued":
|
||||
raise Failure("upload snapshot is not queued")
|
||||
return attachment, content
|
||||
|
||||
|
||||
def main():
|
||||
reset_controls()
|
||||
wait_health()
|
||||
client = Client(BASE, timeout=10)
|
||||
alice, bob, _eve, chat = client.bootstrap()
|
||||
|
||||
attachment, content = upload_sample(client, alice, chat)
|
||||
ready = wait_attachment(client, bob, attachment["id"], "ready")
|
||||
names = {item["name"]: item for item in ready["variants"]}
|
||||
if set(names) != {"thumbnail", "crop", "optimized"}:
|
||||
fail("variants", "unexpected variant set")
|
||||
if (names["thumbnail"]["width"], names["thumbnail"]["height"]) != (256, 160):
|
||||
fail("variants", "thumbnail metadata")
|
||||
if (names["crop"]["width"], names["crop"]["height"]) != (128, 128) or names["crop"]["content_type"] != "image/jpeg":
|
||||
fail("variants", "crop contract")
|
||||
if names["optimized"]["content_type"] != "image/webp" or (names["optimized"]["width"], names["optimized"]["height"]) != (320, 200):
|
||||
fail("variants", "optimized contract")
|
||||
original = client.call("GET", "/api/v1/attachments/" + attachment["id"] + "/content", actor=bob)
|
||||
thumb = client.call("GET", "/api/v1/attachments/" + attachment["id"] + "/content?variant=thumbnail", actor=bob)
|
||||
optimized = client.call("GET", "/api/v1/attachments/" + attachment["id"] + "/content?variant=optimized", actor=bob)
|
||||
if original["bytes"] != content or thumb["headers"].get_content_type() != "image/jpeg":
|
||||
fail("variants", "bytes or thumbnail type")
|
||||
if hashlib.sha256(thumb["bytes"]).hexdigest() != names["thumbnail"]["sha256"] or optimized["bytes"][8:12] != b"WEBP":
|
||||
fail("variants", "checksum or webp header")
|
||||
state = operator("GET", "/internal/attachments/" + attachment["id"])
|
||||
if any(set(message) - ALLOWED_MESSAGE_KEYS or any(isinstance(value, str) and len(value) > 512 for value in message.values()) for message in state["messages"]):
|
||||
fail("variants", "broker payload contains unexpected data")
|
||||
record("grade4 variants and broker payload", {"attachment_id": attachment["id"], "job_id": state["jobs"][0]["job_id"]})
|
||||
|
||||
key = "order-" + uuid.uuid4().hex
|
||||
path = "/api/v1/chats/" + chat["id"] + "/messages"
|
||||
first = client.call("POST", path, 201, actor=alice, raw=b'{"text":"same-body","attachment_ids":[]}', headers={"Content-Type": "application/json", "Idempotency-Key": key}, schema="Message")["json"]
|
||||
second = client.call("POST", path, 201, actor=alice, raw=b'{"attachment_ids":[],"text":"same-body"}', headers={"Content-Type": "application/json", "Idempotency-Key": key}, schema="Message")["json"]
|
||||
if first != second:
|
||||
fail("idempotency-order", "canonical replay differs")
|
||||
record("idempotency key order", {"message_id": first["id"]})
|
||||
|
||||
client.call("POST", "/api/v1/users", 400, body={"username": "abc", "display_name": "Имя", "password": "short", "extra": 1})
|
||||
client.call("POST", "/api/v1/chats/" + chat["id"] + "/attachments?filename=bad.png", 415, actor=alice, raw=b"not-an-image", headers={"Content-Type": "image/png"})
|
||||
import struct
|
||||
import zlib
|
||||
ihdr = struct.pack(">IIBBBBB", 8000, 8000, 8, 2, 0, 0, 0)
|
||||
chunk = struct.pack(">I", len(ihdr)) + b"IHDR" + ihdr + struct.pack(">I", zlib.crc32(b"IHDR" + ihdr) & 0xFFFFFFFF)
|
||||
bomb = b"\x89PNG\r\n\x1a\n" + chunk + struct.pack(">I", 0) + b"IEND" + struct.pack(">I", zlib.crc32(b"IEND") & 0xFFFFFFFF)
|
||||
client.call("POST", "/api/v1/chats/" + chat["id"] + "/attachments?filename=bomb.png", 413, actor=alice, raw=bomb, headers={"Content-Type": "image/png"})
|
||||
record("validation", {"rejected": ["extra_field", "invalid_image", "pixel_limit"]})
|
||||
|
||||
operator("POST", "/internal/control/backpressure", {"max_queued_jobs": 0})
|
||||
try:
|
||||
client.call("POST", "/api/v1/chats/" + chat["id"] + "/attachments?filename=blocked.png", 503, actor=alice, raw=content, headers={"Content-Type": "image/png"})
|
||||
finally:
|
||||
operator("POST", "/internal/control/backpressure", {"max_queued_jobs": None})
|
||||
record("backpressure", {"max_queued_jobs": 0, "status": 503})
|
||||
|
||||
operator("POST", "/internal/control/outbox", {"paused": True})
|
||||
try:
|
||||
paused, _content = upload_sample(client, alice, chat)
|
||||
time.sleep(1.2)
|
||||
metrics = operator("GET", "/internal/metrics")
|
||||
hidden = operator("GET", "/internal/attachments/" + paused["id"])
|
||||
if metrics["outbox_unpublished"] < 1 or hidden["attachment"]["status"] != "queued" or hidden["jobs"][0]["job_id"] is None:
|
||||
fail("outbox-gap", "intent was not retained while publish was paused")
|
||||
if queue_depth() not in {0, None}:
|
||||
fail("outbox-gap", "broker received a message before publish resumed")
|
||||
job_id = hidden["jobs"][0]["job_id"]
|
||||
finally:
|
||||
operator("POST", "/internal/control/outbox", {"paused": False})
|
||||
resumed = wait_attachment(client, alice, paused["id"], "ready")
|
||||
after = operator("GET", "/internal/attachments/" + paused["id"])
|
||||
if after["jobs"][0]["job_id"] != job_id or resumed["status"] != "ready":
|
||||
fail("outbox-gap", "recovery created a different job")
|
||||
record("outbox commit/publish gap", {"attachment_id": paused["id"], "job_id": job_id})
|
||||
|
||||
compose("stop", "worker")
|
||||
try:
|
||||
held, held_bytes = upload_sample(client, alice, chat)
|
||||
deadline = time.monotonic() + 20
|
||||
while time.monotonic() < deadline and operator("GET", "/internal/metrics")["outbox_unpublished"] != 0:
|
||||
time.sleep(0.3)
|
||||
if operator("GET", "/internal/attachments/" + held["id"])["attachment"]["status"] != "queued":
|
||||
fail("worker-stop", "API processed the image synchronously")
|
||||
client.call("GET", "/api/v1/attachments/" + held["id"] + "/content", 409, actor=bob)
|
||||
client.message(bob, chat, "чат жив без worker")
|
||||
depth_before = queue_depth()
|
||||
compose("restart", "api-1", "api-2", "rabbitmq")
|
||||
wait_health()
|
||||
depth_after = None
|
||||
deadline = time.monotonic() + 40
|
||||
while time.monotonic() < deadline:
|
||||
depth_after = queue_depth()
|
||||
if depth_after is not None:
|
||||
break
|
||||
time.sleep(1)
|
||||
if not depth_before or not depth_after:
|
||||
fail("worker-stop", "durable queue did not retain the message across broker restart")
|
||||
finally:
|
||||
compose("start", "worker")
|
||||
held_ready = wait_attachment(client, bob, held["id"], "ready")
|
||||
downloaded = client.call("GET", "/api/v1/attachments/" + held["id"] + "/content?variant=thumbnail", actor=alice)
|
||||
if held_ready["id"] != held["id"] or downloaded["headers"].get_content_type() != "image/jpeg":
|
||||
fail("worker-stop", "restarted worker did not finish the accepted job")
|
||||
if client.call("GET", "/api/v1/attachments/" + held["id"] + "/content", actor=bob)["bytes"] != held_bytes:
|
||||
fail("worker-stop", "original changed")
|
||||
record("worker stop, broker restart, resume", {"attachment_id": held["id"], "queue_before": depth_before, "queue_after_restart": depth_after})
|
||||
|
||||
operator("POST", "/internal/control/fault", {"mode": "permanent"})
|
||||
rejected, _bytes = upload_sample(client, alice, chat)
|
||||
failed = wait_attachment(client, alice, rejected["id"], "failed")
|
||||
if failed["error_code"] != "invalid_image":
|
||||
fail("permanent-replay", "unexpected error code")
|
||||
client.call("GET", "/api/v1/attachments/" + rejected["id"] + "/content", 409, actor=alice)
|
||||
client.call("POST", "/api/v1/chats/" + chat["id"] + "/messages", 409, actor=alice, body={"text": "", "attachment_ids": [rejected["id"]]}, headers={"Idempotency-Key": uuid.uuid4().hex})
|
||||
replay = operator("POST", "/internal/attachments/" + rejected["id"] + "/replay")
|
||||
if not replay["replayed"] or replay["job_id"] != operator("GET", "/internal/attachments/" + rejected["id"])["jobs"][0]["job_id"]:
|
||||
fail("permanent-replay", "replay changed the logical job")
|
||||
restored = wait_attachment(client, alice, rejected["id"], "ready")
|
||||
again = operator("POST", "/internal/attachments/" + rejected["id"] + "/replay")
|
||||
if again["reason"] != "already_ready" or len(restored["variants"]) != 3:
|
||||
fail("permanent-replay", "successful job was replayed as new work")
|
||||
record("permanent failure, DLQ registry, safe replay", {"attachment_id": rejected["id"], "job_id": replay["job_id"]})
|
||||
|
||||
operator("POST", "/internal/control/fault", {"mode": "transient", "remaining": 1})
|
||||
retried, _bytes = upload_sample(client, alice, chat)
|
||||
retried_ready = wait_attachment(client, alice, retried["id"], "ready", timeout=60)
|
||||
retried_state = operator("GET", "/internal/attachments/" + retried["id"])
|
||||
if retried_state["jobs"][0]["attempt"] != 2 or retried_ready["status"] != "ready":
|
||||
fail("backoff", "single transient failure did not retry exactly once")
|
||||
record("transient retry with backoff", {"attachment_id": retried["id"], "attempt": 2})
|
||||
|
||||
operator("POST", "/internal/control/fault", {"mode": "transient", "remaining": 5})
|
||||
exhausted, _bytes = upload_sample(client, alice, chat)
|
||||
exhausted_state = wait_attachment(client, alice, exhausted["id"], "failed", timeout=60)
|
||||
if exhausted_state["error_code"] != "retry_exhausted":
|
||||
fail("retry-budget", "attempts were not capped")
|
||||
if operator("GET", "/internal/metrics")["failed_registry"] < 1:
|
||||
fail("retry-budget", "failed registry is empty")
|
||||
operator("POST", "/internal/attachments/" + exhausted["id"] + "/replay")
|
||||
wait_attachment(client, alice, exhausted["id"], "ready", timeout=60)
|
||||
record("retry budget then explicit replay", {"attachment_id": exhausted["id"]})
|
||||
|
||||
operator("POST", "/internal/control/fault", {"mode": "duplicate"})
|
||||
duplicated, _bytes = upload_sample(client, alice, chat)
|
||||
wait_attachment(client, alice, duplicated["id"], "ready")
|
||||
duplicate_state = operator("GET", "/internal/attachments/" + duplicated["id"])
|
||||
visible = [item for item in duplicate_state["variants"] if item["pipeline_version"] == 1]
|
||||
if len(visible) != 3 or "noop_ready" not in {item["action"] for item in duplicate_state["audits"]}:
|
||||
fail("duplicate", "inline duplicate created another variant set or skipped the no-op")
|
||||
before = {item["sha256"] for item in visible}
|
||||
operator("POST", "/internal/jobs/" + duplicate_state["jobs"][0]["job_id"] + "/redeliver")
|
||||
deadline = time.monotonic() + 20
|
||||
while time.monotonic() < deadline and sum(1 for item in operator("GET", "/internal/attachments/" + duplicated["id"])["audits"] if item["action"] == "noop_ready") < 2:
|
||||
time.sleep(0.4)
|
||||
after_state = operator("GET", "/internal/attachments/" + duplicated["id"])
|
||||
after = {item["sha256"] for item in after_state["variants"] if item["pipeline_version"] == 1}
|
||||
if before != after or len(after_state["variants"]) != 3:
|
||||
fail("duplicate", "broker redelivery rewrote the result")
|
||||
record("duplicate delivery is a no-op", {"attachment_id": duplicated["id"], "job_id": duplicate_state["jobs"][0]["job_id"]})
|
||||
|
||||
for mode, label in (("before_effect", "crash before side effect"), ("after_s3", "crash after object write"), ("after_commit", "crash after commit before ack")):
|
||||
operator("POST", "/internal/control/fault", {"mode": mode})
|
||||
crashed, _bytes = upload_sample(client, alice, chat)
|
||||
wait_attachment(client, alice, crashed["id"], "ready", timeout=90)
|
||||
audits = {item["action"] for item in operator("GET", "/internal/attachments/" + crashed["id"])["audits"]}
|
||||
expected = {"before_effect": "fault_before_effect", "after_s3": "fault_after_s3", "after_commit": "fault_after_commit"}[mode]
|
||||
if expected not in audits or len([item for item in operator("GET", "/internal/attachments/" + crashed["id"])["variants"] if item["pipeline_version"] == 1]) != 3:
|
||||
fail(label, "fault was not recovered idempotently")
|
||||
record(label, {"attachment_id": crashed["id"], "audit": expected})
|
||||
|
||||
baseline, _bytes = upload_sample(client, alice, chat)
|
||||
baseline_ready = wait_attachment(client, alice, baseline["id"], "ready")
|
||||
first_sha = {item["name"]: item["sha256"] for item in baseline_ready["variants"]}
|
||||
reprocess = operator("POST", "/internal/attachments/" + baseline["id"] + "/reprocess", {"pipeline_version": 2})
|
||||
version_two = wait_attachment(client, alice, baseline["id"], "ready")
|
||||
second_sha = {item["name"]: item["sha256"] for item in version_two["variants"]}
|
||||
version_state = operator("GET", "/internal/attachments/" + baseline["id"])
|
||||
if version_state["attachment"]["ready_pipeline_version"] != 2 or first_sha["thumbnail"] == second_sha["thumbnail"]:
|
||||
fail("pipeline-version", "version 2 did not publish a distinct result")
|
||||
if len(version_state["variants"]) != 6:
|
||||
fail("pipeline-version", "versioned variants were overwritten in place")
|
||||
operator("POST", "/internal/jobs/" + version_state["jobs"][0]["job_id"] + "/redeliver")
|
||||
time.sleep(2)
|
||||
stale = operator("GET", "/internal/attachments/" + baseline["id"])
|
||||
public = client.call("GET", "/api/v1/attachments/" + baseline["id"], actor=alice, schema="Attachment")["json"]
|
||||
if stale["attachment"]["ready_pipeline_version"] != 2 or {item["sha256"] for item in public["variants"]} != set(second_sha.values()):
|
||||
fail("pipeline-version", "stale pipeline overwrote the newer result")
|
||||
record("versioned pipeline rejects stale result", {"attachment_id": baseline["id"], "new_job_id": reprocess["job_id"]})
|
||||
|
||||
operator("POST", "/internal/control/outbox", {"paused": True})
|
||||
try:
|
||||
ordered, _bytes = upload_sample(client, alice, chat)
|
||||
operator("POST", "/internal/attachments/" + ordered["id"] + "/reprocess", {"pipeline_version": 2})
|
||||
finally:
|
||||
operator("POST", "/internal/control/outbox", {"paused": False})
|
||||
ordered_ready = wait_attachment(client, alice, ordered["id"], "ready")
|
||||
ordered_state = operator("GET", "/internal/attachments/" + ordered["id"])
|
||||
if ordered_state["attachment"]["ready_pipeline_version"] != 2 or "superseded" not in {job["status"] for job in ordered_state["jobs"]}:
|
||||
fail("reorder", "older queued job was not superseded by the newer pipeline")
|
||||
if {item["name"] for item in ordered_ready["variants"]} != {"thumbnail", "crop", "optimized"}:
|
||||
fail("reorder", "public result is not the complete newer pipeline")
|
||||
record("reordered older job cannot win", {"attachment_id": ordered["id"]})
|
||||
|
||||
persisted = client.message(alice, chat, "пережить рестарт")["json"]
|
||||
compose("restart", "postgres", "minio", "rabbitmq", "api-1", "api-2", "worker", "nginx")
|
||||
wait_health(120)
|
||||
me = client.call("GET", "/api/v1/users/me", actor=alice, schema="User")["json"]
|
||||
history = client.call("GET", "/api/v1/chats/" + chat["id"] + "/messages?limit=100", actor=bob, schema="MessagePage")["json"]
|
||||
if me["id"] != alice["id"] or not any(item == persisted for item in history["items"]):
|
||||
fail("persistence", "session or message did not survive restart")
|
||||
if client.call("GET", "/api/v1/attachments/" + baseline["id"] + "/content", actor=bob)["bytes"] != content:
|
||||
fail("persistence", "object bytes did not survive restart")
|
||||
record("volume restart preserves session message and object", {"message_id": persisted["id"], "attachment_id": baseline["id"]})
|
||||
|
||||
observed = set()
|
||||
for _ in range(20):
|
||||
observed.add(client.call("GET", "/api/v1/users/me", actor=bob, schema="User")["headers"]["X-Instance-Id"])
|
||||
if observed != {"api-1", "api-2"}:
|
||||
fail("replicas", "load balancer did not expose both API replicas")
|
||||
record("two API replicas", {"instances": sorted(observed)})
|
||||
EVIDENCE.parent.mkdir(exist_ok=True)
|
||||
EVIDENCE.write_text(json.dumps({"track": 2, "results": RESULTS}, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
print("PASS: grade 4 and track 2 scenarios")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except Failure as exc:
|
||||
fail("client", str(exc))
|
||||
finally:
|
||||
try:
|
||||
reset_controls()
|
||||
except Exception:
|
||||
pass
|
||||
@@ -0,0 +1,172 @@
|
||||
"""Run the same Docker actions as scripts/restart.sh and scripts/stop-one.sh.
|
||||
|
||||
The public harness executes those POSIX scripts directly. This Windows host
|
||||
cannot spawn a .sh file (WinError 193), so this runner calls the same
|
||||
docker compose / docker stop commands and checks the contract around them.
|
||||
It prints no tokens.
|
||||
"""
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
from client import Client, Failure
|
||||
|
||||
CONTAINERS = [
|
||||
"mtusi-lab5-postgres",
|
||||
"mtusi-lab5-minio",
|
||||
"mtusi-lab5-rabbitmq",
|
||||
"mtusi-lab5-api-1",
|
||||
"mtusi-lab5-api-2",
|
||||
"mtusi-lab5-worker",
|
||||
"mtusi-lab5-nginx",
|
||||
]
|
||||
|
||||
|
||||
def started():
|
||||
completed = subprocess.run(
|
||||
["docker", "inspect", "--format", "{{.Name}} {{.State.StartedAt}}", *CONTAINERS],
|
||||
text=True,
|
||||
capture_output=True,
|
||||
check=True,
|
||||
)
|
||||
return completed.stdout.strip()
|
||||
|
||||
|
||||
def compose(*args):
|
||||
completed = subprocess.run(["docker", "compose", "--env-file", ".env", *args], cwd=ROOT, text=True, capture_output=True)
|
||||
if completed.returncode != 0:
|
||||
raise SystemExit("compose failed: " + completed.stderr[-400:])
|
||||
|
||||
|
||||
def stop_instance(instance):
|
||||
found = subprocess.run(
|
||||
["docker", "ps", "-q", "--filter", "label=lab.instance_id=" + instance],
|
||||
text=True,
|
||||
capture_output=True,
|
||||
check=True,
|
||||
)
|
||||
cid = found.stdout.strip()
|
||||
if not cid:
|
||||
raise SystemExit("no running replica " + instance)
|
||||
completed = subprocess.run(["docker", "stop", "-t", "5", cid], text=True, capture_output=True)
|
||||
if completed.returncode != 0:
|
||||
raise SystemExit("stop failed: " + completed.stderr[-200:])
|
||||
|
||||
|
||||
def wait_ready(client, actor=None, timeout=90):
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
client.call("GET", "/health/ready", schema="Health")
|
||||
if actor is not None:
|
||||
client.call("GET", "/api/v1/users/me", actor=actor, schema="User")
|
||||
return
|
||||
except Failure:
|
||||
time.sleep(0.5)
|
||||
raise SystemExit("stack did not become ready")
|
||||
|
||||
|
||||
def failover(client, alice, chat, target):
|
||||
client.timeout = 2
|
||||
started_at = time.monotonic()
|
||||
outcome = {}
|
||||
|
||||
def stopper():
|
||||
try:
|
||||
stop_instance(target)
|
||||
outcome["ok"] = True
|
||||
except SystemExit as exc:
|
||||
outcome["error"] = str(exc)
|
||||
|
||||
import threading
|
||||
|
||||
thread = threading.Thread(target=stopper)
|
||||
thread.start()
|
||||
successes = []
|
||||
after = []
|
||||
failures = 0
|
||||
first = None
|
||||
key = uuid.uuid4().hex
|
||||
text = "Запись во время отказа " + key
|
||||
while time.monotonic() - started_at < 15:
|
||||
stopped = not thread.is_alive() and outcome.get("ok")
|
||||
try:
|
||||
response = client.message(alice, chat, text, key=key)
|
||||
if stopped:
|
||||
if first is None:
|
||||
first = time.monotonic() - started_at
|
||||
after.append(response["headers"]["X-Instance-Id"])
|
||||
successes.append(response["json"]["id"])
|
||||
key = uuid.uuid4().hex
|
||||
text = "Запись во время отказа " + key
|
||||
except Failure:
|
||||
failures += 1
|
||||
time.sleep(0.1)
|
||||
thread.join(timeout=10)
|
||||
if outcome.get("error"):
|
||||
raise SystemExit(outcome["error"])
|
||||
if first is None or first > 10 or len(after) < 5 or target in after[-5:]:
|
||||
raise SystemExit(
|
||||
"failover " + target + " failed: recovery=" + str(first) + " tail=" + ",".join(after[-5:])
|
||||
)
|
||||
print(
|
||||
"PASS failover",
|
||||
target,
|
||||
"recovery_s",
|
||||
round(first, 3),
|
||||
"writes",
|
||||
len(successes),
|
||||
"failed_attempts",
|
||||
failures,
|
||||
)
|
||||
|
||||
|
||||
def main():
|
||||
print("BEFORE")
|
||||
print(started())
|
||||
client = Client("http://localhost:8080", timeout=10)
|
||||
alice, bob, _eve, chat = client.bootstrap()
|
||||
original = client.message(alice, chat, "Сохранить при отказе")["json"]
|
||||
attachment, content = client.attachment(alice, chat)
|
||||
attachment = client.wait_ready(alice, attachment)
|
||||
compose("restart", "postgres", "minio", "rabbitmq", "api-1", "api-2", "worker", "nginx")
|
||||
print("AFTER")
|
||||
print(started())
|
||||
client.timeout = 10
|
||||
wait_ready(client, alice)
|
||||
me = client.call("GET", "/api/v1/users/me", actor=alice, schema="User")["json"]
|
||||
if me["id"] != alice["id"]:
|
||||
raise SystemExit("session did not survive restart")
|
||||
history = client.call(
|
||||
"GET", "/api/v1/chats/" + chat["id"] + "/messages?limit=100", actor=bob, schema="MessagePage"
|
||||
)["json"]
|
||||
if not any(item == original for item in history["items"]):
|
||||
raise SystemExit("message did not survive restart")
|
||||
downloaded = client.call("GET", "/api/v1/attachments/" + attachment["id"] + "/content", actor=bob)
|
||||
if downloaded["bytes"] != content:
|
||||
raise SystemExit("original bytes changed across restart")
|
||||
print("PASS persistence", original["id"], attachment["id"])
|
||||
|
||||
client.timeout = 10
|
||||
observed = set()
|
||||
for _ in range(40):
|
||||
observed.add(client.call("GET", "/api/v1/users/me", actor=alice, schema="User")["headers"]["X-Instance-Id"])
|
||||
if observed != {"api-1", "api-2"}:
|
||||
raise SystemExit("expected both replicas, saw " + ",".join(sorted(observed)))
|
||||
for target in ("api-1", "api-2"):
|
||||
failover(client, alice, chat, target)
|
||||
compose("start", target)
|
||||
client.timeout = 10
|
||||
wait_ready(client)
|
||||
print("PASS hooks")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
finally:
|
||||
compose("start", "api-1", "api-2")
|
||||
@@ -0,0 +1,65 @@
|
||||
"""Geometry checks for the image contract. Runs without the stand."""
|
||||
import io
|
||||
import struct
|
||||
import sys
|
||||
import zlib
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT))
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
|
||||
from app.images import decode_image, inspect_upload, pipeline_params, render_variants
|
||||
from image_probe import jpeg_size
|
||||
from PIL import Image
|
||||
|
||||
|
||||
def check(condition, message):
|
||||
if not condition:
|
||||
raise SystemExit(message)
|
||||
|
||||
|
||||
def png_chunk(tag, data):
|
||||
crc = zlib.crc32(tag + data) & 0xFFFFFFFF
|
||||
return struct.pack(">I", len(data)) + tag + data + struct.pack(">I", crc)
|
||||
|
||||
|
||||
def main():
|
||||
sample = (ROOT / "tests" / "fixtures" / "sample.png").read_bytes()
|
||||
content_type, width, height = inspect_upload(sample, "image/png", 10485760, 40000000)
|
||||
check(content_type == "image/png" and (width, height) == (320, 200), "sample header mismatch")
|
||||
rendered = render_variants(decode_image(sample, 40000000), pipeline_params(1))
|
||||
check(jpeg_size(rendered["thumbnail"]["bytes"]) == (256, 160), "thumbnail geometry")
|
||||
check((rendered["thumbnail"]["width"], rendered["thumbnail"]["height"]) == (256, 160), "thumbnail metadata")
|
||||
check(jpeg_size(rendered["crop"]["bytes"]) == (128, 128), "crop geometry")
|
||||
check(rendered["optimized"]["bytes"][8:12] == b"WEBP", "optimized is not WebP")
|
||||
check((rendered["optimized"]["width"], rendered["optimized"]["height"]) == (320, 200), "optimized geometry")
|
||||
tiny = io.BytesIO()
|
||||
Image.new("RGB", (20, 10), (1, 2, 3)).save(tiny, format="PNG")
|
||||
tiny_rendered = render_variants(decode_image(tiny.getvalue(), 40000000), pipeline_params(1))
|
||||
check((tiny_rendered["thumbnail"]["width"], tiny_rendered["thumbnail"]["height"]) == (20, 10), "small thumbnail was upscaled")
|
||||
check((tiny_rendered["crop"]["width"], tiny_rendered["crop"]["height"]) == (128, 128), "small crop was not upscaled")
|
||||
oriented = io.BytesIO()
|
||||
source = Image.new("RGB", (320, 200), (255, 0, 0))
|
||||
exif = source.getexif()
|
||||
exif[274] = 6
|
||||
source.save(oriented, format="JPEG", exif=exif)
|
||||
turned = decode_image(oriented.getvalue(), 40000000)
|
||||
check(turned.size == (200, 320), "EXIF orientation was ignored")
|
||||
turned_variants = render_variants(turned, pipeline_params(1))
|
||||
check((turned_variants["thumbnail"]["width"], turned_variants["thumbnail"]["height"]) == (160, 256), "oriented thumbnail geometry")
|
||||
version_one = rendered["thumbnail"]["bytes"]
|
||||
version_two = render_variants(decode_image(sample, 40000000), pipeline_params(2))["thumbnail"]["bytes"]
|
||||
check(version_one != version_two, "pipeline versions are indistinguishable")
|
||||
bomb = b"\x89PNG\r\n\x1a\n" + png_chunk(b"IHDR", struct.pack(">IIBBBBB", 8000, 8000, 8, 2, 0, 0, 0)) + png_chunk(b"IEND", b"")
|
||||
try:
|
||||
inspect_upload(bomb, "image/png", 10485760, 40000000)
|
||||
except Exception as exc:
|
||||
check(getattr(exc, "status", None) == 413, "pixel bomb was not rejected as too large")
|
||||
else:
|
||||
raise SystemExit("pixel bomb was accepted")
|
||||
print("PASS: image contract geometry")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user