forked from mtusi2026/mtusi2026lab5
refactor
This commit is contained in:
+2
-1
@@ -11,7 +11,7 @@ SESSION_STORE=postgres
|
||||
CACHE_URL_FILE=/run/secrets/cache_url
|
||||
API_REPLICAS=2
|
||||
IDEMPOTENCY_TTL_SECONDS=86400
|
||||
S3_ENDPOINT=http://s3:9000
|
||||
S3_ENDPOINT=http://s3:8333
|
||||
S3_REGION=us-east-1
|
||||
S3_BUCKET=messenger-attachments
|
||||
S3_ACCESS_KEY_ID_FILE=/run/secrets/s3_access_key_id
|
||||
@@ -25,3 +25,4 @@ JOB_MAX_ATTEMPTS=5
|
||||
JOB_TIMEOUT_SECONDS=60
|
||||
STAGING_RETENTION_SECONDS=86400
|
||||
MAX_PENDING_JOBS=100
|
||||
RENDER_MEMORY_MB=768
|
||||
|
||||
@@ -1,41 +1,173 @@
|
||||
# Отчёт — лабораторная 5
|
||||
|
||||
- Автор, группа: Matvey Kozyrev, BVT2301
|
||||
- Целевая оценка; выбранный трек 5, если нужен: 5, трек 1 «Очередь под перегрузкой»
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
- Автор: Матвей Козырев, BVT2301.
|
||||
- Целевая оценка: 5, трек 1 «Очередь под перегрузкой».
|
||||
- Ветка: `solution/BVT2301-Kozyrev`. Базовый commit: `4f6e69c` (`add level 5`); исправления критических ошибок и этот отчёт пока не закоммичены. Ссылка на PR не предоставлена.
|
||||
- Дата локальной проверки: 26 октября 2026 года.
|
||||
- Статус: код и локальные проверки подготовлены; выполнение всех требований уровней 3–5 пока не подтверждено.
|
||||
|
||||
## Исходная база, источники и помощь ИИ
|
||||
|
||||
Из собственной lab4 перенесены API на FastAPI, PostgreSQL-миграции, сессии, идемпотентность сообщений, S3-адаптер, две API-реплики и HAProxy. Основание проверки: [README](README.md), [семантика API](contracts/README.md), [состояния вложений](contracts/attachment-states.md), [OpenAPI](contracts/openapi.json), [инфраструктура](contracts/infrastructure.json), [публичные тесты](tests/README.md).
|
||||
|
||||
Использован Codex для переноса исходников, подготовки реализации, аудита, исправлений и оформления отчёта. Локальные проверки не заменяют интеграционную демонстрацию. Справочные первоисточники задания: [RabbitMQ reliability](https://www.rabbitmq.com/docs/reliability), [publisher confirms и consumer acknowledgements](https://www.rabbitmq.com/docs/confirms).
|
||||
|
||||
Зависимости зафиксированы в [pyproject.toml](pyproject.toml) и [uv.lock](uv.lock): FastAPI 0.141.1, Uvicorn 0.40.0, asyncpg 0.31.0, argon2-cffi 25.1.0, boto3 1.43.115, Pillow 13.0.0, pika 1.3.2. Имя пакета пока сохранено от lab4.
|
||||
|
||||
## Запуск из чистого clone
|
||||
|
||||
Python 3.12, PostgreSQL 17.6, SeaweedFS 4.17, RabbitMQ 4.1 и HAProxy 3.0 запускаются через Compose. Для локального синтетического стенда: `python3 scripts/init-secrets.py`, затем `docker compose up -d --build`; дождаться `docker compose ps` со статусом healthy и выполнить `make test`. Локальные секреты создаются в игнорируемом `secrets/`. Остановка без удаления данных: `docker compose down`; удаление учебных данных выполняется оператором отдельно через `docker compose down -v`.
|
||||
Нужны Git, Python 3.12+ для приложения, uv и работающий Docker Engine с Compose v2. REST доступен на `127.0.0.1:8080`; порт должен быть свободен, в том числе от стенда lab4. Используются синтетические данные.
|
||||
|
||||
В [Dockerfile](Dockerfile) и [Compose](compose.yaml) закреплены Python 3.12.12-slim, uv 0.11.28, PostgreSQL 17.6-alpine, SeaweedFS 4.17 с digest, RabbitMQ 4.1.2-alpine, HAProxy 3.0.11-alpine.
|
||||
|
||||
Из корня полученного clone:
|
||||
|
||||
```sh
|
||||
git switch solution/BVT2301-Kozyrev
|
||||
python3 scripts/init-secrets.py
|
||||
docker compose config --quiet
|
||||
docker compose up -d --build
|
||||
docker compose ps -a
|
||||
curl --fail http://localhost:8080/health/ready
|
||||
make test
|
||||
```
|
||||
|
||||
Локальные credentials создаются в игнорируемом `secrets/`. Их содержимое нельзя публиковать. Стандартный Compose использует значения по умолчанию; отдельный .env не обязателен. Сервис migrate применяет SQL из migrations, s3-init создаёт приватный bucket и проверяет запись. API и worker зависят от успешного завершения этих сервисов.
|
||||
|
||||
Эта инструкция пока не подтверждена запуском из чистого clone. Успешный `up -d` сам по себе не доказывает готовность: нужны проверка процессов, readiness и smoke. Worker использует heartbeat AMQP event loop; outbox/reconcile не наследуют HTTP healthcheck.
|
||||
|
||||
Локальные проверки:
|
||||
|
||||
```sh
|
||||
uv sync --locked --no-dev
|
||||
make check
|
||||
.venv/bin/python student-tests/pipeline_critical.py
|
||||
# Для полной OpenAPI-валидации:
|
||||
uv pip install --python .venv/bin/python -r requirements-dev.txt
|
||||
make validate PYTHON=.venv/bin/python
|
||||
```
|
||||
|
||||
Остановка с сохранением volumes: `docker compose down`. Намеренный сброс учебных данных: `docker compose down -v`; удаляет volumes PostgreSQL, S3 и RabbitMQ данного проекта. В ходе проверки сброс не выполнялся.
|
||||
|
||||
## Архитектура и решения
|
||||
|
||||
PostgreSQL хранит метаданные, jobs и transactional outbox; SeaweedFS — private staging/final objects; RabbitMQ — durable сообщения; worker и outbox-dispatcher — отдельные процессы. До HTTP 202 API полностью принимает и валидирует bytes, сохраняет staging-object и в одной DB-транзакции метаданные + job + outbox event. Dispatcher публикует событие с publisher confirm и отмечает event после confirm, поэтому crash между COMMIT и publish восстанавливается без ручной job. Worker ack делает только после durable результата либо после финального failed. Обработка идемпотентна по attachment/pipeline/variant: повтор ready-job не меняет результат. Lease ограничен `JOB_TIMEOUT_SECONDS`; истёкшая processing-job возвращается в очередь dispatcher-ом. Retry ограничен `JOB_MAX_ATTEMPTS`, использует exponential backoff с jitter; исчерпанная job остаётся failed и её служебное сообщение попадает в DLQ. Явный replay: `docker compose run --rm outbox /app/.venv/bin/python -m src.replay <attachment_uuid>`.
|
||||
Клиент → HAProxy → API → PostgreSQL + S3 staging. Фоновый путь: PostgreSQL outbox → dispatcher → RabbitMQ → worker → final S3 objects + метаданные PostgreSQL. Реализация: [attachments.py](src/attachments.py), [postgres.py](src/storage/postgres.py), [outbox.py](src/outbox.py), [broker.py](src/broker.py), [worker.py](src/worker.py).
|
||||
|
||||
На уровне 4 worker создаёт: thumbnail JPEG 256×256 без увеличения; crop JPEG 128×128, центрированный квадрат; optimized WebP до 1280×1280 без увеличения. JPEG quality=85, WebP quality=80; входные EXIF/metadata не сохраняются.
|
||||
PostgreSQL хранит пользователей, чаты, сообщения, сессии, attachments, jobs и outbox. S3 хранит байты. Брокер передаёт служебные ID, без файлов и credentials.
|
||||
|
||||
## Доказательства критериев
|
||||
Перед 202 API полностью принимает HTTP body, проверяет изображение, записывает исходник в staging и фиксирует attachment/job/outbox одной DB-транзакцией. S3 PUT не входит в DB-транзакцию: при rollback возможен непривязанный объект. Resize выполняет worker. Клиент получает состояние через GET метаданных; content неготового вложения возвращает 409.
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Трек 5: очередь под перегрузкой | `make track1 TRACK_ARGS="..."` | TODO: заполнить только результатом выполненного прогона | `evidence/track1-*.json` |
|
||||
Dispatcher декодирует JSONB и публикует persistent сообщение с publisher confirm, затем отмечает event опубликованным. Crash после confirm до отметки допускает дубль. Автоматическое восстановление после реального crash ещё не проверено.
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
Worker захватывает job с lease. Завершение проверяет состояние, время захвата processing_started_at и срок lease. Попытки пишут по разным S3-ключам; только действующая попытка публикует свои ключи в БД. Устаревшее завершение не должно менять актуальный результат. Логический ключ варианта — (attachment_id, pipeline_version, name); физические непривязанные объекты требуют уборки.
|
||||
|
||||
## Эксперименты и отказы
|
||||
Ack выполняется после устойчивого результата или записи retry/failed; при сбое записи используется nack/requeue. Terminal-job повторно не обрабатывается. Это at-least-once доставка, не exactly-once. Реестр failed хранится в PostgreSQL; DLQ используется для некорректных сообщений.
|
||||
|
||||
Гипотеза: увеличение worker concurrency сокращает p95 «202 → ready» до точки насыщения CPU/RAM; prefetch выше concurrency может повысить throughput ценой queue lag и RSS. Проверяются минимум три конфигурации: `1/1`, `2/2`, `4/4` (concurrency/prefetch) и несколько размеров входного PNG. Для каждой конфигурации используются одинаковые CPU/RAM limits Compose, одинаковое число изображений и не менее трёх прогонов после прогрева.
|
||||
Ошибка удаления staging после ready журналируется и не откатывает результат. Сервис reconcile раз в минуту удаляет непривязанные объекты старше retention из staging/originals/variants. Общая advisory-блокировка исключает удаление во время upload и работы worker. Staging failed-задач сохраняется для явного replay; автоматического удаления этих исходников нет. Lease recovery учитывает попытки; retry использует exponential backoff с jitter. Невалидное изображение и превышение памяти рендера переводят задачу в failed без повторов. Ошибки зависимостей и timeout повторяются с ограничением попыток.
|
||||
|
||||
Команда запуска профиля: `WORKER_CONCURRENCY=N WORKER_PREFETCH=P MAX_PENDING_JOBS=20 docker compose up -d --build`. Команда измерения: `make track1 TRACK_ARGS="--run-name cN-pP-sizeS --workers N --prefetch P --max-pending 20 --images 40 --concurrency 8 --size S"`. Скрипт сохраняет raw JSON с successful throughput, p50/p95 времени принятия и ready, queue depth RabbitMQ и RSS worker. Значения будут внесены после реального прогона; сейчас их нет, так как Docker daemon недоступен.
|
||||
Явный операторский replay только failed-job:
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
```sh
|
||||
docker compose run --rm outbox /app/.venv/bin/python -m src.replay ATTACHMENT_UUID
|
||||
```
|
||||
|
||||
Replay сохраняет replayed_at; полного журнала операторских действий нет.
|
||||
|
||||
## Конфигурация и варианты изображений
|
||||
|
||||
| Параметр | Соответствие реализации |
|
||||
| --- | --- |
|
||||
| SESSION_TTL_SECONDS / IDEMPOTENCY_TTL_SECONDS | По умолчанию 3600 / 86400 секунд, PostgreSQL |
|
||||
| S3_ENDPOINT | Compose и .env.example используют http://s3:8333 |
|
||||
| DATABASE_URL_FILE, CURSOR_SECRET_FILE, S3/Broker *_FILE | Credentials подключаются через Compose secrets |
|
||||
| MAX_UPLOAD_BYTES / MAX_IMAGE_PIXELS | 10 485 760 байт / 40 000 000 пикселей; 10 MiB = 10 × 1024 × 1024 байт |
|
||||
| WORKER_CONCURRENCY / WORKER_PREFETCH | По умолчанию 2 / 2, потоки и prefetch |
|
||||
| JOB_MAX_ATTEMPTS / JOB_TIMEOUT_SECONDS | По умолчанию 5 / 60; рендер идёт в отдельном subprocess, принудительно завершаемом по timeout |
|
||||
| MAX_PENDING_JOBS | По умолчанию 100 queued/processing jobs; глобальная DB-блокировка при admission, отказ 429 |
|
||||
| STAGING_RETENTION_SECONDS | Порог возраста непривязанных объектов для плановой уборки, по умолчанию 86400 секунд |
|
||||
| API_REPLICAS, SESSION_STORE, CACHE_URL_FILE | Не управляют текущим Compose: явно заданы api-a/api-b и PostgreSQL без Redis |
|
||||
|
||||
Worker применяет EXIF orientation, композитит alpha на белый, создаёт thumbnail JPEG до 256×256, центрированный crop JPEG 128×128, optimized WebP до 1280×1280. JPEG quality=85, WebP quality=80/method=4; явного переноса входных EXIF нет.
|
||||
|
||||
Округление исправлено: исходник 300×200 даёт 256×170; высота вычисляется как floor(200 × 256 / 300). Маленькие изображения не увеличиваются. Linux renderer ограничен RLIMIT_AS=768 MiB; worker-контейнер ограничен 4 GiB RAM и 2 CPU. RSS собирается отдельно, RLIMIT_AS ограничивает виртуальную память.
|
||||
|
||||
## Выполненные проверки и доказательства критериев
|
||||
|
||||
Повторная локальная проверка 26 октября 2026 года:
|
||||
|
||||
```text
|
||||
make check
|
||||
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||
|
||||
.venv/bin/python student-tests/pipeline_critical.py
|
||||
Ran 8 tests
|
||||
OK
|
||||
|
||||
docker compose config --quiet
|
||||
Завершено с кодом 0.
|
||||
|
||||
docker info --format '{{.ServerVersion}}'
|
||||
Ошибка: Docker socket отсутствует; daemon недоступен.
|
||||
```
|
||||
|
||||
[Регрессионные тесты](student-tests/pipeline_critical.py) используют mocks и проверяют декодирование outbox, ack/nack, confirm DLQ, сохранение ready при ошибке cleanup и отклонение устаревшего завершения/retry. Они не проверяют SQL на реальной БД. Полная OpenAPI-валидация при аудите не запустилась из-за отсутствующего openapi_spec_validator.
|
||||
|
||||
| Критерий README | Состояние / необходимая проверка |
|
||||
| --- | --- |
|
||||
| 3: минимум lab4 и durable upload 202 | Код перенесён; контейнерный smoke не выполнен; failed-вложение отклоняется с 409 |
|
||||
| 3: отдельные broker/worker, durable доставка, ack | Роли/volume/persistent publish есть; локальные ack/nack-тесты прошли; реальная redelivery не проверена |
|
||||
| 3: состояния, оригинал, thumbnail, smoke | Реализованы; округление проверено локально; успешного smoke нет |
|
||||
| 3: ограниченные retries, failed, идемпотентность | Счётчик, terminal states и защита завершения есть; конкурентные проверки на БД не выполнены |
|
||||
| 3: worker stop → upload → restart API/broker → ready | Не выполнено; сохранность при рестарте не доказана |
|
||||
| 4: crop/WebP и лимиты bytes/pixels/time/memory | Варианты и bytes/pixels есть; subprocess timeout и лимиты памяти добавлены; Linux-лимиты на стенде не проверены |
|
||||
| 4: outbox, crash COMMIT/publish | Код и OUTBOX_FAULT_AFTER_COMMIT=1 есть; crash recovery не проверен |
|
||||
| 4: backoff/jitter, классификация, failed/replay | Backoff/replay есть; классификация добавлена; replay на стенде не выполнен |
|
||||
| 4: crash после side effect, lease, concurrency, cleanup | Защита завершения и concurrency есть; crash-тест отсутствует; плановый cleanup реализован; failed staging сохраняется для replay |
|
||||
| 5: три настройки, разные изображения, метрики | Генератор исправлен и проверен с mock-стендом; реальных прогонов нет |
|
||||
| 5: backpressure и восстановление | Лимит jobs есть; демонстрация перегрузки и дренирования не выполнена |
|
||||
|
||||
В evidence пока нет результатов контейнерного smoke или исследования. Наличие кода не считается доказательством выполнения критерия.
|
||||
|
||||
## Эксперимент: очередь под перегрузкой
|
||||
|
||||
Гипотеза: рост concurrency увеличивает throughput до насыщения ресурсов; затем задержки и память могут вырасти. Экспериментальных выводов пока нет.
|
||||
|
||||
План эксперимента: профили concurrency/prefetch 1/1, 2/2, 4/4; изображения 320×320, 1600×1600, 3200×3200; одинаковые CPU/RAM-лимиты; отдельный прогрев и три повторения каждой комбинации. Фиксировать hardware и реальные настройки контейнеров, длительность, accepted/rejected/failed, throughput, queue waiting time, p50/p95/p99 времени 202 → ready и RSS. CPU отдельно фиксировать по контейнерным метрикам.
|
||||
|
||||
Заготовка запуска:
|
||||
|
||||
```sh
|
||||
WORKER_CONCURRENCY=1 WORKER_PREFETCH=1 MAX_PENDING_JOBS=20 docker compose up -d --build
|
||||
make track1 TRACK_ARGS="--run-name c1-p1-s1600 --workers 1 --prefetch 1 --max-pending 20 --images 40 --concurrency 8 --size 1600"
|
||||
```
|
||||
|
||||
Ожидаемый путь результата этой команды: evidence/c1-p1-s1600.json. Измерения ещё не получены.
|
||||
|
||||
Исправленный [генератор](scripts/track1.py) записывает monotonic-время ответа каждой загрузки и опрашивает ready параллельно uploads. Барьер удалён. RSS суммируется по /proc процессов worker-контейнера; shared pages могут учитываться несколько раз, пики между замерами пропускаются. Время ожидания вычисляется в PostgreSQL от создания job до первого claim (миграция 007). Concurrency/prefetch и лимит admission сверяются с работающими контейнерами. Closed-loop и polling вносят ограничения: наблюдаемое время ready является верхней оценкой, а нагрузка зависит от скорости клиента.
|
||||
|
||||
После нагрузки требуется проверить завершение всех принятых задач, отсутствие queued/processing и unacked, затем обработать контрольный upload. Сценарий проверяет дренирование и контрольный upload; при незавершённых задачах или пропущенных метриках завершается ошибкой. Флаг --require-backpressure дополнительно требует наблюдаемый 429. Измеренных RPS, p95, RSS и времени восстановления нет; ресурсы Docker не зафиксированы из-за недоступного daemon.
|
||||
|
||||
## Эксперименты с отказами
|
||||
|
||||
| Воздействие | Ожидаемый эффект | Наблюдение |
|
||||
| --- | --- | --- |
|
||||
| Worker остановлен | 202, metadata, 409 content; ready после запуска | Не выполнено |
|
||||
| API погиб после COMMIT | Dispatcher доставляет intent | Не выполнено |
|
||||
| Worker погиб после S3 до commit/ack | Lease recovery завершает ту же логическую работу | Не выполнено |
|
||||
| Сбой cleanup после ready | Ready сохраняется | Локальный mock-тест прошёл |
|
||||
| Сбой устойчивой записи результата/retry | nack/requeue | Локальный mock-тест прошёл |
|
||||
| Устаревшее завершение | Метаданные не меняются, staging сохраняется | Локальные mock-тесты прошли |
|
||||
| Broker restart с volume | Принятая работа сохраняется | Не выполнено |
|
||||
| Перегрузка | Ограничение приёма, дренирование, восстановление | Не выполнено |
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
Перечисленные в аудите нарушения исправлены в коде. Требуются проверки на реальных PostgreSQL/S3/RabbitMQ: Linux memory limit, конкурентная уборка, рестарты и нагрузочные измерения пока не подтверждены.
|
||||
|
||||
Admission перенесён перед S3 PUT; отклонённые по лимиту uploads не создают S3-объект. Reconciler адаптирован к вариантам и блокировке активных writers. Тест student-tests/attachments.py обновлён для 202 и polling. Другие исторические тесты lab4 требуют отдельной адаптации и не заявляются как пройденные.
|
||||
|
||||
Один Docker-host, одна БД, один брокер, одно S3-хранилище и один LB не доказывают устойчивость к потере host/хранилищ. Две API-реплики покрывают только часть отказов. Порядок завершения: закрыть замечания аудита, запустить smoke и persistence/failover, воспроизвести отказы pipeline, исправить генератор и провести исследование. До этого оценка 5 остаётся целью, а не подтверждённым результатом.
|
||||
|
||||
### Дополнительные локальные регрессии
|
||||
|
||||
`.venv/bin/python student-tests/remaining_regressions.py`: проверены округление/alpha, subprocess render и его timeout, классификация permanent failure, retention активного/failed staging, частичная группа нагрузочного генератора и контроль восстановления на mock-стенде. Числа mock-прогона не являются результатами исследования и не переносятся в evidence.
|
||||
|
||||
+20
-1
@@ -151,10 +151,13 @@ services:
|
||||
memory: 128M
|
||||
|
||||
reconcile:
|
||||
healthcheck:
|
||||
disable: true
|
||||
build:
|
||||
context: .
|
||||
entrypoint: ["/app/.venv/bin/python", "-m", "src.reconcile"]
|
||||
profiles: ["maintenance"]
|
||||
command: ["--apply", "--loop"]
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
CURSOR_SECRET_FILE: /run/secrets/cursor_secret
|
||||
@@ -162,6 +165,7 @@ services:
|
||||
S3_BUCKET: messenger-attachments
|
||||
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_admin_key
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_admin_secret
|
||||
STAGING_RETENTION_SECONDS: ${STAGING_RETENTION_SECONDS:-86400}
|
||||
secrets:
|
||||
- database_url
|
||||
- cursor_secret
|
||||
@@ -185,6 +189,17 @@ services:
|
||||
<<: *api
|
||||
|
||||
worker:
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "/app/.venv/bin/python", "-c", "import time,pathlib; assert time.time()-float(pathlib.Path('/tmp/worker-heartbeat').read_text()) < 15"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 3
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "2.0"
|
||||
memory: 4G
|
||||
build:
|
||||
context: .
|
||||
command: ["/app/.venv/bin/python", "-m", "src.worker"]
|
||||
@@ -198,6 +213,7 @@ services:
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
|
||||
BROKER_URL_FILE: /run/secrets/broker_url
|
||||
WORKER_CONCURRENCY: ${WORKER_CONCURRENCY:-2}
|
||||
RENDER_MEMORY_MB: ${RENDER_MEMORY_MB:-768}
|
||||
WORKER_PREFETCH: ${WORKER_PREFETCH:-2}
|
||||
JOB_MAX_ATTEMPTS: ${JOB_MAX_ATTEMPTS:-5}
|
||||
JOB_TIMEOUT_SECONDS: ${JOB_TIMEOUT_SECONDS:-60}
|
||||
@@ -218,6 +234,8 @@ services:
|
||||
stop_grace_period: 30s
|
||||
|
||||
outbox:
|
||||
healthcheck:
|
||||
disable: true
|
||||
build:
|
||||
context: .
|
||||
command: ["/app/.venv/bin/python", "-m", "src.outbox"]
|
||||
@@ -227,6 +245,7 @@ services:
|
||||
CURSOR_SECRET_FILE: /run/secrets/cursor_secret
|
||||
BROKER_URL_FILE: /run/secrets/broker_url
|
||||
OUTBOX_DISPATCH_INTERVAL_SECONDS: "1"
|
||||
JOB_MAX_ATTEMPTS: ${JOB_MAX_ATTEMPTS:-5}
|
||||
secrets:
|
||||
- database_url
|
||||
- cursor_secret
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
ALTER TABLE attachment_jobs ADD COLUMN first_started_at timestamptz;
|
||||
+2
-2
@@ -6,5 +6,5 @@ cd "$(dirname "$0")/.."
|
||||
docker compose restart postgres s3 broker
|
||||
docker compose up -d --wait postgres s3 broker
|
||||
docker compose run --rm --no-deps s3-init
|
||||
docker compose restart worker api-a api-b load_balancer
|
||||
docker compose up -d --wait worker api-a api-b load_balancer
|
||||
docker compose restart worker outbox reconcile api-a api-b load_balancer
|
||||
docker compose up -d --wait worker outbox reconcile api-a api-b load_balancer
|
||||
|
||||
+172
-101
@@ -1,13 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Repeatable level-5 queue-overload experiment.
|
||||
|
||||
Run a Compose profile first, for example:
|
||||
WORKER_CONCURRENCY=2 MAX_PENDING_JOBS=20 docker compose up -d --build
|
||||
.venv/bin/python scripts/track1.py --run-name c2-p2 --workers 2 --prefetch 2 --images 40 --size 1600
|
||||
|
||||
The script records client-observed acceptance/ready latency, RabbitMQ queue depth
|
||||
and Docker RSS samples. It does not print credentials or session tokens.
|
||||
"""
|
||||
"""Closed-loop upload burst with concurrent readiness observations and drain proof."""
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from io import BytesIO
|
||||
import json
|
||||
@@ -17,11 +9,10 @@ import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from urllib.parse import quote
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
from client import Client, Failure, arguments, check, run # noqa: E402
|
||||
from client import Client, Failure, arguments, check, run
|
||||
|
||||
|
||||
def percentile(values, point):
|
||||
@@ -29,114 +20,194 @@ def percentile(values, point):
|
||||
return round(values[max(0, math.ceil(len(values) * point) - 1)] * 1000, 3) if values else None
|
||||
|
||||
|
||||
def docker_text(*args):
|
||||
def command_json(*args):
|
||||
result = subprocess.run(args, cwd=ROOT, text=True, capture_output=True, timeout=20)
|
||||
if result.returncode:
|
||||
raise Failure("metrics command failed: " + args[0])
|
||||
try:
|
||||
return subprocess.run(args, cwd=ROOT, text=True, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL,
|
||||
timeout=10, check=True).stdout.strip()
|
||||
except (OSError, subprocess.SubprocessError):
|
||||
return None
|
||||
return json.loads(result.stdout)
|
||||
except ValueError:
|
||||
raise Failure("metrics command returned invalid JSON") from None
|
||||
|
||||
|
||||
def metrics(chat_id):
|
||||
return command_json("docker", "compose", "exec", "-T", "worker",
|
||||
"/app/.venv/bin/python", "-m", "src.pipeline_metrics", chat_id)
|
||||
|
||||
|
||||
def broker_depth():
|
||||
text = docker_text("docker", "compose", "exec", "-T", "broker", "rabbitmqctl", "list_queues", "name", "messages_ready", "messages_unacknowledged", "--formatter", "json")
|
||||
if not text:
|
||||
return None
|
||||
try:
|
||||
rows = json.loads(text)
|
||||
return next((row for row in rows if row["name"] == "attachment.process.v1"), None)
|
||||
except (ValueError, KeyError, TypeError):
|
||||
return None
|
||||
rows = command_json("docker", "compose", "exec", "-T", "broker",
|
||||
"rabbitmqctl", "list_queues", "name", "messages_ready",
|
||||
"messages_unacknowledged", "--formatter", "json")
|
||||
return next(row for row in rows if row["name"] == "attachment.process.v1")
|
||||
|
||||
|
||||
def worker_rss():
|
||||
container = docker_text("docker", "compose", "ps", "-q", "worker")
|
||||
if not container:
|
||||
return None
|
||||
text = docker_text("docker", "stats", "--no-stream", "--format", "{{json .}}", container)
|
||||
if not text:
|
||||
return None
|
||||
try:
|
||||
return json.loads(text).get("MemUsage")
|
||||
except ValueError:
|
||||
return None
|
||||
def container_info(service):
|
||||
result = subprocess.run(['docker', 'compose', 'ps', '-q', service], cwd=ROOT,
|
||||
capture_output=True, text=True, timeout=10, check=True)
|
||||
ids = result.stdout.split()
|
||||
check(len(ids) == 1, 'expected exactly one container: ' + service)
|
||||
return command_json('docker', 'inspect', ids[0])[0]
|
||||
|
||||
|
||||
def main():
|
||||
def configure(parser):
|
||||
parser.add_argument("--run-name", required=True)
|
||||
parser.add_argument("--workers", type=int, required=True, help="documented WORKER_CONCURRENCY setting")
|
||||
parser.add_argument("--prefetch", type=int, required=True, help="documented RabbitMQ prefetch setting")
|
||||
parser.add_argument("--max-pending", type=int, required=True, help="documented MAX_PENDING_JOBS setting")
|
||||
parser.add_argument("--images", type=int, default=40)
|
||||
parser.add_argument("--concurrency", type=int, default=4)
|
||||
parser.add_argument("--size", type=int, default=1600, help="square synthetic PNG dimensions")
|
||||
parser.add_argument("--max-ready-seconds", type=float, default=120)
|
||||
parser.add_argument("--output", default=None)
|
||||
args = arguments("Lab 5 track 1 image-pipeline experiment", configure)
|
||||
check(1 <= args.workers <= 64 and 1 <= args.prefetch <= 1024 and 1 <= args.max_pending <= 100000, "invalid worker/prefetch/max-pending")
|
||||
check(1 <= args.images <= 1000 and 1 <= args.concurrency <= 128 and 1 <= args.size <= 6000, "invalid workload")
|
||||
|
||||
def configure(p):
|
||||
p.add_argument("--run-name", required=True)
|
||||
p.add_argument("--workers", type=int, required=True)
|
||||
p.add_argument("--prefetch", type=int, required=True)
|
||||
p.add_argument("--max-pending", type=int, required=True)
|
||||
p.add_argument("--images", type=int, default=40)
|
||||
p.add_argument("--concurrency", type=int, default=8)
|
||||
p.add_argument("--size", type=int, default=1600)
|
||||
p.add_argument("--max-ready-seconds", type=float, default=180)
|
||||
p.add_argument("--require-backpressure", action="store_true")
|
||||
p.add_argument("--output")
|
||||
args = arguments(__doc__, configure)
|
||||
check(1 <= args.images <= 1000 and 1 <= args.concurrency <= 128, "invalid workload")
|
||||
check(1 <= args.size <= 6000 and 0 < args.max_ready_seconds <= 3600, "invalid size/deadline")
|
||||
check(args.workers > 0 and args.prefetch > 0 and args.max_pending > 0, "invalid profile")
|
||||
from PIL import Image
|
||||
image = Image.new("RGB", (args.size, args.size), "#427aa1")
|
||||
raw = BytesIO(); image.save(raw, "PNG", optimize=True); data = raw.getvalue()
|
||||
# Reproducible, nonuniform synthetic image; no real personal data.
|
||||
row = bytes(v for x in range(args.size) for v in ((x * 13) % 256, (x * 37) % 256, (x * 71) % 256))
|
||||
image = Image.frombytes("RGB", (args.size, args.size), row * args.size)
|
||||
stream = BytesIO()
|
||||
image.save(stream, "PNG")
|
||||
data = stream.getvalue()
|
||||
client = Client(args.base_url, args.ca_file, timeout=30)
|
||||
alice, bob, _eve, chat = client.bootstrap()
|
||||
alice, bob, _, chat = client.bootstrap()
|
||||
baseline = metrics(chat["id"])
|
||||
check(baseline["concurrency"] == args.workers and baseline["prefetch"] == args.prefetch,
|
||||
"worker settings do not match requested profile")
|
||||
check(baseline["pending"] == 0, "use an idle isolated stack")
|
||||
for service in ("api-a", "api-b"):
|
||||
env = dict(item.split('=', 1) for item in container_info(service)['Config']['Env'])
|
||||
check(int(env['MAX_PENDING_JOBS']) == args.max_pending,
|
||||
'running API MAX_PENDING_JOBS does not match profile')
|
||||
resources = container_info('worker')['HostConfig']
|
||||
path = "/api/v1/chats/" + chat["id"] + "/attachments?filename=track1.png"
|
||||
barrier = threading.Barrier(args.concurrency)
|
||||
queue_samples, rss_samples = [], []
|
||||
stop_sampling = threading.Event()
|
||||
|
||||
def sampler():
|
||||
while not stop_sampling.wait(1):
|
||||
queue_samples.append({"seconds": round(time.monotonic() - started, 3), "queue": broker_depth()})
|
||||
rss_samples.append({"seconds": round(time.monotonic() - started, 3), "rss": worker_rss()})
|
||||
|
||||
pending, records, samples, errors = {}, [], [], []
|
||||
lock, finished = threading.Lock(), threading.Event()
|
||||
started = time.monotonic()
|
||||
def observe():
|
||||
local = Client(args.base_url, args.ca_file, timeout=5)
|
||||
while not finished.wait(.1):
|
||||
with lock:
|
||||
batch = list(pending.items())
|
||||
for key, record in batch:
|
||||
if finished.is_set():
|
||||
break
|
||||
try:
|
||||
att = local.call("GET", "/api/v1/attachments/" + key, actor=bob, schema="Attachment")["json"]
|
||||
if att["status"] in ("ready", "failed"):
|
||||
with lock:
|
||||
record["terminal"] = att["status"]
|
||||
record["terminal_at"] = time.monotonic()
|
||||
pending.pop(key, None)
|
||||
except Failure:
|
||||
continue
|
||||
def sample():
|
||||
while not finished.is_set():
|
||||
try:
|
||||
sample = metrics(chat["id"])
|
||||
sample["queue"] = broker_depth()
|
||||
sample["seconds"] = time.monotonic() - started
|
||||
with lock:
|
||||
samples.append(sample)
|
||||
except (Failure, OSError, subprocess.TimeoutExpired) as exc:
|
||||
with lock:
|
||||
errors.append(type(exc).__name__)
|
||||
if finished.wait(1):
|
||||
break
|
||||
observer = threading.Thread(target=observe, daemon=True)
|
||||
sampler = threading.Thread(target=sample, daemon=True)
|
||||
observer.start()
|
||||
sampler.start()
|
||||
def upload(index):
|
||||
local = Client(args.base_url, args.ca_file, timeout=30)
|
||||
barrier.wait()
|
||||
started_at = time.monotonic()
|
||||
before = time.monotonic()
|
||||
try:
|
||||
response = local.call("POST", path + "-" + str(index), 202, actor=alice, raw=data,
|
||||
response = local.call("POST", path, (202, 429), actor=alice, raw=data,
|
||||
headers={"Content-Type": "image/png"}, schema="Attachment")
|
||||
return {"accepted": True, "id": response["json"]["id"], "accepted_seconds": time.monotonic() - started_at}
|
||||
except Failure as error:
|
||||
return {"accepted": False, "error": str(error), "accepted_seconds": time.monotonic() - started_at}
|
||||
|
||||
started = time.monotonic()
|
||||
thread = threading.Thread(target=sampler, daemon=True); thread.start()
|
||||
with ThreadPoolExecutor(max_workers=args.concurrency) as pool:
|
||||
results = list(pool.map(upload, range(args.images)))
|
||||
accepted = [item for item in results if item["accepted"]]
|
||||
ready_latencies, failed_ready = [], 0
|
||||
deadline = time.monotonic() + args.max_ready_seconds
|
||||
pending = {item["id"]: started + item["accepted_seconds"] for item in accepted}
|
||||
while pending and time.monotonic() < deadline:
|
||||
for attachment_id, accepted_at in list(pending.items()):
|
||||
try:
|
||||
metadata = client.call("GET", "/api/v1/attachments/" + attachment_id, actor=bob, schema="Attachment")["json"]
|
||||
except Failure:
|
||||
continue
|
||||
if metadata["status"] == "ready":
|
||||
ready_latencies.append(time.monotonic() - accepted_at)
|
||||
del pending[attachment_id]
|
||||
elif metadata["status"] == "failed":
|
||||
failed_ready += 1; del pending[attachment_id]
|
||||
time.sleep(.2)
|
||||
stop_sampling.set(); thread.join(timeout=2)
|
||||
elapsed = time.monotonic() - started
|
||||
received = time.monotonic()
|
||||
record = {"index": index, "status": response["status"], "upload_seconds": received - before}
|
||||
if response["status"] == 202:
|
||||
record.update(id=response["json"]["id"], accepted_at=received)
|
||||
with lock:
|
||||
records.append(record)
|
||||
if response["status"] == 202:
|
||||
pending[record["id"]] = record
|
||||
except Failure:
|
||||
with lock:
|
||||
records.append({"index": index, "status": "transport_or_http_error"})
|
||||
try:
|
||||
with ThreadPoolExecutor(max_workers=args.concurrency) as pool:
|
||||
list(pool.map(upload, range(args.images)))
|
||||
upload_finished = time.monotonic()
|
||||
deadline = upload_finished + args.max_ready_seconds
|
||||
while time.monotonic() < deadline:
|
||||
with lock:
|
||||
empty = not pending
|
||||
if empty:
|
||||
break
|
||||
time.sleep(.1)
|
||||
finally:
|
||||
measured_end = time.monotonic()
|
||||
finished.set()
|
||||
observer.join(timeout=10)
|
||||
sampler.join(timeout=45)
|
||||
elapsed = measured_end - started
|
||||
final = metrics(chat["id"])
|
||||
queue = broker_depth()
|
||||
for _ in range(20):
|
||||
if queue["messages_ready"] == 0 and queue["messages_unacknowledged"] == 0:
|
||||
break
|
||||
time.sleep(.25)
|
||||
queue = broker_depth()
|
||||
accepted = [r for r in records if r["status"] == 202]
|
||||
ready = [r for r in accepted if r.get("terminal") == "ready"]
|
||||
latencies = [r["terminal_at"] - r["accepted_at"] for r in ready]
|
||||
waits = [r["queue_wait_seconds"] for r in final["jobs"] if r["queue_wait_seconds"] is not None]
|
||||
drained = (bool(accepted) and len(ready) == len(accepted) and final["pending"] == 0
|
||||
and queue["messages_ready"] == 0 and queue["messages_unacknowledged"] == 0)
|
||||
recovery = False
|
||||
if drained:
|
||||
try:
|
||||
att, _ = client.attachment(alice, chat)
|
||||
client.wait_ready(bob, att, wait_seconds=args.max_ready_seconds)
|
||||
recovery = True
|
||||
except Failure:
|
||||
pass
|
||||
rejected = sum(r["status"] == 429 for r in records)
|
||||
report = {
|
||||
"track": "queue-overload", "run_name": args.run_name,
|
||||
"settings": {"worker_concurrency": args.workers, "prefetch": args.prefetch, "max_pending_jobs": args.max_pending},
|
||||
"workload": {"images": args.images, "client_concurrency": args.concurrency, "png_dimensions": [args.size, args.size], "input_bytes": len(data)},
|
||||
"results": {"elapsed_seconds": round(elapsed, 3), "accepted": len(accepted), "rejected_or_failed": args.images - len(accepted), "ready": len(ready_latencies), "failed": failed_ready, "timed_out": len(pending), "throughput_ready_per_second": round(len(ready_latencies) / elapsed, 4), "accept_latency_ms": {"p50": percentile([x["accepted_seconds"] for x in accepted], .5), "p95": percentile([x["accepted_seconds"] for x in accepted], .95)}, "time_to_ready_ms": {"p50": percentile(ready_latencies, .5), "p95": percentile(ready_latencies, .95)}},
|
||||
"queue_samples": queue_samples, "worker_rss_samples": rss_samples,
|
||||
"limitations": "Closed-loop client; queue/RSS samples are null when Docker CLI is unavailable. p95 covers ready tasks only.",
|
||||
"run_name": args.run_name,
|
||||
"settings": {"workers": baseline["concurrency"], "prefetch": baseline["prefetch"],
|
||||
"max_pending": args.max_pending,
|
||||
"worker_limits": {key: resources[key] for key in ('Memory', 'NanoCpus')}},
|
||||
"workload": {"images": args.images, "concurrency": args.concurrency,
|
||||
"size": args.size, "bytes": len(data), "type": "closed-loop burst"},
|
||||
"results": {"accepted": len(accepted), "rejected_429": rejected, "ready": len(ready),
|
||||
"failed": sum(r.get("terminal") == "failed" for r in accepted),
|
||||
"timed_out": len(pending), "elapsed_seconds": elapsed,
|
||||
"ready_per_second": len(ready) / elapsed,
|
||||
"ready_latency_ms": {str(p): percentile(latencies, p) for p in (.5, .95, .99)},
|
||||
"queue_wait_ms": {str(p): percentile(waits, p) for p in (.5, .95, .99)},
|
||||
"max_sampled_rss_bytes": max((s["rss_bytes"] for s in samples), default=None),
|
||||
"drained": drained, "recovery_upload_ready": recovery},
|
||||
"records": [{k: v for k, v in r.items() if k not in ("accepted_at", "terminal_at")} |
|
||||
({"ready_seconds": r["terminal_at"] - r["accepted_at"]} if "terminal_at" in r else {})
|
||||
for r in records],
|
||||
"samples": samples, "metric_errors": errors,
|
||||
"limitations": "Closed-loop; ready time is polling upper bound. RSS is sum of worker/container processes, shared pages may be counted twice. Sampling may miss peaks. Queue wait is first claim minus durable job creation.",
|
||||
}
|
||||
text = json.dumps(report, ensure_ascii=False, indent=2)
|
||||
destination = Path(args.output or (ROOT / "evidence" / (args.run_name + ".json")))
|
||||
destination.parent.mkdir(parents=True, exist_ok=True); destination.write_text(text + "\n", encoding="utf-8")
|
||||
print(text)
|
||||
check(bool(accepted), "no accepted uploads")
|
||||
target = Path(args.output or ROOT / "evidence" / (args.run_name + ".json"))
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text(json.dumps(report, indent=2) + "\n")
|
||||
print(json.dumps(report["results"], indent=2))
|
||||
check(not errors and bool(samples), "incomplete metrics; result saved but not valid evidence")
|
||||
check(all(r["status"] in (202, 429) for r in records), "unexpected HTTP/transport failures")
|
||||
check(drained and recovery, "pipeline did not drain/recover; see saved result")
|
||||
if args.require_backpressure:
|
||||
check(rejected > 0, "backpressure was not observed")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
+8
-6
@@ -83,10 +83,6 @@ async def upload(chat_id: UUID, request: Request, filename: str = Query(...),
|
||||
attachment_id, job_id = uuid4(), uuid4()
|
||||
key = f"staging/{attachment_id}/original"
|
||||
async with store.attachment_transaction() as connection:
|
||||
await s3.put(key, file, size, content_type)
|
||||
# Disabled by default; models a crash after PUT, before INSERT.
|
||||
if os.environ.get("UPLOAD_FAULT_AFTER_S3_PUT") == "1":
|
||||
raise ApiError(503, "unavailable", "Инъецирован отказ после S3 PUT")
|
||||
try:
|
||||
max_pending = int(os.environ.get("MAX_PENDING_JOBS", "100"))
|
||||
except ValueError as error:
|
||||
@@ -94,6 +90,9 @@ async def upload(chat_id: UUID, request: Request, filename: str = Query(...),
|
||||
if max_pending < 1:
|
||||
raise ApiError(503, "unavailable", "Некорректный лимит очереди")
|
||||
await store.ensure_attachment_queue_capacity(connection, max_pending)
|
||||
await s3.put(key, file, size, content_type)
|
||||
if os.environ.get("UPLOAD_FAULT_AFTER_S3_PUT") == "1":
|
||||
raise ApiError(503, "unavailable", "Инъецирован отказ после S3 PUT")
|
||||
attachment = await store.create_attachment(
|
||||
attachment_id, chat_id, user["id"], filename, content_type,
|
||||
size, digest.hexdigest(), key, job_id, connection=connection,
|
||||
@@ -130,9 +129,12 @@ async def download(attachment_id: UUID, request: Request,
|
||||
if not matching:
|
||||
raise ApiError(404, "not_found", "Вариант не найден")
|
||||
item = matching[0]
|
||||
suffix = {"thumbnail": "thumbnail.jpg", "crop": "crop.jpg", "optimized": "optimized.webp"}[variant]
|
||||
object_key = await store.pool.fetchval(
|
||||
'SELECT object_key FROM attachment_variants WHERE attachment_id=$1 AND pipeline_version=$2 AND name=$3',
|
||||
attachment_id, attachment['pipeline_version'], variant,
|
||||
)
|
||||
data = await request.app.state.s3.download_key(
|
||||
f"variants/{attachment_id}/{suffix}", item["size_bytes"], item["sha256"]
|
||||
object_key, item["size_bytes"], item["sha256"]
|
||||
)
|
||||
content_type = item["content_type"]
|
||||
return Response(data, media_type=content_type,
|
||||
|
||||
+5
-2
@@ -51,7 +51,7 @@ async def publish(payload: dict) -> None:
|
||||
|
||||
|
||||
async def publish_dlq(payload: dict) -> None:
|
||||
payload = {**payload, "failure": "retry_exhausted"}
|
||||
payload = {**payload, "failure": payload.get('failure', 'retry_exhausted')}
|
||||
try:
|
||||
await run_in_threadpool(_publish_to, DLQ, payload)
|
||||
except (pika.exceptions.AMQPError, OSError, ValueError) as error:
|
||||
@@ -61,9 +61,12 @@ async def publish_dlq(payload: dict) -> None:
|
||||
def _publish_to(queue: str, payload: dict) -> None:
|
||||
connection = _connection()
|
||||
try:
|
||||
connection.channel().basic_publish(
|
||||
channel = connection.channel()
|
||||
channel.confirm_delivery()
|
||||
channel.basic_publish(
|
||||
exchange="", routing_key=queue, body=json.dumps(payload, separators=(",", ":")).encode("utf-8"),
|
||||
properties=pika.BasicProperties(delivery_mode=pika.DeliveryMode.Persistent, content_type="application/json"),
|
||||
mandatory=True,
|
||||
)
|
||||
finally:
|
||||
connection.close()
|
||||
|
||||
@@ -22,6 +22,7 @@ _CONFIGURATION_KEYS = {
|
||||
"MAX_UPLOAD_BYTES", "MAX_IMAGE_PIXELS",
|
||||
"UPLOAD_CONCURRENCY", "UPLOAD_TIMEOUT_SECONDS", "UPLOAD_FAULT_AFTER_S3_PUT",
|
||||
"WORKER_CONCURRENCY", "JOB_MAX_ATTEMPTS", "JOB_TIMEOUT_SECONDS", "STAGING_RETENTION_SECONDS",
|
||||
"WORKER_PREFETCH", "MAX_PENDING_JOBS", "RENDER_MEMORY_MB",
|
||||
}
|
||||
|
||||
|
||||
|
||||
+1
-1
@@ -18,7 +18,7 @@ async def run() -> None:
|
||||
try:
|
||||
while True:
|
||||
try:
|
||||
await store.schedule_due_jobs()
|
||||
await store.schedule_due_jobs(int(os.environ.get('JOB_MAX_ATTEMPTS', '5')))
|
||||
for event in await store.due_outbox_events():
|
||||
await publish(event["payload"])
|
||||
await store.mark_outbox_published(event["id"])
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
"""Operator-only metrics for one synthetic chat; never exposes credentials."""
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import sys
|
||||
from uuid import UUID
|
||||
from .config import load_settings
|
||||
from .storage.postgres import PostgresStore
|
||||
|
||||
|
||||
async def main(chat):
|
||||
store = PostgresStore(load_settings().database_url)
|
||||
await store.connect()
|
||||
try:
|
||||
rows = await store.pool.fetch('SELECT a.id, a.status, extract(epoch from (j.first_started_at-j.created_at))::float8 AS queue_wait_seconds FROM attachments a JOIN attachment_jobs j ON j.attachment_id=a.id WHERE a.chat_id=$1', chat)
|
||||
jobs = [dict(row, id=str(row['id'])) for row in rows]
|
||||
pending = await store.pool.fetchval("SELECT count(*) FROM attachment_jobs WHERE status IN ('queued','processing')")
|
||||
finally:
|
||||
await store.close()
|
||||
rss = 0
|
||||
for path in Path('/proc').glob('[0-9]*/status'):
|
||||
try:
|
||||
if int(path.parent.name) == os.getpid():
|
||||
continue
|
||||
for line in path.read_text().splitlines():
|
||||
if line.startswith('VmRSS:'):
|
||||
rss += int(line.split()[1]) * 1024
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
print(json.dumps({'jobs': jobs, 'pending': pending, 'rss_bytes': rss,
|
||||
'concurrency': int(os.environ['WORKER_CONCURRENCY']),
|
||||
'prefetch': int(os.environ['WORKER_PREFETCH'])}))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main(UUID(sys.argv[1])))
|
||||
+20
-5
@@ -9,6 +9,7 @@ import argparse
|
||||
import asyncio
|
||||
from datetime import datetime, timedelta, timezone
|
||||
import json
|
||||
import os
|
||||
from time import perf_counter
|
||||
|
||||
from .config import load_settings
|
||||
@@ -33,7 +34,11 @@ async def scan(args, db, s3) -> dict:
|
||||
# Age is measured at scan start, not after a potentially slow inventory.
|
||||
cutoff = datetime.now(timezone.utc) - timedelta(seconds=args.grace_seconds)
|
||||
started = perf_counter()
|
||||
objects, s3_pages = await s3.list_with_pages("originals/")
|
||||
objects, s3_pages = {}, 0
|
||||
for prefix in ('staging/', 'originals/', 'variants/'):
|
||||
entries, pages = await s3.list_with_pages(prefix)
|
||||
objects.update(entries)
|
||||
s3_pages += pages
|
||||
listed_at = perf_counter()
|
||||
metadata = await db.attachment_keys()
|
||||
metadata_at = perf_counter()
|
||||
@@ -74,15 +79,25 @@ async def scan(args, db, s3) -> dict:
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--apply", action="store_true", help="delete only aged orphan objects")
|
||||
parser.add_argument("--grace-seconds", type=int, default=3600)
|
||||
parser.add_argument("--grace-seconds", type=int, default=int(os.environ.get('STAGING_RETENTION_SECONDS', '86400')))
|
||||
parser.add_argument('--loop', action='store_true')
|
||||
parser.add_argument("--object-key", action="append", default=[],
|
||||
help="restrict apply to these exact keys; repeatable, inventory stays complete")
|
||||
args = parser.parse_args()
|
||||
if not 0 <= args.grace_seconds <= 7 * 24 * 3600:
|
||||
parser.error("--grace-seconds must be 0..604800")
|
||||
if any(not key.startswith("originals/") for key in args.object_key):
|
||||
parser.error("--object-key must start with originals/")
|
||||
print(json.dumps(asyncio.run(run(args)), ensure_ascii=False, sort_keys=True))
|
||||
if any(not key.startswith(('originals/', 'staging/', 'variants/')) for key in args.object_key):
|
||||
parser.error('unsupported object prefix')
|
||||
if args.loop:
|
||||
import time
|
||||
while True:
|
||||
try:
|
||||
print(json.dumps(asyncio.run(run(args)), ensure_ascii=False, sort_keys=True))
|
||||
except Exception:
|
||||
print('cleanup deferred: dependency unavailable', flush=True)
|
||||
time.sleep(60)
|
||||
else:
|
||||
print(json.dumps(asyncio.run(run(args)), ensure_ascii=False, sort_keys=True))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
"""Isolated image renderer. Parent enforces wall time and kills this process."""
|
||||
import os
|
||||
import pickle
|
||||
import sys
|
||||
|
||||
|
||||
def main():
|
||||
if sys.platform == 'linux':
|
||||
import resource
|
||||
limit = int(os.environ.get('RENDER_MEMORY_MB', '768')) * 1024 * 1024
|
||||
resource.setrlimit(resource.RLIMIT_AS, (limit, limit))
|
||||
from .worker import variants
|
||||
from PIL import UnidentifiedImageError, Image
|
||||
try:
|
||||
data = sys.stdin.buffer.read(10485761)
|
||||
if len(data) > 10485760:
|
||||
raise ValueError('bytes limit')
|
||||
result = variants(data, int(os.environ.get('MAX_IMAGE_PIXELS', '40000000')))
|
||||
pickle.dump(result, sys.stdout.buffer)
|
||||
except MemoryError:
|
||||
sys.exit(21)
|
||||
except (ValueError, OSError, SyntaxError, UnidentifiedImageError, Image.DecompressionBombError):
|
||||
sys.exit(20)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
+35
-23
@@ -247,6 +247,11 @@ class PostgresStore:
|
||||
)
|
||||
if count != len(attachment_ids):
|
||||
raise ApiError(404, "not_found", "Вложение не найдено")
|
||||
failed = await connection.fetchval(
|
||||
"SELECT 1 FROM attachments WHERE id=ANY($1::uuid[]) AND status='failed' LIMIT 1", attachment_ids
|
||||
)
|
||||
if failed:
|
||||
raise ApiError(409, 'conflict', 'Вложение завершилось ошибкой')
|
||||
claimed = await connection.fetchrow(
|
||||
"INSERT INTO idempotency_records (user_id, method, path, idempotency_key, request_hash) "
|
||||
"VALUES ($1, 'POST', $2, $3, $4) ON CONFLICT DO NOTHING RETURNING message_id",
|
||||
@@ -396,16 +401,23 @@ class PostgresStore:
|
||||
attachment_id,
|
||||
)
|
||||
await connection.execute(
|
||||
"UPDATE attachment_jobs SET status = 'processing', lease_until = now() + $2 * interval '1 second', updated_at = now() WHERE attachment_id = $1",
|
||||
"UPDATE attachment_jobs SET status = 'processing', first_started_at=COALESCE(first_started_at, now()), lease_until = now() + $2 * interval '1 second', updated_at = now() WHERE attachment_id = $1",
|
||||
attachment_id, lease_seconds,
|
||||
)
|
||||
return dict(row)
|
||||
result = dict(row)
|
||||
result['claim_time'] = await connection.fetchval('SELECT processing_started_at FROM attachments WHERE id=$1', attachment_id)
|
||||
return result
|
||||
except DB_ERRORS as error:
|
||||
raise RuntimeError("PostgreSQL is unavailable") from error
|
||||
|
||||
async def complete_attachment_job(self, attachment_id: UUID, original_key: str, variants: list[dict]) -> None:
|
||||
async def complete_attachment_job(self, attachment_id: UUID, original_key: str, variants: list[dict], claim_time) -> bool:
|
||||
try:
|
||||
async with self.pool.acquire() as connection, connection.transaction():
|
||||
valid = await connection.fetchval(
|
||||
"SELECT 1 FROM attachments a JOIN attachment_jobs j ON j.attachment_id=a.id WHERE a.id=$1 AND a.status='processing' AND a.processing_started_at=$2 AND j.lease_until > now() FOR UPDATE", attachment_id, claim_time
|
||||
)
|
||||
if not valid:
|
||||
return False
|
||||
for variant in variants:
|
||||
await connection.execute(
|
||||
"INSERT INTO attachment_variants (attachment_id, pipeline_version, name, object_key, content_type, size_bytes, width, height, sha256) "
|
||||
@@ -418,28 +430,20 @@ class PostgresStore:
|
||||
attachment_id, original_key,
|
||||
)
|
||||
await connection.execute("UPDATE attachment_jobs SET status = 'ready', lease_until = NULL, updated_at = now() WHERE attachment_id = $1", attachment_id)
|
||||
return True
|
||||
except DB_ERRORS as error:
|
||||
raise RuntimeError("PostgreSQL is unavailable") from error
|
||||
|
||||
async def fail_attachment_job(self, attachment_id: UUID, code: str) -> None:
|
||||
try:
|
||||
await self.pool.execute(
|
||||
"UPDATE attachments SET status = 'failed', error_code = $2, updated_at = now() WHERE id = $1", attachment_id, code
|
||||
)
|
||||
await self.pool.execute(
|
||||
"UPDATE attachment_jobs SET status = 'failed', last_error = $2, updated_at = now() WHERE attachment_id = $1", attachment_id, code
|
||||
)
|
||||
except DB_ERRORS as error:
|
||||
raise RuntimeError("PostgreSQL is unavailable") from error
|
||||
|
||||
async def retry_attachment_job(self, attachment_id: UUID, max_attempts: int, delay_seconds: float) -> bool:
|
||||
async def retry_attachment_job(self, attachment_id: UUID, max_attempts: int, delay_seconds: float, claim_time, permanent_code=None) -> bool:
|
||||
"""Return true only when a bounded at-least-once retry was recorded."""
|
||||
try:
|
||||
async with self.pool.acquire() as connection, connection.transaction():
|
||||
job = await connection.fetchrow(
|
||||
"SELECT attempt FROM attachment_jobs WHERE attachment_id = $1 FOR UPDATE", attachment_id
|
||||
"SELECT j.attempt FROM attachment_jobs j JOIN attachments a ON a.id=j.attachment_id WHERE a.id=$1 AND a.status='processing' AND a.processing_started_at=$2 FOR UPDATE", attachment_id, claim_time
|
||||
)
|
||||
if job is not None and job["attempt"] < max_attempts:
|
||||
if job is None:
|
||||
return False
|
||||
if permanent_code is None and job["attempt"] < max_attempts:
|
||||
await connection.execute(
|
||||
"UPDATE attachment_jobs SET attempt = attempt + 1, status = 'queued', last_error = 'processing_failed', lease_until = NULL, next_attempt_at = now() + $2 * interval '1 second', updated_at = now() WHERE attachment_id = $1",
|
||||
attachment_id, delay_seconds,
|
||||
@@ -448,14 +452,16 @@ class PostgresStore:
|
||||
"UPDATE attachments SET status = 'queued', updated_at = now() WHERE id = $1", attachment_id
|
||||
)
|
||||
return True
|
||||
await self.fail_attachment_job(attachment_id, "retry_exhausted")
|
||||
return False
|
||||
code = permanent_code or 'retry_exhausted'
|
||||
await connection.execute("UPDATE attachments SET status='failed', error_code=$2, updated_at=now() WHERE id=$1", attachment_id, code)
|
||||
await connection.execute("UPDATE attachment_jobs SET status='failed', last_error=$2, lease_until=NULL, updated_at=now() WHERE attachment_id=$1", attachment_id, code)
|
||||
return False
|
||||
except DB_ERRORS as error:
|
||||
raise RuntimeError("PostgreSQL is unavailable") from error
|
||||
|
||||
async def due_outbox_events(self, limit: int = 20) -> list[dict]:
|
||||
try:
|
||||
return [dict(row) for row in await self.pool.fetch(
|
||||
return [{**dict(row), 'payload': json.loads(row['payload']) if isinstance(row['payload'], str) else row['payload']} for row in await self.pool.fetch(
|
||||
"SELECT id, payload FROM attachment_outbox WHERE published_at IS NULL ORDER BY created_at LIMIT $1", limit
|
||||
)]
|
||||
except DB_ERRORS as error:
|
||||
@@ -464,7 +470,7 @@ class PostgresStore:
|
||||
async def mark_outbox_published(self, event_id: UUID) -> None:
|
||||
await self.pool.execute("UPDATE attachment_outbox SET published_at = now(), attempts = attempts + 1 WHERE id = $1", event_id)
|
||||
|
||||
async def schedule_due_jobs(self) -> None:
|
||||
async def schedule_due_jobs(self, max_attempts: int = 5) -> None:
|
||||
"""Outbox dispatcher also restores expired leases and scheduled retries."""
|
||||
try:
|
||||
async with self.pool.acquire() as connection, connection.transaction():
|
||||
@@ -474,9 +480,15 @@ class PostgresStore:
|
||||
"FOR UPDATE SKIP LOCKED"
|
||||
)
|
||||
for row in rows:
|
||||
row = dict(row)
|
||||
if row["status"] == "processing":
|
||||
if row['attempt'] >= max_attempts:
|
||||
await connection.execute("UPDATE attachment_jobs SET status='failed', lease_until=NULL, last_error='retry_exhausted', updated_at=now() WHERE job_id=$1", row['job_id'])
|
||||
await connection.execute("UPDATE attachments SET status='failed', error_code='retry_exhausted', updated_at=now() WHERE id=$1", row['attachment_id'])
|
||||
continue
|
||||
row['attempt'] += 1
|
||||
await connection.execute(
|
||||
"UPDATE attachment_jobs SET status = 'queued', lease_until = NULL, next_attempt_at = now(), last_error = 'lease_expired', updated_at = now() WHERE job_id = $1", row["job_id"]
|
||||
"UPDATE attachment_jobs SET status = 'queued', attempt=$2, lease_until = NULL, next_attempt_at = now(), last_error = 'lease_expired', updated_at = now() WHERE job_id = $1", row["job_id"], row['attempt']
|
||||
)
|
||||
await connection.execute("UPDATE attachments SET status = 'queued', updated_at = now() WHERE id = $1", row["attachment_id"])
|
||||
payload = {"schema_version": 1, "job_id": str(row["job_id"]), "type": "attachment.process.v1", "attachment_id": str(row["attachment_id"]), "pipeline_version": row["pipeline_version"], "attempt": row["attempt"], "created_at": datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")}
|
||||
@@ -505,7 +517,7 @@ class PostgresStore:
|
||||
async def attachment_keys(self) -> dict[str, UUID]:
|
||||
"""Inventory used only by the maintenance reconciler."""
|
||||
try:
|
||||
rows = await self.pool.fetch("SELECT COALESCE(original_key, staging_key) AS object_key, id FROM attachments")
|
||||
rows = await self.pool.fetch("SELECT staging_key AS object_key, id FROM attachments WHERE status <> 'ready' UNION ALL SELECT original_key, id FROM attachments WHERE original_key IS NOT NULL UNION ALL SELECT object_key, attachment_id AS id FROM attachment_variants")
|
||||
except DB_ERRORS as error:
|
||||
raise RuntimeError("PostgreSQL is unavailable") from error
|
||||
return {row["object_key"]: row["id"] for row in rows}
|
||||
|
||||
+94
-26
@@ -6,8 +6,13 @@ import json
|
||||
import logging
|
||||
import os
|
||||
import random
|
||||
import pickle
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from uuid import UUID
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from PIL import Image, ImageOps
|
||||
|
||||
@@ -19,6 +24,32 @@ from .storage.s3 import S3Settings, S3Store
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s worker %(message)s")
|
||||
|
||||
|
||||
class PermanentFailure(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def fit(image, bound):
|
||||
longest = max(image.size)
|
||||
if longest <= bound:
|
||||
return image.copy()
|
||||
return image.resize((max(1, image.width * bound // longest),
|
||||
max(1, image.height * bound // longest)), Image.Resampling.LANCZOS)
|
||||
|
||||
|
||||
def render_isolated(data, timeout):
|
||||
try:
|
||||
result = subprocess.run([sys.executable, '-m', 'src.render'], input=data,
|
||||
stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, timeout=timeout)
|
||||
except subprocess.TimeoutExpired as error:
|
||||
raise TimeoutError('render deadline exceeded') from error
|
||||
if result.returncode in (20, 21):
|
||||
raise PermanentFailure('invalid_image' if result.returncode == 20 else 'image_resource_limit')
|
||||
if result.returncode:
|
||||
raise RuntimeError('renderer exited')
|
||||
# Only our own child process produces this payload; no external pickle input.
|
||||
return pickle.loads(result.stdout)
|
||||
|
||||
|
||||
def thumbnail(data: bytes) -> tuple[bytes, int, int]:
|
||||
with Image.open(BytesIO(data), formats=("PNG", "JPEG")) as source:
|
||||
image = ImageOps.exif_transpose(source)
|
||||
@@ -28,7 +59,7 @@ def thumbnail(data: bytes) -> tuple[bytes, int, int]:
|
||||
image = background.convert("RGB")
|
||||
else:
|
||||
image = image.convert("RGB")
|
||||
image.thumbnail((256, 256), Image.Resampling.LANCZOS)
|
||||
image = fit(image, 256)
|
||||
output = BytesIO()
|
||||
image.save(output, format="JPEG", quality=85, optimize=True)
|
||||
return output.getvalue(), image.width, image.height
|
||||
@@ -44,8 +75,7 @@ def variants(data: bytes, max_pixels: int = 40_000_000) -> list[tuple[str, bytes
|
||||
white.alpha_composite(image)
|
||||
rgb = white.convert("RGB")
|
||||
|
||||
thumb = rgb.copy()
|
||||
thumb.thumbnail((256, 256), Image.Resampling.LANCZOS)
|
||||
thumb = fit(rgb, 256)
|
||||
thumbnail_out = BytesIO(); thumb.save(thumbnail_out, "JPEG", quality=85, optimize=True)
|
||||
|
||||
side = min(rgb.width, rgb.height)
|
||||
@@ -53,8 +83,7 @@ def variants(data: bytes, max_pixels: int = 40_000_000) -> list[tuple[str, bytes
|
||||
crop = rgb.crop((left, top, left + side, top + side)).resize((128, 128), Image.Resampling.LANCZOS)
|
||||
crop_out = BytesIO(); crop.save(crop_out, "JPEG", quality=85, optimize=True)
|
||||
|
||||
optimized = rgb.copy()
|
||||
optimized.thumbnail((1280, 1280), Image.Resampling.LANCZOS)
|
||||
optimized = fit(rgb, 1280)
|
||||
optimized_out = BytesIO(); optimized.save(optimized_out, "WEBP", quality=80, method=4)
|
||||
return [
|
||||
("thumbnail", thumbnail_out.getvalue(), "image/jpeg", thumb.width, thumb.height),
|
||||
@@ -64,26 +93,50 @@ def variants(data: bytes, max_pixels: int = 40_000_000) -> list[tuple[str, bytes
|
||||
|
||||
|
||||
async def process(store: PostgresStore, s3: S3Store, payload: dict, lease_seconds: int) -> None:
|
||||
# Excludes cleanup for the whole lifetime of S3 effects, including stale attempts.
|
||||
async with store.attachment_transaction():
|
||||
await process_locked(store, s3, payload, lease_seconds)
|
||||
|
||||
|
||||
async def process_locked(store: PostgresStore, s3: S3Store, payload: dict, lease_seconds: int) -> None:
|
||||
attachment_id = UUID(payload["attachment_id"])
|
||||
attachment = await store.claim_attachment_job(attachment_id, lease_seconds)
|
||||
if attachment is None:
|
||||
return
|
||||
original_key = f"originals/{attachment_id}"
|
||||
payload['_claim_time'] = attachment['claim_time']
|
||||
payload['attempt'] = attachment.get('attempt', payload.get('attempt', 1))
|
||||
attempt_key = uuid4().hex
|
||||
original_key = f"originals/{attachment_id}/{attempt_key}"
|
||||
await s3.copy(attachment["staging_key"], original_key, attachment["content_type"])
|
||||
original = await s3.download_key(attachment["staging_key"], attachment["size_bytes"], attachment["sha256"])
|
||||
rendered = await asyncio.wait_for(
|
||||
asyncio.to_thread(variants, original, s3.settings.max_pixels), timeout=lease_seconds
|
||||
)
|
||||
rendered = await asyncio.to_thread(render_isolated, original, lease_seconds)
|
||||
metadata = []
|
||||
for name, content, content_type, width, height in rendered:
|
||||
extension = "webp" if name == "optimized" else "jpg"
|
||||
object_key = f"variants/{attachment_id}/{name}.{extension}"
|
||||
object_key = f"variants/{attachment_id}/{attempt_key}/{name}.{extension}"
|
||||
await s3.put(object_key, BytesIO(content), len(content), content_type)
|
||||
metadata.append({"name": name, "object_key": object_key, "content_type": content_type,
|
||||
"size_bytes": len(content), "width": width, "height": height,
|
||||
"sha256": sha256(content).hexdigest()})
|
||||
await store.complete_attachment_job(attachment_id, original_key, metadata)
|
||||
await s3.delete(attachment["staging_key"])
|
||||
committed = await store.complete_attachment_job(attachment_id, original_key, metadata, attachment['claim_time'])
|
||||
if committed:
|
||||
try:
|
||||
await s3.delete(attachment["staging_key"])
|
||||
except Exception:
|
||||
logging.warning('staging cleanup deferred attachment_id=%s', attachment_id)
|
||||
|
||||
|
||||
def settle_delivery(channel, delivery_tag, future):
|
||||
"""Only a successful durable disposition permits acknowledgement."""
|
||||
try:
|
||||
success = future.result() is True
|
||||
except Exception:
|
||||
success = False
|
||||
if channel.is_open:
|
||||
if success:
|
||||
channel.basic_ack(delivery_tag)
|
||||
else:
|
||||
channel.basic_nack(delivery_tag, requeue=True)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
@@ -104,46 +157,61 @@ def main() -> None:
|
||||
s3.client.close()
|
||||
await store.close()
|
||||
|
||||
async def schedule_retry(attachment_id, delay):
|
||||
async def schedule_retry(attachment_id, delay, claim_time, permanent_code=None):
|
||||
retry_store = PostgresStore(settings.database_url)
|
||||
await retry_store.connect()
|
||||
try:
|
||||
return await retry_store.retry_attachment_job(
|
||||
attachment_id, max(1, int(os.environ.get("JOB_MAX_ATTEMPTS", "5"))), delay
|
||||
attachment_id, max(1, int(os.environ.get("JOB_MAX_ATTEMPTS", "5"))), delay, claim_time, permanent_code
|
||||
)
|
||||
finally:
|
||||
await retry_store.close()
|
||||
|
||||
def handle(body):
|
||||
payload = None
|
||||
try:
|
||||
payload = json.loads(body)
|
||||
if not isinstance(payload, dict):
|
||||
raise ValueError('job must be an object')
|
||||
if payload["type"] != "attachment.process.v1" or payload["schema_version"] != 1:
|
||||
raise ValueError("unexpected broker payload")
|
||||
UUID(payload['attachment_id'])
|
||||
UUID(payload['job_id'])
|
||||
except (ValueError, KeyError, TypeError):
|
||||
try:
|
||||
asyncio.run(publish_dlq({'failure': 'invalid_job'}))
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
try:
|
||||
asyncio.run(handle_async(payload))
|
||||
except Exception:
|
||||
logging.exception("job failed")
|
||||
except Exception as error:
|
||||
logging.warning("job failed attachment_id=%s", payload['attachment_id'])
|
||||
if '_claim_time' not in payload:
|
||||
return False
|
||||
try:
|
||||
attachment_id = UUID(payload["attachment_id"])
|
||||
attempt = int(payload.get("attempt", 1))
|
||||
delay = min(60.0, float(2 ** min(attempt, 6))) * random.uniform(0.5, 1.5)
|
||||
retry = asyncio.run(schedule_retry(attachment_id, delay))
|
||||
asyncio.run(schedule_retry(attachment_id, delay, payload['_claim_time'],
|
||||
str(error) if isinstance(error, PermanentFailure) else None))
|
||||
except Exception:
|
||||
retry = False
|
||||
if not retry:
|
||||
try:
|
||||
asyncio.run(publish_dlq(payload))
|
||||
except Exception:
|
||||
logging.exception("could not publish failed job to DLQ")
|
||||
return None
|
||||
return False
|
||||
return True
|
||||
|
||||
def consume(ch, method, properties, body):
|
||||
del properties
|
||||
future = executor.submit(handle, body)
|
||||
def acknowledged(_future):
|
||||
connection.add_callback_threadsafe(lambda: ch.basic_ack(method.delivery_tag))
|
||||
if connection.is_open:
|
||||
connection.add_callback_threadsafe(lambda: settle_delivery(ch, method.delivery_tag, _future))
|
||||
future.add_done_callback(acknowledged)
|
||||
|
||||
try:
|
||||
def heartbeat():
|
||||
Path('/tmp/worker-heartbeat').write_text(str(time.time()))
|
||||
connection.call_later(2, heartbeat)
|
||||
heartbeat()
|
||||
with ThreadPoolExecutor(max_workers=workers, thread_name_prefix="attachment-worker") as executor:
|
||||
channel.basic_consume(queue=QUEUE, on_message_callback=consume, auto_ack=False)
|
||||
channel.start_consuming()
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Attachment contract checks against the running lab4 stack."""
|
||||
"""Attachment contract checks against the running lab5 stack."""
|
||||
import hashlib
|
||||
import http.client
|
||||
from io import BytesIO
|
||||
@@ -23,16 +23,17 @@ def main():
|
||||
path = '/api/v1/chats/' + chat['id'] + '/attachments?filename='
|
||||
png = (Path(__file__).resolve().parents[1] / 'tests/fixtures/sample.png').read_bytes()
|
||||
|
||||
def upload(data, mime='image/png', filename='photo.png', status=201, actor=alice):
|
||||
def upload(data, mime='image/png', filename='photo.png', status=202, actor=alice):
|
||||
return c.call('POST', path + quote(filename, safe=''), status, actor=actor, raw=data,
|
||||
headers={'Content-Type': mime}, schema='Attachment' if status == 201 else None)
|
||||
headers={'Content-Type': mime}, schema='Attachment' if status == 202 else None)
|
||||
|
||||
for fmt, mime in [('PNG', 'image/png'), ('JPEG', 'image/jpeg')]:
|
||||
buf = BytesIO()
|
||||
Image.new('RGB', (32, 20), 'blue').save(buf, fmt)
|
||||
data = buf.getvalue()
|
||||
att = upload(data, mime)['json']
|
||||
check(att['variants'] == [] and att['status'] == 'ready', 'wrong attachment state')
|
||||
check(att['variants'] == [] and att['status'] == 'queued', 'wrong attachment state')
|
||||
att = c.wait_ready(bob, att)
|
||||
check(att['sha256'] == hashlib.sha256(data).hexdigest(), 'wrong checksum')
|
||||
content_path = '/api/v1/attachments/' + att['id']
|
||||
check(c.call('GET', content_path, actor=bob, schema='Attachment')['json'] == att,
|
||||
@@ -41,7 +42,7 @@ def main():
|
||||
c.call('GET', content_path, 404, actor=eve)
|
||||
c.call('GET', content_path + '/content', 404, actor=eve)
|
||||
c.call('GET', content_path + '/content', 401)
|
||||
c.call('GET', content_path + '/content?variant=thumbnail', 404, actor=bob)
|
||||
c.call('GET', content_path + '/content?variant=thumbnail', actor=bob)
|
||||
message = c.message(bob, chat, '', attachments=[att['id']])['json']
|
||||
history = c.call('GET', '/api/v1/chats/' + chat['id'] + '/messages',
|
||||
actor=alice, schema='MessagePage')['json']['items']
|
||||
@@ -98,6 +99,7 @@ def main():
|
||||
# Exact limits are accepted; original bytes (including PNG trailing data) survive.
|
||||
exact = png + b'\0' * (10485760 - len(png))
|
||||
att = upload(exact)['json']
|
||||
c.wait_ready(bob, att)
|
||||
check(c.call('GET', '/api/v1/attachments/' + att['id'] + '/content', actor=bob)['bytes'] == exact,
|
||||
'10 MiB original changed')
|
||||
buf = BytesIO()
|
||||
|
||||
@@ -0,0 +1,107 @@
|
||||
"""Offline regression checks for delivery and terminal-state safety."""
|
||||
import asyncio
|
||||
from contextlib import asynccontextmanager
|
||||
from datetime import datetime, timezone
|
||||
from io import BytesIO
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sys
|
||||
from types import SimpleNamespace
|
||||
import unittest
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
from uuid import uuid4
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
from src.storage.postgres import PostgresStore
|
||||
from src.worker import process_locked as process, settle_delivery
|
||||
from concurrent.futures import Future
|
||||
from src.broker import _publish_to
|
||||
|
||||
|
||||
class CriticalTests(unittest.IsolatedAsyncioTestCase):
|
||||
def test_failed_or_unrecorded_work_is_requeued(self):
|
||||
for result in (False, None, RuntimeError('database unavailable')):
|
||||
future, channel = Future(), MagicMock()
|
||||
if isinstance(result, Exception):
|
||||
future.set_exception(result)
|
||||
else:
|
||||
future.set_result(result)
|
||||
settle_delivery(channel, 7, future)
|
||||
channel.basic_ack.assert_not_called()
|
||||
channel.basic_nack.assert_called_once_with(7, requeue=True)
|
||||
|
||||
def test_durable_success_is_acked(self):
|
||||
future, channel = Future(), MagicMock()
|
||||
future.set_result(True)
|
||||
settle_delivery(channel, 7, future)
|
||||
channel.basic_ack.assert_called_once_with(7)
|
||||
channel.basic_nack.assert_not_called()
|
||||
|
||||
async def test_outbox_decodes_jsonb(self):
|
||||
store = PostgresStore('unused')
|
||||
store._pool = AsyncMock()
|
||||
payload = {'type': 'attachment.process.v1', 'attachment_id': str(uuid4())}
|
||||
store.pool.fetch.return_value = [{'id': uuid4(), 'payload': json.dumps(payload)}]
|
||||
event, = await store.due_outbox_events()
|
||||
self.assertEqual(json.loads(json.dumps(event['payload'])), payload)
|
||||
|
||||
async def run_processing(self, committed):
|
||||
store, s3 = AsyncMock(), AsyncMock()
|
||||
stamp = datetime.now(timezone.utc)
|
||||
store.claim_attachment_job.return_value = {
|
||||
'claim_time': stamp, 'staging_key': 'staging/test',
|
||||
'content_type': 'image/png', 'size_bytes': 1, 'sha256': 'unused',
|
||||
}
|
||||
store.complete_attachment_job.return_value = committed
|
||||
s3.settings = SimpleNamespace(max_pixels=40000000)
|
||||
s3.download_key.return_value = b'fixture'
|
||||
s3.delete.side_effect = RuntimeError('storage temporarily unavailable')
|
||||
payload = {'attachment_id': str(uuid4())}
|
||||
with patch('src.worker.render_isolated', return_value=[('thumbnail', b'jpeg', 'image/jpeg', 1, 1)]):
|
||||
await process(store, s3, payload, 60)
|
||||
self.assertEqual(store.complete_attachment_job.call_args.args[-1], stamp)
|
||||
return store, s3
|
||||
|
||||
async def test_cleanup_error_does_not_retry_ready(self):
|
||||
store, s3 = await self.run_processing(True)
|
||||
s3.delete.assert_awaited_once()
|
||||
store.retry_attachment_job.assert_not_awaited()
|
||||
|
||||
async def test_stale_completion_keeps_staging(self):
|
||||
_, s3 = await self.run_processing(False)
|
||||
s3.delete.assert_not_awaited()
|
||||
|
||||
async def test_stale_claim_cannot_publish_metadata(self):
|
||||
store = PostgresStore('unused')
|
||||
connection = AsyncMock()
|
||||
connection.fetchval.return_value = None
|
||||
@asynccontextmanager
|
||||
async def context():
|
||||
yield connection
|
||||
connection.transaction = context
|
||||
store._pool = SimpleNamespace(acquire=context)
|
||||
self.assertFalse(await store.complete_attachment_job(uuid4(), 'original', [], datetime.now(timezone.utc)))
|
||||
connection.execute.assert_not_awaited()
|
||||
|
||||
async def test_retry_of_terminal_or_stale_attempt_is_noop(self):
|
||||
store = PostgresStore('unused')
|
||||
connection = AsyncMock()
|
||||
connection.fetchrow.return_value = None
|
||||
@asynccontextmanager
|
||||
async def context():
|
||||
yield connection
|
||||
connection.transaction = context
|
||||
store._pool = SimpleNamespace(acquire=context)
|
||||
self.assertFalse(await store.retry_attachment_job(uuid4(), 5, 1, datetime.now(timezone.utc)))
|
||||
connection.execute.assert_not_awaited()
|
||||
|
||||
def test_dlq_waits_for_publisher_confirm(self):
|
||||
connection = MagicMock()
|
||||
with patch('src.broker._connection', return_value=connection):
|
||||
_publish_to('dlq', {'failure': 'invalid_job'})
|
||||
connection.channel.return_value.confirm_delivery.assert_called_once()
|
||||
self.assertTrue(connection.channel.return_value.basic_publish.call_args.kwargs['mandatory'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -0,0 +1,127 @@
|
||||
"""Contract, isolated-render and benchmark regressions without Docker."""
|
||||
import asyncio
|
||||
from contextlib import asynccontextmanager
|
||||
from datetime import datetime, timedelta, timezone
|
||||
import importlib.util
|
||||
from io import BytesIO
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from types import SimpleNamespace
|
||||
import unittest
|
||||
from unittest.mock import AsyncMock, patch
|
||||
from uuid import uuid4
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT))
|
||||
from PIL import Image
|
||||
from src.worker import variants, render_isolated, PermanentFailure
|
||||
from src.storage.postgres import PostgresStore
|
||||
from src.reconcile import scan
|
||||
from src.errors import ApiError
|
||||
|
||||
|
||||
def png(width, height):
|
||||
buffer = BytesIO()
|
||||
Image.new('RGBA', (width, height), (255, 0, 0, 0)).save(buffer, 'PNG')
|
||||
return buffer.getvalue()
|
||||
|
||||
|
||||
class Regressions(unittest.IsolatedAsyncioTestCase):
|
||||
async def test_failed_attachment_is_rejected(self):
|
||||
store, connection = PostgresStore('unused'), AsyncMock()
|
||||
connection.fetchval.side_effect = [1, 1, 1]
|
||||
@asynccontextmanager
|
||||
async def context():
|
||||
yield connection
|
||||
connection.transaction = context
|
||||
store._pool = SimpleNamespace(acquire=context)
|
||||
with self.assertRaises(ApiError) as error:
|
||||
await store.create_message_idempotent(uuid4(), uuid4(), '', 'key', 86400, [uuid4()])
|
||||
self.assertEqual(error.exception.status_code, 409)
|
||||
connection.fetchrow.assert_not_awaited()
|
||||
|
||||
def test_floor_and_small_images(self):
|
||||
self.assertEqual(variants(png(300, 200))[0][3:], (256, 170))
|
||||
self.assertEqual(variants(png(200, 300))[0][3:], (170, 256))
|
||||
self.assertEqual(variants(png(10, 8))[0][3:], (10, 8))
|
||||
for name, data, mime, width, height in variants(png(300, 200)):
|
||||
with Image.open(BytesIO(data)) as image:
|
||||
self.assertEqual(image.size, (width, height))
|
||||
self.assertTrue(all(value > 240 for value in image.convert('RGB').getpixel((0, 0))))
|
||||
|
||||
def test_renderer_success_and_permanent_error(self):
|
||||
self.assertEqual(render_isolated(png(300, 200), 10)[0][3:], (256, 170))
|
||||
with self.assertRaises(PermanentFailure):
|
||||
render_isolated(b'not an image', 10)
|
||||
|
||||
def test_renderer_is_killed_on_deadline(self):
|
||||
started = time.monotonic()
|
||||
with self.assertRaises(TimeoutError):
|
||||
render_isolated(png(300, 200), .0001)
|
||||
self.assertLess(time.monotonic() - started, 2)
|
||||
|
||||
async def test_cleanup_retains_active_and_failed_staging(self):
|
||||
old = datetime.now(timezone.utc) - timedelta(days=2)
|
||||
db, s3 = AsyncMock(), AsyncMock()
|
||||
db.attachment_keys.return_value = {'staging/active': uuid4(), 'staging/failed': uuid4(), 'originals/ready': uuid4()}
|
||||
s3.list_with_pages.side_effect = [({'staging/active': old, 'staging/failed': old, 'staging/orphan': old}, 1),
|
||||
({'originals/ready': old}, 1), ({'variants/orphan': old}, 1)]
|
||||
report = await scan(SimpleNamespace(grace_seconds=60, apply=True, object_key=[]), db, s3)
|
||||
self.assertEqual(report['deleted'], ['staging/orphan', 'variants/orphan'])
|
||||
|
||||
async def test_permanent_failure_is_atomic_and_not_retried(self):
|
||||
store = PostgresStore('unused')
|
||||
connection = AsyncMock()
|
||||
connection.fetchrow.return_value = {'attempt': 1}
|
||||
@asynccontextmanager
|
||||
async def context():
|
||||
yield connection
|
||||
connection.transaction = context
|
||||
store._pool = SimpleNamespace(acquire=context)
|
||||
result = await store.retry_attachment_job(uuid4(), 5, 1, datetime.now(timezone.utc), 'invalid_image')
|
||||
self.assertFalse(result)
|
||||
self.assertEqual(connection.execute.await_count, 2)
|
||||
for call in connection.execute.call_args_list:
|
||||
self.assertIn("status='failed'", call.args[0])
|
||||
self.assertEqual(call.args[2], 'invalid_image')
|
||||
|
||||
def test_benchmark_partial_batch_and_recovery(self):
|
||||
spec = importlib.util.spec_from_file_location('track1', ROOT / 'scripts/track1.py')
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
jobs, lock = {}, threading.Lock()
|
||||
class FakeClient:
|
||||
def __init__(self, *args, **kwargs): pass
|
||||
def bootstrap(self): return {}, {}, {}, {'id': str(uuid4())}
|
||||
def call(self, method, path, *args, **kwargs):
|
||||
if method == 'POST':
|
||||
key = str(uuid4())
|
||||
with lock: jobs[key] = {'id': key, 'status': 'ready', 'queue_wait_seconds': .01}
|
||||
return {'status': 202, 'json': {'id': key}}
|
||||
return {'json': {'status': 'ready'}}
|
||||
def attachment(self, *args): return {}, b''
|
||||
def wait_ready(self, *args, **kwargs): return {}
|
||||
def metrics(_):
|
||||
with lock:
|
||||
return {'jobs': list(jobs.values()), 'pending': 0, 'rss_bytes': 1000, 'concurrency': 2, 'prefetch': 2}
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
destination = str(Path(directory) / 'result.json')
|
||||
args = SimpleNamespace(run_name='test', workers=2, prefetch=2, max_pending=20, images=10,
|
||||
concurrency=8, size=8, max_ready_seconds=2, require_backpressure=False,
|
||||
output=destination, base_url=None, ca_file=None)
|
||||
info = {'Config': {'Env': ['MAX_PENDING_JOBS=20']}, 'HostConfig': {'Memory': 4096, 'NanoCpus': 2}}
|
||||
with patch.object(module, 'arguments', return_value=args), patch.object(module, 'Client', FakeClient), \
|
||||
patch.object(module, 'metrics', side_effect=metrics), patch.object(module, 'container_info', return_value=info), \
|
||||
patch.object(module, 'broker_depth', return_value={'messages_ready': 0, 'messages_unacknowledged': 0}):
|
||||
module.main()
|
||||
report = json.loads(Path(destination).read_text())
|
||||
self.assertEqual(report['results']['ready'], 10)
|
||||
self.assertTrue(report['results']['recovery_upload_ready'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user