ЛР 5 - БВТ2304 - Соколова - на 4 #89
+8
-1
@@ -19,7 +19,14 @@ S3_SECRET_ACCESS_KEY_FILE=/run/secrets/s3_secret_access_key
|
||||
MAX_UPLOAD_BYTES=10485760
|
||||
MAX_IMAGE_PIXELS=40000000
|
||||
BROKER_URL_FILE=/run/secrets/broker_url
|
||||
WORKER_CONCURRENCY=2
|
||||
# One task per worker process; values other than 1 are rejected.
|
||||
WORKER_CONCURRENCY=1
|
||||
JOB_MAX_ATTEMPTS=5
|
||||
JOB_TIMEOUT_SECONDS=60
|
||||
STAGING_RETENTION_SECONDS=86400
|
||||
|
||||
# Additional settings used by compose.queue.yaml:
|
||||
JOB_LEASE_SECONDS=120
|
||||
JOB_MEMORY_MB=512
|
||||
# Local credentials are generated by scripts/setup_lab5.py.
|
||||
# See REPORT.md for mappings and settings fixed by the contract.
|
||||
|
||||
+26
@@ -0,0 +1,26 @@
|
||||
FROM python:3.12.10-slim-bookworm
|
||||
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
PIP_NO_CACHE_DIR=1
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY requirements.txt ./
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
RUN addgroup --system appgroup \
|
||||
&& adduser --system --ingroup appgroup --no-create-home appuser \
|
||||
&& chown appuser:appgroup /app
|
||||
|
||||
COPY --chown=appuser:appgroup app ./app
|
||||
COPY --chown=appuser:appgroup migrations ./migrations
|
||||
|
||||
USER appuser
|
||||
|
||||
EXPOSE 8080
|
||||
|
||||
HEALTHCHECK --interval=10s --timeout=3s --start-period=5s --retries=3 \
|
||||
CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=2).read()" || exit 1
|
||||
|
||||
CMD ["python", "-m", "uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8080", "--no-access-log"]
|
||||
@@ -1,37 +1,312 @@
|
||||
# Отчёт — лабораторная 5
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
- Автор, группа: Полина Соколова, БВТ2304.
|
||||
- Целевая оценка: 4. Исследовательский трек уровня 5 не заявляется.
|
||||
- Commit/PR: будут указаны при публикации.
|
||||
- Проверки выполнены 17.11.2026 на локальном учебном стенде.
|
||||
- Использованы собственные исходники lab4, базовый коммит dcaebee.
|
||||
Контракт, публичные tests и Makefile взяты из skeleton lab5.
|
||||
|
||||
## Запуск из чистого clone
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
Требуются Linux, Docker Engine с Docker Compose plugin, Python 3.9+
|
||||
для публичных проверок, make. Приложение работает в Python 3.12.10
|
||||
в контейнере. Локальные проверки запускались также на Python 3.11.
|
||||
|
||||
Из корня репозитория:
|
||||
|
||||
```sh
|
||||
python3 scripts/setup_lab5.py
|
||||
./scripts/dc config --quiet
|
||||
./scripts/dc up -d --build
|
||||
./scripts/dc ps -a
|
||||
```
|
||||
|
||||
setup_lab5.py генерирует локальные .env и secrets/, не выводя значения
|
||||
секретов. При существующей конфигурации скрипт отказывается её перезаписывать.
|
||||
Повторный обычный запуск не требует повторной генерации секретов.
|
||||
|
||||
scripts/dc явно выбирает проект mtusi2026lab5, .env и три Compose-файла:
|
||||
compose.yaml, compose.s3.yaml, compose.queue.yaml.
|
||||
|
||||
Перед тестами дождаться HTTP 200:
|
||||
|
||||
```sh
|
||||
curl --fail http://127.0.0.1:18085/health/ready
|
||||
make check
|
||||
make test BASE_URL=http://127.0.0.1:18085
|
||||
```
|
||||
|
||||
Если curl ещё не получил 200, повторить проверку после завершения старта.
|
||||
Не выполнять тесты через внутренние порты API в обход LB.
|
||||
|
||||
Миграции выполняет одноразовый сервис migrate.
|
||||
Существующие миграции проверяются по checksum.
|
||||
s3-check проверяет доступ к приватному bucket перед запуском API/worker.
|
||||
|
||||
Для локальных тестов преобразования:
|
||||
|
||||
```sh
|
||||
python3 -m venv .venv
|
||||
.venv/bin/python -m pip install -r requirements-dev.txt 'Pillow==13.0.0'
|
||||
.venv/bin/python -m unittest discover \
|
||||
-s student-tests -p 'test_image_pipeline.py' -v
|
||||
```
|
||||
|
||||
Тесты уборщика запускаются с зависимостями приложения:
|
||||
|
||||
```sh
|
||||
./scripts/dc exec -T api python - \
|
||||
< student-tests/test_cleanup_staging.py
|
||||
```
|
||||
|
||||
Полная OpenAPI-валидация:
|
||||
|
||||
```sh
|
||||
make validate PYTHON=.venv/bin/python
|
||||
```
|
||||
|
||||
### Версии
|
||||
|
||||
- Python image: 3.12.10-slim-bookworm.
|
||||
- PostgreSQL: 16.4-bookworm.
|
||||
- Redis: 7.2.5-bookworm.
|
||||
- HAProxy: 3.0.5-bookworm.
|
||||
- SeaweedFS: 4.44.
|
||||
- RabbitMQ: 4.0.5-management.
|
||||
- Python-зависимости закреплены в requirements.txt,
|
||||
включая Pillow 13.0.0 и Pika 1.3.2.
|
||||
|
||||
### Остановка и удаление данных
|
||||
|
||||
Безопасная остановка без удаления данных:
|
||||
|
||||
```sh
|
||||
./scripts/dc stop
|
||||
```
|
||||
|
||||
Повторный запуск:
|
||||
|
||||
```sh
|
||||
./scripts/dc up -d
|
||||
```
|
||||
|
||||
Только для намеренного полного удаления учебных данных lab5:
|
||||
|
||||
```sh
|
||||
./scripts/dc down --volumes
|
||||
```
|
||||
|
||||
Последняя команда удаляет volumes проекта, включая БД, S3 и очередь.
|
||||
Не использовать её в проверках сохранности и отказов.
|
||||
|
||||
## Архитектура и решения
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab4, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
```text
|
||||
клиент → HAProxy → две API-реплики
|
||||
├→ PostgreSQL: метаданные + job + outbox
|
||||
└→ приватный S3 staging
|
||||
|
||||
outbox publisher → RabbitMQ → worker → дочерний job_runner
|
||||
├→ оригинал и варианты в S3
|
||||
└→ COMMIT результата в PostgreSQL
|
||||
```
|
||||
|
||||
PostgreSQL — источник истины для пользователей, сессий, чатов,
|
||||
сообщений, метаданных и логических заданий. Redis используется
|
||||
перенесённой реализацией кэша; не является единственным хранилищем сессий.
|
||||
|
||||
API принимает сырые bytes PNG/JPEG и проверяет входное изображение.
|
||||
До 202 оригинал записан в приватный S3-prefix staging/, а вложение,
|
||||
job и outbox закоммичены одной транзакцией PostgreSQL.
|
||||
Это не общая транзакция PostgreSQL/S3: запись без успешного COMMIT
|
||||
может оставить непривязанный staging-объект. Его удаляет уборщик.
|
||||
|
||||
API возвращает снимок queued. Он не создаёт thumbnail синхронно.
|
||||
Валидация исходного изображения остаётся синхронной частью upload.
|
||||
|
||||
Publisher читает outbox, публикует persistent-сообщение в durable
|
||||
очередь attachment.process.v1 с mandatory и publisher confirms.
|
||||
published_at фиксируется после подтверждения брокера.
|
||||
Crash после publish до фиксации published_at допускает дубль.
|
||||
|
||||
В сообщении только идентификаторы и служебные поля по jobs.schema.json.
|
||||
Credentials, bytes, клиентский filename и произвольный URL туда не входят.
|
||||
Worker получает доверенные storage metadata из PostgreSQL.
|
||||
|
||||
Worker захватывает job с lease token. Преобразование и операции одной
|
||||
попытки выполняются в дочернем процессе с внешним wall-clock timeout.
|
||||
Результат публикуется транзакционно после записи всех объектов и проверки
|
||||
текущего lease. Ack отправляется после устойчивого результата либо
|
||||
устойчивой фиксации retry/failed.
|
||||
|
||||
Доставка at least once, не exactly once.
|
||||
Повтор ready-job не откатывает состояние и не запускает преобразование.
|
||||
Для текущего pipeline_version=1 ключи вариантов детерминированы.
|
||||
Ключ метаданных варианта: attachment_id + pipeline_version + name.
|
||||
Исследование конкурирующих версий pipeline не проводилось.
|
||||
|
||||
### Варианты изображений
|
||||
|
||||
- Original возвращается побайтно без перекодирования.
|
||||
- Thumbnail: JPEG, вписан в 256×256, без увеличения, округление вниз.
|
||||
- Crop: JPEG, центрированный квадрат 128×128, увеличение допускается.
|
||||
- Optimized: WebP, вписан в 1280×1280 без увеличения.
|
||||
- EXIF orientation применяется перед преобразованиями.
|
||||
- Alpha вариантов компонуется на белый фон.
|
||||
- JPEG quality=85, WebP quality=82, method=4.
|
||||
- Метаданные EXIF/ICC/XMP исходника не переносятся в варианты.
|
||||
- WebP не обязан быть меньше каждого исходного файла.
|
||||
|
||||
### Retry, ограничения и failed
|
||||
|
||||
- По умолчанию 5 попыток.
|
||||
- Задержка retry: uniform(0.5, min(30, 2**min(attempt, 5))) секунд.
|
||||
- InvalidImage и пойманный MemoryError — постоянные ошибки.
|
||||
- Прочие ошибки дочернего исполнения и timeout — временные;
|
||||
после исчерпания попыток — failed/retry_exhausted.
|
||||
- Реестр failed находится в attachment_jobs.
|
||||
- Один worker обрабатывает одну задачу одновременно.
|
||||
- Worker задаёт prefetch_count=1 и использует basic_get по одной доставке.
|
||||
- Фактическое ограничение параллелизма обеспечивается кодом worker;
|
||||
basic_qos не заявляется отдельной гарантией для polling basic_get.
|
||||
- Upload: максимум 10 485 760 bytes и 40 000 000 пикселей.
|
||||
- По умолчанию timeout попытки 60 секунд, lease 120 секунд.
|
||||
- Дочерний процесс: RLIMIT_AS 512 MiB и RLIMIT_CPU.
|
||||
RLIMIT_AS ограничивает адресное пространство, не точный RSS.
|
||||
- Контейнер worker: mem_limit 768m, cpus=1.
|
||||
- Исчерпание памяти или аварийный сигнал, не пойманный как MemoryError,
|
||||
могут обрабатываться как временная ошибка до исчерпания попыток.
|
||||
- Верхние границы времени и памяти установлены в коде/Compose;
|
||||
отдельный эксперимент на предельных изображениях не проводился.
|
||||
|
||||
### Операторский replay
|
||||
|
||||
```sh
|
||||
./scripts/dc exec -T worker python -m app.replay --list
|
||||
```
|
||||
|
||||
Для failed-job сначала dry-run:
|
||||
|
||||
```sh
|
||||
./scripts/dc exec -T worker python -m app.replay \
|
||||
--job-id UUID --actor operator --reason "Причина устранена"
|
||||
```
|
||||
|
||||
После проверки — та же команда с --apply.
|
||||
Apply проверяет checksum staging, сохраняет job_id, создаёт аудит и
|
||||
новую попытку/outbox одной транзакцией. По умолчанию выдаётся бюджет
|
||||
пяти дополнительных попыток. Счётчик попыток не обнуляется.
|
||||
Ready/queued/processing или устаревшая работа дают noop.
|
||||
Dry-run показывает план, но не проверяет чтение staging; это делает apply.
|
||||
|
||||
### Retention staging
|
||||
|
||||
Worker удаляет staging после COMMIT результата.
|
||||
Периодический cleanup проходит раз в час с grace=86400 секунд:
|
||||
|
||||
- queued/processing не удаляются по возрасту;
|
||||
- ready-остатки и непривязанные объекты удаляются после grace;
|
||||
- failed сохраняется минимум grace после изменения состояния
|
||||
и минимум grace после последней записи объекта;
|
||||
- после удаления failed-staging replay недоступен;
|
||||
- метаданные failed и аудит автоматически не удаляются.
|
||||
|
||||
Уборщик использует ту же advisory-блокировку ключа, что upload,
|
||||
и блокировку строки вложения для согласования с replay.
|
||||
Originals и variants этот уборщик не удаляет.
|
||||
Нулевой/короткий grace разрешён только явно и для одного attachment_id.
|
||||
|
||||
### Соответствие конфигурации
|
||||
|
||||
scripts/setup_lab5.py, а не копирование .env.example, создаёт рабочую
|
||||
локальную конфигурацию. Исходный .env.example содержит также предложения
|
||||
логического интерфейса skeleton.
|
||||
|
||||
- POSTGRES_PASSWORD, RABBITMQ_PASSWORD — локальные значения из .env.
|
||||
- SESSION_TTL_SECONDS, USER_CACHE_TTL_SECONDS — передаются API.
|
||||
- S3_*_FILE и BROKER_URL_FILE — пути подключённых Compose secrets.
|
||||
- JOB_MAX_ATTEMPTS — бюджет новых job и операторского replay.
|
||||
- JOB_TIMEOUT_SECONDS, JOB_LEASE_SECONDS, JOB_MEMORY_MB — настройки worker.
|
||||
- WORKER_CONCURRENCY — поддерживается только 1; другое значение отклоняется.
|
||||
- STAGING_RETENTION_SECONDS — срок фоновой уборки.
|
||||
- Две API-реплики заданы в compose.yaml, не через API_REPLICAS.
|
||||
- MAX_UPLOAD_BYTES/MAX_IMAGE_PIXELS закреплены кодом по контракту.
|
||||
- DATABASE_URL_FILE, CACHE_URL_FILE, SESSION_STORE и
|
||||
IDEMPOTENCY_TTL_SECONDS из предложений skeleton не являются
|
||||
реализованными переключателями этой конфигурации.
|
||||
- Подключение PostgreSQL использует PGHOST/PGPORT/PGDATABASE/PGUSER/PGPASSWORD.
|
||||
|
||||
Readiness API проверяет PostgreSQL и S3. Broker/worker не входят
|
||||
в синхронный readiness: принятая работа может ждать в durable outbox.
|
||||
Это не означает неограниченную ёмкость хранилищ.
|
||||
Liveness проверяет только жизнь API-процесса.
|
||||
|
||||
## Доказательства критериев
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| Критерий | Команда | Наблюдение | Доказательство |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
| Публичный API | make test BASE_URL=http://127.0.0.1:18085 | PASS, 202 и настоящий thumbnail | evidence/smoke.txt |
|
||||
| Worker остановлен, рестарт API/broker | .venv/bin/python student-tests/queued_restart.py | То же вложение стало ready, оригинал и варианты доступны | evidence/queued_restart.json |
|
||||
| Разрыв COMMIT/publish | .venv/bin/python student-tests/outbox_gap.py | После SIGKILL API существующий outbox опубликован без новой job | evidence/outbox_gap.json |
|
||||
| После COMMIT результата до ack | .venv/bin/python student-tests/before_ack.py | Redelivered=True, attempt=1, варианты=3, объекты=4 | evidence/before_ack.json |
|
||||
| После записи S3 до COMMIT результата | .venv/bin/python student-tests/after_objects.py | Сначала processing и 0 строк вариантов; после lease та же job ready на попытке 2 | evidence/after_objects.json |
|
||||
| Failed и replay | .venv/bin/python student-tests/failed_replay.py | Проверены отказ при повреждении, dry-run, аудит, replay и ready-noop | evidence/failed_replay.json |
|
||||
| Исчерпание retry | .venv/bin/python student-tests/retry_exhaustion.py | 5 опубликованных попыток, failed/retry_exhausted, без вариантов | evidence/retry_exhaustion.json |
|
||||
| Уборка staging | .venv/bin/python student-tests/staging_cleanup.py | Активный queued сохранён, orphan и ready-остаток удалены | evidence/staging_cleanup.json |
|
||||
| Преобразования | unittest test_image_pipeline.py | 6 тестов PASS | evidence/image_pipeline_tests.txt |
|
||||
| Защитные ветки cleanup | API python через stdin с test_cleanup_staging.py | 8 тестов PASS, подставные БД/S3 | evidence/cleanup_tests.txt |
|
||||
| Сохранность стенда | make test-persistence с scripts/restart.sh | Данные и прежняя сессия сохранены, хранилища перезапущены с прежними volumes | evidence/persistence.txt; evidence/restart_containers.json |
|
||||
| Отказ API | make test-failover с scripts/stop-one.sh | 2 реплики до отказа, 130 успешных записей, 2 ошибки, восстановление 2.089 с | evidence/failover.txt; evidence/failover_containers.json |
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
PASS в таблице относится к выполненному сценарию, не к универсальной
|
||||
гарантии при любых отказах. Для воспроизведения failure-тестов нужны
|
||||
работающие publisher и worker и один worker-контейнер.
|
||||
|
||||
## Эксперименты и отказы
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
Исследовательская нагрузка уровня 5 не проводилась.
|
||||
Failover-цифры — результат публичного сценария отказа, не benchmark
|
||||
максимального throughput и не SLA.
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
Fault injection before_ack и after_objects включается только явной
|
||||
записью оператора для конкретной job в pipeline_fault_injections.
|
||||
Воздействие одноразовое, с consumed_at. Тесты используют exit code 75.
|
||||
Эти точки оставлены для локальной учебной защиты.
|
||||
|
||||
В after_objects проверено восстановление по реальному lease=120 секунд.
|
||||
В retry_exhaustion проверены сохранённые границы задержек, но не
|
||||
статистическое распределение jitter и не точность времени доставки.
|
||||
|
||||
Исправлена настройка healthcheck RabbitMQ: две диагностические команды
|
||||
не укладывались в общий timeout=5s. Проверки вручную завершались успешно.
|
||||
Установлены interval=20s, timeout=30s, start_period=40s, retries=3.
|
||||
Проверка не отключена: проверяются работа приложения RabbitMQ
|
||||
и listener 5672.
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
- Один Docker-host, единичные PostgreSQL, S3, RabbitMQ и LB:
|
||||
отказ хоста и HA этих компонентов не обеспечиваются.
|
||||
- HTTP допустим только для локальных синтетических данных.
|
||||
- Compose file-backed secrets не заменяют промышленное secret-хранилище.
|
||||
- Администратор учебного брокера/S3 имеет широкие права.
|
||||
- Очередь и staging не имеют реализованной квоты принятой работы.
|
||||
- Отдельный эксперимент перегрузки, RSS и p95 до ready не проводился.
|
||||
- Нет доказательства exactly-once доставки.
|
||||
- Несколько конкурирующих pipeline_version не исследовались.
|
||||
- Подставные тесты cleanup не доказывают реальную конкурентную блокировку.
|
||||
- Удаление непривязанных конечных originals/variants не реализовано
|
||||
новым staging-уборщиком.
|
||||
- Финальное выполнение всех тестов и публикация PR указываются отдельно.
|
||||
|
||||
## Источники и библиотеки
|
||||
|
||||
- Контракт и README skeleton lab5.
|
||||
- RabbitMQ: Reliability Guide; Consumer Acknowledgements and Publisher Confirms.
|
||||
- Pika: BlockingConnection, delivery confirmations, mandatory publishing.
|
||||
- Pillow: ImageOps, EXIF transpose, resize и сохранение JPEG/WebP.
|
||||
- Python: subprocess и resource.
|
||||
- PostgreSQL: транзакции, row/advisory locks, ограничения.
|
||||
- Docker Compose: services, secrets, healthchecks, persistence volumes.
|
||||
- Полный список закреплённых Python-библиотек: requirements.txt.
|
||||
|
||||
+108
@@ -0,0 +1,108 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import binascii
|
||||
import hashlib
|
||||
import re
|
||||
import secrets
|
||||
|
||||
from argon2 import PasswordHasher, Type
|
||||
from argon2.exceptions import VerificationError
|
||||
from starlette.concurrency import run_in_threadpool
|
||||
|
||||
|
||||
USERNAME_RE = re.compile(r"[a-z0-9_]{3,32}", re.ASCII)
|
||||
|
||||
password_hasher = PasswordHasher(
|
||||
time_cost=3,
|
||||
memory_cost=65536,
|
||||
parallelism=4,
|
||||
hash_len=32,
|
||||
salt_len=16,
|
||||
type=Type.ID,
|
||||
)
|
||||
|
||||
DUMMY_PASSWORD_HASH = password_hasher.hash(
|
||||
secrets.token_urlsafe(32)
|
||||
)
|
||||
|
||||
|
||||
async def hash_password(password: str) -> str:
|
||||
return await run_in_threadpool(
|
||||
password_hasher.hash,
|
||||
password,
|
||||
)
|
||||
|
||||
|
||||
async def verify_password(
|
||||
password_hash: str,
|
||||
password: str,
|
||||
) -> bool:
|
||||
try:
|
||||
return await run_in_threadpool(
|
||||
password_hasher.verify,
|
||||
password_hash,
|
||||
password,
|
||||
)
|
||||
except VerificationError:
|
||||
return False
|
||||
|
||||
|
||||
def parse_basic_credentials(
|
||||
authorization: str | None,
|
||||
) -> tuple[str, str] | None:
|
||||
if authorization is None or len(authorization) > 2048:
|
||||
return None
|
||||
|
||||
parts = authorization.split()
|
||||
|
||||
if len(parts) != 2 or parts[0].lower() != "basic":
|
||||
return None
|
||||
|
||||
try:
|
||||
decoded = base64.b64decode(
|
||||
parts[1],
|
||||
validate=True,
|
||||
).decode("utf-8")
|
||||
except (binascii.Error, UnicodeError, ValueError):
|
||||
return None
|
||||
|
||||
username, separator, password = decoded.partition(":")
|
||||
|
||||
if not separator:
|
||||
return None
|
||||
|
||||
if USERNAME_RE.fullmatch(username) is None:
|
||||
return None
|
||||
|
||||
if not 12 <= len(password) <= 128:
|
||||
return None
|
||||
|
||||
return username, password
|
||||
|
||||
|
||||
def parse_bearer_token(
|
||||
authorization: str | None,
|
||||
) -> str | None:
|
||||
if authorization is None or len(authorization) > 2048:
|
||||
return None
|
||||
|
||||
parts = authorization.split()
|
||||
|
||||
if len(parts) != 2 or parts[0].lower() != "bearer":
|
||||
return None
|
||||
|
||||
token = parts[1]
|
||||
|
||||
if re.fullmatch(r"[A-Za-z0-9_-]{43}", token) is None:
|
||||
return None
|
||||
|
||||
return token
|
||||
|
||||
|
||||
def generate_token() -> str:
|
||||
return secrets.token_urlsafe(32)
|
||||
|
||||
|
||||
def hash_token(token: str) -> str:
|
||||
return hashlib.sha256(token.encode("ascii")).hexdigest()
|
||||
@@ -0,0 +1,36 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import pika
|
||||
|
||||
|
||||
QUEUE_NAME = "attachment.process.v1"
|
||||
|
||||
|
||||
def connect_broker():
|
||||
url = Path(os.environ["BROKER_URL_FILE"]).read_text().strip()
|
||||
if not url:
|
||||
raise ValueError("Empty broker URL file")
|
||||
|
||||
parameters = pika.URLParameters(url)
|
||||
parameters.heartbeat = 15
|
||||
parameters.blocked_connection_timeout = 5
|
||||
parameters.socket_timeout = 5
|
||||
parameters.stack_timeout = 10
|
||||
parameters.connection_attempts = 1
|
||||
|
||||
connection = pika.BlockingConnection(parameters)
|
||||
try:
|
||||
channel = connection.channel()
|
||||
channel.queue_declare(
|
||||
queue=QUEUE_NAME,
|
||||
durable=True,
|
||||
exclusive=False,
|
||||
auto_delete=False,
|
||||
)
|
||||
return connection, channel
|
||||
except Exception:
|
||||
connection.close()
|
||||
raise
|
||||
+129
@@ -0,0 +1,129 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from datetime import datetime, timezone
|
||||
from uuid import UUID
|
||||
|
||||
from redis.asyncio import Redis
|
||||
from redis.exceptions import RedisError
|
||||
|
||||
from app.store import store
|
||||
|
||||
|
||||
logger = logging.getLogger("messenger.cache")
|
||||
|
||||
CACHE_TTL_SECONDS = int(
|
||||
os.getenv("USER_CACHE_TTL_SECONDS", "60")
|
||||
)
|
||||
|
||||
if not 1 <= CACHE_TTL_SECONDS <= 3600:
|
||||
raise ValueError(
|
||||
"USER_CACHE_TTL_SECONDS must be between 1 and 3600"
|
||||
)
|
||||
|
||||
|
||||
redis_client = Redis(
|
||||
host=os.getenv("REDIS_HOST", "redis"),
|
||||
port=int(os.getenv("REDIS_PORT", "6379")),
|
||||
decode_responses=True,
|
||||
socket_connect_timeout=0.2,
|
||||
socket_timeout=0.2,
|
||||
retry_on_timeout=False,
|
||||
max_connections=10,
|
||||
)
|
||||
|
||||
|
||||
def serialize_user(user: dict) -> str:
|
||||
created_at = user["created_at"]
|
||||
|
||||
return json.dumps(
|
||||
{
|
||||
"id": str(user["id"]),
|
||||
"username": user["username"],
|
||||
"display_name": user["display_name"],
|
||||
"created_at": created_at.astimezone(
|
||||
timezone.utc
|
||||
).isoformat().replace("+00:00", "Z"),
|
||||
},
|
||||
ensure_ascii=False,
|
||||
separators=(",", ":"),
|
||||
)
|
||||
|
||||
|
||||
def deserialize_user(
|
||||
raw: str,
|
||||
expected_id: UUID,
|
||||
) -> dict:
|
||||
data = json.loads(raw)
|
||||
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError("Invalid cached user")
|
||||
|
||||
expected_fields = {
|
||||
"id",
|
||||
"username",
|
||||
"display_name",
|
||||
"created_at",
|
||||
}
|
||||
|
||||
if set(data) != expected_fields:
|
||||
raise ValueError("Invalid cached user fields")
|
||||
|
||||
if not all(isinstance(value, str) for value in data.values()):
|
||||
raise ValueError("Invalid cached user types")
|
||||
|
||||
user_id = UUID(data["id"])
|
||||
|
||||
if user_id != expected_id:
|
||||
raise ValueError("Cached user ID mismatch")
|
||||
|
||||
created_at = datetime.fromisoformat(
|
||||
data["created_at"].replace("Z", "+00:00")
|
||||
)
|
||||
|
||||
if created_at.tzinfo is None:
|
||||
raise ValueError("Cached timestamp has no timezone")
|
||||
|
||||
return {
|
||||
"id": user_id,
|
||||
"username": data["username"],
|
||||
"display_name": data["display_name"],
|
||||
"created_at": created_at,
|
||||
}
|
||||
|
||||
|
||||
async def get_cached_user(user_id: UUID) -> dict | None:
|
||||
key = f"user:v1:{user_id}"
|
||||
|
||||
try:
|
||||
raw = await redis_client.get(key)
|
||||
|
||||
if raw is not None:
|
||||
return deserialize_user(raw, user_id)
|
||||
|
||||
except RedisError:
|
||||
logger.warning("User cache read unavailable")
|
||||
except (ValueError, TypeError, OverflowError):
|
||||
logger.warning("Invalid user cache entry ignored")
|
||||
|
||||
user = await store.get_user(user_id)
|
||||
|
||||
if user is None:
|
||||
return None
|
||||
|
||||
try:
|
||||
await redis_client.set(
|
||||
key,
|
||||
serialize_user(user),
|
||||
ex=CACHE_TTL_SECONDS,
|
||||
)
|
||||
except RedisError:
|
||||
logger.warning("User cache write unavailable")
|
||||
|
||||
return user
|
||||
|
||||
|
||||
async def close_cache() -> None:
|
||||
await redis_client.aclose()
|
||||
@@ -0,0 +1,208 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from collections import Counter
|
||||
from datetime import datetime, timedelta, timezone
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from uuid import UUID
|
||||
|
||||
import psycopg
|
||||
from psycopg.rows import dict_row
|
||||
from botocore.exceptions import ClientError
|
||||
|
||||
from app.object_locks import object_lock_id
|
||||
from app.storage import bucket_name, get_client
|
||||
|
||||
|
||||
def managed_key(key: str) -> bool:
|
||||
prefix = "originals/"
|
||||
if not key.startswith(prefix):
|
||||
return False
|
||||
|
||||
suffix = key[len(prefix):]
|
||||
try:
|
||||
return str(UUID(suffix)) == suffix
|
||||
except ValueError:
|
||||
return False
|
||||
|
||||
|
||||
def database_connection():
|
||||
return psycopg.connect(
|
||||
host=os.getenv("PGHOST", "postgres"),
|
||||
port=int(os.getenv("PGPORT", "5432")),
|
||||
dbname=os.getenv("PGDATABASE", "messenger"),
|
||||
user=os.getenv("PGUSER", "messenger"),
|
||||
password=os.environ["PGPASSWORD"],
|
||||
connect_timeout=3,
|
||||
options=(
|
||||
"-c statement_timeout=3000 "
|
||||
"-c lock_timeout=2000 "
|
||||
"-c idle_in_transaction_session_timeout=10000"
|
||||
),
|
||||
autocommit=True,
|
||||
row_factory=dict_row,
|
||||
)
|
||||
|
||||
|
||||
def sweep(*, apply: bool, grace_seconds: int) -> dict:
|
||||
cutoff = datetime.now(timezone.utc) - timedelta(
|
||||
seconds=grace_seconds
|
||||
)
|
||||
counts = Counter({
|
||||
"scanned": 0,
|
||||
"kept": 0,
|
||||
"fresh": 0,
|
||||
"busy": 0,
|
||||
"ignored": 0,
|
||||
"gone": 0,
|
||||
"candidates": 0,
|
||||
"deleted": 0,
|
||||
})
|
||||
|
||||
with database_connection() as conn:
|
||||
client = get_client()
|
||||
bucket = bucket_name()
|
||||
pages = client.get_paginator("list_objects_v2")
|
||||
|
||||
for page in pages.paginate(
|
||||
Bucket=bucket,
|
||||
Prefix="originals/",
|
||||
PaginationConfig={"PageSize": 200},
|
||||
):
|
||||
for item in page.get("Contents", []):
|
||||
counts["scanned"] += 1
|
||||
key = item["Key"]
|
||||
|
||||
if not managed_key(key):
|
||||
counts["ignored"] += 1
|
||||
continue
|
||||
|
||||
if item["LastModified"] >= cutoff:
|
||||
counts["fresh"] += 1
|
||||
continue
|
||||
|
||||
with conn.transaction():
|
||||
conn.execute(
|
||||
"SET LOCAL "
|
||||
"idle_in_transaction_session_timeout = '60s'"
|
||||
)
|
||||
|
||||
locked = conn.execute(
|
||||
"SELECT pg_try_advisory_xact_lock(%s) AS locked",
|
||||
(object_lock_id(bucket, key),),
|
||||
).fetchone()["locked"]
|
||||
|
||||
if not locked:
|
||||
counts["busy"] += 1
|
||||
continue
|
||||
|
||||
metadata = conn.execute(
|
||||
"""
|
||||
SELECT 1
|
||||
FROM attachments
|
||||
WHERE object_key = %s
|
||||
""",
|
||||
(key,),
|
||||
).fetchone()
|
||||
|
||||
if metadata is not None:
|
||||
counts["kept"] += 1
|
||||
continue
|
||||
|
||||
try:
|
||||
current = client.head_object(
|
||||
Bucket=bucket,
|
||||
Key=key,
|
||||
)
|
||||
except ClientError as exc:
|
||||
status = exc.response.get(
|
||||
"ResponseMetadata", {}
|
||||
).get("HTTPStatusCode")
|
||||
if status == 404:
|
||||
counts["gone"] += 1
|
||||
continue
|
||||
raise
|
||||
|
||||
if current["LastModified"] >= cutoff:
|
||||
counts["fresh"] += 1
|
||||
continue
|
||||
|
||||
counts["candidates"] += 1
|
||||
print(
|
||||
json.dumps({
|
||||
"action": "delete" if apply else "would_delete",
|
||||
"key": key,
|
||||
}),
|
||||
flush=True,
|
||||
)
|
||||
|
||||
if apply:
|
||||
client.delete_object(
|
||||
Bucket=bucket,
|
||||
Key=key,
|
||||
)
|
||||
counts["deleted"] += 1
|
||||
|
||||
return {
|
||||
"mode": "apply" if apply else "dry-run",
|
||||
"grace_seconds": grace_seconds,
|
||||
**dict(counts),
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--apply", action="store_true")
|
||||
parser.add_argument("--grace-seconds", type=int, default=86400)
|
||||
parser.add_argument("--allow-short-grace", action="store_true")
|
||||
parser.add_argument("--interval-seconds", type=int, default=0)
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.grace_seconds < 60:
|
||||
parser.error("grace must be at least 60 seconds")
|
||||
|
||||
if args.grace_seconds < 3600 and not args.allow_short_grace:
|
||||
parser.error(
|
||||
"short test grace requires --allow-short-grace"
|
||||
)
|
||||
|
||||
if args.interval_seconds != 0 and args.interval_seconds < 60:
|
||||
parser.error("interval must be zero or at least 60 seconds")
|
||||
|
||||
if args.grace_seconds < 3600:
|
||||
print(
|
||||
"WARNING: short grace is for controlled local tests",
|
||||
file=sys.stderr,
|
||||
flush=True,
|
||||
)
|
||||
|
||||
while True:
|
||||
try:
|
||||
result = sweep(
|
||||
apply=args.apply,
|
||||
grace_seconds=args.grace_seconds,
|
||||
)
|
||||
except Exception as exc:
|
||||
print(
|
||||
json.dumps({
|
||||
"status": "failed",
|
||||
"error_type": type(exc).__name__,
|
||||
}),
|
||||
file=sys.stderr,
|
||||
flush=True,
|
||||
)
|
||||
return 1
|
||||
|
||||
print(json.dumps(result, sort_keys=True), flush=True)
|
||||
|
||||
if args.interval_seconds == 0:
|
||||
return 0
|
||||
|
||||
time.sleep(args.interval_seconds)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,198 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from collections import Counter
|
||||
from datetime import datetime, timedelta, timezone
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import time
|
||||
from uuid import UUID
|
||||
|
||||
from botocore.exceptions import ClientError
|
||||
|
||||
from app.job_db import connect_db
|
||||
from app.object_locks import object_lock_id
|
||||
from app.storage import bucket_name, get_client
|
||||
|
||||
|
||||
KEY_PATTERN = re.compile(
|
||||
r"staging/([0-9a-f-]{36})/v([1-9][0-9]*)"
|
||||
)
|
||||
|
||||
|
||||
def managed_key(key):
|
||||
match = KEY_PATTERN.fullmatch(key)
|
||||
if match is None:
|
||||
return False
|
||||
try:
|
||||
return str(UUID(match.group(1))) == match.group(1)
|
||||
except ValueError:
|
||||
return False
|
||||
|
||||
|
||||
def sweep(*, apply, grace_seconds, attachment_id=None):
|
||||
cutoff = datetime.now(timezone.utc) - timedelta(seconds=grace_seconds)
|
||||
counts = Counter({
|
||||
"scanned": 0,
|
||||
"ignored": 0,
|
||||
"fresh": 0,
|
||||
"busy": 0,
|
||||
"active": 0,
|
||||
"gone": 0,
|
||||
"candidates": 0,
|
||||
"deleted": 0,
|
||||
})
|
||||
|
||||
bucket = bucket_name()
|
||||
client = get_client()
|
||||
prefix = (
|
||||
f"staging/{attachment_id}/"
|
||||
if attachment_id is not None
|
||||
else "staging/"
|
||||
)
|
||||
|
||||
with connect_db() as db:
|
||||
for page in client.get_paginator("list_objects_v2").paginate(
|
||||
Bucket=bucket,
|
||||
Prefix=prefix,
|
||||
PaginationConfig={"PageSize": 200},
|
||||
):
|
||||
for item in page.get("Contents", []):
|
||||
counts["scanned"] += 1
|
||||
key = item["Key"]
|
||||
|
||||
if not managed_key(key):
|
||||
counts["ignored"] += 1
|
||||
continue
|
||||
|
||||
if item["LastModified"] >= cutoff:
|
||||
counts["fresh"] += 1
|
||||
continue
|
||||
|
||||
with db.transaction():
|
||||
db.execute(
|
||||
"SET LOCAL idle_in_transaction_session_timeout = '60s'"
|
||||
)
|
||||
locked = db.execute(
|
||||
"SELECT pg_try_advisory_xact_lock(%s) AS locked",
|
||||
(object_lock_id(bucket, key),),
|
||||
).fetchone()["locked"]
|
||||
|
||||
if not locked:
|
||||
counts["busy"] += 1
|
||||
continue
|
||||
|
||||
row = db.execute(
|
||||
"""
|
||||
SELECT id, status, updated_at
|
||||
FROM attachments
|
||||
WHERE staging_key = %s
|
||||
FOR UPDATE
|
||||
""",
|
||||
(key,),
|
||||
).fetchone()
|
||||
|
||||
if row is not None:
|
||||
if row["status"] in ("queued", "processing"):
|
||||
counts["active"] += 1
|
||||
continue
|
||||
|
||||
if row["status"] == "failed" and row["updated_at"] >= cutoff:
|
||||
counts["fresh"] += 1
|
||||
continue
|
||||
|
||||
try:
|
||||
current = client.head_object(Bucket=bucket, Key=key)
|
||||
except ClientError as exc:
|
||||
status = exc.response.get(
|
||||
"ResponseMetadata", {}
|
||||
).get("HTTPStatusCode")
|
||||
if status == 404:
|
||||
counts["gone"] += 1
|
||||
continue
|
||||
raise
|
||||
|
||||
if current["LastModified"] >= cutoff:
|
||||
counts["fresh"] += 1
|
||||
continue
|
||||
|
||||
counts["candidates"] += 1
|
||||
|
||||
if apply:
|
||||
client.delete_object(Bucket=bucket, Key=key)
|
||||
|
||||
if row is not None:
|
||||
db.execute(
|
||||
"""
|
||||
UPDATE attachments
|
||||
SET staging_key = NULL
|
||||
WHERE id = %s
|
||||
AND staging_key = %s
|
||||
AND status IN ('ready', 'failed')
|
||||
""",
|
||||
(row["id"], key),
|
||||
)
|
||||
|
||||
counts["deleted"] += 1
|
||||
|
||||
return {
|
||||
"mode": "apply" if apply else "dry-run",
|
||||
"grace_seconds": grace_seconds,
|
||||
"attachment_id": (
|
||||
str(attachment_id) if attachment_id is not None else None
|
||||
),
|
||||
**dict(counts),
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--apply", action="store_true")
|
||||
parser.add_argument(
|
||||
"--grace-seconds",
|
||||
type=int,
|
||||
default=int(os.getenv("STAGING_RETENTION_SECONDS", "86400")),
|
||||
)
|
||||
parser.add_argument("--attachment-id", type=UUID)
|
||||
parser.add_argument("--allow-short-grace", action="store_true")
|
||||
parser.add_argument("--interval-seconds", type=int, default=0)
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.grace_seconds < 0:
|
||||
parser.error("grace must not be negative")
|
||||
|
||||
if args.grace_seconds < 3600 and (
|
||||
not args.allow_short_grace or args.attachment_id is None
|
||||
):
|
||||
parser.error(
|
||||
"short grace requires --allow-short-grace "
|
||||
"and a specific --attachment-id"
|
||||
)
|
||||
|
||||
if args.interval_seconds != 0 and args.interval_seconds < 60:
|
||||
parser.error("interval must be zero or at least 60 seconds")
|
||||
|
||||
while True:
|
||||
try:
|
||||
result = sweep(
|
||||
apply=args.apply,
|
||||
grace_seconds=args.grace_seconds,
|
||||
attachment_id=args.attachment_id,
|
||||
)
|
||||
except Exception as exc:
|
||||
print(json.dumps({
|
||||
"result": "error",
|
||||
"error_type": type(exc).__name__,
|
||||
}), flush=True)
|
||||
raise SystemExit(1)
|
||||
|
||||
print(json.dumps(result), flush=True)
|
||||
|
||||
if args.interval_seconds == 0:
|
||||
return
|
||||
time.sleep(args.interval_seconds)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,12 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
|
||||
|
||||
SESSION_TTL_SECONDS = int(
|
||||
os.getenv("SESSION_TTL_SECONDS", "3600")
|
||||
)
|
||||
|
||||
|
||||
if SESSION_TTL_SECONDS < 1:
|
||||
raise ValueError("SESSION_TTL_SECONDS must be positive")
|
||||
@@ -0,0 +1,76 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
from psycopg import AsyncConnection, OperationalError, waiting
|
||||
from psycopg.rows import dict_row
|
||||
from psycopg_pool import AsyncConnectionPool
|
||||
|
||||
|
||||
DB_IO_TIMEOUT_SECONDS = float(
|
||||
os.getenv("DB_IO_TIMEOUT_SECONDS", "3")
|
||||
)
|
||||
|
||||
if DB_IO_TIMEOUT_SECONDS <= 0:
|
||||
raise ValueError("DB_IO_TIMEOUT_SECONDS must be positive")
|
||||
|
||||
|
||||
class DeadlineConnection(AsyncConnection):
|
||||
async def wait(self, gen, interval=0.1):
|
||||
try:
|
||||
async with asyncio.timeout(DB_IO_TIMEOUT_SECONDS):
|
||||
return await waiting.wait_async(
|
||||
gen,
|
||||
self.pgconn.socket,
|
||||
interval=interval,
|
||||
)
|
||||
|
||||
except TimeoutError:
|
||||
await self.close()
|
||||
raise OperationalError(
|
||||
"PostgreSQL response deadline exceeded"
|
||||
) from None
|
||||
|
||||
except asyncio.CancelledError:
|
||||
await self.close()
|
||||
raise
|
||||
|
||||
|
||||
pool = AsyncConnectionPool(
|
||||
conninfo="",
|
||||
connection_class=DeadlineConnection,
|
||||
kwargs={
|
||||
"host": os.getenv("PGHOST", "postgres"),
|
||||
"port": int(os.getenv("PGPORT", "5432")),
|
||||
"dbname": os.getenv("PGDATABASE", "messenger"),
|
||||
"user": os.getenv("PGUSER", "messenger"),
|
||||
"password": os.environ.get("PGPASSWORD", ""),
|
||||
"connect_timeout": 3,
|
||||
"options": (
|
||||
"-c statement_timeout=3000 "
|
||||
"-c lock_timeout=2000 "
|
||||
"-c idle_in_transaction_session_timeout=10000"
|
||||
),
|
||||
"row_factory": dict_row,
|
||||
},
|
||||
min_size=1,
|
||||
max_size=5,
|
||||
timeout=3,
|
||||
max_waiting=20,
|
||||
open=False,
|
||||
check=AsyncConnectionPool.check_connection,
|
||||
)
|
||||
|
||||
|
||||
async def open_db() -> None:
|
||||
await pool.open(wait=True, timeout=10)
|
||||
|
||||
|
||||
async def close_db() -> None:
|
||||
await pool.close()
|
||||
|
||||
|
||||
async def check_db() -> None:
|
||||
async with pool.connection() as conn:
|
||||
await conn.execute("SELECT 1")
|
||||
@@ -0,0 +1,48 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from uuid import UUID
|
||||
|
||||
from fastapi import Request
|
||||
|
||||
from app.auth import hash_token, parse_bearer_token
|
||||
from app.cache import get_cached_user
|
||||
from app.errors import not_found, unauthorized
|
||||
from app.store import store
|
||||
from app.validation import parse_uuid
|
||||
|
||||
|
||||
async def current_user(request: Request) -> tuple[dict, str]:
|
||||
token = parse_bearer_token(
|
||||
request.headers.get("authorization")
|
||||
)
|
||||
|
||||
if token is None:
|
||||
raise unauthorized()
|
||||
|
||||
token_digest = hash_token(token)
|
||||
|
||||
user_id = await store.get_session_user_id(token_digest)
|
||||
|
||||
if user_id is None:
|
||||
raise unauthorized()
|
||||
|
||||
user = await get_cached_user(user_id)
|
||||
|
||||
if user is None:
|
||||
raise unauthorized()
|
||||
|
||||
return user, token_digest
|
||||
|
||||
|
||||
async def accessible_chat(
|
||||
raw_chat_id: str,
|
||||
user_id: UUID,
|
||||
) -> tuple[UUID, dict]:
|
||||
chat_id = parse_uuid(raw_chat_id)
|
||||
|
||||
chat = await store.get_accessible_chat(chat_id, user_id)
|
||||
|
||||
if chat is None:
|
||||
raise not_found()
|
||||
|
||||
return chat_id, chat
|
||||
@@ -0,0 +1,71 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import Request
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
|
||||
class ApiError(Exception):
|
||||
def __init__(
|
||||
self,
|
||||
status: int,
|
||||
code: str,
|
||||
message: str,
|
||||
challenge: str | None = None,
|
||||
):
|
||||
super().__init__(message)
|
||||
self.status = status
|
||||
self.code = code
|
||||
self.message = message
|
||||
self.challenge = challenge
|
||||
|
||||
|
||||
def invalid(message: str = "Некорректный запрос") -> ApiError:
|
||||
return ApiError(400, "invalid_request", message)
|
||||
|
||||
|
||||
def unauthorized(scheme: str = "Bearer") -> ApiError:
|
||||
if scheme == "Basic":
|
||||
challenge = 'Basic realm="messenger", charset="UTF-8"'
|
||||
else:
|
||||
challenge = 'Bearer realm="messenger"'
|
||||
|
||||
return ApiError(
|
||||
401,
|
||||
"unauthorized",
|
||||
"Недействительная аутентификация",
|
||||
challenge,
|
||||
)
|
||||
|
||||
|
||||
def not_found() -> ApiError:
|
||||
return ApiError(404, "not_found", "Объект не найден")
|
||||
|
||||
|
||||
def error_response(
|
||||
request: Request,
|
||||
error: ApiError,
|
||||
) -> JSONResponse:
|
||||
request_id = getattr(
|
||||
request.state,
|
||||
"request_id",
|
||||
str(uuid4()),
|
||||
)
|
||||
|
||||
headers = {"X-Request-Id": request_id}
|
||||
|
||||
if error.challenge is not None:
|
||||
headers["WWW-Authenticate"] = error.challenge
|
||||
|
||||
return JSONResponse(
|
||||
status_code=error.status,
|
||||
content={
|
||||
"error": {
|
||||
"code": error.code,
|
||||
"message": error.message,
|
||||
"request_id": request_id,
|
||||
}
|
||||
},
|
||||
headers=headers,
|
||||
)
|
||||
@@ -0,0 +1,34 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
|
||||
from app.job_db import connect_db
|
||||
|
||||
|
||||
logger = logging.getLogger("messenger.faults")
|
||||
|
||||
|
||||
def trip_fault(job_id, stage: str) -> None:
|
||||
with connect_db() as db:
|
||||
row = db.execute(
|
||||
"""
|
||||
UPDATE pipeline_fault_injections
|
||||
SET consumed_at = clock_timestamp()
|
||||
WHERE job_id = %s
|
||||
AND stage = %s
|
||||
AND consumed_at IS NULL
|
||||
RETURNING job_id
|
||||
""",
|
||||
(job_id, stage),
|
||||
).fetchone()
|
||||
|
||||
if row is None:
|
||||
return
|
||||
|
||||
logger.warning(
|
||||
"fault_injection job_id=%s stage=%s exit_code=75",
|
||||
job_id,
|
||||
stage,
|
||||
)
|
||||
os._exit(75)
|
||||
@@ -0,0 +1,180 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from app.db import pool
|
||||
from app.errors import ApiError, invalid, not_found
|
||||
|
||||
|
||||
async def create_message_once(
|
||||
chat_id: UUID,
|
||||
sender_id: UUID,
|
||||
text: str,
|
||||
key: str,
|
||||
attachment_ids: list[UUID] | None = None,
|
||||
) -> dict:
|
||||
attachment_ids = list(
|
||||
attachment_ids if attachment_ids is not None else []
|
||||
)
|
||||
|
||||
if (
|
||||
len(attachment_ids) > 10
|
||||
or len(set(attachment_ids)) != len(attachment_ids)
|
||||
):
|
||||
raise invalid("Некорректный список вложений")
|
||||
|
||||
if not isinstance(text, str) or len(text) > 4000:
|
||||
raise invalid("Некорректный текст сообщения")
|
||||
|
||||
if not text.strip() and not attachment_ids:
|
||||
raise invalid("Сообщение должно содержать текст или вложение")
|
||||
|
||||
scope = json.dumps(
|
||||
[
|
||||
str(sender_id),
|
||||
"POST",
|
||||
f"/api/v1/chats/{chat_id}/messages",
|
||||
key,
|
||||
],
|
||||
separators=(",", ":"),
|
||||
)
|
||||
|
||||
lock_id = int.from_bytes(
|
||||
hashlib.sha256(scope.encode("utf-8")).digest()[:8],
|
||||
byteorder="big",
|
||||
signed=True,
|
||||
)
|
||||
|
||||
async with pool.connection() as conn:
|
||||
await conn.execute(
|
||||
"SELECT pg_advisory_xact_lock(%s)",
|
||||
(lock_id,),
|
||||
)
|
||||
|
||||
cursor = await conn.execute(
|
||||
"""
|
||||
SELECT i.request_text,
|
||||
i.request_attachment_ids,
|
||||
m.id, m.chat_id, m.sender_id,
|
||||
m.text, m.created_at,
|
||||
ARRAY(
|
||||
SELECT ma.attachment_id
|
||||
FROM message_attachments AS ma
|
||||
WHERE ma.message_id = m.id
|
||||
ORDER BY ma.position
|
||||
) AS attachment_ids
|
||||
FROM message_idempotency AS i
|
||||
JOIN messages AS m ON m.id = i.message_id
|
||||
WHERE i.user_id = %s
|
||||
AND i.chat_id = %s
|
||||
AND i.http_method = 'POST'
|
||||
AND i.idempotency_key = %s
|
||||
""",
|
||||
(sender_id, chat_id, key),
|
||||
)
|
||||
previous = await cursor.fetchone()
|
||||
|
||||
if previous is not None:
|
||||
if (
|
||||
previous["request_text"] != text
|
||||
or previous["request_attachment_ids"] != attachment_ids
|
||||
):
|
||||
raise ApiError(
|
||||
409,
|
||||
"conflict",
|
||||
"Ключ уже использован с другим сообщением",
|
||||
)
|
||||
|
||||
message = {
|
||||
field: previous[field]
|
||||
for field in (
|
||||
"id",
|
||||
"chat_id",
|
||||
"sender_id",
|
||||
"text",
|
||||
"created_at",
|
||||
"attachment_ids",
|
||||
)
|
||||
}
|
||||
else:
|
||||
if attachment_ids:
|
||||
cursor = await conn.execute(
|
||||
"""
|
||||
SELECT id, status
|
||||
FROM attachments
|
||||
WHERE id = ANY(%s::uuid[])
|
||||
AND chat_id = %s
|
||||
ORDER BY id
|
||||
FOR SHARE
|
||||
""",
|
||||
(attachment_ids, chat_id),
|
||||
)
|
||||
available = await cursor.fetchall()
|
||||
|
||||
if len(available) != len(attachment_ids):
|
||||
raise not_found()
|
||||
|
||||
if any(row["status"] == "failed" for row in available):
|
||||
raise ApiError(
|
||||
409,
|
||||
"conflict",
|
||||
"Нельзя прикрепить вложение с ошибкой обработки",
|
||||
)
|
||||
|
||||
cursor = await conn.execute(
|
||||
"""
|
||||
INSERT INTO messages (
|
||||
id, chat_id, sender_id, text
|
||||
)
|
||||
VALUES (%s, %s, %s, %s)
|
||||
RETURNING id, chat_id, sender_id,
|
||||
text, created_at
|
||||
""",
|
||||
(uuid4(), chat_id, sender_id, text),
|
||||
)
|
||||
message = await cursor.fetchone()
|
||||
|
||||
if attachment_ids:
|
||||
async with conn.cursor() as cursor:
|
||||
await cursor.executemany(
|
||||
"""
|
||||
INSERT INTO message_attachments (
|
||||
message_id, attachment_id, position
|
||||
)
|
||||
VALUES (%s, %s, %s)
|
||||
""",
|
||||
[
|
||||
(message["id"], attachment_id, position)
|
||||
for position, attachment_id
|
||||
in enumerate(attachment_ids)
|
||||
],
|
||||
)
|
||||
|
||||
await conn.execute(
|
||||
"""
|
||||
INSERT INTO message_idempotency (
|
||||
user_id,
|
||||
chat_id,
|
||||
http_method,
|
||||
idempotency_key,
|
||||
request_text,
|
||||
request_attachment_ids,
|
||||
message_id
|
||||
)
|
||||
VALUES (%s, %s, 'POST', %s, %s, %s::uuid[], %s)
|
||||
""",
|
||||
(
|
||||
sender_id,
|
||||
chat_id,
|
||||
key,
|
||||
text,
|
||||
attachment_ids,
|
||||
message["id"],
|
||||
),
|
||||
)
|
||||
|
||||
message["attachment_ids"] = attachment_ids
|
||||
|
||||
return message
|
||||
@@ -0,0 +1,133 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import io
|
||||
import warnings
|
||||
|
||||
from PIL import Image, ImageOps, UnidentifiedImageError
|
||||
|
||||
|
||||
MAX_BYTES = 10_485_760
|
||||
MAX_PIXELS = 40_000_000
|
||||
JPEG_QUALITY = 85
|
||||
WEBP_QUALITY = 82
|
||||
|
||||
|
||||
class InvalidImage(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
def fitted_size(width: int, height: int, limit: int) -> tuple[int, int]:
|
||||
longest = max(width, height)
|
||||
if longest <= limit:
|
||||
return width, height
|
||||
return (
|
||||
max(1, width * limit // longest),
|
||||
max(1, height * limit // longest),
|
||||
)
|
||||
|
||||
|
||||
def encode_variant(name: str, image: Image.Image, fmt: str) -> dict:
|
||||
image.info.clear()
|
||||
output = io.BytesIO()
|
||||
|
||||
if fmt == "JPEG":
|
||||
image.save(
|
||||
output,
|
||||
format="JPEG",
|
||||
quality=JPEG_QUALITY,
|
||||
optimize=True,
|
||||
exif=b"",
|
||||
icc_profile=None,
|
||||
)
|
||||
content_type = "image/jpeg"
|
||||
else:
|
||||
image.save(
|
||||
output,
|
||||
format="WEBP",
|
||||
quality=WEBP_QUALITY,
|
||||
method=4,
|
||||
exif=b"",
|
||||
icc_profile=b"",
|
||||
xmp=b"",
|
||||
)
|
||||
content_type = "image/webp"
|
||||
|
||||
data = output.getvalue()
|
||||
return {
|
||||
"name": name,
|
||||
"content_type": content_type,
|
||||
"size_bytes": len(data),
|
||||
"width": image.width,
|
||||
"height": image.height,
|
||||
"sha256": hashlib.sha256(data).hexdigest(),
|
||||
"data": data,
|
||||
}
|
||||
|
||||
|
||||
def render_variants(data: bytes) -> list[dict]:
|
||||
if not 1 <= len(data) <= MAX_BYTES:
|
||||
raise InvalidImage("invalid_size")
|
||||
|
||||
try:
|
||||
with warnings.catch_warnings():
|
||||
warnings.simplefilter("error", Image.DecompressionBombWarning)
|
||||
|
||||
with Image.open(io.BytesIO(data)) as source:
|
||||
if source.format not in ("PNG", "JPEG"):
|
||||
raise InvalidImage("invalid_format")
|
||||
width, height = source.size
|
||||
if width < 1 or height < 1 or width * height > MAX_PIXELS:
|
||||
raise InvalidImage("invalid_dimensions")
|
||||
source.verify()
|
||||
|
||||
with Image.open(io.BytesIO(data)) as source:
|
||||
oriented = ImageOps.exif_transpose(source)
|
||||
try:
|
||||
rgba = oriented.convert("RGBA")
|
||||
try:
|
||||
rgb = Image.new("RGB", rgba.size, "white")
|
||||
alpha = rgba.getchannel("A")
|
||||
try:
|
||||
rgb.paste(rgba, mask=alpha)
|
||||
finally:
|
||||
alpha.close()
|
||||
finally:
|
||||
rgba.close()
|
||||
finally:
|
||||
oriented.close()
|
||||
|
||||
try:
|
||||
variants = []
|
||||
|
||||
size = fitted_size(rgb.width, rgb.height, 256)
|
||||
with rgb.resize(size, Image.Resampling.LANCZOS) as thumb:
|
||||
variants.append(encode_variant("thumbnail", thumb, "JPEG"))
|
||||
|
||||
with ImageOps.fit(
|
||||
rgb,
|
||||
(128, 128),
|
||||
method=Image.Resampling.LANCZOS,
|
||||
centering=(0.5, 0.5),
|
||||
) as crop:
|
||||
variants.append(encode_variant("crop", crop, "JPEG"))
|
||||
|
||||
size = fitted_size(rgb.width, rgb.height, 1280)
|
||||
with rgb.resize(size, Image.Resampling.LANCZOS) as optimized:
|
||||
variants.append(encode_variant("optimized", optimized, "WEBP"))
|
||||
|
||||
return variants
|
||||
finally:
|
||||
rgb.close()
|
||||
|
||||
except (
|
||||
UnidentifiedImageError,
|
||||
Image.DecompressionBombError,
|
||||
Image.DecompressionBombWarning,
|
||||
OSError,
|
||||
SyntaxError,
|
||||
ValueError,
|
||||
) as exc:
|
||||
if isinstance(exc, InvalidImage):
|
||||
raise
|
||||
raise InvalidImage("invalid_image") from None
|
||||
@@ -0,0 +1,95 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import unicodedata
|
||||
|
||||
from PIL import Image, UnidentifiedImageError
|
||||
|
||||
from app.errors import ApiError, invalid
|
||||
|
||||
|
||||
MAX_BYTES = 10 * 1024 * 1024
|
||||
MAX_PIXELS = 40_000_000
|
||||
|
||||
FORMAT_TO_MIME = {
|
||||
"PNG": "image/png",
|
||||
"JPEG": "image/jpeg",
|
||||
}
|
||||
|
||||
|
||||
def validate_filename(value: str | None) -> str:
|
||||
if (
|
||||
value is None
|
||||
or not 1 <= len(value) <= 255
|
||||
or not value.strip()
|
||||
):
|
||||
raise invalid("Некорректное имя файла")
|
||||
|
||||
for char in value:
|
||||
if char in "/\\" or unicodedata.category(char) == "Cc":
|
||||
raise invalid("Некорректное имя файла")
|
||||
|
||||
return value
|
||||
|
||||
|
||||
def inspect_image(data: bytes, declared_type: str) -> str:
|
||||
if len(data) > MAX_BYTES:
|
||||
raise ApiError(
|
||||
413,
|
||||
"payload_too_large",
|
||||
"Файл превышает 10 MiB",
|
||||
)
|
||||
|
||||
if declared_type not in FORMAT_TO_MIME.values():
|
||||
raise ApiError(
|
||||
415,
|
||||
"unsupported_media_type",
|
||||
"Допустимы только image/png и image/jpeg",
|
||||
)
|
||||
|
||||
try:
|
||||
with Image.open(io.BytesIO(data)) as image:
|
||||
detected = FORMAT_TO_MIME.get(image.format or "")
|
||||
width, height = image.size
|
||||
pixels = width * height
|
||||
except Image.DecompressionBombError:
|
||||
raise ApiError(
|
||||
413,
|
||||
"payload_too_large",
|
||||
"Слишком много пикселей",
|
||||
) from None
|
||||
except (UnidentifiedImageError, OSError, SyntaxError, ValueError):
|
||||
raise ApiError(
|
||||
415,
|
||||
"unsupported_media_type",
|
||||
"Файл не является корректным PNG/JPEG",
|
||||
) from None
|
||||
|
||||
if detected is None or detected != declared_type:
|
||||
raise ApiError(
|
||||
415,
|
||||
"unsupported_media_type",
|
||||
"Содержимое не соответствует Content-Type",
|
||||
)
|
||||
|
||||
if width < 1 or height < 1 or pixels > MAX_PIXELS:
|
||||
raise ApiError(
|
||||
413,
|
||||
"payload_too_large",
|
||||
"Слишком много пикселей",
|
||||
)
|
||||
|
||||
try:
|
||||
with Image.open(io.BytesIO(data)) as image:
|
||||
image.verify()
|
||||
|
||||
with Image.open(io.BytesIO(data)) as image:
|
||||
image.load()
|
||||
except (Image.DecompressionBombError, OSError, SyntaxError, ValueError):
|
||||
raise ApiError(
|
||||
415,
|
||||
"unsupported_media_type",
|
||||
"Файл повреждён",
|
||||
) from None
|
||||
|
||||
return detected
|
||||
@@ -0,0 +1,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import socket
|
||||
from uuid import uuid4
|
||||
|
||||
|
||||
INSTANCE_ID = f"{socket.gethostname()}:{uuid4().hex[:12]}"
|
||||
@@ -0,0 +1,24 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
|
||||
import psycopg
|
||||
from psycopg.rows import dict_row
|
||||
|
||||
|
||||
def connect_db():
|
||||
return psycopg.connect(
|
||||
host=os.getenv("PGHOST", "postgres"),
|
||||
port=int(os.getenv("PGPORT", "5432")),
|
||||
dbname=os.getenv("PGDATABASE", "messenger"),
|
||||
user=os.getenv("PGUSER", "messenger"),
|
||||
password=os.environ["PGPASSWORD"],
|
||||
connect_timeout=3,
|
||||
autocommit=True,
|
||||
row_factory=dict_row,
|
||||
options=(
|
||||
"-c statement_timeout=3000 "
|
||||
"-c lock_timeout=2000 "
|
||||
"-c idle_in_transaction_session_timeout=60000"
|
||||
),
|
||||
)
|
||||
@@ -0,0 +1,194 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import resource
|
||||
import sys
|
||||
from uuid import UUID
|
||||
|
||||
|
||||
def execute(job_id: UUID, lease_token: UUID) -> str:
|
||||
from app.image_pipeline import InvalidImage, render_variants
|
||||
from app.faults import trip_fault
|
||||
from app.job_db import connect_db
|
||||
from app.storage import _get, _put, bucket_name, get_client
|
||||
|
||||
with connect_db() as db:
|
||||
row = db.execute(
|
||||
"""
|
||||
SELECT j.attachment_id, j.pipeline_version,
|
||||
a.staging_key, a.object_key,
|
||||
a.content_type, a.size_bytes, a.sha256
|
||||
FROM attachment_jobs AS j
|
||||
JOIN attachments AS a ON a.id = j.attachment_id
|
||||
WHERE j.id = %s
|
||||
AND j.status = 'processing'
|
||||
AND j.lease_token = %s
|
||||
AND j.lease_until > clock_timestamp()
|
||||
AND a.pipeline_version = j.pipeline_version
|
||||
AND a.status = 'processing'
|
||||
""",
|
||||
(job_id, lease_token),
|
||||
).fetchone()
|
||||
|
||||
if row is None:
|
||||
return "stale"
|
||||
|
||||
data = _get(row["staging_key"])
|
||||
if (
|
||||
len(data) != row["size_bytes"]
|
||||
or hashlib.sha256(data).hexdigest() != row["sha256"]
|
||||
):
|
||||
raise InvalidImage("original_integrity_error")
|
||||
|
||||
variants = render_variants(data)
|
||||
|
||||
_put(row["object_key"], data, row["content_type"])
|
||||
|
||||
for variant in variants:
|
||||
variant["object_key"] = (
|
||||
f"variants/{row['attachment_id']}/"
|
||||
f"v{row['pipeline_version']}/{variant['name']}"
|
||||
)
|
||||
_put(
|
||||
variant["object_key"],
|
||||
variant["data"],
|
||||
variant["content_type"],
|
||||
)
|
||||
|
||||
trip_fault(job_id, "after_objects")
|
||||
|
||||
with connect_db() as db:
|
||||
with db.transaction():
|
||||
job = db.execute(
|
||||
"""
|
||||
SELECT id
|
||||
FROM attachment_jobs
|
||||
WHERE id = %s
|
||||
AND status = 'processing'
|
||||
AND lease_token = %s
|
||||
AND lease_until > clock_timestamp()
|
||||
FOR UPDATE
|
||||
""",
|
||||
(job_id, lease_token),
|
||||
).fetchone()
|
||||
|
||||
if job is None:
|
||||
return "stale"
|
||||
|
||||
attachment = db.execute(
|
||||
"""
|
||||
SELECT id
|
||||
FROM attachments
|
||||
WHERE id = %s
|
||||
AND pipeline_version = %s
|
||||
AND status = 'processing'
|
||||
FOR UPDATE
|
||||
""",
|
||||
(row["attachment_id"], row["pipeline_version"]),
|
||||
).fetchone()
|
||||
|
||||
if attachment is None:
|
||||
return "stale"
|
||||
|
||||
for variant in variants:
|
||||
db.execute(
|
||||
"""
|
||||
INSERT INTO attachment_variants (
|
||||
attachment_id, pipeline_version,
|
||||
name, object_key, content_type,
|
||||
size_bytes, width, height, sha256
|
||||
)
|
||||
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)
|
||||
ON CONFLICT (attachment_id, pipeline_version, name)
|
||||
DO UPDATE SET
|
||||
object_key = EXCLUDED.object_key,
|
||||
content_type = EXCLUDED.content_type,
|
||||
size_bytes = EXCLUDED.size_bytes,
|
||||
width = EXCLUDED.width,
|
||||
height = EXCLUDED.height,
|
||||
sha256 = EXCLUDED.sha256
|
||||
""",
|
||||
(
|
||||
row["attachment_id"], row["pipeline_version"],
|
||||
variant["name"], variant["object_key"],
|
||||
variant["content_type"], variant["size_bytes"],
|
||||
variant["width"], variant["height"],
|
||||
variant["sha256"],
|
||||
),
|
||||
)
|
||||
|
||||
db.execute(
|
||||
"""
|
||||
UPDATE attachments
|
||||
SET status = 'ready',
|
||||
error_code = NULL,
|
||||
updated_at = clock_timestamp()
|
||||
WHERE id = %s
|
||||
""",
|
||||
(row["attachment_id"],),
|
||||
)
|
||||
|
||||
db.execute(
|
||||
"""
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'ready',
|
||||
lease_token = NULL,
|
||||
lease_until = NULL,
|
||||
error_code = NULL,
|
||||
updated_at = clock_timestamp()
|
||||
WHERE id = %s
|
||||
""",
|
||||
(job_id,),
|
||||
)
|
||||
|
||||
# Результат уже зафиксирован. Ошибка очистки не отменяет ready.
|
||||
try:
|
||||
get_client().delete_object(
|
||||
Bucket=bucket_name(),
|
||||
Key=row["staging_key"],
|
||||
)
|
||||
with connect_db() as db:
|
||||
db.execute(
|
||||
"""
|
||||
UPDATE attachments
|
||||
SET staging_key = NULL
|
||||
WHERE id = %s
|
||||
AND staging_key = %s
|
||||
AND status = 'ready'
|
||||
""",
|
||||
(row["attachment_id"], row["staging_key"]),
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return "ready"
|
||||
|
||||
|
||||
def main():
|
||||
memory_bytes = int(os.getenv("JOB_MEMORY_MB", "512")) * 1024 * 1024
|
||||
cpu_seconds = int(os.getenv("JOB_TIMEOUT_SECONDS", "60"))
|
||||
resource.setrlimit(resource.RLIMIT_AS, (memory_bytes, memory_bytes))
|
||||
resource.setrlimit(
|
||||
resource.RLIMIT_CPU,
|
||||
(cpu_seconds, cpu_seconds + 1),
|
||||
)
|
||||
|
||||
from app.image_pipeline import InvalidImage
|
||||
|
||||
try:
|
||||
outcome = execute(UUID(sys.argv[1]), UUID(sys.argv[2]))
|
||||
result = {"outcome": outcome}
|
||||
except InvalidImage:
|
||||
result = {"outcome": "failed", "error_code": "invalid_image"}
|
||||
except MemoryError:
|
||||
result = {"outcome": "failed", "error_code": "resource_limit"}
|
||||
except Exception:
|
||||
result = {"outcome": "retry", "error_code": "processing_failed"}
|
||||
|
||||
print(json.dumps(result), flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+90
@@ -0,0 +1,90 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import os
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from app.db import pool
|
||||
from app.object_locks import object_lock_id
|
||||
from app.storage import bucket_name, put_object
|
||||
|
||||
|
||||
PIPELINE_VERSION = 1
|
||||
JOB_MAX_ATTEMPTS = int(os.getenv("JOB_MAX_ATTEMPTS", "5"))
|
||||
|
||||
if not 1 <= JOB_MAX_ATTEMPTS <= 100:
|
||||
raise ValueError("JOB_MAX_ATTEMPTS must be between 1 and 100")
|
||||
|
||||
|
||||
async def accept_upload(
|
||||
*,
|
||||
chat_id: UUID,
|
||||
owner_id: UUID,
|
||||
filename: str,
|
||||
content_type: str,
|
||||
data: bytes,
|
||||
) -> dict:
|
||||
attachment_id = uuid4()
|
||||
job_id = uuid4()
|
||||
staging_key = f"staging/{attachment_id}/v{PIPELINE_VERSION}"
|
||||
original_key = f"originals/{attachment_id}"
|
||||
digest = hashlib.sha256(data).hexdigest()
|
||||
|
||||
async with pool.connection() as conn:
|
||||
await conn.execute(
|
||||
"SET LOCAL idle_in_transaction_session_timeout = '60s'"
|
||||
)
|
||||
await conn.execute(
|
||||
"SELECT pg_advisory_xact_lock(%s)",
|
||||
(object_lock_id(bucket_name(), staging_key),),
|
||||
)
|
||||
|
||||
await put_object(staging_key, data, content_type)
|
||||
|
||||
cursor = await conn.execute(
|
||||
"""
|
||||
INSERT INTO attachments (
|
||||
id, chat_id, owner_id, filename,
|
||||
content_type, size_bytes, sha256,
|
||||
object_key, staging_key, pipeline_version, status
|
||||
)
|
||||
VALUES (
|
||||
%s, %s, %s, %s,
|
||||
%s, %s, %s,
|
||||
%s, %s, %s, 'queued'
|
||||
)
|
||||
RETURNING id, chat_id, owner_id, filename,
|
||||
content_type, size_bytes, sha256,
|
||||
status, error_code, created_at, updated_at
|
||||
""",
|
||||
(
|
||||
attachment_id, chat_id, owner_id, filename,
|
||||
content_type, len(data), digest,
|
||||
original_key, staging_key, PIPELINE_VERSION,
|
||||
),
|
||||
)
|
||||
row = await cursor.fetchone()
|
||||
|
||||
await conn.execute(
|
||||
"""
|
||||
INSERT INTO attachment_jobs (
|
||||
id, attachment_id, pipeline_version, max_attempts
|
||||
)
|
||||
VALUES (%s, %s, %s, %s)
|
||||
""",
|
||||
(
|
||||
job_id, attachment_id,
|
||||
PIPELINE_VERSION, JOB_MAX_ATTEMPTS,
|
||||
),
|
||||
)
|
||||
|
||||
await conn.execute(
|
||||
"""
|
||||
INSERT INTO attachment_outbox (id, job_id, attempt)
|
||||
VALUES (%s, %s, 1)
|
||||
""",
|
||||
(uuid4(), job_id),
|
||||
)
|
||||
|
||||
row["variants"] = []
|
||||
return row
|
||||
+116
@@ -0,0 +1,116 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from contextlib import asynccontextmanager
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import FastAPI, Request
|
||||
from fastapi.exceptions import RequestValidationError
|
||||
from psycopg import InterfaceError, OperationalError
|
||||
from psycopg.errors import LockNotAvailable, QueryCanceled
|
||||
from psycopg_pool import PoolTimeout, TooManyRequests
|
||||
from starlette.exceptions import HTTPException
|
||||
|
||||
from app.cache import close_cache
|
||||
from app.instance import INSTANCE_ID
|
||||
from app.db import close_db, open_db
|
||||
from app.errors import ApiError, error_response, invalid, not_found
|
||||
from app.routers import attachments, chats, health, messages, sessions, users
|
||||
|
||||
|
||||
logger = logging.getLogger("messenger.api")
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
try:
|
||||
await open_db()
|
||||
yield
|
||||
finally:
|
||||
await close_cache()
|
||||
await close_db()
|
||||
|
||||
|
||||
app = FastAPI(
|
||||
title="Replicated PostgreSQL messenger — Lab 3",
|
||||
docs_url=None,
|
||||
redoc_url=None,
|
||||
openapi_url=None,
|
||||
lifespan=lifespan,
|
||||
)
|
||||
|
||||
|
||||
@app.middleware("http")
|
||||
async def request_middleware(request: Request, call_next):
|
||||
request.state.request_id = str(uuid4())
|
||||
|
||||
try:
|
||||
response = await call_next(request)
|
||||
|
||||
except ApiError as exc:
|
||||
response = error_response(request, exc)
|
||||
|
||||
except (
|
||||
OperationalError,
|
||||
InterfaceError,
|
||||
PoolTimeout,
|
||||
TooManyRequests,
|
||||
QueryCanceled,
|
||||
LockNotAvailable,
|
||||
) as exc:
|
||||
logger.warning(
|
||||
"Database unavailable request_id=%s error_type=%s",
|
||||
request.state.request_id,
|
||||
type(exc).__name__,
|
||||
)
|
||||
response = error_response(
|
||||
request,
|
||||
ApiError(
|
||||
503,
|
||||
"unavailable",
|
||||
"Сервис временно недоступен",
|
||||
),
|
||||
)
|
||||
|
||||
except Exception as exc:
|
||||
logger.error(
|
||||
"Unhandled error request_id=%s error_type=%s",
|
||||
request.state.request_id,
|
||||
type(exc).__name__,
|
||||
)
|
||||
response = error_response(
|
||||
request,
|
||||
ApiError(
|
||||
500,
|
||||
"internal_error",
|
||||
"Внутренняя ошибка сервера",
|
||||
),
|
||||
)
|
||||
|
||||
response.headers["X-Request-Id"] = request.state.request_id
|
||||
response.headers["X-Instance-Id"] = INSTANCE_ID
|
||||
return response
|
||||
|
||||
|
||||
@app.exception_handler(RequestValidationError)
|
||||
async def validation_error_handler(request: Request, exc):
|
||||
return error_response(request, invalid())
|
||||
|
||||
|
||||
@app.exception_handler(HTTPException)
|
||||
async def http_error_handler(request: Request, exc: HTTPException):
|
||||
if exc.status_code == 404:
|
||||
error = not_found()
|
||||
else:
|
||||
error = invalid("Недопустимый HTTP-запрос")
|
||||
|
||||
return error_response(request, error)
|
||||
|
||||
|
||||
app.include_router(health.router)
|
||||
app.include_router(users.router)
|
||||
app.include_router(sessions.router)
|
||||
app.include_router(chats.router)
|
||||
app.include_router(messages.router)
|
||||
|
||||
app.include_router(attachments.router)
|
||||
@@ -0,0 +1,84 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import psycopg
|
||||
|
||||
|
||||
MIGRATIONS_DIR = Path(__file__).resolve().parent.parent / "migrations"
|
||||
MIGRATION_LOCK_ID = 202602001
|
||||
|
||||
|
||||
def apply_migrations() -> None:
|
||||
files = sorted(MIGRATIONS_DIR.glob("[0-9]*_*.sql"))
|
||||
|
||||
if not files:
|
||||
raise RuntimeError("Migration files not found")
|
||||
|
||||
with psycopg.connect(
|
||||
host=os.getenv("PGHOST", "postgres"),
|
||||
port=int(os.getenv("PGPORT", "5432")),
|
||||
dbname=os.getenv("PGDATABASE", "messenger"),
|
||||
user=os.getenv("PGUSER", "messenger"),
|
||||
password=os.environ["PGPASSWORD"],
|
||||
connect_timeout=5,
|
||||
options="-c statement_timeout=30000 -c lock_timeout=10000",
|
||||
) as conn:
|
||||
conn.execute(
|
||||
"SELECT pg_advisory_xact_lock(%s)",
|
||||
(MIGRATION_LOCK_ID,),
|
||||
)
|
||||
|
||||
conn.execute(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS schema_migrations (
|
||||
version TEXT PRIMARY KEY,
|
||||
checksum TEXT NOT NULL,
|
||||
applied_at TIMESTAMPTZ NOT NULL
|
||||
DEFAULT clock_timestamp()
|
||||
)
|
||||
"""
|
||||
)
|
||||
|
||||
for path in files:
|
||||
version = path.name
|
||||
content = path.read_bytes()
|
||||
checksum = hashlib.sha256(content).hexdigest()
|
||||
|
||||
row = conn.execute(
|
||||
"""
|
||||
SELECT checksum
|
||||
FROM schema_migrations
|
||||
WHERE version = %s
|
||||
""",
|
||||
(version,),
|
||||
).fetchone()
|
||||
|
||||
if row is not None:
|
||||
if row[0] != checksum:
|
||||
raise RuntimeError(
|
||||
f"Applied migration was modified: {version}"
|
||||
)
|
||||
|
||||
print(f"Already applied: {version}", flush=True)
|
||||
continue
|
||||
|
||||
conn.execute(content.decode("utf-8"), prepare=False)
|
||||
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO schema_migrations (version, checksum)
|
||||
VALUES (%s, %s)
|
||||
""",
|
||||
(version, checksum),
|
||||
)
|
||||
|
||||
print(f"Executed: {version} (pending commit)", flush=True)
|
||||
|
||||
print("Migrations committed successfully", flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
apply_migrations()
|
||||
@@ -0,0 +1,12 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
|
||||
|
||||
def object_lock_id(bucket: str, object_key: str) -> int:
|
||||
value = f"s3-object\0{bucket}\0{object_key}".encode("utf-8")
|
||||
return int.from_bytes(
|
||||
hashlib.sha256(value).digest()[:8],
|
||||
byteorder="big",
|
||||
signed=True,
|
||||
)
|
||||
@@ -0,0 +1,128 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import binascii
|
||||
import json
|
||||
import re
|
||||
from datetime import datetime
|
||||
from uuid import UUID
|
||||
|
||||
from fastapi import Request
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
from app.errors import invalid
|
||||
from app.responses import json_response, utc_text
|
||||
|
||||
|
||||
def encode_cursor(scope: str, item: dict) -> str:
|
||||
raw = json.dumps(
|
||||
{
|
||||
"scope": scope,
|
||||
"created_at": utc_text(item["created_at"]),
|
||||
"id": str(item["id"]),
|
||||
},
|
||||
separators=(",", ":"),
|
||||
).encode("utf-8")
|
||||
|
||||
return (
|
||||
base64.urlsafe_b64encode(raw)
|
||||
.decode("ascii")
|
||||
.rstrip("=")
|
||||
)
|
||||
|
||||
|
||||
def decode_cursor(
|
||||
cursor: str,
|
||||
scope: str,
|
||||
) -> tuple[datetime, UUID]:
|
||||
if not cursor or len(cursor) > 2048:
|
||||
raise invalid("Некорректный cursor")
|
||||
|
||||
try:
|
||||
padded = cursor + "=" * (-len(cursor) % 4)
|
||||
|
||||
raw = base64.b64decode(
|
||||
padded,
|
||||
altchars=b"-_",
|
||||
validate=True,
|
||||
)
|
||||
|
||||
data = json.loads(raw.decode("utf-8"))
|
||||
|
||||
if (
|
||||
not isinstance(data, dict)
|
||||
or set(data) != {"scope", "created_at", "id"}
|
||||
or data["scope"] != scope
|
||||
):
|
||||
raise ValueError
|
||||
|
||||
timestamp = data["created_at"]
|
||||
|
||||
if (
|
||||
not isinstance(timestamp, str)
|
||||
or not timestamp.endswith("Z")
|
||||
or not isinstance(data["id"], str)
|
||||
):
|
||||
raise ValueError
|
||||
|
||||
created_at = datetime.fromisoformat(
|
||||
timestamp[:-1] + "+00:00"
|
||||
)
|
||||
|
||||
item_id = UUID(data["id"])
|
||||
|
||||
return created_at, item_id
|
||||
|
||||
except (
|
||||
ValueError,
|
||||
TypeError,
|
||||
UnicodeError,
|
||||
binascii.Error,
|
||||
OverflowError,
|
||||
):
|
||||
raise invalid("Некорректный cursor") from None
|
||||
|
||||
|
||||
def pagination(
|
||||
request: Request,
|
||||
scope: str,
|
||||
) -> tuple[int, tuple[datetime, UUID] | None]:
|
||||
raw_limit = request.query_params.get("limit", "50")
|
||||
|
||||
if (
|
||||
len(raw_limit) > 10
|
||||
or re.fullmatch(r"[0-9]+", raw_limit) is None
|
||||
):
|
||||
raise invalid("Некорректный limit")
|
||||
|
||||
limit = int(raw_limit)
|
||||
|
||||
if not 1 <= limit <= 100:
|
||||
raise invalid("Некорректный limit")
|
||||
|
||||
raw_cursor = request.query_params.get("cursor")
|
||||
after = None
|
||||
|
||||
if raw_cursor is not None:
|
||||
after = decode_cursor(raw_cursor, scope)
|
||||
|
||||
return limit, after
|
||||
|
||||
|
||||
def page_response(
|
||||
rows: list[dict],
|
||||
limit: int,
|
||||
scope: str,
|
||||
) -> JSONResponse:
|
||||
items = rows[:limit]
|
||||
next_cursor = None
|
||||
|
||||
if len(rows) > limit:
|
||||
next_cursor = encode_cursor(scope, items[-1])
|
||||
|
||||
return json_response(
|
||||
{
|
||||
"items": items,
|
||||
"next_cursor": next_cursor,
|
||||
}
|
||||
)
|
||||
@@ -0,0 +1,126 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
|
||||
import pika
|
||||
|
||||
from app.broker import QUEUE_NAME, connect_broker
|
||||
from app.job_db import connect_db
|
||||
from app.responses import utc_text
|
||||
|
||||
|
||||
logger = logging.getLogger("messenger.publisher")
|
||||
|
||||
|
||||
def publish_one(db, channel) -> bool:
|
||||
with db.transaction():
|
||||
row = db.execute(
|
||||
"""
|
||||
SELECT o.id AS outbox_id,
|
||||
o.job_id,
|
||||
o.attempt,
|
||||
j.attachment_id,
|
||||
j.pipeline_version,
|
||||
j.created_at
|
||||
FROM attachment_outbox AS o
|
||||
JOIN attachment_jobs AS j ON j.id = o.job_id
|
||||
JOIN attachments AS a ON a.id = j.attachment_id
|
||||
WHERE o.published_at IS NULL
|
||||
AND o.available_at <= clock_timestamp()
|
||||
AND j.available_at <= clock_timestamp()
|
||||
AND j.status = 'queued'
|
||||
AND j.attempt = o.attempt
|
||||
AND a.status = 'queued'
|
||||
AND a.pipeline_version = j.pipeline_version
|
||||
ORDER BY o.available_at, o.id
|
||||
LIMIT 1
|
||||
FOR UPDATE OF o SKIP LOCKED
|
||||
"""
|
||||
).fetchone()
|
||||
|
||||
if row is None:
|
||||
return False
|
||||
|
||||
payload = {
|
||||
"schema_version": 1,
|
||||
"job_id": str(row["job_id"]),
|
||||
"type": "attachment.process.v1",
|
||||
"attachment_id": str(row["attachment_id"]),
|
||||
"pipeline_version": row["pipeline_version"],
|
||||
"attempt": row["attempt"],
|
||||
"created_at": utc_text(row["created_at"]),
|
||||
}
|
||||
|
||||
channel.basic_publish(
|
||||
exchange="",
|
||||
routing_key=QUEUE_NAME,
|
||||
body=json.dumps(
|
||||
payload,
|
||||
separators=(",", ":"),
|
||||
).encode("utf-8"),
|
||||
properties=pika.BasicProperties(
|
||||
content_type="application/json",
|
||||
delivery_mode=2,
|
||||
message_id=str(row["outbox_id"]),
|
||||
type="attachment.process.v1",
|
||||
),
|
||||
mandatory=True,
|
||||
)
|
||||
|
||||
db.execute(
|
||||
"""
|
||||
UPDATE attachment_outbox
|
||||
SET published_at = clock_timestamp()
|
||||
WHERE id = %s
|
||||
""",
|
||||
(row["outbox_id"],),
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"published job_id=%s attempt=%s",
|
||||
row["job_id"],
|
||||
row["attempt"],
|
||||
)
|
||||
return True
|
||||
|
||||
|
||||
def main():
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s %(levelname)s %(message)s",
|
||||
)
|
||||
|
||||
while True:
|
||||
broker = None
|
||||
try:
|
||||
broker, channel = connect_broker()
|
||||
channel.confirm_delivery()
|
||||
|
||||
with connect_db() as db:
|
||||
logger.info("publisher connected")
|
||||
while True:
|
||||
if not publish_one(db, channel):
|
||||
broker.process_data_events(time_limit=0.5)
|
||||
|
||||
except KeyboardInterrupt:
|
||||
return
|
||||
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"publisher reconnect error_type=%s",
|
||||
type(exc).__name__,
|
||||
)
|
||||
time.sleep(2)
|
||||
|
||||
finally:
|
||||
if broker is not None and broker.is_open:
|
||||
try:
|
||||
broker.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+188
@@ -0,0 +1,188 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from app.job_db import connect_db
|
||||
from app.responses import to_json
|
||||
from app.storage import _get
|
||||
|
||||
|
||||
def list_failed():
|
||||
with connect_db() as db:
|
||||
return db.execute(
|
||||
"""
|
||||
SELECT id AS job_id, attachment_id,
|
||||
pipeline_version, attempt, max_attempts,
|
||||
error_code, updated_at
|
||||
FROM attachment_jobs
|
||||
WHERE status = 'failed'
|
||||
ORDER BY updated_at, id
|
||||
LIMIT 100
|
||||
"""
|
||||
).fetchall()
|
||||
|
||||
|
||||
def replay(job_id, *, actor, reason, apply):
|
||||
budget = int(os.getenv("JOB_MAX_ATTEMPTS", "5"))
|
||||
if not 1 <= budget <= 100:
|
||||
raise ValueError("Invalid retry budget")
|
||||
|
||||
with connect_db() as db:
|
||||
with db.transaction():
|
||||
row = db.execute(
|
||||
"""
|
||||
SELECT j.*,
|
||||
a.status AS attachment_status,
|
||||
a.pipeline_version AS current_version,
|
||||
a.staging_key, a.size_bytes, a.sha256
|
||||
FROM attachment_jobs j
|
||||
JOIN attachments a ON a.id = j.attachment_id
|
||||
WHERE j.id = %s
|
||||
FOR UPDATE OF j, a
|
||||
""",
|
||||
(job_id,),
|
||||
).fetchone()
|
||||
|
||||
if row is None:
|
||||
raise ValueError("Unknown job")
|
||||
|
||||
if (
|
||||
row["status"] != "failed"
|
||||
or row["attachment_status"] != "failed"
|
||||
or row["pipeline_version"] != row["current_version"]
|
||||
):
|
||||
return {
|
||||
"result": "noop",
|
||||
"job_id": str(job_id),
|
||||
"status": row["status"],
|
||||
"reason": "job_is_not_current_failed_work",
|
||||
}
|
||||
|
||||
if not row["staging_key"]:
|
||||
raise ValueError("Staging is unavailable")
|
||||
|
||||
plan = {
|
||||
"result": "replayed" if apply else "dry_run",
|
||||
"job_id": str(job_id),
|
||||
"previous_attempt": row["attempt"],
|
||||
"new_attempt": row["attempt"] + 1,
|
||||
"new_max_attempts": row["attempt"] + budget,
|
||||
"previous_error_code": row["error_code"],
|
||||
}
|
||||
|
||||
if not apply:
|
||||
return plan
|
||||
|
||||
# Не ставим задачу в очередь, если исходник отсутствует
|
||||
# или не совпадает с устойчиво принятыми метаданными.
|
||||
db.execute(
|
||||
"SET LOCAL idle_in_transaction_session_timeout = '60s'"
|
||||
)
|
||||
original = _get(row["staging_key"])
|
||||
if (
|
||||
len(original) != row["size_bytes"]
|
||||
or hashlib.sha256(original).hexdigest() != row["sha256"]
|
||||
):
|
||||
raise ValueError("Staging integrity check failed")
|
||||
|
||||
db.execute(
|
||||
"""
|
||||
INSERT INTO attachment_job_replays (
|
||||
id, job_id, actor, reason,
|
||||
previous_attempt, new_attempt, previous_error_code
|
||||
)
|
||||
VALUES (%s, %s, %s, %s, %s, %s, %s)
|
||||
""",
|
||||
(
|
||||
uuid4(), job_id, actor, reason,
|
||||
row["attempt"], plan["new_attempt"], row["error_code"],
|
||||
),
|
||||
)
|
||||
|
||||
db.execute(
|
||||
"""
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'queued',
|
||||
attempt = %s,
|
||||
max_attempts = %s,
|
||||
error_code = NULL,
|
||||
lease_token = NULL,
|
||||
lease_until = NULL,
|
||||
available_at = clock_timestamp(),
|
||||
updated_at = clock_timestamp()
|
||||
WHERE id = %s
|
||||
""",
|
||||
(
|
||||
plan["new_attempt"],
|
||||
plan["new_max_attempts"],
|
||||
job_id,
|
||||
),
|
||||
)
|
||||
|
||||
db.execute(
|
||||
"""
|
||||
UPDATE attachments
|
||||
SET status = 'queued',
|
||||
error_code = NULL,
|
||||
updated_at = clock_timestamp()
|
||||
WHERE id = %s
|
||||
""",
|
||||
(row["attachment_id"],),
|
||||
)
|
||||
|
||||
db.execute(
|
||||
"""
|
||||
INSERT INTO attachment_outbox (id, job_id, attempt)
|
||||
VALUES (%s, %s, %s)
|
||||
""",
|
||||
(uuid4(), job_id, plan["new_attempt"]),
|
||||
)
|
||||
|
||||
return plan
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
selection = parser.add_mutually_exclusive_group(required=True)
|
||||
selection.add_argument("--list", action="store_true")
|
||||
selection.add_argument("--job-id", type=UUID)
|
||||
parser.add_argument("--actor")
|
||||
parser.add_argument("--reason")
|
||||
parser.add_argument("--apply", action="store_true")
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.list and args.apply:
|
||||
parser.error("--apply requires --job-id")
|
||||
|
||||
if args.job_id:
|
||||
if not args.actor or not 1 <= len(args.actor.strip()) <= 100:
|
||||
parser.error("--actor must contain 1–100 characters")
|
||||
if not args.reason or not 1 <= len(args.reason.strip()) <= 500:
|
||||
parser.error("--reason must contain 1–500 characters")
|
||||
|
||||
try:
|
||||
if args.list:
|
||||
result = list_failed()
|
||||
else:
|
||||
result = replay(
|
||||
args.job_id,
|
||||
actor=args.actor.strip(),
|
||||
reason=args.reason.strip(),
|
||||
apply=args.apply,
|
||||
)
|
||||
except Exception as exc:
|
||||
print(json.dumps({
|
||||
"result": "error",
|
||||
"error_type": type(exc).__name__,
|
||||
}))
|
||||
raise SystemExit(1)
|
||||
|
||||
print(json.dumps(to_json(result), ensure_ascii=False, indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,40 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from uuid import UUID
|
||||
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
|
||||
def utc_text(value: datetime) -> str:
|
||||
return (
|
||||
value.astimezone(timezone.utc)
|
||||
.isoformat(timespec="microseconds")
|
||||
.replace("+00:00", "Z")
|
||||
)
|
||||
|
||||
|
||||
def to_json(value):
|
||||
if isinstance(value, UUID):
|
||||
return str(value)
|
||||
|
||||
if isinstance(value, datetime):
|
||||
return utc_text(value)
|
||||
|
||||
if isinstance(value, dict):
|
||||
return {
|
||||
key: to_json(item)
|
||||
for key, item in value.items()
|
||||
}
|
||||
|
||||
if isinstance(value, (list, tuple)):
|
||||
return [to_json(item) for item in value]
|
||||
|
||||
return value
|
||||
|
||||
|
||||
def json_response(value, status: int = 200) -> JSONResponse:
|
||||
return JSONResponse(
|
||||
status_code=status,
|
||||
content=to_json(value),
|
||||
)
|
||||
@@ -0,0 +1,275 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
from contextlib import asynccontextmanager
|
||||
from uuid import UUID
|
||||
|
||||
from fastapi import APIRouter, Request
|
||||
from fastapi.responses import Response
|
||||
from starlette.concurrency import run_in_threadpool
|
||||
from starlette.requests import ClientDisconnect
|
||||
|
||||
from app.db import pool
|
||||
from app.jobs import accept_upload
|
||||
from app.dependencies import accessible_chat, current_user
|
||||
from app.errors import ApiError, invalid, not_found
|
||||
from app.images import MAX_BYTES, inspect_image, validate_filename
|
||||
from app.responses import json_response
|
||||
from app.storage import get_object
|
||||
from app.validation import parse_uuid
|
||||
|
||||
|
||||
router = APIRouter()
|
||||
binary_slot = asyncio.Semaphore(1)
|
||||
|
||||
PUBLIC_FIELDS = (
|
||||
"id",
|
||||
"chat_id",
|
||||
"owner_id",
|
||||
"filename",
|
||||
"content_type",
|
||||
"size_bytes",
|
||||
"sha256",
|
||||
"status",
|
||||
"created_at",
|
||||
"updated_at",
|
||||
)
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def binary_operation():
|
||||
try:
|
||||
await asyncio.wait_for(binary_slot.acquire(), timeout=3)
|
||||
except TimeoutError:
|
||||
raise ApiError(
|
||||
503,
|
||||
"unavailable",
|
||||
"Обработка файлов временно занята",
|
||||
) from None
|
||||
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
binary_slot.release()
|
||||
|
||||
|
||||
def public_attachment(row: dict) -> dict:
|
||||
result = {field: row[field] for field in PUBLIC_FIELDS}
|
||||
result["variants"] = row.get("variants", [])
|
||||
result["error_code"] = row.get("error_code")
|
||||
return result
|
||||
|
||||
|
||||
async def read_image_body(request: Request) -> bytes:
|
||||
lengths = request.headers.getlist("content-length")
|
||||
|
||||
if len(lengths) > 1:
|
||||
raise invalid("Некорректный Content-Length")
|
||||
|
||||
expected = None
|
||||
if lengths:
|
||||
value = lengths[0]
|
||||
if not value.isascii() or not value.isdecimal():
|
||||
raise invalid("Некорректный Content-Length")
|
||||
|
||||
if len(value) > 20:
|
||||
raise ApiError(
|
||||
413, "payload_too_large", "Файл превышает 10 MiB"
|
||||
)
|
||||
|
||||
expected = int(value)
|
||||
if expected > MAX_BYTES:
|
||||
raise ApiError(
|
||||
413, "payload_too_large", "Файл превышает 10 MiB"
|
||||
)
|
||||
|
||||
data = bytearray()
|
||||
|
||||
async for chunk in request.stream():
|
||||
if len(data) + len(chunk) > MAX_BYTES:
|
||||
raise ApiError(
|
||||
413, "payload_too_large", "Файл превышает 10 MiB"
|
||||
)
|
||||
data.extend(chunk)
|
||||
|
||||
if expected is not None and len(data) != expected:
|
||||
raise invalid("Размер тела не соответствует Content-Length")
|
||||
|
||||
return bytes(data)
|
||||
|
||||
|
||||
async def accessible_attachment(
|
||||
attachment_id: UUID,
|
||||
user_id: UUID,
|
||||
) -> dict:
|
||||
async with pool.connection() as conn:
|
||||
cursor = await conn.execute(
|
||||
"""
|
||||
SELECT a.id, a.chat_id, a.owner_id,
|
||||
a.filename, a.content_type, a.size_bytes,
|
||||
a.sha256, a.object_key, a.status,
|
||||
a.error_code, a.pipeline_version,
|
||||
a.created_at, a.updated_at
|
||||
FROM attachments AS a
|
||||
WHERE a.id = %s
|
||||
AND EXISTS (
|
||||
SELECT 1
|
||||
FROM chat_members AS cm
|
||||
WHERE cm.chat_id = a.chat_id
|
||||
AND cm.user_id = %s
|
||||
)
|
||||
""",
|
||||
(attachment_id, user_id),
|
||||
)
|
||||
row = await cursor.fetchone()
|
||||
|
||||
if row is not None:
|
||||
cursor = await conn.execute(
|
||||
"""
|
||||
SELECT name, content_type, size_bytes,
|
||||
width, height, sha256, object_key
|
||||
FROM attachment_variants
|
||||
WHERE attachment_id = %s
|
||||
AND pipeline_version = %s
|
||||
ORDER BY name
|
||||
""",
|
||||
(row["id"], row["pipeline_version"]),
|
||||
)
|
||||
variants = await cursor.fetchall()
|
||||
row["_variant_objects"] = {
|
||||
variant["name"]: variant
|
||||
for variant in variants
|
||||
}
|
||||
row["variants"] = [
|
||||
{
|
||||
key: value
|
||||
for key, value in variant.items()
|
||||
if key != "object_key"
|
||||
}
|
||||
for variant in variants
|
||||
] if row["status"] == "ready" else []
|
||||
|
||||
if row is None:
|
||||
raise not_found()
|
||||
|
||||
return row
|
||||
|
||||
|
||||
@router.post("/api/v1/chats/{chat_id}/attachments")
|
||||
async def upload_attachment(chat_id: str, request: Request):
|
||||
user, _ = await current_user(request)
|
||||
chat_uuid, _ = await accessible_chat(chat_id, user["id"])
|
||||
|
||||
filenames = request.query_params.getlist("filename")
|
||||
if len(filenames) != 1:
|
||||
raise invalid("Требуется одно имя файла")
|
||||
|
||||
filename = validate_filename(filenames[0])
|
||||
|
||||
content_types = request.headers.getlist("content-type")
|
||||
if len(content_types) != 1:
|
||||
raise ApiError(
|
||||
415,
|
||||
"unsupported_media_type",
|
||||
"Требуется Content-Type изображения",
|
||||
)
|
||||
|
||||
declared_type = content_types[0].split(";", 1)[0].strip().lower()
|
||||
if declared_type not in ("image/png", "image/jpeg"):
|
||||
raise ApiError(
|
||||
415,
|
||||
"unsupported_media_type",
|
||||
"Допустимы только image/png и image/jpeg",
|
||||
)
|
||||
|
||||
async with binary_operation():
|
||||
try:
|
||||
data = await asyncio.wait_for(
|
||||
read_image_body(request),
|
||||
timeout=60,
|
||||
)
|
||||
except ClientDisconnect:
|
||||
raise invalid("Загрузка файла прервана") from None
|
||||
except TimeoutError:
|
||||
raise invalid("Превышено время загрузки файла") from None
|
||||
|
||||
content_type = await run_in_threadpool(
|
||||
inspect_image,
|
||||
data,
|
||||
declared_type,
|
||||
)
|
||||
|
||||
row = await accept_upload(
|
||||
chat_id=chat_uuid,
|
||||
owner_id=user["id"],
|
||||
filename=filename,
|
||||
content_type=content_type,
|
||||
data=data,
|
||||
)
|
||||
|
||||
return json_response(public_attachment(row), 202)
|
||||
|
||||
|
||||
@router.get("/api/v1/attachments/{attachment_id}")
|
||||
async def get_attachment(attachment_id: str, request: Request):
|
||||
user, _ = await current_user(request)
|
||||
|
||||
row = await accessible_attachment(
|
||||
parse_uuid(attachment_id),
|
||||
user["id"],
|
||||
)
|
||||
|
||||
return json_response(public_attachment(row))
|
||||
|
||||
|
||||
@router.get("/api/v1/attachments/{attachment_id}/content")
|
||||
async def download_attachment(attachment_id: str, request: Request):
|
||||
user, _ = await current_user(request)
|
||||
|
||||
row = await accessible_attachment(
|
||||
parse_uuid(attachment_id),
|
||||
user["id"],
|
||||
)
|
||||
|
||||
variants = request.query_params.getlist("variant")
|
||||
if len(variants) > 1:
|
||||
raise invalid("Допускается один вариант изображения")
|
||||
|
||||
variant_name = variants[0] if variants else "original"
|
||||
|
||||
if row["status"] != "ready":
|
||||
raise ApiError(
|
||||
409,
|
||||
"conflict",
|
||||
"Вложение не готово к скачиванию",
|
||||
)
|
||||
|
||||
if variant_name == "original":
|
||||
selected = row
|
||||
else:
|
||||
selected = row["_variant_objects"].get(variant_name)
|
||||
if selected is None:
|
||||
raise not_found()
|
||||
|
||||
async with binary_operation():
|
||||
data = await get_object(selected["object_key"])
|
||||
|
||||
if (
|
||||
len(data) != selected["size_bytes"]
|
||||
or hashlib.sha256(data).hexdigest() != selected["sha256"]
|
||||
):
|
||||
raise ApiError(
|
||||
503,
|
||||
"unavailable",
|
||||
"Изображение временно недоступно",
|
||||
)
|
||||
|
||||
return Response(
|
||||
content=data,
|
||||
media_type=selected["content_type"],
|
||||
headers={
|
||||
"X-Content-Type-Options": "nosniff",
|
||||
"Cache-Control": "private, no-store",
|
||||
},
|
||||
)
|
||||
@@ -0,0 +1,74 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter, Request
|
||||
|
||||
from app.dependencies import accessible_chat, current_user
|
||||
from app.errors import invalid
|
||||
from app.pagination import page_response, pagination
|
||||
from app.responses import json_response
|
||||
from app.store import InvalidMembersError, store
|
||||
from app.validation import parse_uuid, read_payload, valid_text
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.post("/api/v1/chats")
|
||||
async def create_chat(request: Request):
|
||||
user, _ = await current_user(request)
|
||||
|
||||
payload = await read_payload(
|
||||
request,
|
||||
{"title", "member_ids"},
|
||||
)
|
||||
|
||||
if not valid_text(payload["title"], 100):
|
||||
raise invalid("Некорректный title")
|
||||
|
||||
raw_members = payload["member_ids"]
|
||||
|
||||
if (
|
||||
not isinstance(raw_members, list)
|
||||
or not 1 <= len(raw_members) <= 100
|
||||
):
|
||||
raise invalid("Некорректный member_ids")
|
||||
|
||||
member_ids = [parse_uuid(item) for item in raw_members]
|
||||
|
||||
if len(set(member_ids)) != len(member_ids):
|
||||
raise invalid("В member_ids есть дубликаты")
|
||||
|
||||
if user["id"] not in member_ids:
|
||||
raise invalid("Создатель должен входить в member_ids")
|
||||
|
||||
try:
|
||||
chat = await store.create_chat(
|
||||
payload["title"],
|
||||
member_ids,
|
||||
)
|
||||
except InvalidMembersError:
|
||||
raise invalid(
|
||||
"В member_ids есть неизвестный пользователь"
|
||||
) from None
|
||||
|
||||
return json_response(chat, 201)
|
||||
|
||||
|
||||
@router.get("/api/v1/chats")
|
||||
async def list_chats(request: Request):
|
||||
user, _ = await current_user(request)
|
||||
|
||||
scope = f"chats:{user['id']}"
|
||||
limit, after = pagination(request, scope)
|
||||
|
||||
rows = await store.list_chats(user["id"], limit, after)
|
||||
|
||||
return page_response(rows, limit, scope)
|
||||
|
||||
|
||||
@router.get("/api/v1/chats/{chat_id}")
|
||||
async def get_chat(chat_id: str, request: Request):
|
||||
user, _ = await current_user(request)
|
||||
|
||||
_, chat = await accessible_chat(chat_id, user["id"])
|
||||
|
||||
return json_response(chat)
|
||||
@@ -0,0 +1,20 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter
|
||||
|
||||
from app.db import check_db
|
||||
from app.storage import check_storage
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/health/live")
|
||||
async def health_live():
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
@router.get("/health/ready")
|
||||
async def health_ready():
|
||||
await check_db()
|
||||
await check_storage()
|
||||
return {"status": "ok"}
|
||||
@@ -0,0 +1,90 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
from fastapi import APIRouter, Request
|
||||
|
||||
from app.dependencies import accessible_chat, current_user
|
||||
from app.errors import invalid
|
||||
from app.idempotency import create_message_once
|
||||
from app.pagination import page_response, pagination
|
||||
from app.responses import json_response
|
||||
from app.store import store
|
||||
from app.validation import parse_uuid, read_payload
|
||||
|
||||
|
||||
router = APIRouter()
|
||||
KEY_RE = re.compile(r"[A-Za-z0-9._:-]{1,128}", re.ASCII)
|
||||
|
||||
|
||||
@router.post("/api/v1/chats/{chat_id}/messages")
|
||||
async def create_message(chat_id: str, request: Request):
|
||||
user, _ = await current_user(request)
|
||||
|
||||
chat_uuid, _ = await accessible_chat(
|
||||
chat_id,
|
||||
user["id"],
|
||||
)
|
||||
|
||||
keys = request.headers.getlist("idempotency-key")
|
||||
if len(keys) != 1 or KEY_RE.fullmatch(keys[0]) is None:
|
||||
raise invalid("Требуется корректный Idempotency-Key")
|
||||
|
||||
payload = await read_payload(
|
||||
request,
|
||||
{"text"},
|
||||
optional_fields={"attachment_ids"},
|
||||
)
|
||||
|
||||
text = payload["text"]
|
||||
if not isinstance(text, str) or len(text) > 4000:
|
||||
raise invalid("Некорректный текст сообщения")
|
||||
|
||||
raw_attachment_ids = payload.get("attachment_ids", [])
|
||||
if (
|
||||
not isinstance(raw_attachment_ids, list)
|
||||
or len(raw_attachment_ids) > 10
|
||||
):
|
||||
raise invalid("Допускается список максимум из 10 вложений")
|
||||
|
||||
attachment_ids = [
|
||||
parse_uuid(value)
|
||||
for value in raw_attachment_ids
|
||||
]
|
||||
|
||||
if len(set(attachment_ids)) != len(attachment_ids):
|
||||
raise invalid("Вложения не должны повторяться")
|
||||
|
||||
if not text.strip() and not attachment_ids:
|
||||
raise invalid("Сообщение должно содержать текст или вложение")
|
||||
|
||||
message = await create_message_once(
|
||||
chat_uuid,
|
||||
user["id"],
|
||||
text,
|
||||
keys[0],
|
||||
attachment_ids=attachment_ids,
|
||||
)
|
||||
|
||||
return json_response(message, 201)
|
||||
|
||||
|
||||
@router.get("/api/v1/chats/{chat_id}/messages")
|
||||
async def list_messages(chat_id: str, request: Request):
|
||||
user, _ = await current_user(request)
|
||||
|
||||
chat_uuid, _ = await accessible_chat(
|
||||
chat_id,
|
||||
user["id"],
|
||||
)
|
||||
|
||||
scope = f"messages:{chat_uuid}"
|
||||
limit, after = pagination(request, scope)
|
||||
|
||||
rows = await store.list_messages(
|
||||
chat_uuid,
|
||||
limit,
|
||||
after,
|
||||
)
|
||||
|
||||
return page_response(rows, limit, scope)
|
||||
@@ -0,0 +1,82 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter, Request
|
||||
from fastapi.responses import Response
|
||||
|
||||
from app.auth import (
|
||||
DUMMY_PASSWORD_HASH,
|
||||
generate_token,
|
||||
hash_token,
|
||||
parse_basic_credentials,
|
||||
verify_password,
|
||||
)
|
||||
from app.config import SESSION_TTL_SECONDS
|
||||
from app.dependencies import current_user
|
||||
from app.errors import invalid, unauthorized
|
||||
from app.responses import json_response
|
||||
from app.store import store
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.post("/api/v1/auth/sessions")
|
||||
async def create_session(request: Request):
|
||||
credentials = parse_basic_credentials(
|
||||
request.headers.get("authorization")
|
||||
)
|
||||
|
||||
if credentials is None:
|
||||
raise unauthorized("Basic")
|
||||
|
||||
username, password = credentials
|
||||
|
||||
user = await store.get_user_for_login(username)
|
||||
|
||||
stored_hash = (
|
||||
DUMMY_PASSWORD_HASH
|
||||
if user is None
|
||||
else user["password_hash"]
|
||||
)
|
||||
|
||||
password_ok = await verify_password(stored_hash, password)
|
||||
|
||||
if user is None or not password_ok:
|
||||
raise unauthorized("Basic")
|
||||
|
||||
if await request.body():
|
||||
raise invalid("Тело запроса при входе не допускается")
|
||||
|
||||
token = generate_token()
|
||||
|
||||
session = await store.create_session(
|
||||
user["id"],
|
||||
hash_token(token),
|
||||
SESSION_TTL_SECONDS,
|
||||
)
|
||||
|
||||
public_user = {
|
||||
field: user[field]
|
||||
for field in ("id", "username", "display_name", "created_at")
|
||||
}
|
||||
|
||||
return json_response(
|
||||
{
|
||||
"token": token,
|
||||
"token_type": "Bearer",
|
||||
"expires_at": session["expires_at"],
|
||||
"user": public_user,
|
||||
},
|
||||
201,
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/api/v1/auth/sessions/current")
|
||||
async def revoke_session(request: Request):
|
||||
_, token_digest = await current_user(request)
|
||||
|
||||
revoked = await store.revoke_session(token_digest)
|
||||
|
||||
if not revoked:
|
||||
raise unauthorized()
|
||||
|
||||
return Response(status_code=204)
|
||||
@@ -0,0 +1,59 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter, Request
|
||||
|
||||
from app.auth import USERNAME_RE, hash_password
|
||||
from app.dependencies import current_user
|
||||
from app.errors import ApiError, invalid
|
||||
from app.responses import json_response
|
||||
from app.store import store
|
||||
from app.validation import read_payload, valid_text
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.post("/api/v1/users")
|
||||
async def create_user(request: Request):
|
||||
payload = await read_payload(
|
||||
request,
|
||||
{"username", "display_name", "password"},
|
||||
)
|
||||
|
||||
username = payload["username"]
|
||||
display_name = payload["display_name"]
|
||||
password = payload["password"]
|
||||
|
||||
if (
|
||||
not isinstance(username, str)
|
||||
or USERNAME_RE.fullmatch(username) is None
|
||||
):
|
||||
raise invalid("Некорректный username")
|
||||
|
||||
if not valid_text(display_name, 100):
|
||||
raise invalid("Некорректное display_name")
|
||||
|
||||
if not valid_text(password, 128) or len(password) < 12:
|
||||
raise invalid("Пароль должен содержать от 12 до 128 символов")
|
||||
|
||||
password_digest = await hash_password(password)
|
||||
|
||||
user = await store.create_user(
|
||||
username,
|
||||
display_name,
|
||||
password_digest,
|
||||
)
|
||||
|
||||
if user is None:
|
||||
raise ApiError(
|
||||
409,
|
||||
"conflict",
|
||||
"Username уже занят",
|
||||
)
|
||||
|
||||
return json_response(user, 201)
|
||||
|
||||
|
||||
@router.get("/api/v1/users/me")
|
||||
async def get_current_user(request: Request):
|
||||
user, _ = await current_user(request)
|
||||
return json_response(user)
|
||||
@@ -0,0 +1,34 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
|
||||
from app.errors import ApiError
|
||||
from app.storage import check_storage
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
deadline = time.monotonic() + 90
|
||||
|
||||
while True:
|
||||
try:
|
||||
await check_storage()
|
||||
except ApiError:
|
||||
if time.monotonic() >= deadline:
|
||||
raise SystemExit(
|
||||
"S3 readiness failed: bucket unavailable "
|
||||
"or API credentials rejected"
|
||||
) from None
|
||||
|
||||
print("Waiting for S3 bucket...", flush=True)
|
||||
await asyncio.sleep(2)
|
||||
else:
|
||||
print(
|
||||
"S3 bucket is accessible with API credentials",
|
||||
flush=True,
|
||||
)
|
||||
return
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
+123
@@ -0,0 +1,123 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import boto3
|
||||
from botocore.config import Config
|
||||
from botocore.exceptions import (
|
||||
BotoCoreError,
|
||||
ClientError,
|
||||
)
|
||||
|
||||
from app.errors import ApiError
|
||||
|
||||
|
||||
_client = None
|
||||
|
||||
|
||||
def _unavailable() -> ApiError:
|
||||
return ApiError(
|
||||
503,
|
||||
"unavailable",
|
||||
"Сервис временно недоступен",
|
||||
)
|
||||
|
||||
|
||||
|
||||
def _credential(file_variable: str) -> str:
|
||||
file_path = os.environ[file_variable]
|
||||
value = Path(file_path).read_text().rstrip("\r\n")
|
||||
if not value:
|
||||
raise ValueError("Empty S3 credential file")
|
||||
return value
|
||||
|
||||
def bucket_name() -> str:
|
||||
return os.environ["S3_BUCKET"]
|
||||
|
||||
|
||||
def get_client():
|
||||
global _client
|
||||
|
||||
if _client is None:
|
||||
_client = boto3.client(
|
||||
"s3",
|
||||
endpoint_url=os.environ["S3_ENDPOINT"],
|
||||
aws_access_key_id=_credential("S3_ACCESS_KEY_ID_FILE"),
|
||||
aws_secret_access_key=_credential("S3_SECRET_ACCESS_KEY_FILE"),
|
||||
region_name=os.getenv("S3_REGION", "us-east-1"),
|
||||
config=Config(
|
||||
signature_version="s3v4",
|
||||
s3={"addressing_style": "path"},
|
||||
connect_timeout=3,
|
||||
read_timeout=10,
|
||||
retries={
|
||||
"mode": "standard",
|
||||
"total_max_attempts": 2,
|
||||
},
|
||||
),
|
||||
)
|
||||
|
||||
return _client
|
||||
|
||||
|
||||
def _put(key: str, data: bytes, content_type: str) -> None:
|
||||
get_client().put_object(
|
||||
Bucket=bucket_name(),
|
||||
Key=key,
|
||||
Body=data,
|
||||
ContentType=content_type,
|
||||
)
|
||||
|
||||
|
||||
def _get(key: str) -> bytes:
|
||||
maximum = 10 * 1024 * 1024
|
||||
response = get_client().get_object(
|
||||
Bucket=bucket_name(),
|
||||
Key=key,
|
||||
)
|
||||
body = response["Body"]
|
||||
|
||||
try:
|
||||
expected_size = response["ContentLength"]
|
||||
if not 1 <= expected_size <= maximum:
|
||||
raise _unavailable()
|
||||
|
||||
data = body.read(maximum + 1)
|
||||
|
||||
if len(data) != expected_size or len(data) > maximum:
|
||||
raise _unavailable()
|
||||
|
||||
return data
|
||||
finally:
|
||||
body.close()
|
||||
|
||||
|
||||
async def put_object(
|
||||
key: str,
|
||||
data: bytes,
|
||||
content_type: str,
|
||||
) -> None:
|
||||
try:
|
||||
await asyncio.to_thread(_put, key, data, content_type)
|
||||
except (BotoCoreError, ClientError, OSError, KeyError, ValueError):
|
||||
raise _unavailable() from None
|
||||
|
||||
|
||||
async def get_object(key: str) -> bytes:
|
||||
try:
|
||||
return await asyncio.to_thread(_get, key)
|
||||
except (BotoCoreError, ClientError, OSError, KeyError, ValueError):
|
||||
raise _unavailable() from None
|
||||
|
||||
|
||||
def _check() -> None:
|
||||
get_client().head_bucket(Bucket=bucket_name())
|
||||
|
||||
|
||||
async def check_storage() -> None:
|
||||
try:
|
||||
await asyncio.to_thread(_check)
|
||||
except (BotoCoreError, ClientError, OSError, KeyError, ValueError):
|
||||
raise _unavailable() from None
|
||||
+280
@@ -0,0 +1,280 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from psycopg.errors import ForeignKeyViolation, UniqueViolation
|
||||
|
||||
from app.db import pool
|
||||
|
||||
|
||||
class InvalidMembersError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class PostgresStore:
|
||||
async def create_user(
|
||||
self,
|
||||
username: str,
|
||||
display_name: str,
|
||||
password_hash: str,
|
||||
) -> dict | None:
|
||||
try:
|
||||
async with pool.connection() as conn:
|
||||
cursor = await conn.execute(
|
||||
"""
|
||||
INSERT INTO users (
|
||||
id, username, display_name, password_hash
|
||||
)
|
||||
VALUES (%s, %s, %s, %s)
|
||||
RETURNING id, username, display_name, created_at
|
||||
""",
|
||||
(uuid4(), username, display_name, password_hash),
|
||||
)
|
||||
user = await cursor.fetchone()
|
||||
|
||||
return user
|
||||
except UniqueViolation as exc:
|
||||
if exc.diag.constraint_name == "users_username_unique":
|
||||
return None
|
||||
raise
|
||||
|
||||
async def get_user(self, user_id: UUID) -> dict | None:
|
||||
async with pool.connection() as conn:
|
||||
cursor = await conn.execute(
|
||||
"""
|
||||
SELECT id, username, display_name, created_at
|
||||
FROM users
|
||||
WHERE id = %s
|
||||
""",
|
||||
(user_id,),
|
||||
)
|
||||
return await cursor.fetchone()
|
||||
|
||||
async def get_user_for_login(self, username: str) -> dict | None:
|
||||
async with pool.connection() as conn:
|
||||
cursor = await conn.execute(
|
||||
"""
|
||||
SELECT id, username, display_name,
|
||||
password_hash, created_at
|
||||
FROM users
|
||||
WHERE username = %s
|
||||
""",
|
||||
(username,),
|
||||
)
|
||||
return await cursor.fetchone()
|
||||
|
||||
async def create_chat(
|
||||
self,
|
||||
title: str,
|
||||
member_ids: list[UUID],
|
||||
) -> dict:
|
||||
chat_id = uuid4()
|
||||
|
||||
try:
|
||||
async with pool.connection() as conn:
|
||||
cursor = await conn.execute(
|
||||
"""
|
||||
INSERT INTO chats (id, title)
|
||||
VALUES (%s, %s)
|
||||
RETURNING id, title, created_at
|
||||
""",
|
||||
(chat_id, title),
|
||||
)
|
||||
chat = await cursor.fetchone()
|
||||
|
||||
async with conn.cursor() as cursor:
|
||||
await cursor.executemany(
|
||||
"""
|
||||
INSERT INTO chat_members (chat_id, user_id)
|
||||
VALUES (%s, %s)
|
||||
""",
|
||||
[(chat_id, user_id) for user_id in member_ids],
|
||||
)
|
||||
|
||||
chat["member_ids"] = member_ids
|
||||
|
||||
return chat
|
||||
except ForeignKeyViolation as exc:
|
||||
raise InvalidMembersError from exc
|
||||
|
||||
async def get_accessible_chat(
|
||||
self,
|
||||
chat_id: UUID,
|
||||
user_id: UUID,
|
||||
) -> dict | None:
|
||||
async with pool.connection() as conn:
|
||||
cursor = await conn.execute(
|
||||
"""
|
||||
SELECT c.id, c.title, c.created_at,
|
||||
ARRAY(
|
||||
SELECT cm.user_id
|
||||
FROM chat_members AS cm
|
||||
WHERE cm.chat_id = c.id
|
||||
ORDER BY cm.user_id
|
||||
) AS member_ids
|
||||
FROM chats AS c
|
||||
WHERE c.id = %s
|
||||
AND EXISTS (
|
||||
SELECT 1
|
||||
FROM chat_members AS access
|
||||
WHERE access.chat_id = c.id
|
||||
AND access.user_id = %s
|
||||
)
|
||||
""",
|
||||
(chat_id, user_id),
|
||||
)
|
||||
return await cursor.fetchone()
|
||||
|
||||
async def list_chats(
|
||||
self,
|
||||
user_id: UUID,
|
||||
limit: int,
|
||||
after: tuple[datetime, UUID] | None = None,
|
||||
) -> list[dict]:
|
||||
params: list[object] = [user_id]
|
||||
after_sql = ""
|
||||
|
||||
if after is not None:
|
||||
after_sql = "AND (c.created_at, c.id) > (%s, %s)"
|
||||
params.extend(after)
|
||||
|
||||
params.append(limit + 1)
|
||||
|
||||
async with pool.connection() as conn:
|
||||
cursor = await conn.execute(
|
||||
f"""
|
||||
SELECT c.id, c.title, c.created_at,
|
||||
ARRAY(
|
||||
SELECT cm.user_id
|
||||
FROM chat_members AS cm
|
||||
WHERE cm.chat_id = c.id
|
||||
ORDER BY cm.user_id
|
||||
) AS member_ids
|
||||
FROM chats AS c
|
||||
JOIN chat_members AS access
|
||||
ON access.chat_id = c.id
|
||||
WHERE access.user_id = %s
|
||||
{after_sql}
|
||||
ORDER BY c.created_at, c.id
|
||||
LIMIT %s
|
||||
""",
|
||||
params,
|
||||
)
|
||||
return await cursor.fetchall()
|
||||
|
||||
async def create_message(
|
||||
self,
|
||||
chat_id: UUID,
|
||||
sender_id: UUID,
|
||||
text: str,
|
||||
) -> dict:
|
||||
async with pool.connection() as conn:
|
||||
cursor = await conn.execute(
|
||||
"""
|
||||
INSERT INTO messages (id, chat_id, sender_id, text)
|
||||
VALUES (%s, %s, %s, %s)
|
||||
RETURNING id, chat_id, sender_id, text, created_at
|
||||
""",
|
||||
(uuid4(), chat_id, sender_id, text),
|
||||
)
|
||||
message = await cursor.fetchone()
|
||||
|
||||
return message
|
||||
|
||||
async def list_messages(
|
||||
self,
|
||||
chat_id: UUID,
|
||||
limit: int,
|
||||
after: tuple[datetime, UUID] | None = None,
|
||||
) -> list[dict]:
|
||||
params: list[object] = [chat_id]
|
||||
after_sql = ""
|
||||
|
||||
if after is not None:
|
||||
after_sql = "AND (created_at, id) > (%s, %s)"
|
||||
params.extend(after)
|
||||
|
||||
params.append(limit + 1)
|
||||
|
||||
async with pool.connection() as conn:
|
||||
cursor = await conn.execute(
|
||||
f"""
|
||||
SELECT m.id, m.chat_id, m.sender_id,
|
||||
m.text, m.created_at,
|
||||
ARRAY(
|
||||
SELECT ma.attachment_id
|
||||
FROM message_attachments AS ma
|
||||
WHERE ma.message_id = m.id
|
||||
ORDER BY ma.position
|
||||
) AS attachment_ids
|
||||
FROM messages AS m
|
||||
WHERE m.chat_id = %s
|
||||
{after_sql}
|
||||
ORDER BY created_at, id
|
||||
LIMIT %s
|
||||
""",
|
||||
params,
|
||||
)
|
||||
return await cursor.fetchall()
|
||||
|
||||
async def create_session(
|
||||
self,
|
||||
user_id: UUID,
|
||||
token_hash: str,
|
||||
ttl_seconds: int,
|
||||
) -> dict:
|
||||
async with pool.connection() as conn:
|
||||
cursor = await conn.execute(
|
||||
"""
|
||||
INSERT INTO sessions (
|
||||
token_hash, user_id, created_at, expires_at
|
||||
)
|
||||
VALUES (
|
||||
%s, %s, statement_timestamp(),
|
||||
statement_timestamp()
|
||||
+ (%s * INTERVAL '1 second')
|
||||
)
|
||||
RETURNING expires_at
|
||||
""",
|
||||
(token_hash, user_id, ttl_seconds),
|
||||
)
|
||||
session = await cursor.fetchone()
|
||||
|
||||
return session
|
||||
|
||||
async def get_session_user_id(
|
||||
self,
|
||||
token_hash: str,
|
||||
) -> UUID | None:
|
||||
async with pool.connection() as conn:
|
||||
cursor = await conn.execute(
|
||||
"""
|
||||
SELECT user_id
|
||||
FROM sessions
|
||||
WHERE token_hash = %s
|
||||
AND expires_at > statement_timestamp()
|
||||
""",
|
||||
(token_hash,),
|
||||
)
|
||||
row = await cursor.fetchone()
|
||||
|
||||
return None if row is None else row["user_id"]
|
||||
|
||||
async def revoke_session(self, token_hash: str) -> bool:
|
||||
async with pool.connection() as conn:
|
||||
cursor = await conn.execute(
|
||||
"""
|
||||
DELETE FROM sessions
|
||||
WHERE token_hash = %s
|
||||
AND expires_at > statement_timestamp()
|
||||
RETURNING token_hash
|
||||
""",
|
||||
(token_hash,),
|
||||
)
|
||||
row = await cursor.fetchone()
|
||||
|
||||
return row is not None
|
||||
|
||||
|
||||
store = PostgresStore()
|
||||
@@ -0,0 +1,61 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from uuid import UUID
|
||||
|
||||
from fastapi import Request
|
||||
|
||||
from app.errors import ApiError, invalid
|
||||
|
||||
|
||||
def parse_uuid(value: object) -> UUID:
|
||||
if not isinstance(value, str):
|
||||
raise invalid("Некорректный UUID")
|
||||
|
||||
try:
|
||||
return UUID(value)
|
||||
except ValueError:
|
||||
raise invalid("Некорректный UUID") from None
|
||||
|
||||
|
||||
def valid_text(value: object, maximum: int) -> bool:
|
||||
return (
|
||||
isinstance(value, str)
|
||||
and 1 <= len(value) <= maximum
|
||||
and bool(value.strip())
|
||||
)
|
||||
|
||||
|
||||
async def read_payload(
|
||||
request: Request,
|
||||
expected_fields: set[str],
|
||||
optional_fields: set[str] | None = None,
|
||||
) -> dict:
|
||||
content_type = (
|
||||
request.headers.get("content-type", "")
|
||||
.split(";", 1)[0]
|
||||
.strip()
|
||||
.lower()
|
||||
)
|
||||
|
||||
if content_type != "application/json":
|
||||
raise ApiError(
|
||||
415,
|
||||
"unsupported_media_type",
|
||||
"Требуется Content-Type: application/json",
|
||||
)
|
||||
|
||||
try:
|
||||
payload = await request.json()
|
||||
except (ValueError, UnicodeError):
|
||||
raise invalid("Некорректный JSON") from None
|
||||
|
||||
allowed_fields = expected_fields | (optional_fields or set())
|
||||
|
||||
if (
|
||||
not isinstance(payload, dict)
|
||||
or not expected_fields.issubset(payload)
|
||||
or not set(payload).issubset(allowed_fields)
|
||||
):
|
||||
raise invalid("Некорректные поля запроса")
|
||||
|
||||
return payload
|
||||
+341
@@ -0,0 +1,341 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import random
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from app.broker import QUEUE_NAME, connect_broker
|
||||
from app.job_db import connect_db
|
||||
from app.faults import trip_fault
|
||||
|
||||
|
||||
logger = logging.getLogger("messenger.worker")
|
||||
|
||||
WORKER_CONCURRENCY = int(os.getenv("WORKER_CONCURRENCY", "1"))
|
||||
if WORKER_CONCURRENCY != 1:
|
||||
raise ValueError("This worker supports WORKER_CONCURRENCY=1 only")
|
||||
|
||||
JOB_TIMEOUT = int(os.getenv("JOB_TIMEOUT_SECONDS", "60"))
|
||||
LEASE_SECONDS = int(os.getenv("JOB_LEASE_SECONDS", "120"))
|
||||
|
||||
if JOB_TIMEOUT < 1 or LEASE_SECONDS < JOB_TIMEOUT + 15:
|
||||
raise ValueError("Lease must exceed job timeout by at least 15 seconds")
|
||||
|
||||
|
||||
def claim_job(payload: dict):
|
||||
job_id = UUID(payload["job_id"])
|
||||
attachment_id = UUID(payload["attachment_id"])
|
||||
attempt = payload["attempt"]
|
||||
version = payload["pipeline_version"]
|
||||
|
||||
if (
|
||||
payload.get("schema_version") != 1
|
||||
or payload.get("type") != "attachment.process.v1"
|
||||
or type(attempt) is not int
|
||||
or attempt < 1
|
||||
or type(version) is not int
|
||||
or version < 1
|
||||
):
|
||||
raise ValueError("Invalid job message")
|
||||
|
||||
token = uuid4()
|
||||
|
||||
with connect_db() as db:
|
||||
with db.transaction():
|
||||
row = db.execute(
|
||||
"""
|
||||
SELECT j.id, j.status, j.attempt,
|
||||
j.attachment_id, j.pipeline_version,
|
||||
a.status AS attachment_status,
|
||||
a.pipeline_version AS current_version
|
||||
FROM attachment_jobs AS j
|
||||
JOIN attachments AS a ON a.id = j.attachment_id
|
||||
WHERE j.id = %s
|
||||
FOR UPDATE OF j, a
|
||||
""",
|
||||
(job_id,),
|
||||
).fetchone()
|
||||
|
||||
if (
|
||||
row is None
|
||||
or row["attachment_id"] != attachment_id
|
||||
or row["pipeline_version"] != version
|
||||
or row["current_version"] != version
|
||||
or row["attempt"] != attempt
|
||||
or row["status"] != "queued"
|
||||
or row["attachment_status"] != "queued"
|
||||
):
|
||||
return None
|
||||
|
||||
db.execute(
|
||||
"""
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'processing',
|
||||
lease_token = %s,
|
||||
lease_until = clock_timestamp()
|
||||
+ make_interval(secs => %s),
|
||||
updated_at = clock_timestamp()
|
||||
WHERE id = %s
|
||||
""",
|
||||
(token, LEASE_SECONDS, job_id),
|
||||
)
|
||||
|
||||
db.execute(
|
||||
"""
|
||||
UPDATE attachments
|
||||
SET status = 'processing',
|
||||
updated_at = clock_timestamp()
|
||||
WHERE id = %s
|
||||
""",
|
||||
(attachment_id,),
|
||||
)
|
||||
|
||||
return job_id, token
|
||||
|
||||
|
||||
def finish_unsuccessful(job_id, token, *, permanent, error_code):
|
||||
with connect_db() as db:
|
||||
with db.transaction():
|
||||
row = db.execute(
|
||||
"""
|
||||
SELECT *
|
||||
FROM attachment_jobs
|
||||
WHERE id = %s
|
||||
FOR UPDATE
|
||||
""",
|
||||
(job_id,),
|
||||
).fetchone()
|
||||
|
||||
if (
|
||||
row is None
|
||||
or row["status"] != "processing"
|
||||
or row["lease_token"] != token
|
||||
):
|
||||
return
|
||||
|
||||
exhausted = row["attempt"] >= row["max_attempts"]
|
||||
terminal = permanent or exhausted
|
||||
status = "failed" if terminal else "queued"
|
||||
code = (
|
||||
error_code if permanent else "retry_exhausted"
|
||||
) if terminal else None
|
||||
|
||||
delay = random.uniform(
|
||||
0.5,
|
||||
min(30.0, 2.0 ** min(row["attempt"], 5)),
|
||||
)
|
||||
next_attempt = row["attempt"] if terminal else row["attempt"] + 1
|
||||
|
||||
db.execute(
|
||||
"""
|
||||
UPDATE attachment_jobs
|
||||
SET status = %s,
|
||||
attempt = %s,
|
||||
error_code = %s,
|
||||
lease_token = NULL,
|
||||
lease_until = NULL,
|
||||
available_at = clock_timestamp()
|
||||
+ make_interval(secs => %s),
|
||||
updated_at = clock_timestamp()
|
||||
WHERE id = %s
|
||||
""",
|
||||
(status, next_attempt, code, delay, job_id),
|
||||
)
|
||||
|
||||
db.execute(
|
||||
"""
|
||||
UPDATE attachments
|
||||
SET status = %s,
|
||||
error_code = %s,
|
||||
updated_at = clock_timestamp()
|
||||
WHERE id = %s
|
||||
AND pipeline_version = %s
|
||||
AND status <> 'ready'
|
||||
""",
|
||||
(
|
||||
status, code,
|
||||
row["attachment_id"], row["pipeline_version"],
|
||||
),
|
||||
)
|
||||
|
||||
if not terminal:
|
||||
db.execute(
|
||||
"""
|
||||
INSERT INTO attachment_outbox (
|
||||
id, job_id, attempt, available_at
|
||||
)
|
||||
VALUES (
|
||||
%s, %s, %s,
|
||||
clock_timestamp() + make_interval(secs => %s)
|
||||
)
|
||||
""",
|
||||
(uuid4(), job_id, next_attempt, delay),
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"job transition job_id=%s status=%s reason=%s",
|
||||
job_id, status, error_code,
|
||||
)
|
||||
|
||||
|
||||
def recover_expired():
|
||||
with connect_db() as db:
|
||||
rows = db.execute(
|
||||
"""
|
||||
SELECT id, lease_token
|
||||
FROM attachment_jobs
|
||||
WHERE status = 'processing'
|
||||
AND lease_until < clock_timestamp()
|
||||
ORDER BY lease_until
|
||||
LIMIT 10
|
||||
"""
|
||||
).fetchall()
|
||||
|
||||
for row in rows:
|
||||
finish_unsuccessful(
|
||||
row["id"],
|
||||
row["lease_token"],
|
||||
permanent=False,
|
||||
error_code="lease_expired",
|
||||
)
|
||||
|
||||
|
||||
def run_child(broker, job_id, token):
|
||||
process = subprocess.Popen(
|
||||
[
|
||||
sys.executable,
|
||||
"-m",
|
||||
"app.job_runner",
|
||||
str(job_id),
|
||||
str(token),
|
||||
],
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.DEVNULL,
|
||||
text=True,
|
||||
)
|
||||
deadline = time.monotonic() + JOB_TIMEOUT
|
||||
|
||||
try:
|
||||
while process.poll() is None:
|
||||
if time.monotonic() >= deadline:
|
||||
process.kill()
|
||||
process.communicate()
|
||||
return {"outcome": "retry", "error_code": "processing_timeout"}
|
||||
|
||||
broker.process_data_events(time_limit=0.2)
|
||||
|
||||
output, _ = process.communicate()
|
||||
|
||||
if process.returncode == 75:
|
||||
logger.warning(
|
||||
"child_fault job_id=%s stage=after_objects exit_code=75",
|
||||
job_id,
|
||||
)
|
||||
os._exit(75)
|
||||
|
||||
if process.returncode != 0:
|
||||
return {"outcome": "retry", "error_code": "processing_failed"}
|
||||
|
||||
result = json.loads(output)
|
||||
if result.get("outcome") not in ("ready", "stale", "failed", "retry"):
|
||||
raise ValueError("Invalid child outcome")
|
||||
return result
|
||||
|
||||
finally:
|
||||
if process.poll() is None:
|
||||
process.kill()
|
||||
process.communicate()
|
||||
|
||||
|
||||
def main():
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s %(levelname)s %(message)s",
|
||||
)
|
||||
|
||||
while True:
|
||||
broker = None
|
||||
try:
|
||||
broker, channel = connect_broker()
|
||||
channel.basic_qos(prefetch_count=1)
|
||||
logger.info("worker connected")
|
||||
|
||||
next_recovery = 0.0
|
||||
|
||||
while True:
|
||||
if time.monotonic() >= next_recovery:
|
||||
recover_expired()
|
||||
next_recovery = time.monotonic() + 5
|
||||
|
||||
method, properties, body = channel.basic_get(
|
||||
queue=QUEUE_NAME,
|
||||
auto_ack=False,
|
||||
)
|
||||
|
||||
if method is None:
|
||||
broker.process_data_events(time_limit=0.5)
|
||||
continue
|
||||
|
||||
try:
|
||||
if len(body) > 16_384:
|
||||
raise ValueError("Oversized job message")
|
||||
payload = json.loads(body)
|
||||
if not isinstance(payload, dict):
|
||||
raise ValueError("Job must be an object")
|
||||
claimed = claim_job(payload)
|
||||
logger.info(
|
||||
"delivery job_id=%s redelivered=%s",
|
||||
payload["job_id"],
|
||||
method.redelivered,
|
||||
)
|
||||
except (ValueError, KeyError, TypeError):
|
||||
logger.warning("invalid broker message rejected")
|
||||
channel.basic_reject(
|
||||
delivery_tag=method.delivery_tag,
|
||||
requeue=False,
|
||||
)
|
||||
continue
|
||||
|
||||
if claimed is not None:
|
||||
job_id, token = claimed
|
||||
result = run_child(broker, job_id, token)
|
||||
|
||||
if result["outcome"] in ("failed", "retry"):
|
||||
finish_unsuccessful(
|
||||
job_id,
|
||||
token,
|
||||
permanent=result["outcome"] == "failed",
|
||||
error_code=result["error_code"],
|
||||
)
|
||||
|
||||
if claimed is not None and result["outcome"] == "ready":
|
||||
trip_fault(job_id, "before_ack")
|
||||
|
||||
channel.basic_ack(delivery_tag=method.delivery_tag)
|
||||
logger.info("ack job_id=%s", payload["job_id"])
|
||||
|
||||
except KeyboardInterrupt:
|
||||
return
|
||||
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"worker reconnect error_type=%s",
|
||||
type(exc).__name__,
|
||||
)
|
||||
time.sleep(2)
|
||||
|
||||
finally:
|
||||
if broker is not None and broker.is_open:
|
||||
try:
|
||||
broker.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,112 @@
|
||||
x-job-environment: &job-environment
|
||||
PGHOST: postgres
|
||||
PGPORT: "5432"
|
||||
PGDATABASE: messenger
|
||||
PGUSER: messenger
|
||||
PGPASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}
|
||||
BROKER_URL_FILE: /run/secrets/broker_url
|
||||
|
||||
services:
|
||||
broker:
|
||||
image: rabbitmq:4.0.5-management
|
||||
hostname: broker
|
||||
environment:
|
||||
RABBITMQ_DEFAULT_USER: messenger
|
||||
RABBITMQ_DEFAULT_PASS: ${RABBITMQ_PASSWORD:?Set RABBITMQ_PASSWORD in .env}
|
||||
volumes:
|
||||
- broker_data:/var/lib/rabbitmq
|
||||
healthcheck:
|
||||
test:
|
||||
- CMD-SHELL
|
||||
- rabbitmq-diagnostics -q check_running && rabbitmq-diagnostics -q check_port_listener 5672
|
||||
interval: 20s
|
||||
timeout: 30s
|
||||
retries: 3
|
||||
start_period: 40s
|
||||
cpus: "0.75"
|
||||
mem_limit: 512m
|
||||
stop_grace_period: 30s
|
||||
restart: unless-stopped
|
||||
|
||||
publisher:
|
||||
build:
|
||||
context: .
|
||||
image: mtusi-messenger:lab5
|
||||
command: ["python", "-m", "app.publisher"]
|
||||
environment: *job-environment
|
||||
secrets:
|
||||
- broker_url
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
broker:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
disable: true
|
||||
cpus: "0.25"
|
||||
mem_limit: 192m
|
||||
stop_grace_period: 10s
|
||||
restart: unless-stopped
|
||||
|
||||
worker:
|
||||
build:
|
||||
context: .
|
||||
image: mtusi-messenger:lab5
|
||||
command: ["python", "-m", "app.worker"]
|
||||
environment:
|
||||
<<: *job-environment
|
||||
S3_ENDPOINT: http://s3:9000
|
||||
S3_REGION: us-east-1
|
||||
S3_BUCKET: messenger-attachments
|
||||
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
|
||||
JOB_TIMEOUT_SECONDS: "${JOB_TIMEOUT_SECONDS:-60}"
|
||||
JOB_LEASE_SECONDS: "${JOB_LEASE_SECONDS:-120}"
|
||||
JOB_MEMORY_MB: "${JOB_MEMORY_MB:-512}"
|
||||
WORKER_CONCURRENCY: "${WORKER_CONCURRENCY:-1}"
|
||||
JOB_MAX_ATTEMPTS: "${JOB_MAX_ATTEMPTS:-5}"
|
||||
secrets:
|
||||
- broker_url
|
||||
- s3_access_key_id
|
||||
- s3_secret_access_key
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
broker:
|
||||
condition: service_healthy
|
||||
s3-check:
|
||||
condition: service_completed_successfully
|
||||
healthcheck:
|
||||
disable: true
|
||||
cpus: "1.0"
|
||||
mem_limit: 768m
|
||||
stop_grace_period: 10s
|
||||
restart: unless-stopped
|
||||
|
||||
api:
|
||||
environment:
|
||||
JOB_MAX_ATTEMPTS: "${JOB_MAX_ATTEMPTS:-5}"
|
||||
|
||||
# Старый cleanup пока запускается только явно:
|
||||
# уборку staging и результатов новых попыток доработаем отдельно.
|
||||
cleanup:
|
||||
command:
|
||||
- python
|
||||
- -m
|
||||
- app.cleanup_staging
|
||||
- --apply
|
||||
- --interval-seconds
|
||||
- "3600"
|
||||
environment:
|
||||
STAGING_RETENTION_SECONDS: "${STAGING_RETENTION_SECONDS:-86400}"
|
||||
|
||||
volumes:
|
||||
broker_data:
|
||||
|
||||
secrets:
|
||||
broker_url:
|
||||
file: ./secrets/broker_url
|
||||
+120
@@ -0,0 +1,120 @@
|
||||
services:
|
||||
s3:
|
||||
image: chrislusf/seaweedfs:4.44
|
||||
entrypoint: ["weed"]
|
||||
command:
|
||||
- mini
|
||||
- -dir=/data
|
||||
- -ip=s3
|
||||
- -ip.bind=0.0.0.0
|
||||
- -admin.ui=false
|
||||
- -bucket=messenger-attachments
|
||||
- -s3.port=9000
|
||||
- -s3.config=/run/secrets/s3_config
|
||||
- -s3.iam=false
|
||||
- -s3.port.iceberg=0
|
||||
- -s3.port.lance=0
|
||||
volumes:
|
||||
- s3_data:/data
|
||||
secrets:
|
||||
- s3_config
|
||||
cpus: "1.0"
|
||||
mem_limit: 768m
|
||||
stop_grace_period: 20s
|
||||
restart: unless-stopped
|
||||
|
||||
s3-check:
|
||||
build:
|
||||
context: .
|
||||
image: mtusi-messenger:lab5
|
||||
command: ["python", "-m", "app.s3_check"]
|
||||
environment:
|
||||
S3_ENDPOINT: http://s3:9000
|
||||
S3_REGION: us-east-1
|
||||
S3_BUCKET: messenger-attachments
|
||||
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
|
||||
secrets:
|
||||
- s3_access_key_id
|
||||
- s3_secret_access_key
|
||||
depends_on:
|
||||
s3:
|
||||
condition: service_started
|
||||
healthcheck:
|
||||
disable: true
|
||||
cpus: "0.5"
|
||||
mem_limit: 256m
|
||||
restart: "no"
|
||||
|
||||
migrate:
|
||||
image: mtusi-messenger:lab5
|
||||
|
||||
api:
|
||||
image: mtusi-messenger:lab5
|
||||
environment:
|
||||
S3_ENDPOINT: http://s3:9000
|
||||
S3_REGION: us-east-1
|
||||
S3_BUCKET: messenger-attachments
|
||||
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
|
||||
secrets:
|
||||
- s3_access_key_id
|
||||
- s3_secret_access_key
|
||||
depends_on:
|
||||
s3-check:
|
||||
condition: service_completed_successfully
|
||||
|
||||
|
||||
cleanup:
|
||||
build:
|
||||
context: .
|
||||
image: mtusi-messenger:lab5
|
||||
command:
|
||||
- python
|
||||
- -m
|
||||
- app.cleanup_orphans
|
||||
- --apply
|
||||
- --grace-seconds
|
||||
- "86400"
|
||||
- --interval-seconds
|
||||
- "3600"
|
||||
environment:
|
||||
PGHOST: postgres
|
||||
PGPORT: "5432"
|
||||
PGDATABASE: messenger
|
||||
PGUSER: messenger
|
||||
PGPASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}
|
||||
S3_ENDPOINT: http://s3:9000
|
||||
S3_REGION: us-east-1
|
||||
S3_BUCKET: messenger-attachments
|
||||
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
|
||||
secrets:
|
||||
- s3_access_key_id
|
||||
- s3_secret_access_key
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
s3-check:
|
||||
condition: service_completed_successfully
|
||||
api:
|
||||
condition: service_started
|
||||
healthcheck:
|
||||
disable: true
|
||||
cpus: "0.25"
|
||||
mem_limit: 256m
|
||||
stop_grace_period: 30s
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
s3_data:
|
||||
|
||||
secrets:
|
||||
s3_config:
|
||||
file: ./secrets/s3_config.json
|
||||
s3_access_key_id:
|
||||
file: ./secrets/s3_access_key_id
|
||||
s3_secret_access_key:
|
||||
file: ./secrets/s3_secret_access_key
|
||||
+132
@@ -0,0 +1,132 @@
|
||||
name: mtusi2026lab5
|
||||
|
||||
x-db-environment: &db-environment
|
||||
PGHOST: postgres
|
||||
PGPORT: "5432"
|
||||
PGDATABASE: messenger
|
||||
PGUSER: messenger
|
||||
PGPASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16.4-bookworm
|
||||
environment:
|
||||
POSTGRES_DB: messenger
|
||||
POSTGRES_USER: messenger
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test:
|
||||
- CMD-SHELL
|
||||
- pg_isready -U "$$POSTGRES_USER" -d "$$POSTGRES_DB"
|
||||
interval: 2s
|
||||
timeout: 2s
|
||||
retries: 15
|
||||
start_period: 20s
|
||||
cpus: "1.0"
|
||||
mem_limit: 512m
|
||||
restart: unless-stopped
|
||||
|
||||
redis:
|
||||
image: redis:7.2.5-bookworm
|
||||
command:
|
||||
- redis-server
|
||||
- --appendonly
|
||||
- "yes"
|
||||
- --appendfsync
|
||||
- everysec
|
||||
- --maxmemory
|
||||
- 64mb
|
||||
- --maxmemory-policy
|
||||
- allkeys-lru
|
||||
volumes:
|
||||
- redis_data:/data
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 2s
|
||||
timeout: 2s
|
||||
retries: 10
|
||||
start_period: 5s
|
||||
cpus: "0.25"
|
||||
mem_limit: 128m
|
||||
restart: unless-stopped
|
||||
|
||||
migrate:
|
||||
build:
|
||||
context: .
|
||||
image: mtusi-messenger:lab5
|
||||
command: ["python", "-m", "app.migrate"]
|
||||
environment: *db-environment
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
disable: true
|
||||
cpus: "0.5"
|
||||
mem_limit: 256m
|
||||
restart: "no"
|
||||
|
||||
api:
|
||||
build:
|
||||
context: .
|
||||
image: mtusi-messenger:lab5
|
||||
command:
|
||||
- python
|
||||
- -m
|
||||
- uvicorn
|
||||
- app.main:app
|
||||
- --host
|
||||
- 0.0.0.0
|
||||
- --port
|
||||
- "8080"
|
||||
- --workers
|
||||
- "1"
|
||||
- --timeout-graceful-shutdown
|
||||
- "5"
|
||||
- --no-access-log
|
||||
environment:
|
||||
<<: *db-environment
|
||||
REDIS_HOST: redis
|
||||
REDIS_PORT: "6379"
|
||||
USER_CACHE_TTL_SECONDS: "${USER_CACHE_TTL_SECONDS:-60}"
|
||||
SESSION_TTL_SECONDS: "${SESSION_TTL_SECONDS:-3600}"
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
healthcheck:
|
||||
test:
|
||||
- CMD
|
||||
- python
|
||||
- -c
|
||||
- "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=2).read()"
|
||||
interval: 2s
|
||||
timeout: 3s
|
||||
retries: 3
|
||||
start_period: 10s
|
||||
deploy:
|
||||
replicas: 2
|
||||
cpus: "0.75"
|
||||
mem_limit: 512m
|
||||
stop_grace_period: 10s
|
||||
restart: unless-stopped
|
||||
|
||||
lb:
|
||||
image: haproxy:3.0.5-bookworm
|
||||
ports:
|
||||
- "127.0.0.1:18085:8080"
|
||||
volumes:
|
||||
- ./infra/haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
|
||||
depends_on:
|
||||
api:
|
||||
condition: service_started
|
||||
cpus: "0.25"
|
||||
mem_limit: 128m
|
||||
stop_grace_period: 10s
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
redis_data:
|
||||
@@ -0,0 +1,58 @@
|
||||
{
|
||||
"scenario": "worker crash after S3 writes, before result COMMIT",
|
||||
"attachment_id": "5f4ea8f0-0efb-4fa2-bd7a-d4c07fa3a469",
|
||||
"initial_job": {
|
||||
"job_id": "48d3eef7-6769-4bbd-902e-55e52b89f67a",
|
||||
"status": "queued",
|
||||
"attempt": 1,
|
||||
"outbox_count": 1,
|
||||
"published": false
|
||||
},
|
||||
"after_crash": {
|
||||
"job_id": "48d3eef7-6769-4bbd-902e-55e52b89f67a",
|
||||
"status": "processing",
|
||||
"attempt": 1,
|
||||
"lease_remaining_seconds": 116.606905,
|
||||
"fault_consumed": true,
|
||||
"variant_rows": 0
|
||||
},
|
||||
"objects_before_recovery": [
|
||||
"originals/5f4ea8f0-0efb-4fa2-bd7a-d4c07fa3a469",
|
||||
"variants/5f4ea8f0-0efb-4fa2-bd7a-d4c07fa3a469/v1/crop",
|
||||
"variants/5f4ea8f0-0efb-4fa2-bd7a-d4c07fa3a469/v1/optimized",
|
||||
"variants/5f4ea8f0-0efb-4fa2-bd7a-d4c07fa3a469/v1/thumbnail"
|
||||
],
|
||||
"confirmed_intermediate_state": {
|
||||
"job_id": "48d3eef7-6769-4bbd-902e-55e52b89f67a",
|
||||
"status": "processing",
|
||||
"attempt": 1,
|
||||
"lease_remaining_seconds": 111.482557,
|
||||
"fault_consumed": true,
|
||||
"variant_rows": 0
|
||||
},
|
||||
"final_job": {
|
||||
"job_id": "48d3eef7-6769-4bbd-902e-55e52b89f67a",
|
||||
"status": "ready",
|
||||
"attempt": 2,
|
||||
"outbox_count": 2,
|
||||
"published": true
|
||||
},
|
||||
"variant_rows": 3,
|
||||
"objects_after_recovery": [
|
||||
"originals/5f4ea8f0-0efb-4fa2-bd7a-d4c07fa3a469",
|
||||
"variants/5f4ea8f0-0efb-4fa2-bd7a-d4c07fa3a469/v1/crop",
|
||||
"variants/5f4ea8f0-0efb-4fa2-bd7a-d4c07fa3a469/v1/optimized",
|
||||
"variants/5f4ea8f0-0efb-4fa2-bd7a-d4c07fa3a469/v1/thumbnail"
|
||||
],
|
||||
"worker_events": [
|
||||
"worker-1 | 2026-11-17 14:49:11,362 INFO delivery job_id=48d3eef7-6769-4bbd-902e-55e52b89f67a redelivered=False",
|
||||
"worker-1 | 2026-11-17 14:49:14,573 WARNING child_fault job_id=48d3eef7-6769-4bbd-902e-55e52b89f67a stage=after_objects exit_code=75",
|
||||
"worker-1 | 2026-11-17 14:49:18,242 INFO delivery job_id=48d3eef7-6769-4bbd-902e-55e52b89f67a redelivered=True",
|
||||
"worker-1 | 2026-11-17 14:49:18,244 INFO ack job_id=48d3eef7-6769-4bbd-902e-55e52b89f67a",
|
||||
"worker-1 | 2026-11-17 14:51:14,782 INFO job transition job_id=48d3eef7-6769-4bbd-902e-55e52b89f67a status=queued reason=lease_expired",
|
||||
"worker-1 | 2026-11-17 14:51:17,341 INFO delivery job_id=48d3eef7-6769-4bbd-902e-55e52b89f67a redelivered=False",
|
||||
"worker-1 | 2026-11-17 14:51:20,984 INFO ack job_id=48d3eef7-6769-4bbd-902e-55e52b89f67a"
|
||||
],
|
||||
"elapsed_s": 135.979,
|
||||
"result": "PASS"
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
{
|
||||
"scenario": "worker crash after ready commit, before ack",
|
||||
"attachment_id": "821b61c9-7251-491b-ad51-c29870dd43ab",
|
||||
"job_id": "92915f04-938a-4253-823f-8c299eb87788",
|
||||
"initial_job": {
|
||||
"job_id": "92915f04-938a-4253-823f-8c299eb87788",
|
||||
"status": "queued",
|
||||
"attempt": 1,
|
||||
"outbox_count": 1,
|
||||
"published": false
|
||||
},
|
||||
"worker_before_fault": {
|
||||
"started_at": "2026-11-17T14:39:01.016850053Z",
|
||||
"restart_count": 0
|
||||
},
|
||||
"delivery_events": [
|
||||
"worker-1 | 2026-11-17 14:39:02,540 INFO delivery job_id=92915f04-938a-4253-823f-8c299eb87788 redelivered=False",
|
||||
"worker-1 | 2026-11-17 14:39:05,774 WARNING fault_injection job_id=92915f04-938a-4253-823f-8c299eb87788 stage=before_ack exit_code=75",
|
||||
"worker-1 | 2026-11-17 14:39:08,149 INFO delivery job_id=92915f04-938a-4253-823f-8c299eb87788 redelivered=True",
|
||||
"worker-1 | 2026-11-17 14:39:08,150 INFO ack job_id=92915f04-938a-4253-823f-8c299eb87788"
|
||||
],
|
||||
"worker_after_fault": {
|
||||
"started_at": "2026-11-17T14:39:06.865588951Z",
|
||||
"restart_count": 1
|
||||
},
|
||||
"final_job": {
|
||||
"job_id": "92915f04-938a-4253-823f-8c299eb87788",
|
||||
"status": "ready",
|
||||
"attempt": 1,
|
||||
"outbox_count": 1,
|
||||
"published": true
|
||||
},
|
||||
"fault": {
|
||||
"consumed": true
|
||||
},
|
||||
"variant_rows": 3,
|
||||
"final_object_keys": [
|
||||
"originals/821b61c9-7251-491b-ad51-c29870dd43ab",
|
||||
"variants/821b61c9-7251-491b-ad51-c29870dd43ab/v1/crop",
|
||||
"variants/821b61c9-7251-491b-ad51-c29870dd43ab/v1/optimized",
|
||||
"variants/821b61c9-7251-491b-ad51-c29870dd43ab/v1/thumbnail"
|
||||
],
|
||||
"result": "PASS"
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
test_busy_upload_lock_prevents_deletion (__main__.CleanupStagingTests.test_busy_upload_lock_prevents_deletion) ... ok
|
||||
test_expired_failed_dry_run_is_non_destructive (__main__.CleanupStagingTests.test_expired_failed_dry_run_is_non_destructive) ... ok
|
||||
test_expired_failed_staging_is_deleted_and_reference_cleared (__main__.CleanupStagingTests.test_expired_failed_staging_is_deleted_and_reference_cleared) ... ok
|
||||
test_old_orphan_is_deleted_without_attachment_update (__main__.CleanupStagingTests.test_old_orphan_is_deleted_without_attachment_update) ... ok
|
||||
test_processing_is_preserved_even_when_old (__main__.CleanupStagingTests.test_processing_is_preserved_even_when_old) ... ok
|
||||
test_queued_is_preserved_even_when_old (__main__.CleanupStagingTests.test_queued_is_preserved_even_when_old) ... ok
|
||||
test_recent_failed_state_preserves_old_object (__main__.CleanupStagingTests.test_recent_failed_state_preserves_old_object) ... ok
|
||||
test_recent_object_is_preserved (__main__.CleanupStagingTests.test_recent_object_is_preserved) ... ok
|
||||
|
||||
----------------------------------------------------------------------
|
||||
Ran 8 tests in 0.054s
|
||||
|
||||
OK
|
||||
@@ -0,0 +1,48 @@
|
||||
{
|
||||
"scenario": "failed staging integrity and audited replay",
|
||||
"attachment_id": "9bcd1157-a974-444f-bba6-ec4fc558d287",
|
||||
"job_id": "c48f14d0-586d-4c76-9b0e-a1b90e2e0cd8",
|
||||
"failed_job": {
|
||||
"job_id": "c48f14d0-586d-4c76-9b0e-a1b90e2e0cd8",
|
||||
"status": "failed",
|
||||
"attempt": 1,
|
||||
"outbox_count": 1,
|
||||
"published": true
|
||||
},
|
||||
"rejected_replay": {
|
||||
"result": "error",
|
||||
"error_type": "ValueError"
|
||||
},
|
||||
"dry_run": {
|
||||
"result": "dry_run",
|
||||
"job_id": "c48f14d0-586d-4c76-9b0e-a1b90e2e0cd8",
|
||||
"previous_attempt": 1,
|
||||
"new_attempt": 2,
|
||||
"new_max_attempts": 6,
|
||||
"previous_error_code": "invalid_image"
|
||||
},
|
||||
"applied_replay": {
|
||||
"result": "replayed",
|
||||
"job_id": "c48f14d0-586d-4c76-9b0e-a1b90e2e0cd8",
|
||||
"previous_attempt": 1,
|
||||
"new_attempt": 2,
|
||||
"new_max_attempts": 6,
|
||||
"previous_error_code": "invalid_image"
|
||||
},
|
||||
"final_job": {
|
||||
"job_id": "c48f14d0-586d-4c76-9b0e-a1b90e2e0cd8",
|
||||
"status": "ready",
|
||||
"attempt": 2,
|
||||
"outbox_count": 2,
|
||||
"published": true
|
||||
},
|
||||
"ready_replay": {
|
||||
"result": "noop",
|
||||
"job_id": "c48f14d0-586d-4c76-9b0e-a1b90e2e0cd8",
|
||||
"status": "ready",
|
||||
"reason": "job_is_not_current_failed_work"
|
||||
},
|
||||
"audit_rows": 1,
|
||||
"object_count": 4,
|
||||
"result": "PASS"
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:18085"
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||
{"observed_instances_before": 2, "successful_writes": 130, "failed_attempts": 2, "first_recovery_seconds": 2.089}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
@@ -0,0 +1,18 @@
|
||||
[
|
||||
{
|
||||
"container_id": "bfa59f929e063a129517dd173a77bd12a619f6e5bac20f0215f9fdafb92c300b",
|
||||
"status": "running",
|
||||
"running": true,
|
||||
"started_at": "2026-11-17T15:28:48.449382903Z",
|
||||
"finished_at": "2026-11-17T15:28:43.450087124Z",
|
||||
"exit_code": 0
|
||||
},
|
||||
{
|
||||
"container_id": "ff57dc533c533c0f3121e8c3377c2aa697cbd8da304adbebf4dce43b0a0998b8",
|
||||
"status": "exited",
|
||||
"running": false,
|
||||
"started_at": "2026-11-17T15:28:47.804844296Z",
|
||||
"finished_at": "2026-11-17T15:30:39.004319199Z",
|
||||
"exit_code": 0
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,11 @@
|
||||
test_contract_dimensions_and_original (test_image_pipeline.ImagePipelineTests.test_contract_dimensions_and_original) ... ok
|
||||
test_exif_orientation (test_image_pipeline.ImagePipelineTests.test_exif_orientation) ... ok
|
||||
test_invalid_input (test_image_pipeline.ImagePipelineTests.test_invalid_input) ... ok
|
||||
test_rounding_down (test_image_pipeline.ImagePipelineTests.test_rounding_down) ... ok
|
||||
test_small_image_not_upscaled_except_crop (test_image_pipeline.ImagePipelineTests.test_small_image_not_upscaled_except_crop) ... ok
|
||||
test_transparency_becomes_white (test_image_pipeline.ImagePipelineTests.test_transparency_becomes_white) ... ok
|
||||
|
||||
----------------------------------------------------------------------
|
||||
Ran 6 tests in 0.100s
|
||||
|
||||
OK
|
||||
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"scenario": "API crash after COMMIT while publisher is stopped",
|
||||
"attachment_id": "b5d15121-0129-4bf5-a249-2000aa16e911",
|
||||
"before_crash": {
|
||||
"job_id": "0010fe3a-2555-4ddf-a8da-9ce29abd9dd5",
|
||||
"status": "queued",
|
||||
"attempt": 1,
|
||||
"outbox_count": 1,
|
||||
"published": false
|
||||
},
|
||||
"after_api_kill": {
|
||||
"job_id": "0010fe3a-2555-4ddf-a8da-9ce29abd9dd5",
|
||||
"status": "queued",
|
||||
"attempt": 1,
|
||||
"outbox_count": 1,
|
||||
"published": false
|
||||
},
|
||||
"after_publisher_start": {
|
||||
"job_id": "0010fe3a-2555-4ddf-a8da-9ce29abd9dd5",
|
||||
"status": "queued",
|
||||
"attempt": 1,
|
||||
"outbox_count": 1,
|
||||
"published": true
|
||||
},
|
||||
"final": {
|
||||
"job_id": "0010fe3a-2555-4ddf-a8da-9ce29abd9dd5",
|
||||
"status": "ready",
|
||||
"attempt": 1,
|
||||
"outbox_count": 1,
|
||||
"published": true
|
||||
},
|
||||
"ready_after_worker_start_s": 4.038,
|
||||
"result": "PASS"
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://127.0.0.1:18085"
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"scenario": "queued upload survives API and broker restart",
|
||||
"attachment_id": "6876fe3c-1544-4e70-8011-b82c8b446ac9",
|
||||
"message_id": "244c0f7c-c17b-4c17-867c-47e5da0e799a",
|
||||
"job_before_restart": {
|
||||
"status": "queued",
|
||||
"attempt": 1,
|
||||
"published": true
|
||||
},
|
||||
"queue_before_restart": {
|
||||
"name": "attachment.process.v1",
|
||||
"durable": true,
|
||||
"messages_ready": 1
|
||||
},
|
||||
"container_starts_before": {
|
||||
"ea6e598f8967caf02b45629157b7156d60952001b4c73647212bb0124e0094f2": "2026-11-17T13:52:02.655967981Z",
|
||||
"cd7a5131f262d163ca99084295bf4c291e971fb087af1dbd2020e13dd8e4d91f": "2026-11-17T13:52:02.172273085Z",
|
||||
"d8f9d07940fea84f297d8562c8a4721cc5eac6b9bd684b0ea1edf9f51ffed1fd": "2026-11-17T13:56:52.523576495Z"
|
||||
},
|
||||
"container_starts_after": {
|
||||
"ea6e598f8967caf02b45629157b7156d60952001b4c73647212bb0124e0094f2": "2026-11-17T13:57:33.504017667Z",
|
||||
"cd7a5131f262d163ca99084295bf4c291e971fb087af1dbd2020e13dd8e4d91f": "2026-11-17T13:57:32.869870682Z",
|
||||
"d8f9d07940fea84f297d8562c8a4721cc5eac6b9bd684b0ea1edf9f51ffed1fd": "2026-11-17T13:57:34.831360909Z"
|
||||
},
|
||||
"queue_after_restart": {
|
||||
"name": "attachment.process.v1",
|
||||
"durable": true,
|
||||
"messages_ready": 1
|
||||
},
|
||||
"ready_after_worker_start_s": 5.857,
|
||||
"original_sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
|
||||
"variants": [
|
||||
"crop",
|
||||
"optimized",
|
||||
"thumbnail"
|
||||
],
|
||||
"result": "PASS"
|
||||
}
|
||||
@@ -0,0 +1,163 @@
|
||||
{
|
||||
"before": [
|
||||
{
|
||||
"service": "api",
|
||||
"container_id": "bfa59f929e063a129517dd173a77bd12a619f6e5bac20f0215f9fdafb92c300b",
|
||||
"started_at": "2026-11-17T15:18:11.387840892Z",
|
||||
"status": "running",
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"service": "api",
|
||||
"container_id": "ff57dc533c533c0f3121e8c3377c2aa697cbd8da304adbebf4dce43b0a0998b8",
|
||||
"started_at": "2026-11-17T15:18:10.988865407Z",
|
||||
"status": "running",
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"service": "broker",
|
||||
"container_id": "26183862d5be1a9677d3981cdd4aa7ab41583720ea6ff51664b4b6ca1a30df54",
|
||||
"started_at": "2026-11-17T14:37:29.569999892Z",
|
||||
"status": "running",
|
||||
"volumes": [
|
||||
"mtusi2026lab5_broker_data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "cleanup",
|
||||
"container_id": "d29213849d9d711eb711621a965f2f6720c6c4351b10705f06a07969fb43bc2d",
|
||||
"started_at": "2026-11-17T15:22:37.758977743Z",
|
||||
"status": "running",
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"service": "lb",
|
||||
"container_id": "c7d31b9c6c1fb456d4b79b50521f00c6690e9fdf016fc53c9245f009296b242c",
|
||||
"started_at": "2026-11-17T13:52:03.2138386Z",
|
||||
"status": "running",
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"service": "postgres",
|
||||
"container_id": "ad1aa87a07ef37319bb04c6e1caa05c5d7e0455bab6a8156603cb93ae3ed1638",
|
||||
"started_at": "2026-11-17T13:51:56.606984187Z",
|
||||
"status": "running",
|
||||
"volumes": [
|
||||
"mtusi2026lab5_postgres_data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "publisher",
|
||||
"container_id": "123cc8ab2c6227e49096b6b41e3d59fc2b69d8e091a63e44b65662d72d13857b",
|
||||
"started_at": "2026-11-17T15:04:26.1288457Z",
|
||||
"status": "running",
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"service": "redis",
|
||||
"container_id": "bc013b9a224e5d3aae5a3afed9880af92db0921acee34e9187ea7467eb5cafd4",
|
||||
"started_at": "2026-11-17T13:51:56.57427508Z",
|
||||
"status": "running",
|
||||
"volumes": [
|
||||
"mtusi2026lab5_redis_data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "s3",
|
||||
"container_id": "4731679b480bde3edf68aaac640f2108a864efd437541891986480a404aefbf6",
|
||||
"started_at": "2026-11-17T13:51:56.614820161Z",
|
||||
"status": "running",
|
||||
"volumes": [
|
||||
"mtusi2026lab5_s3_data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "worker",
|
||||
"container_id": "168ec66f5667bb59a99acd51a13fb36fcabc8d726d935007b3d4a605a8f0e798",
|
||||
"started_at": "2026-11-17T15:18:59.941835146Z",
|
||||
"status": "running",
|
||||
"volumes": []
|
||||
}
|
||||
],
|
||||
"result": "HOOK_COMPLETED",
|
||||
"after": [
|
||||
{
|
||||
"service": "api",
|
||||
"container_id": "bfa59f929e063a129517dd173a77bd12a619f6e5bac20f0215f9fdafb92c300b",
|
||||
"started_at": "2026-11-17T15:28:48.449382903Z",
|
||||
"status": "running",
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"service": "api",
|
||||
"container_id": "ff57dc533c533c0f3121e8c3377c2aa697cbd8da304adbebf4dce43b0a0998b8",
|
||||
"started_at": "2026-11-17T15:28:47.804844296Z",
|
||||
"status": "running",
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"service": "broker",
|
||||
"container_id": "26183862d5be1a9677d3981cdd4aa7ab41583720ea6ff51664b4b6ca1a30df54",
|
||||
"started_at": "2026-11-17T15:28:45.809717435Z",
|
||||
"status": "running",
|
||||
"volumes": [
|
||||
"mtusi2026lab5_broker_data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "cleanup",
|
||||
"container_id": "fe25b5c8dff54d1283675dcb8a31136f727b83e1bedef3a073df68c9084bd67c",
|
||||
"started_at": "2026-11-17T15:28:49.052588548Z",
|
||||
"status": "running",
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"service": "lb",
|
||||
"container_id": "c7d31b9c6c1fb456d4b79b50521f00c6690e9fdf016fc53c9245f009296b242c",
|
||||
"started_at": "2026-11-17T15:28:49.152009495Z",
|
||||
"status": "running",
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"service": "postgres",
|
||||
"container_id": "ad1aa87a07ef37319bb04c6e1caa05c5d7e0455bab6a8156603cb93ae3ed1638",
|
||||
"started_at": "2026-11-17T15:28:44.877894569Z",
|
||||
"status": "running",
|
||||
"volumes": [
|
||||
"mtusi2026lab5_postgres_data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "publisher",
|
||||
"container_id": "8b583aba3eadce6ff307738c148164901325c1b763a7bfcd7a8cad1fbbc0df21",
|
||||
"started_at": "2026-11-17T15:28:47.928076896Z",
|
||||
"status": "running",
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"service": "redis",
|
||||
"container_id": "bc013b9a224e5d3aae5a3afed9880af92db0921acee34e9187ea7467eb5cafd4",
|
||||
"started_at": "2026-11-17T15:28:44.875553344Z",
|
||||
"status": "running",
|
||||
"volumes": [
|
||||
"mtusi2026lab5_redis_data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "s3",
|
||||
"container_id": "4731679b480bde3edf68aaac640f2108a864efd437541891986480a404aefbf6",
|
||||
"started_at": "2026-11-17T15:28:46.590899487Z",
|
||||
"status": "running",
|
||||
"volumes": [
|
||||
"mtusi2026lab5_s3_data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "worker",
|
||||
"container_id": "168ec66f5667bb59a99acd51a13fb36fcabc8d726d935007b3d4a605a8f0e798",
|
||||
"started_at": "2026-11-17T15:28:47.957685083Z",
|
||||
"status": "running",
|
||||
"volumes": []
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
{
|
||||
"scenario": "temporary S3 read failure exhausts bounded retries",
|
||||
"attachment_id": "aa0f40d2-b242-4f79-95d6-3dd758e5f094",
|
||||
"initial_job": {
|
||||
"job_id": "e01d1f5b-9e96-4d61-b7cc-84320920a6dc",
|
||||
"status": "queued",
|
||||
"attempt": 1,
|
||||
"max_attempts": 5,
|
||||
"error_code": null,
|
||||
"lease_cleared": true,
|
||||
"outbox": [
|
||||
{
|
||||
"attempt": 1,
|
||||
"published": false,
|
||||
"delay_seconds": 0.0
|
||||
}
|
||||
]
|
||||
},
|
||||
"observed_attempts": [
|
||||
1,
|
||||
2,
|
||||
3,
|
||||
4,
|
||||
5
|
||||
],
|
||||
"final_job": {
|
||||
"job_id": "e01d1f5b-9e96-4d61-b7cc-84320920a6dc",
|
||||
"status": "failed",
|
||||
"attempt": 5,
|
||||
"max_attempts": 5,
|
||||
"error_code": "retry_exhausted",
|
||||
"lease_cleared": true,
|
||||
"outbox": [
|
||||
{
|
||||
"attempt": 1,
|
||||
"published": true,
|
||||
"delay_seconds": 0.0
|
||||
},
|
||||
{
|
||||
"attempt": 2,
|
||||
"published": true,
|
||||
"delay_seconds": 1.238091
|
||||
},
|
||||
{
|
||||
"attempt": 3,
|
||||
"published": true,
|
||||
"delay_seconds": 0.55709
|
||||
},
|
||||
{
|
||||
"attempt": 4,
|
||||
"published": true,
|
||||
"delay_seconds": 3.659677
|
||||
},
|
||||
{
|
||||
"attempt": 5,
|
||||
"published": true,
|
||||
"delay_seconds": 4.988969
|
||||
}
|
||||
]
|
||||
},
|
||||
"terminal_observation_seconds": 5,
|
||||
"elapsed_s": 26.369,
|
||||
"result": "PASS"
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
python3 tests/smoke.py --base-url "http://127.0.0.1:18085"
|
||||
PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.
|
||||
@@ -0,0 +1,62 @@
|
||||
{
|
||||
"scenario": "scoped staging cleanup safety",
|
||||
"queued_dry_run": {
|
||||
"mode": "dry-run",
|
||||
"grace_seconds": 0,
|
||||
"attachment_id": "b8f27860-a4b8-49e4-a2a7-34a151b9c5bb",
|
||||
"scanned": 2,
|
||||
"ignored": 0,
|
||||
"fresh": 0,
|
||||
"busy": 0,
|
||||
"active": 1,
|
||||
"gone": 0,
|
||||
"candidates": 1,
|
||||
"deleted": 0
|
||||
},
|
||||
"queued_apply": {
|
||||
"mode": "apply",
|
||||
"grace_seconds": 0,
|
||||
"attachment_id": "b8f27860-a4b8-49e4-a2a7-34a151b9c5bb",
|
||||
"scanned": 2,
|
||||
"ignored": 0,
|
||||
"fresh": 0,
|
||||
"busy": 0,
|
||||
"active": 1,
|
||||
"gone": 0,
|
||||
"candidates": 1,
|
||||
"deleted": 1
|
||||
},
|
||||
"attachment_id": "b8f27860-a4b8-49e4-a2a7-34a151b9c5bb",
|
||||
"ready_dry_run": {
|
||||
"mode": "dry-run",
|
||||
"grace_seconds": 0,
|
||||
"attachment_id": "b8f27860-a4b8-49e4-a2a7-34a151b9c5bb",
|
||||
"scanned": 1,
|
||||
"ignored": 0,
|
||||
"fresh": 0,
|
||||
"busy": 0,
|
||||
"active": 0,
|
||||
"gone": 0,
|
||||
"candidates": 1,
|
||||
"deleted": 0
|
||||
},
|
||||
"ready_apply": {
|
||||
"mode": "apply",
|
||||
"grace_seconds": 0,
|
||||
"attachment_id": "b8f27860-a4b8-49e4-a2a7-34a151b9c5bb",
|
||||
"scanned": 1,
|
||||
"ignored": 0,
|
||||
"fresh": 0,
|
||||
"busy": 0,
|
||||
"active": 0,
|
||||
"gone": 0,
|
||||
"candidates": 1,
|
||||
"deleted": 1
|
||||
},
|
||||
"final_storage_state": {
|
||||
"keys": [],
|
||||
"status": "ready",
|
||||
"staging_reference": false
|
||||
},
|
||||
"result": "PASS"
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
global
|
||||
log stdout format raw local0
|
||||
maxconn 512
|
||||
|
||||
defaults
|
||||
log global
|
||||
mode http
|
||||
option httplog
|
||||
option dontlognull
|
||||
timeout connect 1s
|
||||
timeout client 30s
|
||||
timeout server 30s
|
||||
timeout http-request 10s
|
||||
timeout http-keep-alive 5s
|
||||
timeout queue 3s
|
||||
timeout check 1s
|
||||
retries 0
|
||||
|
||||
resolvers docker
|
||||
nameserver dns 127.0.0.11:53
|
||||
accepted_payload_size 8192
|
||||
resolve_retries 3
|
||||
timeout resolve 1s
|
||||
timeout retry 1s
|
||||
hold valid 1s
|
||||
hold obsolete 1s
|
||||
hold nx 1s
|
||||
hold timeout 1s
|
||||
hold refused 1s
|
||||
hold other 1s
|
||||
|
||||
frontend public_api
|
||||
bind :8080
|
||||
default_backend api_replicas
|
||||
|
||||
backend api_replicas
|
||||
balance roundrobin
|
||||
option httpchk
|
||||
http-check send meth GET uri /health/ready ver HTTP/1.1 hdr Host localhost
|
||||
http-check expect status 200
|
||||
default-server inter 1s fastinter 500ms downinter 1s fall 2 rise 1
|
||||
server-template api 1-4 api:8080 check resolvers docker resolve-prefer ipv4 init-addr none
|
||||
@@ -0,0 +1,71 @@
|
||||
CREATE TABLE users (
|
||||
id UUID PRIMARY KEY,
|
||||
username TEXT NOT NULL,
|
||||
display_name TEXT NOT NULL,
|
||||
password_hash TEXT NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
|
||||
|
||||
CONSTRAINT users_username_unique UNIQUE (username),
|
||||
CONSTRAINT users_username_format
|
||||
CHECK (username ~ '^[a-z0-9_]{3,32}$'),
|
||||
CONSTRAINT users_display_name_length
|
||||
CHECK (char_length(display_name) BETWEEN 1 AND 100)
|
||||
);
|
||||
|
||||
CREATE TABLE chats (
|
||||
id UUID PRIMARY KEY,
|
||||
title TEXT NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
|
||||
|
||||
CONSTRAINT chats_title_length
|
||||
CHECK (char_length(title) BETWEEN 1 AND 100)
|
||||
);
|
||||
|
||||
CREATE TABLE chat_members (
|
||||
chat_id UUID NOT NULL REFERENCES chats(id),
|
||||
user_id UUID NOT NULL REFERENCES users(id),
|
||||
|
||||
PRIMARY KEY (chat_id, user_id)
|
||||
);
|
||||
|
||||
CREATE INDEX chat_members_user_chat_idx
|
||||
ON chat_members (user_id, chat_id);
|
||||
|
||||
CREATE INDEX chats_created_id_idx
|
||||
ON chats (created_at, id);
|
||||
|
||||
CREATE TABLE messages (
|
||||
id UUID PRIMARY KEY,
|
||||
chat_id UUID NOT NULL,
|
||||
sender_id UUID NOT NULL,
|
||||
text TEXT NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
|
||||
|
||||
CONSTRAINT messages_sender_membership_fk
|
||||
FOREIGN KEY (chat_id, sender_id)
|
||||
REFERENCES chat_members (chat_id, user_id),
|
||||
|
||||
CONSTRAINT messages_text_length
|
||||
CHECK (char_length(text) BETWEEN 1 AND 4000)
|
||||
);
|
||||
|
||||
CREATE INDEX messages_chat_created_id_idx
|
||||
ON messages (chat_id, created_at, id);
|
||||
|
||||
CREATE TABLE sessions (
|
||||
token_hash TEXT PRIMARY KEY,
|
||||
user_id UUID NOT NULL REFERENCES users(id),
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
|
||||
expires_at TIMESTAMPTZ NOT NULL,
|
||||
|
||||
CONSTRAINT sessions_token_hash_format
|
||||
CHECK (token_hash ~ '^[0-9a-f]{64}$'),
|
||||
CONSTRAINT sessions_expiry_order
|
||||
CHECK (expires_at > created_at)
|
||||
);
|
||||
|
||||
CREATE INDEX sessions_expires_at_idx
|
||||
ON sessions (expires_at);
|
||||
|
||||
CREATE INDEX sessions_user_id_idx
|
||||
ON sessions (user_id);
|
||||
@@ -0,0 +1,22 @@
|
||||
CREATE TABLE message_idempotency (
|
||||
user_id UUID NOT NULL REFERENCES users(id),
|
||||
chat_id UUID NOT NULL REFERENCES chats(id),
|
||||
http_method TEXT NOT NULL DEFAULT 'POST',
|
||||
idempotency_key TEXT NOT NULL,
|
||||
request_text TEXT NOT NULL,
|
||||
message_id UUID NOT NULL REFERENCES messages(id),
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
|
||||
|
||||
PRIMARY KEY (
|
||||
user_id, chat_id, http_method, idempotency_key
|
||||
),
|
||||
|
||||
CONSTRAINT message_idempotency_method
|
||||
CHECK (http_method = 'POST'),
|
||||
|
||||
CONSTRAINT message_idempotency_key_format
|
||||
CHECK (
|
||||
char_length(idempotency_key) BETWEEN 1 AND 128
|
||||
AND idempotency_key ~ '^[A-Za-z0-9._:-]+$'
|
||||
)
|
||||
);
|
||||
@@ -0,0 +1,67 @@
|
||||
CREATE TABLE attachments (
|
||||
id UUID PRIMARY KEY,
|
||||
chat_id UUID NOT NULL REFERENCES chats(id),
|
||||
owner_id UUID NOT NULL REFERENCES users(id),
|
||||
filename TEXT NOT NULL,
|
||||
content_type TEXT NOT NULL,
|
||||
size_bytes BIGINT NOT NULL,
|
||||
sha256 TEXT NOT NULL,
|
||||
object_key TEXT NOT NULL UNIQUE,
|
||||
status TEXT NOT NULL DEFAULT 'ready',
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
|
||||
|
||||
CONSTRAINT attachments_filename_length
|
||||
CHECK (char_length(filename) BETWEEN 1 AND 255),
|
||||
|
||||
CONSTRAINT attachments_content_type
|
||||
CHECK (content_type IN ('image/png', 'image/jpeg')),
|
||||
|
||||
CONSTRAINT attachments_size_bytes
|
||||
CHECK (size_bytes BETWEEN 1 AND 10485760),
|
||||
|
||||
CONSTRAINT attachments_sha256_format
|
||||
CHECK (sha256 ~ '^[a-f0-9]{64}$'),
|
||||
|
||||
CONSTRAINT attachments_status
|
||||
CHECK (status = 'ready'),
|
||||
|
||||
CONSTRAINT attachments_owner_membership_fk
|
||||
FOREIGN KEY (chat_id, owner_id)
|
||||
REFERENCES chat_members (chat_id, user_id)
|
||||
);
|
||||
|
||||
CREATE INDEX attachments_chat_created_id_idx
|
||||
ON attachments (chat_id, created_at, id);
|
||||
|
||||
CREATE TABLE message_attachments (
|
||||
message_id UUID NOT NULL REFERENCES messages(id),
|
||||
attachment_id UUID NOT NULL REFERENCES attachments(id),
|
||||
position SMALLINT NOT NULL,
|
||||
|
||||
PRIMARY KEY (message_id, attachment_id),
|
||||
|
||||
CONSTRAINT message_attachments_position_unique
|
||||
UNIQUE (message_id, position),
|
||||
|
||||
CONSTRAINT message_attachments_position_range
|
||||
CHECK (position BETWEEN 0 AND 9)
|
||||
);
|
||||
|
||||
CREATE INDEX message_attachments_attachment_idx
|
||||
ON message_attachments (attachment_id);
|
||||
|
||||
ALTER TABLE messages
|
||||
DROP CONSTRAINT messages_text_length;
|
||||
|
||||
ALTER TABLE messages
|
||||
ADD CONSTRAINT messages_text_length
|
||||
CHECK (char_length(text) BETWEEN 0 AND 4000);
|
||||
|
||||
ALTER TABLE message_idempotency
|
||||
ADD COLUMN request_attachment_ids UUID[] NOT NULL
|
||||
DEFAULT ARRAY[]::UUID[];
|
||||
|
||||
ALTER TABLE message_idempotency
|
||||
ADD CONSTRAINT message_idempotency_attachment_count
|
||||
CHECK (cardinality(request_attachment_ids) <= 10);
|
||||
@@ -0,0 +1,101 @@
|
||||
ALTER TABLE attachments
|
||||
DROP CONSTRAINT attachments_status;
|
||||
|
||||
ALTER TABLE attachments
|
||||
ADD CONSTRAINT attachments_status
|
||||
CHECK (status IN ('queued', 'processing', 'ready', 'failed')),
|
||||
ADD COLUMN staging_key TEXT,
|
||||
ADD COLUMN pipeline_version INTEGER NOT NULL DEFAULT 1
|
||||
CHECK (pipeline_version >= 1),
|
||||
ADD COLUMN error_code TEXT;
|
||||
|
||||
ALTER TABLE attachments
|
||||
ADD CONSTRAINT attachments_error_state
|
||||
CHECK (
|
||||
(status = 'failed' AND error_code IS NOT NULL)
|
||||
OR
|
||||
(status <> 'failed' AND error_code IS NULL)
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX attachments_staging_key_unique
|
||||
ON attachments (staging_key)
|
||||
WHERE staging_key IS NOT NULL;
|
||||
|
||||
CREATE TABLE attachment_variants (
|
||||
attachment_id UUID NOT NULL REFERENCES attachments(id),
|
||||
pipeline_version INTEGER NOT NULL CHECK (pipeline_version >= 1),
|
||||
name TEXT NOT NULL CHECK (name IN ('thumbnail', 'crop', 'optimized')),
|
||||
object_key TEXT NOT NULL UNIQUE,
|
||||
content_type TEXT NOT NULL
|
||||
CHECK (content_type IN ('image/jpeg', 'image/png', 'image/webp')),
|
||||
size_bytes BIGINT NOT NULL CHECK (size_bytes > 0),
|
||||
width INTEGER NOT NULL CHECK (width > 0),
|
||||
height INTEGER NOT NULL CHECK (height > 0),
|
||||
sha256 TEXT NOT NULL CHECK (sha256 ~ '^[a-f0-9]{64}$'),
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
|
||||
|
||||
PRIMARY KEY (attachment_id, pipeline_version, name)
|
||||
);
|
||||
|
||||
CREATE TABLE attachment_jobs (
|
||||
id UUID PRIMARY KEY,
|
||||
attachment_id UUID NOT NULL REFERENCES attachments(id),
|
||||
pipeline_version INTEGER NOT NULL CHECK (pipeline_version >= 1),
|
||||
status TEXT NOT NULL DEFAULT 'queued'
|
||||
CHECK (status IN ('queued', 'processing', 'ready', 'failed')),
|
||||
attempt INTEGER NOT NULL DEFAULT 1 CHECK (attempt >= 1),
|
||||
max_attempts INTEGER NOT NULL CHECK (max_attempts >= 1),
|
||||
available_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
|
||||
lease_token UUID,
|
||||
lease_until TIMESTAMPTZ,
|
||||
error_code TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
|
||||
|
||||
UNIQUE (attachment_id, pipeline_version),
|
||||
|
||||
CHECK (attempt <= max_attempts),
|
||||
|
||||
CHECK (
|
||||
(
|
||||
status = 'processing'
|
||||
AND lease_token IS NOT NULL
|
||||
AND lease_until IS NOT NULL
|
||||
)
|
||||
OR
|
||||
(
|
||||
status <> 'processing'
|
||||
AND lease_token IS NULL
|
||||
AND lease_until IS NULL
|
||||
)
|
||||
),
|
||||
|
||||
CHECK (
|
||||
(status = 'failed' AND error_code IS NOT NULL)
|
||||
OR
|
||||
(status <> 'failed' AND error_code IS NULL)
|
||||
)
|
||||
);
|
||||
|
||||
CREATE INDEX attachment_jobs_recovery_idx
|
||||
ON attachment_jobs (lease_until)
|
||||
WHERE status = 'processing';
|
||||
|
||||
CREATE INDEX attachment_jobs_failed_idx
|
||||
ON attachment_jobs (updated_at, id)
|
||||
WHERE status = 'failed';
|
||||
|
||||
CREATE TABLE attachment_outbox (
|
||||
id UUID PRIMARY KEY,
|
||||
job_id UUID NOT NULL REFERENCES attachment_jobs(id),
|
||||
attempt INTEGER NOT NULL CHECK (attempt >= 1),
|
||||
available_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
|
||||
published_at TIMESTAMPTZ,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
|
||||
|
||||
UNIQUE (job_id, attempt)
|
||||
);
|
||||
|
||||
CREATE INDEX attachment_outbox_pending_idx
|
||||
ON attachment_outbox (available_at, id)
|
||||
WHERE published_at IS NULL;
|
||||
@@ -0,0 +1,8 @@
|
||||
CREATE TABLE pipeline_fault_injections (
|
||||
job_id UUID NOT NULL REFERENCES attachment_jobs(id),
|
||||
stage TEXT NOT NULL CHECK (stage IN ('before_ack')),
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
|
||||
consumed_at TIMESTAMPTZ,
|
||||
|
||||
PRIMARY KEY (job_id, stage)
|
||||
);
|
||||
@@ -0,0 +1,6 @@
|
||||
ALTER TABLE pipeline_fault_injections
|
||||
DROP CONSTRAINT pipeline_fault_injections_stage_check;
|
||||
|
||||
ALTER TABLE pipeline_fault_injections
|
||||
ADD CONSTRAINT pipeline_fault_injections_stage_check
|
||||
CHECK (stage IN ('before_ack', 'after_objects'));
|
||||
@@ -0,0 +1,13 @@
|
||||
CREATE TABLE attachment_job_replays (
|
||||
id UUID PRIMARY KEY,
|
||||
job_id UUID NOT NULL REFERENCES attachment_jobs(id),
|
||||
actor TEXT NOT NULL CHECK (char_length(actor) BETWEEN 1 AND 100),
|
||||
reason TEXT NOT NULL CHECK (char_length(reason) BETWEEN 1 AND 500),
|
||||
previous_attempt INTEGER NOT NULL CHECK (previous_attempt >= 1),
|
||||
new_attempt INTEGER NOT NULL,
|
||||
previous_error_code TEXT NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
|
||||
|
||||
CHECK (new_attempt = previous_attempt + 1),
|
||||
UNIQUE (job_id, new_attempt)
|
||||
);
|
||||
@@ -0,0 +1,9 @@
|
||||
fastapi==0.115.12
|
||||
uvicorn[standard]==0.34.2
|
||||
psycopg[binary]==3.2.6
|
||||
psycopg-pool==3.2.6
|
||||
redis==5.2.1
|
||||
argon2-cffi==23.1.0
|
||||
boto3==1.43.124
|
||||
Pillow==13.0.0
|
||||
pika==1.3.2
|
||||
Executable
+10
@@ -0,0 +1,10 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
exec docker compose \
|
||||
--project-name mtusi2026lab5 \
|
||||
--env-file .env \
|
||||
-f compose.yaml \
|
||||
-f compose.s3.yaml \
|
||||
-f compose.queue.yaml \
|
||||
"$@"
|
||||
Executable
+124
@@ -0,0 +1,124 @@
|
||||
#!/usr/bin/env python3
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import time
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
DC = str(ROOT / "scripts/dc")
|
||||
|
||||
STORES = ["postgres", "redis", "s3", "broker"]
|
||||
PROCESSES = ["api", "lb", "publisher", "worker", "cleanup"]
|
||||
SERVICES = STORES + PROCESSES
|
||||
|
||||
|
||||
def compose(*args, timeout=60):
|
||||
result = subprocess.run(
|
||||
[DC, *args],
|
||||
cwd=ROOT,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=timeout,
|
||||
)
|
||||
if result.returncode:
|
||||
raise RuntimeError("Compose failed: " + " ".join(args))
|
||||
return result.stdout.strip()
|
||||
|
||||
|
||||
def snapshot():
|
||||
ids = compose("ps", "-a", "-q", *SERVICES).split()
|
||||
if not ids:
|
||||
raise RuntimeError("No lab5 containers found")
|
||||
|
||||
result = subprocess.run(
|
||||
["docker", "inspect", *ids],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=15,
|
||||
)
|
||||
|
||||
records = []
|
||||
for item in json.loads(result.stdout):
|
||||
records.append({
|
||||
"service": item["Config"]["Labels"]["com.docker.compose.service"],
|
||||
"container_id": item["Id"],
|
||||
"started_at": item["State"]["StartedAt"],
|
||||
"status": item["State"]["Status"],
|
||||
"volumes": sorted(
|
||||
mount["Name"]
|
||||
for mount in item["Mounts"]
|
||||
if mount["Type"] == "volume"
|
||||
),
|
||||
})
|
||||
return sorted(records, key=lambda item: (item["service"], item["container_id"]))
|
||||
|
||||
|
||||
def main():
|
||||
report = {"before": snapshot(), "result": "INCOMPLETE"}
|
||||
try:
|
||||
print("Stopping API, LB and background processes...", flush=True)
|
||||
compose("stop", "--timeout", "5", *PROCESSES)
|
||||
|
||||
print("Restarting PostgreSQL, Redis, S3 and RabbitMQ...", flush=True)
|
||||
compose("restart", "--no-deps", "--timeout", "10", *STORES)
|
||||
|
||||
deadline = time.monotonic() + 20
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
compose(
|
||||
"exec", "-T", "postgres",
|
||||
"pg_isready", "-U", "messenger", "-d", "messenger",
|
||||
timeout=4,
|
||||
)
|
||||
break
|
||||
except (RuntimeError, subprocess.TimeoutExpired):
|
||||
time.sleep(1)
|
||||
else:
|
||||
raise RuntimeError("PostgreSQL readiness timeout")
|
||||
|
||||
print("Starting API, LB and background processes...", flush=True)
|
||||
compose("up", "-d", "--no-deps", "--no-build", *PROCESSES)
|
||||
|
||||
after = snapshot()
|
||||
report["after"] = after
|
||||
|
||||
for service in SERVICES:
|
||||
old = [item for item in report["before"] if item["service"] == service]
|
||||
new = [item for item in after if item["service"] == service]
|
||||
|
||||
if not old or len(old) != len(new):
|
||||
raise RuntimeError(f"Unexpected container count: {service}")
|
||||
|
||||
old_starts = {item["started_at"] for item in old}
|
||||
if any(item["started_at"] in old_starts for item in new):
|
||||
raise RuntimeError(f"Restart not observed: {service}")
|
||||
|
||||
for service in STORES:
|
||||
old_volumes = {
|
||||
volume
|
||||
for item in report["before"] if item["service"] == service
|
||||
for volume in item["volumes"]
|
||||
}
|
||||
new_volumes = {
|
||||
volume
|
||||
for item in after if item["service"] == service
|
||||
for volume in item["volumes"]
|
||||
}
|
||||
if not old_volumes or old_volumes != new_volumes:
|
||||
raise RuntimeError(f"Persistent volumes changed: {service}")
|
||||
|
||||
report["result"] = "HOOK_COMPLETED"
|
||||
print("Restart hook completed; persistent volume names preserved.", flush=True)
|
||||
|
||||
finally:
|
||||
path = ROOT / "evidence" / "restart_containers.json"
|
||||
path.parent.mkdir(exist_ok=True)
|
||||
path.write_text(json.dumps(report, indent=2) + "\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,67 @@
|
||||
from pathlib import Path
|
||||
import json
|
||||
import secrets
|
||||
|
||||
|
||||
root = Path(__file__).resolve().parent.parent
|
||||
directory = root / "secrets"
|
||||
|
||||
targets = [
|
||||
root / ".env",
|
||||
directory / "s3_config.json",
|
||||
directory / "s3_access_key_id",
|
||||
directory / "s3_secret_access_key",
|
||||
directory / "broker_url",
|
||||
]
|
||||
|
||||
existing = [path for path in targets if path.exists()]
|
||||
if existing:
|
||||
raise SystemExit(
|
||||
"STOP: local configuration already exists; "
|
||||
"nothing changed. Do not delete or rotate it blindly."
|
||||
)
|
||||
|
||||
directory.mkdir(mode=0o700, exist_ok=True)
|
||||
directory.chmod(0o700)
|
||||
|
||||
postgres_password = secrets.token_hex(24)
|
||||
rabbit_password = secrets.token_hex(24)
|
||||
access_key = "lab5" + secrets.token_hex(12)
|
||||
secret_key = secrets.token_hex(32)
|
||||
|
||||
config = {
|
||||
"identities": [
|
||||
{
|
||||
"name": "lab5",
|
||||
"credentials": [
|
||||
{
|
||||
"accessKey": access_key,
|
||||
"secretKey": secret_key,
|
||||
}
|
||||
],
|
||||
"actions": ["Admin", "Read", "Write", "List", "Tagging"],
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
contents = {
|
||||
root / ".env": (
|
||||
f"POSTGRES_PASSWORD={postgres_password}\n"
|
||||
f"RABBITMQ_PASSWORD={rabbit_password}\n"
|
||||
"SESSION_TTL_SECONDS=3600\n"
|
||||
"USER_CACHE_TTL_SECONDS=60\n"
|
||||
),
|
||||
directory / "s3_config.json": json.dumps(config, indent=2) + "\n",
|
||||
directory / "s3_access_key_id": access_key + "\n",
|
||||
directory / "s3_secret_access_key": secret_key + "\n",
|
||||
directory / "broker_url": (
|
||||
f"amqp://messenger:{rabbit_password}@broker:5672/%2F\n"
|
||||
),
|
||||
}
|
||||
|
||||
for path, content in contents.items():
|
||||
with path.open("x") as stream:
|
||||
path.chmod(0o600 if path.name == ".env" else 0o644)
|
||||
stream.write(content)
|
||||
|
||||
print("Created local lab5 configuration; secret values were not printed.")
|
||||
@@ -0,0 +1,126 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
import secrets
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
DIRECTORY = ROOT / "secrets"
|
||||
BUCKET = "messenger-attachments"
|
||||
|
||||
KEY_FILES = (
|
||||
"s3_admin_access_key",
|
||||
"s3_admin_secret_key",
|
||||
"s3_access_key_id",
|
||||
"s3_secret_access_key",
|
||||
)
|
||||
|
||||
|
||||
def configuration(values):
|
||||
return {
|
||||
"identities": [
|
||||
{
|
||||
"name": "bootstrap-admin",
|
||||
"credentials": [{
|
||||
"accessKey": values["s3_admin_access_key"],
|
||||
"secretKey": values["s3_admin_secret_key"],
|
||||
}],
|
||||
"actions": ["Admin", "Read", "Write", "List", "Tagging"],
|
||||
},
|
||||
{
|
||||
"name": "messenger-api",
|
||||
"credentials": [{
|
||||
"accessKey": values["s3_access_key_id"],
|
||||
"secretKey": values["s3_secret_access_key"],
|
||||
}],
|
||||
"actions": [
|
||||
f"Read:{BUCKET}",
|
||||
f"Write:{BUCKET}",
|
||||
f"List:{BUCKET}",
|
||||
],
|
||||
},
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
def ensure_ignores():
|
||||
patterns = (
|
||||
".env",
|
||||
"/secrets/",
|
||||
"*.bak",
|
||||
"__pycache__/",
|
||||
"*.py[cod]",
|
||||
)
|
||||
for filename in (".gitignore", ".dockerignore"):
|
||||
path = ROOT / filename
|
||||
text = path.read_text() if path.exists() else ""
|
||||
existing = set(text.splitlines())
|
||||
missing = [item for item in patterns if item not in existing]
|
||||
if missing:
|
||||
path.write_text(
|
||||
text.rstrip("\n") + "\n" + "\n".join(missing) + "\n"
|
||||
)
|
||||
|
||||
|
||||
def main():
|
||||
if DIRECTORY.is_symlink():
|
||||
raise SystemExit("Refusing a symlink at secrets/")
|
||||
|
||||
paths = [DIRECTORY / name for name in KEY_FILES]
|
||||
paths.append(DIRECTORY / "s3_config.json")
|
||||
present = [path.exists() for path in paths]
|
||||
|
||||
if any(present):
|
||||
if not all(path.is_file() and not path.is_symlink() for path in paths):
|
||||
raise SystemExit(
|
||||
"Incomplete or unsafe secret files; nothing regenerated"
|
||||
)
|
||||
|
||||
values = {
|
||||
name: (DIRECTORY / name).read_text().rstrip("\r\n")
|
||||
for name in KEY_FILES
|
||||
}
|
||||
if not all(values.values()):
|
||||
raise SystemExit("Empty credential file; nothing regenerated")
|
||||
|
||||
actual = json.loads(
|
||||
(DIRECTORY / "s3_config.json").read_text()
|
||||
)
|
||||
if actual != configuration(values):
|
||||
raise SystemExit(
|
||||
"Credentials and S3 configuration differ; nothing regenerated"
|
||||
)
|
||||
message = "Existing credentials verified; keys unchanged."
|
||||
else:
|
||||
DIRECTORY.mkdir(mode=0o700, exist_ok=True)
|
||||
DIRECTORY.chmod(0o700)
|
||||
|
||||
values = {
|
||||
"s3_admin_access_key": "admin-" + secrets.token_hex(12),
|
||||
"s3_admin_secret_key": secrets.token_hex(32),
|
||||
"s3_access_key_id": "api-" + secrets.token_hex(12),
|
||||
"s3_secret_access_key": secrets.token_hex(32),
|
||||
}
|
||||
|
||||
for name, value in values.items():
|
||||
with (DIRECTORY / name).open("x") as stream:
|
||||
stream.write(value + "\n")
|
||||
|
||||
with (DIRECTORY / "s3_config.json").open("x") as stream:
|
||||
json.dump(configuration(values), stream, indent=2)
|
||||
stream.write("\n")
|
||||
|
||||
message = "New credentials created; values are not printed."
|
||||
|
||||
DIRECTORY.chmod(0o700)
|
||||
for path in paths:
|
||||
path.chmod(0o644)
|
||||
|
||||
ensure_ignores()
|
||||
print(message)
|
||||
print("Secrets, .env, Python cache and .bak files are excluded.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+66
@@ -0,0 +1,66 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
exec python3 - <<'PY'
|
||||
import os
|
||||
import subprocess
|
||||
|
||||
target = os.environ.get("TARGET_INSTANCE_ID")
|
||||
if not target:
|
||||
raise SystemExit("TARGET_INSTANCE_ID is required")
|
||||
|
||||
compose = ["./scripts/dc"]
|
||||
|
||||
containers = subprocess.check_output(
|
||||
compose + ["ps", "-q", "api"],
|
||||
text=True,
|
||||
timeout=10,
|
||||
).split()
|
||||
|
||||
if len(containers) < 2:
|
||||
raise SystemExit("At least two running API containers are required")
|
||||
|
||||
probe = (
|
||||
"import urllib.request; "
|
||||
"r=urllib.request.urlopen("
|
||||
"'http://127.0.0.1:8080/health/ready', timeout=2); "
|
||||
"print(r.headers['X-Instance-Id'])"
|
||||
)
|
||||
|
||||
matches = []
|
||||
for container in containers:
|
||||
instance = subprocess.check_output(
|
||||
["docker", "exec", container, "python", "-c", probe],
|
||||
text=True,
|
||||
timeout=4,
|
||||
).strip()
|
||||
|
||||
if instance == target:
|
||||
matches.append(container)
|
||||
|
||||
if len(matches) != 1:
|
||||
raise SystemExit("Target instance was not resolved uniquely")
|
||||
|
||||
container = matches[0]
|
||||
print("Stopping target API container:", container[:12], flush=True)
|
||||
|
||||
subprocess.run(
|
||||
["docker", "stop", "--time", "5", container],
|
||||
check=True,
|
||||
stdout=subprocess.DEVNULL,
|
||||
timeout=15,
|
||||
)
|
||||
|
||||
running = subprocess.check_output(
|
||||
["docker", "inspect", "-f", "{{.State.Running}}", container],
|
||||
text=True,
|
||||
timeout=5,
|
||||
).strip()
|
||||
|
||||
if running != "false":
|
||||
raise SystemExit("Target API container is still running")
|
||||
|
||||
print("Target API container is stopped", flush=True)
|
||||
PY
|
||||
@@ -0,0 +1,266 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
import json
|
||||
import secrets
|
||||
import time
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from queued_restart import ROOT, api, dc, http, wait_ready
|
||||
from outbox_gap import snapshot
|
||||
from before_ack import sql, object_keys
|
||||
|
||||
|
||||
EVIDENCE = ROOT / "evidence" / "after_objects.json"
|
||||
|
||||
|
||||
def intermediate_state(job_id):
|
||||
return json.loads(sql(f"""
|
||||
SELECT json_build_object(
|
||||
'job_id', j.id,
|
||||
'status', j.status,
|
||||
'attempt', j.attempt,
|
||||
'lease_remaining_seconds',
|
||||
extract(epoch FROM (j.lease_until - clock_timestamp())),
|
||||
'fault_consumed', f.consumed_at IS NOT NULL,
|
||||
'variant_rows', (
|
||||
SELECT count(*)
|
||||
FROM attachment_variants v
|
||||
WHERE v.attachment_id = j.attachment_id
|
||||
)
|
||||
)
|
||||
FROM attachment_jobs j
|
||||
JOIN pipeline_fault_injections f ON f.job_id = j.id
|
||||
WHERE j.id = '{job_id}'::uuid
|
||||
AND f.stage = 'after_objects';
|
||||
"""))
|
||||
|
||||
|
||||
def main():
|
||||
evidence = {
|
||||
"scenario": "worker crash after S3 writes, before result COMMIT",
|
||||
}
|
||||
job_id = None
|
||||
restore = False
|
||||
|
||||
try:
|
||||
wait_ready()
|
||||
for service in ("publisher", "worker"):
|
||||
if not dc("ps", "--status", "running", "-q", service):
|
||||
raise AssertionError(f"Start {service} before this test")
|
||||
|
||||
username = "obj_" + uuid4().hex[:16]
|
||||
password = secrets.token_hex(16)
|
||||
user = api(
|
||||
"POST", "/api/v1/users", 201,
|
||||
payload={
|
||||
"username": username,
|
||||
"display_name": "After objects test",
|
||||
"password": password,
|
||||
},
|
||||
)
|
||||
basic = base64.b64encode(
|
||||
f"{username}:{password}".encode()
|
||||
).decode()
|
||||
token = api(
|
||||
"POST", "/api/v1/auth/sessions", 201,
|
||||
headers={"Authorization": "Basic " + basic},
|
||||
)["token"]
|
||||
|
||||
chat = api(
|
||||
"POST", "/api/v1/chats", 201,
|
||||
token=token,
|
||||
payload={
|
||||
"title": "After objects",
|
||||
"member_ids": [user["id"]],
|
||||
},
|
||||
)
|
||||
|
||||
restore = True
|
||||
dc("stop", "worker")
|
||||
if dc("ps", "--status", "running", "-q", "worker"):
|
||||
raise AssertionError("Worker is still running")
|
||||
|
||||
original = (ROOT / "tests/fixtures/sample.png").read_bytes()
|
||||
attachment = api(
|
||||
"POST",
|
||||
f"/api/v1/chats/{chat['id']}/attachments?filename=sample.png",
|
||||
202,
|
||||
token=token,
|
||||
data=original,
|
||||
headers={"Content-Type": "image/png"},
|
||||
)
|
||||
|
||||
attachment_id = str(UUID(attachment["id"]))
|
||||
path = f"/api/v1/attachments/{attachment_id}"
|
||||
initial = snapshot(attachment_id)
|
||||
job_id = str(UUID(initial["job_id"]))
|
||||
|
||||
sql(f"""
|
||||
INSERT INTO pipeline_fault_injections (job_id, stage)
|
||||
VALUES ('{job_id}'::uuid, 'after_objects');
|
||||
""")
|
||||
|
||||
evidence["attachment_id"] = attachment_id
|
||||
evidence["initial_job"] = initial
|
||||
print(f"Armed one-shot after_objects fault for job {job_id}")
|
||||
|
||||
dc("start", "worker")
|
||||
started = time.monotonic()
|
||||
|
||||
deadline = started + 45
|
||||
while time.monotonic() < deadline:
|
||||
middle = intermediate_state(job_id)
|
||||
if middle["fault_consumed"]:
|
||||
break
|
||||
time.sleep(0.5)
|
||||
else:
|
||||
raise AssertionError("after_objects fault was not consumed")
|
||||
|
||||
assert middle["status"] == "processing"
|
||||
assert middle["attempt"] == 1
|
||||
assert middle["lease_remaining_seconds"] > 0
|
||||
assert middle["variant_rows"] == 0
|
||||
evidence["after_crash"] = middle
|
||||
|
||||
current = api("GET", path, 200, token=token)
|
||||
assert current["status"] == "processing"
|
||||
assert current["variants"] == []
|
||||
api("GET", path + "/content", 409, token=token)
|
||||
|
||||
expected_keys = sorted([
|
||||
f"originals/{attachment_id}",
|
||||
f"variants/{attachment_id}/v1/thumbnail",
|
||||
f"variants/{attachment_id}/v1/crop",
|
||||
f"variants/{attachment_id}/v1/optimized",
|
||||
])
|
||||
|
||||
# Во время автоматического рестарта docker exec может быть недоступен.
|
||||
deadline = time.monotonic() + 25
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
keys_before_recovery = object_keys(attachment_id)
|
||||
break
|
||||
except RuntimeError:
|
||||
time.sleep(1)
|
||||
else:
|
||||
raise AssertionError("Could not inspect S3 after worker restart")
|
||||
|
||||
assert keys_before_recovery == expected_keys
|
||||
|
||||
# Подтверждаем, что осмотр S3 всё ещё был до восстановления lease.
|
||||
confirmed_middle = intermediate_state(job_id)
|
||||
assert confirmed_middle["status"] == "processing"
|
||||
assert confirmed_middle["attempt"] == 1
|
||||
assert confirmed_middle["lease_remaining_seconds"] > 0
|
||||
assert confirmed_middle["variant_rows"] == 0
|
||||
|
||||
evidence["objects_before_recovery"] = keys_before_recovery
|
||||
evidence["confirmed_intermediate_state"] = confirmed_middle
|
||||
print(
|
||||
"Fault consumed: processing, attempt=1, active lease, "
|
||||
"S3 objects=4, variant rows=0, content=409"
|
||||
)
|
||||
print("Waiting for automatic lease recovery...")
|
||||
|
||||
deadline = started + 240
|
||||
while time.monotonic() < deadline:
|
||||
current = api("GET", path, 200, token=token)
|
||||
if current["status"] == "ready":
|
||||
break
|
||||
if current["status"] == "failed":
|
||||
raise AssertionError(
|
||||
"Attachment failed: " + str(current["error_code"])
|
||||
)
|
||||
time.sleep(1)
|
||||
else:
|
||||
raise AssertionError("Lease recovery did not reach ready")
|
||||
|
||||
final = snapshot(attachment_id)
|
||||
assert final["job_id"] == job_id
|
||||
assert final["status"] == "ready"
|
||||
assert final["attempt"] == 2
|
||||
assert final["outbox_count"] == 2
|
||||
assert current["error_code"] is None
|
||||
|
||||
status, body = http("GET", path + "/content", token=token)
|
||||
assert status == 200 and body == original
|
||||
|
||||
variants = {item["name"]: item for item in current["variants"]}
|
||||
assert len(current["variants"]) == 3
|
||||
assert set(variants) == {"thumbnail", "crop", "optimized"}
|
||||
|
||||
for name, metadata in variants.items():
|
||||
status, body = http(
|
||||
"GET", path + "/content?variant=" + name,
|
||||
token=token,
|
||||
)
|
||||
assert status == 200
|
||||
assert len(body) == metadata["size_bytes"]
|
||||
assert hashlib.sha256(body).hexdigest() == metadata["sha256"]
|
||||
|
||||
count = int(sql(f"""
|
||||
SELECT count(*)
|
||||
FROM attachment_variants
|
||||
WHERE attachment_id = '{attachment_id}'::uuid;
|
||||
"""))
|
||||
assert count == 3
|
||||
|
||||
keys_after_recovery = object_keys(attachment_id)
|
||||
assert keys_after_recovery == keys_before_recovery
|
||||
|
||||
logs = dc("logs", "--no-color", "--tail=400", "worker")
|
||||
events = [
|
||||
line for line in logs.splitlines()
|
||||
if job_id in line
|
||||
]
|
||||
assert any(
|
||||
"child_fault" in line and "stage=after_objects" in line
|
||||
for line in events
|
||||
)
|
||||
assert any(
|
||||
"status=queued" in line and "reason=lease_expired" in line
|
||||
for line in events
|
||||
)
|
||||
|
||||
evidence["final_job"] = final
|
||||
evidence["variant_rows"] = count
|
||||
evidence["objects_after_recovery"] = keys_after_recovery
|
||||
evidence["worker_events"] = events
|
||||
evidence["elapsed_s"] = round(time.monotonic() - started, 3)
|
||||
evidence["result"] = "PASS"
|
||||
|
||||
except BaseException as exc:
|
||||
evidence["result"] = "FAIL"
|
||||
evidence["error_type"] = type(exc).__name__
|
||||
raise
|
||||
|
||||
finally:
|
||||
try:
|
||||
if job_id is not None:
|
||||
sql(f"""
|
||||
UPDATE pipeline_fault_injections
|
||||
SET consumed_at = clock_timestamp()
|
||||
WHERE job_id = '{job_id}'::uuid
|
||||
AND stage = 'after_objects'
|
||||
AND consumed_at IS NULL;
|
||||
""")
|
||||
finally:
|
||||
try:
|
||||
if restore:
|
||||
dc("start", "worker")
|
||||
finally:
|
||||
EVIDENCE.parent.mkdir(exist_ok=True)
|
||||
EVIDENCE.write_text(
|
||||
json.dumps(evidence, ensure_ascii=False, indent=2) + "\n"
|
||||
)
|
||||
|
||||
print(
|
||||
"PASS: crash after S3 writes recovered via expired lease; "
|
||||
"same job, attempt=2, variants=3, objects=4"
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,262 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
import json
|
||||
import secrets
|
||||
import subprocess
|
||||
import time
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from queued_restart import ROOT, api, dc, http, wait_ready
|
||||
from outbox_gap import snapshot
|
||||
|
||||
|
||||
EVIDENCE = ROOT / "evidence" / "before_ack.json"
|
||||
|
||||
|
||||
def sql(query):
|
||||
return dc(
|
||||
"exec", "-T", "postgres",
|
||||
"psql", "-U", "messenger", "-d", "messenger",
|
||||
"-X", "-A", "-t", "-v", "ON_ERROR_STOP=1",
|
||||
"-c", query,
|
||||
)
|
||||
|
||||
|
||||
def worker_state():
|
||||
ids = dc("ps", "-a", "-q", "worker").split()
|
||||
if len(ids) != 1:
|
||||
raise AssertionError("This test expects exactly one worker container")
|
||||
|
||||
template = (
|
||||
'{"started_at":{{json .State.StartedAt}},'
|
||||
'"restart_count":{{.RestartCount}}}'
|
||||
)
|
||||
result = subprocess.run(
|
||||
["docker", "inspect", "--format", template, ids[0]],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
return json.loads(result.stdout)
|
||||
|
||||
|
||||
def object_keys(attachment_id):
|
||||
code = """
|
||||
import json
|
||||
import sys
|
||||
from uuid import UUID
|
||||
from app.storage import bucket_name, get_client
|
||||
|
||||
attachment_id = str(UUID(sys.argv[1]))
|
||||
client = get_client()
|
||||
keys = []
|
||||
for prefix in (
|
||||
f"originals/{attachment_id}",
|
||||
f"variants/{attachment_id}/",
|
||||
):
|
||||
for page in client.get_paginator("list_objects_v2").paginate(
|
||||
Bucket=bucket_name(), Prefix=prefix,
|
||||
):
|
||||
keys.extend(item["Key"] for item in page.get("Contents", []))
|
||||
print(json.dumps(sorted(keys)))
|
||||
"""
|
||||
return json.loads(
|
||||
dc("exec", "-T", "worker", "python", "-c", code, attachment_id)
|
||||
)
|
||||
|
||||
|
||||
def main():
|
||||
evidence = {"scenario": "worker crash after ready commit, before ack"}
|
||||
job_id = None
|
||||
restore = False
|
||||
|
||||
try:
|
||||
wait_ready()
|
||||
if not dc("ps", "--status", "running", "-q", "worker"):
|
||||
raise AssertionError("Start worker before this test")
|
||||
|
||||
username = "ack_" + uuid4().hex[:16]
|
||||
password = secrets.token_hex(16)
|
||||
user = api(
|
||||
"POST", "/api/v1/users", 201,
|
||||
payload={
|
||||
"username": username,
|
||||
"display_name": "Before ack test",
|
||||
"password": password,
|
||||
},
|
||||
)
|
||||
basic = base64.b64encode(
|
||||
f"{username}:{password}".encode()
|
||||
).decode()
|
||||
token = api(
|
||||
"POST", "/api/v1/auth/sessions", 201,
|
||||
headers={"Authorization": "Basic " + basic},
|
||||
)["token"]
|
||||
|
||||
chat = api(
|
||||
"POST", "/api/v1/chats", 201,
|
||||
token=token,
|
||||
payload={
|
||||
"title": "Before ack",
|
||||
"member_ids": [user["id"]],
|
||||
},
|
||||
)
|
||||
|
||||
restore = True
|
||||
dc("stop", "worker")
|
||||
if dc("ps", "--status", "running", "-q", "worker"):
|
||||
raise AssertionError("Worker is still running")
|
||||
|
||||
original = (ROOT / "tests/fixtures/sample.png").read_bytes()
|
||||
attachment = api(
|
||||
"POST",
|
||||
f"/api/v1/chats/{chat['id']}/attachments?filename=sample.png",
|
||||
202,
|
||||
token=token,
|
||||
data=original,
|
||||
headers={"Content-Type": "image/png"},
|
||||
)
|
||||
attachment_id = str(UUID(attachment["id"]))
|
||||
path = f"/api/v1/attachments/{attachment_id}"
|
||||
before = snapshot(attachment_id)
|
||||
job_id = str(UUID(before["job_id"]))
|
||||
|
||||
sql(f"""
|
||||
INSERT INTO pipeline_fault_injections (job_id, stage)
|
||||
VALUES ('{job_id}'::uuid, 'before_ack');
|
||||
""")
|
||||
|
||||
evidence["attachment_id"] = attachment_id
|
||||
evidence["job_id"] = job_id
|
||||
evidence["initial_job"] = before
|
||||
print(f"Armed one-shot before_ack fault for job {job_id}")
|
||||
|
||||
dc("start", "worker")
|
||||
started_state = worker_state()
|
||||
evidence["worker_before_fault"] = started_state
|
||||
|
||||
fault_line = f"fault_injection job_id={job_id} stage=before_ack"
|
||||
redelivery_line = f"delivery job_id={job_id} redelivered=True"
|
||||
ack_line = f"ack job_id={job_id}"
|
||||
|
||||
deadline = time.monotonic() + 120
|
||||
while time.monotonic() < deadline:
|
||||
logs = dc("logs", "--no-color", "--tail=300", "worker")
|
||||
fault_at = logs.find(fault_line)
|
||||
redelivery_at = logs.find(redelivery_line)
|
||||
ack_at = logs.find(ack_line)
|
||||
|
||||
if 0 <= fault_at < redelivery_at < ack_at:
|
||||
break
|
||||
time.sleep(1)
|
||||
else:
|
||||
raise AssertionError(
|
||||
"Expected fault -> redelivered=True -> ack was not observed"
|
||||
)
|
||||
|
||||
evidence["delivery_events"] = [
|
||||
line for line in logs.splitlines()
|
||||
if job_id in line
|
||||
]
|
||||
print("Observed: before_ack crash -> redelivered=True -> ack")
|
||||
|
||||
after_state = worker_state()
|
||||
assert (
|
||||
after_state["started_at"] != started_state["started_at"]
|
||||
or after_state["restart_count"] > started_state["restart_count"]
|
||||
)
|
||||
evidence["worker_after_fault"] = after_state
|
||||
|
||||
consumed = json.loads(sql(f"""
|
||||
SELECT json_build_object(
|
||||
'consumed', consumed_at IS NOT NULL
|
||||
)
|
||||
FROM pipeline_fault_injections
|
||||
WHERE job_id = '{job_id}'::uuid
|
||||
AND stage = 'before_ack';
|
||||
"""))
|
||||
assert consumed["consumed"] is True
|
||||
|
||||
final = snapshot(attachment_id)
|
||||
assert final["job_id"] == job_id
|
||||
assert final["status"] == "ready"
|
||||
assert final["attempt"] == 1
|
||||
assert final["outbox_count"] == 1
|
||||
|
||||
current = api("GET", path, 200, token=token)
|
||||
assert current["status"] == "ready"
|
||||
assert current["error_code"] is None
|
||||
|
||||
variants = {item["name"]: item for item in current["variants"]}
|
||||
assert len(current["variants"]) == 3
|
||||
assert set(variants) == {"thumbnail", "crop", "optimized"}
|
||||
|
||||
status, body = http("GET", path + "/content", token=token)
|
||||
assert status == 200 and body == original
|
||||
|
||||
for name, metadata in variants.items():
|
||||
status, body = http(
|
||||
"GET", path + "/content?variant=" + name,
|
||||
token=token,
|
||||
)
|
||||
assert status == 200
|
||||
assert len(body) == metadata["size_bytes"]
|
||||
assert hashlib.sha256(body).hexdigest() == metadata["sha256"]
|
||||
|
||||
variant_count = int(sql(f"""
|
||||
SELECT count(*)
|
||||
FROM attachment_variants
|
||||
WHERE attachment_id = '{attachment_id}'::uuid;
|
||||
"""))
|
||||
assert variant_count == 3
|
||||
|
||||
keys = object_keys(attachment_id)
|
||||
expected_keys = sorted([
|
||||
f"originals/{attachment_id}",
|
||||
f"variants/{attachment_id}/v1/thumbnail",
|
||||
f"variants/{attachment_id}/v1/crop",
|
||||
f"variants/{attachment_id}/v1/optimized",
|
||||
])
|
||||
assert keys == expected_keys
|
||||
|
||||
evidence["final_job"] = final
|
||||
evidence["fault"] = consumed
|
||||
evidence["variant_rows"] = variant_count
|
||||
evidence["final_object_keys"] = keys
|
||||
evidence["result"] = "PASS"
|
||||
|
||||
except BaseException as exc:
|
||||
evidence["result"] = "FAIL"
|
||||
evidence["error_type"] = type(exc).__name__
|
||||
raise
|
||||
|
||||
finally:
|
||||
try:
|
||||
if job_id is not None:
|
||||
sql(f"""
|
||||
UPDATE pipeline_fault_injections
|
||||
SET consumed_at = clock_timestamp()
|
||||
WHERE job_id = '{job_id}'::uuid
|
||||
AND stage = 'before_ack'
|
||||
AND consumed_at IS NULL;
|
||||
""")
|
||||
finally:
|
||||
try:
|
||||
if restore:
|
||||
dc("start", "worker")
|
||||
finally:
|
||||
EVIDENCE.parent.mkdir(exist_ok=True)
|
||||
EVIDENCE.write_text(
|
||||
json.dumps(evidence, ensure_ascii=False, indent=2) + "\n"
|
||||
)
|
||||
|
||||
print(
|
||||
"PASS: crash before ack caused broker redelivery; "
|
||||
"ready result preserved, attempt=1, variants=3, objects=4"
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,308 @@
|
||||
from collections import Counter
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
|
||||
from client import Client, arguments, check, run
|
||||
from resilience import history
|
||||
|
||||
|
||||
WORKERS = 12
|
||||
ROUNDS = 3
|
||||
|
||||
|
||||
def main():
|
||||
args = arguments("Concurrent cross-replica idempotency")
|
||||
|
||||
control = Client(
|
||||
args.base_url,
|
||||
args.ca_file,
|
||||
timeout=10,
|
||||
)
|
||||
alice, bob, eve, chat = control.bootstrap()
|
||||
path = f"/api/v1/chats/{chat['id']}/messages"
|
||||
|
||||
check(
|
||||
history(control, alice, chat) == [],
|
||||
"fresh test chat is not empty",
|
||||
)
|
||||
|
||||
clients = [
|
||||
Client(args.base_url, args.ca_file, timeout=10)
|
||||
for _ in range(WORKERS)
|
||||
]
|
||||
|
||||
saved_messages = []
|
||||
round_results = []
|
||||
first_key = None
|
||||
|
||||
for round_number in range(1, ROUNDS + 1):
|
||||
key = uuid.uuid4().hex
|
||||
text = f"Один ключ — одно сообщение: {round_number} {key}"
|
||||
|
||||
if first_key is None:
|
||||
first_key = key
|
||||
|
||||
barrier = threading.Barrier(WORKERS)
|
||||
|
||||
def send(index):
|
||||
if index % 3 == 0:
|
||||
body = {"text": text}
|
||||
elif index % 3 == 1:
|
||||
body = {
|
||||
"attachment_ids": [],
|
||||
"text": text,
|
||||
}
|
||||
else:
|
||||
body = {
|
||||
"text": text,
|
||||
"attachment_ids": [],
|
||||
}
|
||||
|
||||
raw = json.dumps(
|
||||
body,
|
||||
ensure_ascii=False,
|
||||
indent=2 if index % 2 else None,
|
||||
).encode("utf-8")
|
||||
|
||||
barrier.wait(timeout=15)
|
||||
|
||||
return clients[index].call(
|
||||
"POST",
|
||||
path,
|
||||
expected=201,
|
||||
actor=alice,
|
||||
raw=raw,
|
||||
headers={
|
||||
"Content-Type": "application/json",
|
||||
"Idempotency-Key": key,
|
||||
},
|
||||
schema="Message",
|
||||
)
|
||||
|
||||
started = time.monotonic()
|
||||
|
||||
with ThreadPoolExecutor(max_workers=WORKERS) as executor:
|
||||
responses = list(executor.map(send, range(WORKERS)))
|
||||
|
||||
elapsed = time.monotonic() - started
|
||||
|
||||
original = responses[0]["json"]
|
||||
|
||||
check(
|
||||
all(
|
||||
response["json"] == original
|
||||
for response in responses
|
||||
),
|
||||
"same key returned different Message objects",
|
||||
)
|
||||
|
||||
instances = Counter(
|
||||
response["headers"]["X-Instance-Id"]
|
||||
for response in responses
|
||||
)
|
||||
|
||||
check(
|
||||
len(instances) >= 2,
|
||||
"concurrent requests did not reach two replicas",
|
||||
)
|
||||
|
||||
messages = history(control, bob, chat)
|
||||
same_text = [
|
||||
message
|
||||
for message in messages
|
||||
if message["text"] == text
|
||||
]
|
||||
|
||||
check(
|
||||
same_text == [original],
|
||||
"concurrent repeats created zero or multiple messages",
|
||||
)
|
||||
check(
|
||||
len(messages) == round_number,
|
||||
"unexpected message count after concurrent round",
|
||||
)
|
||||
|
||||
saved_messages.append(original)
|
||||
|
||||
result = {
|
||||
"round": round_number,
|
||||
"requests": WORKERS,
|
||||
"responses_201": len(responses),
|
||||
"observed_instances": len(instances),
|
||||
"responses_by_instance": dict(instances),
|
||||
"unique_message_ids": len(
|
||||
{
|
||||
response["json"]["id"]
|
||||
for response in responses
|
||||
}
|
||||
),
|
||||
"matching_history_messages": len(same_text),
|
||||
"duration_seconds": round(elapsed, 3),
|
||||
}
|
||||
round_results.append(result)
|
||||
print(json.dumps(result, ensure_ascii=False))
|
||||
|
||||
conflict = control.call(
|
||||
"POST",
|
||||
path,
|
||||
expected=409,
|
||||
actor=alice,
|
||||
body={"text": "Другой текст с прежним ключом"},
|
||||
headers={"Idempotency-Key": first_key},
|
||||
)
|
||||
check(
|
||||
conflict["json"]["error"]["code"] == "conflict",
|
||||
"wrong error code for body conflict",
|
||||
)
|
||||
|
||||
control.call(
|
||||
"POST",
|
||||
path,
|
||||
expected=400,
|
||||
actor=alice,
|
||||
body={"text": "Без ключа"},
|
||||
)
|
||||
|
||||
for invalid_key in ("bad key", "x" * 129):
|
||||
control.call(
|
||||
"POST",
|
||||
path,
|
||||
expected=400,
|
||||
actor=alice,
|
||||
body={"text": "Некорректный ключ"},
|
||||
headers={"Idempotency-Key": invalid_key},
|
||||
)
|
||||
|
||||
control.call(
|
||||
"POST",
|
||||
path,
|
||||
expected=404,
|
||||
actor=eve,
|
||||
body={"text": saved_messages[0]["text"]},
|
||||
headers={"Idempotency-Key": first_key},
|
||||
)
|
||||
|
||||
bob_message = control.message(
|
||||
bob,
|
||||
chat,
|
||||
"Тот же ключ другого пользователя",
|
||||
key=first_key,
|
||||
)["json"]
|
||||
|
||||
check(
|
||||
bob_message["sender_id"] == bob["id"],
|
||||
"wrong sender for another user's scope",
|
||||
)
|
||||
check(
|
||||
bob_message["id"] != saved_messages[0]["id"],
|
||||
"key incorrectly shared between users",
|
||||
)
|
||||
|
||||
second_chat = control.call(
|
||||
"POST",
|
||||
"/api/v1/chats",
|
||||
expected=201,
|
||||
actor=alice,
|
||||
body={
|
||||
"title": "Другой чат для области ключа",
|
||||
"member_ids": [alice["id"], bob["id"]],
|
||||
},
|
||||
schema="Chat",
|
||||
)["json"]
|
||||
|
||||
other_chat_message = control.message(
|
||||
alice,
|
||||
second_chat,
|
||||
"Тот же ключ в другом чате",
|
||||
key=first_key,
|
||||
)["json"]
|
||||
|
||||
check(
|
||||
other_chat_message["chat_id"] == second_chat["id"],
|
||||
"wrong chat in scoped result",
|
||||
)
|
||||
check(
|
||||
other_chat_message["id"] != saved_messages[0]["id"],
|
||||
"key incorrectly shared between chats",
|
||||
)
|
||||
|
||||
replay = control.message(
|
||||
alice,
|
||||
chat,
|
||||
saved_messages[0]["text"],
|
||||
key=first_key,
|
||||
)["json"]
|
||||
|
||||
check(
|
||||
replay == saved_messages[0],
|
||||
"original result changed after scope tests",
|
||||
)
|
||||
|
||||
messages = history(control, bob, chat)
|
||||
expected = saved_messages + [bob_message]
|
||||
|
||||
check(
|
||||
len(messages) == len(expected),
|
||||
"unexpected final message count",
|
||||
)
|
||||
check(
|
||||
{message["id"] for message in messages}
|
||||
== {message["id"] for message in expected},
|
||||
"missing or extra messages in first chat",
|
||||
)
|
||||
check(
|
||||
history(control, bob, second_chat) == [other_chat_message],
|
||||
"unexpected history in second chat",
|
||||
)
|
||||
|
||||
control.call(
|
||||
"DELETE",
|
||||
"/api/v1/auth/sessions/current",
|
||||
expected=204,
|
||||
actor=alice,
|
||||
)
|
||||
|
||||
control.call(
|
||||
"POST",
|
||||
path,
|
||||
expected=401,
|
||||
actor=alice,
|
||||
body={"text": saved_messages[0]["text"]},
|
||||
headers={"Idempotency-Key": first_key},
|
||||
)
|
||||
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"rounds": ROUNDS,
|
||||
"concurrent_requests_per_round": WORKERS,
|
||||
"concurrent_requests_total": WORKERS * ROUNDS,
|
||||
"body_conflict_409": True,
|
||||
"missing_and_invalid_key_400": True,
|
||||
"foreign_user_404": True,
|
||||
"user_scope_independent": True,
|
||||
"chat_scope_independent": True,
|
||||
"replay_preserved": True,
|
||||
"revoked_session_401": True,
|
||||
"unexpected_messages": 0,
|
||||
},
|
||||
ensure_ascii=False,
|
||||
)
|
||||
)
|
||||
|
||||
print(
|
||||
"PASS: concurrent cross-replica idempotency, "
|
||||
"scope, validation and authorization checks."
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
run(main)
|
||||
@@ -0,0 +1,294 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
import json
|
||||
import secrets
|
||||
import subprocess
|
||||
import time
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from queued_restart import ROOT, DC, api, dc, http, wait_ready
|
||||
from outbox_gap import snapshot
|
||||
from before_ack import sql, object_keys
|
||||
|
||||
|
||||
EVIDENCE = ROOT / "evidence" / "failed_replay.json"
|
||||
|
||||
|
||||
def staging_write(attachment_id, *, original=None):
|
||||
code = """
|
||||
import sys
|
||||
from uuid import UUID
|
||||
from app.job_db import connect_db
|
||||
from app.storage import _put
|
||||
|
||||
attachment_id = UUID(sys.argv[1])
|
||||
mode = sys.argv[2]
|
||||
|
||||
with connect_db() as db:
|
||||
row = db.execute(
|
||||
'''
|
||||
SELECT staging_key, content_type
|
||||
FROM attachments
|
||||
WHERE id = %s
|
||||
''',
|
||||
(attachment_id,),
|
||||
).fetchone()
|
||||
|
||||
if row is None or not row["staging_key"]:
|
||||
raise SystemExit(2)
|
||||
|
||||
data = (
|
||||
b"deliberate synthetic staging corruption"
|
||||
if mode == "corrupt"
|
||||
else sys.stdin.buffer.read()
|
||||
)
|
||||
_put(row["staging_key"], data, row["content_type"])
|
||||
"""
|
||||
mode = "corrupt" if original is None else "restore"
|
||||
result = subprocess.run(
|
||||
[
|
||||
DC, "exec", "-T", "api",
|
||||
"python", "-c", code, attachment_id, mode,
|
||||
],
|
||||
cwd=ROOT,
|
||||
input=b"" if original is None else original,
|
||||
capture_output=True,
|
||||
timeout=90,
|
||||
)
|
||||
if result.returncode:
|
||||
raise RuntimeError(f"Staging operation failed: {mode}")
|
||||
|
||||
|
||||
def replay_command(job_id, *, apply=False):
|
||||
args = [
|
||||
DC, "exec", "-T", "worker",
|
||||
"python", "-m", "app.replay",
|
||||
"--job-id", job_id,
|
||||
"--actor", "student-test",
|
||||
"--reason", "Synthetic staging integrity fault repaired",
|
||||
]
|
||||
if apply:
|
||||
args.append("--apply")
|
||||
|
||||
result = subprocess.run(
|
||||
args,
|
||||
cwd=ROOT,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=90,
|
||||
)
|
||||
return result.returncode, json.loads(result.stdout)
|
||||
|
||||
|
||||
def audit_count(job_id):
|
||||
return int(sql(f"""
|
||||
SELECT count(*)
|
||||
FROM attachment_job_replays
|
||||
WHERE job_id = '{job_id}'::uuid;
|
||||
"""))
|
||||
|
||||
|
||||
def wait_status(path, token, expected, timeout=90):
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
current = api("GET", path, 200, token=token)
|
||||
if current["status"] == expected:
|
||||
return current
|
||||
if expected == "ready" and current["status"] == "failed":
|
||||
raise AssertionError(
|
||||
"Replay failed: " + str(current["error_code"])
|
||||
)
|
||||
time.sleep(0.5)
|
||||
raise AssertionError(f"Timeout waiting for {expected}")
|
||||
|
||||
|
||||
def main():
|
||||
evidence = {"scenario": "failed staging integrity and audited replay"}
|
||||
restore_worker = False
|
||||
corrupted = False
|
||||
attachment_id = None
|
||||
original = None
|
||||
|
||||
try:
|
||||
assert sql(
|
||||
"SELECT to_regclass('attachment_job_replays') IS NOT NULL;"
|
||||
) == "t", "Apply migration 007 first"
|
||||
|
||||
wait_ready()
|
||||
if not dc("ps", "--status", "running", "-q", "worker"):
|
||||
raise AssertionError("Start worker before this test")
|
||||
|
||||
username = "rep_" + uuid4().hex[:16]
|
||||
password = secrets.token_hex(16)
|
||||
|
||||
user = api(
|
||||
"POST", "/api/v1/users", 201,
|
||||
payload={
|
||||
"username": username,
|
||||
"display_name": "Failed replay test",
|
||||
"password": password,
|
||||
},
|
||||
)
|
||||
basic = base64.b64encode(
|
||||
f"{username}:{password}".encode()
|
||||
).decode()
|
||||
token = api(
|
||||
"POST", "/api/v1/auth/sessions", 201,
|
||||
headers={"Authorization": "Basic " + basic},
|
||||
)["token"]
|
||||
|
||||
chat = api(
|
||||
"POST", "/api/v1/chats", 201,
|
||||
token=token,
|
||||
payload={
|
||||
"title": "Failed replay",
|
||||
"member_ids": [user["id"]],
|
||||
},
|
||||
)
|
||||
|
||||
restore_worker = True
|
||||
dc("stop", "worker")
|
||||
if dc("ps", "--status", "running", "-q", "worker"):
|
||||
raise AssertionError("Worker is still running")
|
||||
|
||||
original = (ROOT / "tests/fixtures/sample.png").read_bytes()
|
||||
attachment = api(
|
||||
"POST",
|
||||
f"/api/v1/chats/{chat['id']}/attachments?filename=sample.png",
|
||||
202,
|
||||
token=token,
|
||||
data=original,
|
||||
headers={"Content-Type": "image/png"},
|
||||
)
|
||||
attachment_id = str(UUID(attachment["id"]))
|
||||
path = f"/api/v1/attachments/{attachment_id}"
|
||||
job_id = str(UUID(snapshot(attachment_id)["job_id"]))
|
||||
evidence["attachment_id"] = attachment_id
|
||||
evidence["job_id"] = job_id
|
||||
|
||||
# Отмечаем до вызова, чтобы finally попытался восстановить
|
||||
# исходник даже при неопределённом результате S3-записи.
|
||||
corrupted = True
|
||||
staging_write(attachment_id)
|
||||
dc("start", "worker")
|
||||
|
||||
failed = wait_status(path, token, "failed")
|
||||
assert failed["error_code"] == "invalid_image"
|
||||
assert failed["variants"] == []
|
||||
|
||||
failed_job = snapshot(attachment_id)
|
||||
assert failed_job["status"] == "failed"
|
||||
assert failed_job["attempt"] == 1
|
||||
assert failed_job["outbox_count"] == 1
|
||||
assert audit_count(job_id) == 0
|
||||
|
||||
registry = json.loads(
|
||||
dc("exec", "-T", "worker", "python", "-m", "app.replay", "--list")
|
||||
)
|
||||
assert any(item["job_id"] == job_id for item in registry)
|
||||
|
||||
api("GET", path + "/content", 409, token=token)
|
||||
api(
|
||||
"POST",
|
||||
f"/api/v1/chats/{chat['id']}/messages",
|
||||
409,
|
||||
token=token,
|
||||
payload={"text": "", "attachment_ids": [attachment_id]},
|
||||
headers={"Idempotency-Key": str(uuid4())},
|
||||
)
|
||||
evidence["failed_job"] = failed_job
|
||||
print("Corrupted staging produced failed/invalid_image; API returns 409")
|
||||
|
||||
code, rejected = replay_command(job_id, apply=True)
|
||||
assert code != 0
|
||||
assert rejected["result"] == "error"
|
||||
assert snapshot(attachment_id) == failed_job
|
||||
assert audit_count(job_id) == 0
|
||||
evidence["rejected_replay"] = rejected
|
||||
print("Replay rejected corrupted staging without state changes or audit")
|
||||
|
||||
staging_write(attachment_id, original=original)
|
||||
corrupted = False
|
||||
|
||||
code, dry_run = replay_command(job_id)
|
||||
assert code == 0 and dry_run["result"] == "dry_run"
|
||||
assert snapshot(attachment_id) == failed_job
|
||||
assert audit_count(job_id) == 0
|
||||
evidence["dry_run"] = dry_run
|
||||
print("Original restored; dry-run left job unchanged")
|
||||
|
||||
code, applied = replay_command(job_id, apply=True)
|
||||
assert code == 0 and applied["result"] == "replayed"
|
||||
assert applied["new_attempt"] == 2
|
||||
|
||||
ready = wait_status(path, token, "ready")
|
||||
final = snapshot(attachment_id)
|
||||
assert final["job_id"] == job_id
|
||||
assert final["status"] == "ready"
|
||||
assert final["attempt"] == 2
|
||||
assert final["outbox_count"] == 2
|
||||
assert audit_count(job_id) == 1
|
||||
|
||||
status, body = http("GET", path + "/content", token=token)
|
||||
assert status == 200 and body == original
|
||||
|
||||
variants = {item["name"]: item for item in ready["variants"]}
|
||||
assert len(ready["variants"]) == 3
|
||||
assert set(variants) == {"thumbnail", "crop", "optimized"}
|
||||
|
||||
for name, metadata in variants.items():
|
||||
status, body = http(
|
||||
"GET", path + "/content?variant=" + name,
|
||||
token=token,
|
||||
)
|
||||
assert status == 200
|
||||
assert len(body) == metadata["size_bytes"]
|
||||
assert hashlib.sha256(body).hexdigest() == metadata["sha256"]
|
||||
|
||||
keys_before_noop = object_keys(attachment_id)
|
||||
assert len(keys_before_noop) == 4
|
||||
|
||||
code, noop = replay_command(job_id, apply=True)
|
||||
assert code == 0
|
||||
assert noop["result"] == "noop"
|
||||
assert noop["status"] == "ready"
|
||||
assert snapshot(attachment_id) == final
|
||||
assert audit_count(job_id) == 1
|
||||
assert object_keys(attachment_id) == keys_before_noop
|
||||
|
||||
evidence["applied_replay"] = applied
|
||||
evidence["final_job"] = final
|
||||
evidence["ready_replay"] = noop
|
||||
evidence["audit_rows"] = 1
|
||||
evidence["object_count"] = len(keys_before_noop)
|
||||
evidence["result"] = "PASS"
|
||||
|
||||
except BaseException as exc:
|
||||
evidence["result"] = "FAIL"
|
||||
evidence["error_type"] = type(exc).__name__
|
||||
raise
|
||||
|
||||
finally:
|
||||
try:
|
||||
if corrupted and attachment_id is not None:
|
||||
staging_write(attachment_id, original=original)
|
||||
finally:
|
||||
try:
|
||||
if restore_worker:
|
||||
dc("start", "worker")
|
||||
finally:
|
||||
EVIDENCE.parent.mkdir(exist_ok=True)
|
||||
EVIDENCE.write_text(
|
||||
json.dumps(evidence, ensure_ascii=False, indent=2) + "\n"
|
||||
)
|
||||
|
||||
print(
|
||||
"PASS: failed registry, integrity guard, dry-run, audited replay "
|
||||
"and ready noop verified; same job, attempt=2"
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,228 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import secrets
|
||||
import time
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from PIL import Image
|
||||
|
||||
from queued_restart import ROOT, api, dc, http, wait_ready
|
||||
|
||||
|
||||
EVIDENCE = ROOT / "evidence" / "outbox_gap.json"
|
||||
|
||||
|
||||
def snapshot(attachment_id):
|
||||
attachment_id = str(UUID(attachment_id))
|
||||
query = f"""
|
||||
SELECT json_build_object(
|
||||
'job_id', j.id,
|
||||
'status', j.status,
|
||||
'attempt', j.attempt,
|
||||
'outbox_count', (
|
||||
SELECT count(*)
|
||||
FROM attachment_outbox o
|
||||
WHERE o.job_id = j.id
|
||||
),
|
||||
'published', EXISTS (
|
||||
SELECT 1
|
||||
FROM attachment_outbox o
|
||||
WHERE o.job_id = j.id
|
||||
AND o.published_at IS NOT NULL
|
||||
)
|
||||
)
|
||||
FROM attachment_jobs j
|
||||
WHERE j.attachment_id = '{attachment_id}'::uuid
|
||||
AND j.pipeline_version = 1;
|
||||
"""
|
||||
result = dc(
|
||||
"exec", "-T", "postgres",
|
||||
"psql", "-U", "messenger", "-d", "messenger",
|
||||
"-X", "-A", "-t", "-v", "ON_ERROR_STOP=1",
|
||||
"-c", query,
|
||||
)
|
||||
if not result:
|
||||
raise AssertionError("Job is missing")
|
||||
return json.loads(result)
|
||||
|
||||
|
||||
def main():
|
||||
evidence = {
|
||||
"scenario": "API crash after COMMIT while publisher is stopped",
|
||||
}
|
||||
restore = False
|
||||
|
||||
try:
|
||||
wait_ready()
|
||||
for service in ("publisher", "worker"):
|
||||
if not dc("ps", "--status", "running", "-q", service):
|
||||
raise AssertionError(f"Start {service} before this test")
|
||||
|
||||
username = "gap_" + uuid4().hex[:16]
|
||||
password = secrets.token_hex(16)
|
||||
user = api(
|
||||
"POST", "/api/v1/users", 201,
|
||||
payload={
|
||||
"username": username,
|
||||
"display_name": "Outbox gap test",
|
||||
"password": password,
|
||||
},
|
||||
)
|
||||
basic = base64.b64encode(
|
||||
f"{username}:{password}".encode()
|
||||
).decode()
|
||||
session = api(
|
||||
"POST", "/api/v1/auth/sessions", 201,
|
||||
headers={"Authorization": "Basic " + basic},
|
||||
)
|
||||
token = session["token"]
|
||||
chat = api(
|
||||
"POST", "/api/v1/chats", 201,
|
||||
token=token,
|
||||
payload={
|
||||
"title": "Outbox gap",
|
||||
"member_ids": [user["id"]],
|
||||
},
|
||||
)
|
||||
|
||||
restore = True
|
||||
dc("stop", "publisher", "worker")
|
||||
for service in ("publisher", "worker"):
|
||||
if dc("ps", "--status", "running", "-q", service):
|
||||
raise AssertionError(f"{service} is still running")
|
||||
print("Publisher and worker stopped")
|
||||
|
||||
original = (ROOT / "tests/fixtures/sample.png").read_bytes()
|
||||
attachment = api(
|
||||
"POST",
|
||||
f"/api/v1/chats/{chat['id']}/attachments?filename=sample.png",
|
||||
202,
|
||||
token=token,
|
||||
data=original,
|
||||
headers={"Content-Type": "image/png"},
|
||||
)
|
||||
attachment_id = attachment["id"]
|
||||
path = f"/api/v1/attachments/{attachment_id}"
|
||||
assert attachment["status"] == "queued"
|
||||
|
||||
before = snapshot(attachment_id)
|
||||
assert before["status"] == "queued"
|
||||
assert before["attempt"] == 1
|
||||
assert before["outbox_count"] == 1
|
||||
assert before["published"] is False
|
||||
|
||||
evidence["attachment_id"] = attachment_id
|
||||
evidence["before_crash"] = before
|
||||
print("Upload 202; job committed; outbox is not published")
|
||||
|
||||
dc("kill", "--signal", "SIGKILL", "api")
|
||||
|
||||
after_crash = snapshot(attachment_id)
|
||||
assert after_crash == before
|
||||
evidence["after_api_kill"] = after_crash
|
||||
print("API killed with SIGKILL; committed job remains in PostgreSQL")
|
||||
|
||||
dc("up", "-d", "--no-deps", "api")
|
||||
wait_ready()
|
||||
|
||||
# Прежняя сессия и то же вложение должны сохраниться.
|
||||
current = api("GET", path, 200, token=token)
|
||||
assert current["status"] == "queued"
|
||||
api("GET", path + "/content", 409, token=token)
|
||||
|
||||
dc("start", "publisher")
|
||||
deadline = time.monotonic() + 45
|
||||
while time.monotonic() < deadline:
|
||||
published = snapshot(attachment_id)
|
||||
if published["published"]:
|
||||
break
|
||||
time.sleep(1)
|
||||
else:
|
||||
raise AssertionError("Pending outbox was not published")
|
||||
|
||||
assert published["job_id"] == before["job_id"]
|
||||
assert published["attempt"] == 1
|
||||
assert published["outbox_count"] == 1
|
||||
evidence["after_publisher_start"] = published
|
||||
print("Publisher recovered the existing outbox without a new job")
|
||||
|
||||
dc("start", "worker")
|
||||
started = time.monotonic()
|
||||
deadline = started + 90
|
||||
|
||||
while time.monotonic() < deadline:
|
||||
current = api("GET", path, 200, token=token)
|
||||
if current["status"] == "ready":
|
||||
break
|
||||
if current["status"] == "failed":
|
||||
raise AssertionError(
|
||||
"Processing failed: " + str(current["error_code"])
|
||||
)
|
||||
time.sleep(0.5)
|
||||
else:
|
||||
raise AssertionError("Ready timeout")
|
||||
|
||||
status, body = http("GET", path + "/content", token=token)
|
||||
assert status == 200
|
||||
assert body == original
|
||||
|
||||
variants = {item["name"]: item for item in current["variants"]}
|
||||
assert set(variants) == {"thumbnail", "crop", "optimized"}
|
||||
|
||||
for name, expected_format in (
|
||||
("thumbnail", "JPEG"),
|
||||
("crop", "JPEG"),
|
||||
("optimized", "WEBP"),
|
||||
):
|
||||
status, body = http(
|
||||
"GET", path + "/content?variant=" + name,
|
||||
token=token,
|
||||
)
|
||||
metadata = variants[name]
|
||||
assert status == 200
|
||||
assert len(body) == metadata["size_bytes"]
|
||||
assert hashlib.sha256(body).hexdigest() == metadata["sha256"]
|
||||
with Image.open(io.BytesIO(body)) as image:
|
||||
image.load()
|
||||
assert image.format == expected_format
|
||||
assert image.size == (
|
||||
metadata["width"], metadata["height"]
|
||||
)
|
||||
|
||||
final = snapshot(attachment_id)
|
||||
assert final["job_id"] == before["job_id"]
|
||||
assert final["status"] == "ready"
|
||||
assert final["attempt"] == 1
|
||||
assert final["outbox_count"] == 1
|
||||
|
||||
evidence["final"] = final
|
||||
evidence["ready_after_worker_start_s"] = round(
|
||||
time.monotonic() - started, 3
|
||||
)
|
||||
evidence["result"] = "PASS"
|
||||
|
||||
except BaseException as exc:
|
||||
evidence["result"] = "FAIL"
|
||||
evidence["error_type"] = type(exc).__name__
|
||||
raise
|
||||
|
||||
finally:
|
||||
try:
|
||||
if restore:
|
||||
dc("up", "-d", "--no-deps", "api")
|
||||
dc("start", "publisher", "worker")
|
||||
finally:
|
||||
EVIDENCE.parent.mkdir(exist_ok=True)
|
||||
EVIDENCE.write_text(
|
||||
json.dumps(evidence, ensure_ascii=False, indent=2) + "\n"
|
||||
)
|
||||
|
||||
print("PASS: committed outbox survived API crash and reached ready without a new job")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,176 @@
|
||||
#!/usr/bin/env python3
|
||||
from __future__ import annotations
|
||||
|
||||
import concurrent.futures
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
|
||||
from client import Client
|
||||
|
||||
|
||||
BASE_URL = "http://127.0.0.1:18082"
|
||||
MAX_RESPONSE_SECONDS = 6.0
|
||||
WATCHDOG_SECONDS = 8.0
|
||||
|
||||
COMPOSE = [
|
||||
"docker", "compose",
|
||||
"-f", "compose.yaml",
|
||||
"-f", "compose.s3.yaml",
|
||||
"--env-file", ".env",
|
||||
]
|
||||
|
||||
|
||||
def docker(*args):
|
||||
return subprocess.check_output(
|
||||
["docker", *args],
|
||||
cwd=ROOT,
|
||||
text=True,
|
||||
timeout=20,
|
||||
stderr=subprocess.PIPE,
|
||||
).strip()
|
||||
|
||||
|
||||
def main():
|
||||
pg_id = subprocess.check_output(
|
||||
[*COMPOSE, "ps", "-q", "postgres"],
|
||||
cwd=ROOT,
|
||||
text=True,
|
||||
timeout=20,
|
||||
).strip()
|
||||
assert pg_id, "PostgreSQL container not found"
|
||||
assert docker(
|
||||
"inspect", "-f", "{{.State.Paused}}", pg_id
|
||||
) == "false", "PostgreSQL is already paused"
|
||||
|
||||
c = Client(BASE_URL, None)
|
||||
alice, _, _, chat = c.bootstrap()
|
||||
message = c.message(alice, chat, "postgres-pause regression")["json"]
|
||||
path = "/api/v1/chats/" + chat["id"] + "/messages?limit=100"
|
||||
|
||||
baseline_instances = set()
|
||||
for _ in range(6):
|
||||
response = c.call(
|
||||
"GET", path, actor=alice, schema="MessagePage"
|
||||
)
|
||||
assert any(
|
||||
item["id"] == message["id"]
|
||||
for item in response["json"]["items"]
|
||||
), "Baseline message is missing"
|
||||
instance = response["headers"].get("X-Instance-Id")
|
||||
assert instance, "Baseline response has no X-Instance-Id"
|
||||
baseline_instances.add(instance)
|
||||
|
||||
assert len(baseline_instances) >= 2, (
|
||||
"Baseline did not reach both API replicas"
|
||||
)
|
||||
print("Baseline: history is available through both replicas", flush=True)
|
||||
|
||||
def probe():
|
||||
started = time.monotonic()
|
||||
try:
|
||||
client = Client(BASE_URL, None)
|
||||
response = client.call("GET", path, 503, actor=alice)
|
||||
elapsed = time.monotonic() - started
|
||||
instance = response["headers"].get("X-Instance-Id")
|
||||
code = response["json"]["error"]["code"]
|
||||
return {
|
||||
"ok": bool(instance) and code == "unavailable",
|
||||
"status": 503,
|
||||
"code": code,
|
||||
"instance": instance,
|
||||
"elapsed_s": round(elapsed, 4),
|
||||
}
|
||||
except Exception as exc:
|
||||
return {
|
||||
"ok": False,
|
||||
"error_type": type(exc).__name__,
|
||||
"elapsed_s": round(time.monotonic() - started, 4),
|
||||
}
|
||||
|
||||
executor = concurrent.futures.ThreadPoolExecutor(max_workers=4)
|
||||
paused_by_test = False
|
||||
try:
|
||||
paused_by_test = True
|
||||
docker("pause", pg_id)
|
||||
assert docker(
|
||||
"inspect", "-f", "{{.State.Paused}}", pg_id
|
||||
) == "true"
|
||||
print("PostgreSQL paused; requesting history", flush=True)
|
||||
|
||||
futures = [executor.submit(probe) for _ in range(4)]
|
||||
_, pending = concurrent.futures.wait(
|
||||
futures, timeout=WATCHDOG_SECONDS
|
||||
)
|
||||
assert not pending, "History requests exceeded the test watchdog"
|
||||
|
||||
results = [future.result() for future in futures]
|
||||
print(json.dumps(results, indent=2), flush=True)
|
||||
|
||||
assert all(item["ok"] for item in results), (
|
||||
"Expected API JSON 503 unavailable with X-Instance-Id"
|
||||
)
|
||||
assert all(
|
||||
item["elapsed_s"] < MAX_RESPONSE_SECONDS
|
||||
for item in results
|
||||
), "API response exceeded the declared deadline"
|
||||
|
||||
failed_instances = {item["instance"] for item in results}
|
||||
assert baseline_instances.issubset(failed_instances), (
|
||||
"Paused-DB requests did not exercise both baseline replicas"
|
||||
)
|
||||
finally:
|
||||
try:
|
||||
if paused_by_test:
|
||||
state = docker(
|
||||
"inspect", "-f", "{{.State.Paused}}", pg_id
|
||||
)
|
||||
if state == "true":
|
||||
docker("unpause", pg_id)
|
||||
print("PostgreSQL unpaused", flush=True)
|
||||
finally:
|
||||
executor.shutdown(wait=False, cancel_futures=True)
|
||||
|
||||
recovery_started = time.monotonic()
|
||||
recovery_deadline = recovery_started + 60
|
||||
while True:
|
||||
try:
|
||||
response = c.call(
|
||||
"GET", path, actor=alice, schema="MessagePage"
|
||||
)
|
||||
assert any(
|
||||
item["id"] == message["id"]
|
||||
for item in response["json"]["items"]
|
||||
), "Confirmed message disappeared"
|
||||
break
|
||||
except Exception:
|
||||
if time.monotonic() >= recovery_deadline:
|
||||
raise AssertionError("History did not recover") from None
|
||||
time.sleep(1)
|
||||
|
||||
report = {
|
||||
"scenario": "postgres_pause_history",
|
||||
"passed": True,
|
||||
"max_response_seconds": MAX_RESPONSE_SECONDS,
|
||||
"baseline_instances": sorted(baseline_instances),
|
||||
"paused_requests": results,
|
||||
"recovery_observed_after_s": round(
|
||||
time.monotonic() - recovery_started, 4
|
||||
),
|
||||
"confirmed_message_preserved": True,
|
||||
}
|
||||
target = ROOT / "evidence" / "postgres_pause.json"
|
||||
target.parent.mkdir(exist_ok=True)
|
||||
target.write_text(json.dumps(report, indent=2) + "\n")
|
||||
|
||||
print("PASS: bounded API 503 on both replicas; history recovered", flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,345 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
from pathlib import Path
|
||||
import secrets
|
||||
import subprocess
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from uuid import uuid4
|
||||
|
||||
from PIL import Image
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
BASE = "http://127.0.0.1:18085"
|
||||
DC = str(ROOT / "scripts/dc")
|
||||
EVIDENCE = ROOT / "evidence" / "queued_restart.json"
|
||||
|
||||
|
||||
def dc(*args, timeout=150):
|
||||
result = subprocess.run(
|
||||
[DC, *args],
|
||||
cwd=ROOT,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=timeout,
|
||||
)
|
||||
if result.returncode:
|
||||
details = "\n".join(
|
||||
part.strip()
|
||||
for part in (result.stdout, result.stderr)
|
||||
if part.strip()
|
||||
)
|
||||
raise RuntimeError(
|
||||
"Compose command failed: "
|
||||
+ " ".join(args)
|
||||
+ f" (exit={result.returncode})"
|
||||
+ "\n"
|
||||
+ details[-6000:]
|
||||
)
|
||||
return result.stdout.strip()
|
||||
|
||||
|
||||
def http(method, path, *, token=None, payload=None, data=None, headers=None):
|
||||
request_headers = dict(headers or {})
|
||||
if token:
|
||||
request_headers["Authorization"] = "Bearer " + token
|
||||
if payload is not None:
|
||||
data = json.dumps(payload).encode()
|
||||
request_headers["Content-Type"] = "application/json"
|
||||
|
||||
request = urllib.request.Request(
|
||||
BASE + path,
|
||||
data=data,
|
||||
headers=request_headers,
|
||||
method=method,
|
||||
)
|
||||
try:
|
||||
response = urllib.request.urlopen(request, timeout=15)
|
||||
except urllib.error.HTTPError as exc:
|
||||
response = exc
|
||||
|
||||
with response:
|
||||
return response.status, response.read()
|
||||
|
||||
|
||||
def api(method, path, expected, **kwargs):
|
||||
status, body = http(method, path, **kwargs)
|
||||
if status != expected:
|
||||
raise AssertionError(
|
||||
f"{method} {path}: expected {expected}, got {status}"
|
||||
)
|
||||
return json.loads(body) if body else None
|
||||
|
||||
|
||||
def wait_ready(timeout=90):
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
if http("GET", "/health/ready")[0] == 200:
|
||||
return
|
||||
except (OSError, TimeoutError):
|
||||
pass
|
||||
time.sleep(1)
|
||||
raise AssertionError("API readiness timeout")
|
||||
|
||||
|
||||
def container_starts():
|
||||
result = {}
|
||||
for service in ("api", "broker"):
|
||||
ids = dc("ps", "-q", service).split()
|
||||
if not ids:
|
||||
raise AssertionError(f"No running containers for {service}")
|
||||
for container_id in ids:
|
||||
inspected = subprocess.run(
|
||||
[
|
||||
"docker", "inspect",
|
||||
"--format", "{{.State.StartedAt}}",
|
||||
container_id,
|
||||
],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
result[container_id] = inspected.stdout.strip()
|
||||
return result
|
||||
|
||||
|
||||
def job_snapshot(attachment_id):
|
||||
code = """
|
||||
import json
|
||||
import sys
|
||||
from uuid import UUID
|
||||
from app.job_db import connect_db
|
||||
|
||||
with connect_db() as db:
|
||||
row = db.execute(
|
||||
'''
|
||||
SELECT j.status, j.attempt,
|
||||
EXISTS (
|
||||
SELECT 1 FROM attachment_outbox o
|
||||
WHERE o.job_id = j.id
|
||||
AND o.attempt = j.attempt
|
||||
AND o.published_at IS NOT NULL
|
||||
) AS published
|
||||
FROM attachment_jobs j
|
||||
WHERE j.attachment_id = %s
|
||||
AND j.pipeline_version = 1
|
||||
''',
|
||||
(UUID(sys.argv[1]),),
|
||||
).fetchone()
|
||||
print(json.dumps(row))
|
||||
"""
|
||||
return json.loads(
|
||||
dc("exec", "-T", "publisher", "python", "-c", code, attachment_id)
|
||||
)
|
||||
|
||||
|
||||
def queue_snapshot(timeout=90):
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
output = dc(
|
||||
"exec", "-T", "broker",
|
||||
"rabbitmqctl", "-q", "list_queues",
|
||||
"name", "durable", "messages_ready",
|
||||
timeout=20,
|
||||
)
|
||||
for line in output.splitlines():
|
||||
fields = line.split()
|
||||
if len(fields) == 3 and fields[0] == "attachment.process.v1":
|
||||
return {
|
||||
"name": fields[0],
|
||||
"durable": fields[1].lower() == "true",
|
||||
"messages_ready": int(fields[2]),
|
||||
}
|
||||
except (RuntimeError, subprocess.TimeoutExpired):
|
||||
pass
|
||||
time.sleep(2)
|
||||
raise AssertionError("Queue inspection timeout")
|
||||
|
||||
|
||||
def main():
|
||||
evidence = {"scenario": "queued upload survives API and broker restart"}
|
||||
stopped = False
|
||||
|
||||
try:
|
||||
wait_ready()
|
||||
if not dc("ps", "--status", "running", "-q", "worker"):
|
||||
raise AssertionError("Start worker before running this test")
|
||||
|
||||
username = "lab5_" + uuid4().hex[:16]
|
||||
password = secrets.token_hex(16)
|
||||
|
||||
user = api(
|
||||
"POST", "/api/v1/users", 201,
|
||||
payload={
|
||||
"username": username,
|
||||
"display_name": "Queue restart test",
|
||||
"password": password,
|
||||
},
|
||||
)
|
||||
basic = base64.b64encode(
|
||||
f"{username}:{password}".encode()
|
||||
).decode()
|
||||
session = api(
|
||||
"POST", "/api/v1/auth/sessions", 201,
|
||||
headers={"Authorization": "Basic " + basic},
|
||||
)
|
||||
token = session["token"]
|
||||
|
||||
chat = api(
|
||||
"POST", "/api/v1/chats", 201,
|
||||
token=token,
|
||||
payload={
|
||||
"title": "Queued restart",
|
||||
"member_ids": [user["id"]],
|
||||
},
|
||||
)
|
||||
|
||||
dc("stop", "worker")
|
||||
stopped = True
|
||||
if dc("ps", "--status", "running", "-q", "worker"):
|
||||
raise AssertionError("Worker is still running")
|
||||
print("Worker stopped")
|
||||
|
||||
original = (ROOT / "tests/fixtures/sample.png").read_bytes()
|
||||
attachment = api(
|
||||
"POST",
|
||||
f"/api/v1/chats/{chat['id']}/attachments?filename=sample.png",
|
||||
202,
|
||||
token=token,
|
||||
data=original,
|
||||
headers={"Content-Type": "image/png"},
|
||||
)
|
||||
attachment_id = attachment["id"]
|
||||
path = f"/api/v1/attachments/{attachment_id}"
|
||||
assert attachment["status"] == "queued"
|
||||
evidence["attachment_id"] = attachment_id
|
||||
print(f"Upload accepted: {attachment_id}, status=queued")
|
||||
|
||||
api("GET", path + "/content", 409, token=token)
|
||||
|
||||
message = api(
|
||||
"POST",
|
||||
f"/api/v1/chats/{chat['id']}/messages",
|
||||
201,
|
||||
token=token,
|
||||
payload={"text": "", "attachment_ids": [attachment_id]},
|
||||
headers={"Idempotency-Key": str(uuid4())},
|
||||
)
|
||||
evidence["message_id"] = message["id"]
|
||||
|
||||
deadline = time.monotonic() + 30
|
||||
while time.monotonic() < deadline:
|
||||
snapshot = job_snapshot(attachment_id)
|
||||
if snapshot and snapshot["published"]:
|
||||
break
|
||||
time.sleep(1)
|
||||
else:
|
||||
raise AssertionError("Outbox was not confirmed as published")
|
||||
|
||||
evidence["job_before_restart"] = snapshot
|
||||
before_queue = queue_snapshot()
|
||||
assert before_queue["durable"]
|
||||
assert before_queue["messages_ready"] >= 1
|
||||
evidence["queue_before_restart"] = before_queue
|
||||
|
||||
before = container_starts()
|
||||
dc("restart", "api", "broker")
|
||||
wait_ready()
|
||||
after_queue = queue_snapshot()
|
||||
after = container_starts()
|
||||
|
||||
assert set(before) == set(after)
|
||||
assert all(before[key] != after[key] for key in before)
|
||||
assert after_queue["durable"]
|
||||
assert after_queue["messages_ready"] >= 1
|
||||
evidence["container_starts_before"] = before
|
||||
evidence["container_starts_after"] = after
|
||||
evidence["queue_after_restart"] = after_queue
|
||||
print("API and broker restarted; queued message remains in broker")
|
||||
|
||||
current = api("GET", path, 200, token=token)
|
||||
assert current["status"] == "queued"
|
||||
api("GET", path + "/content", 409, token=token)
|
||||
|
||||
history = api(
|
||||
"GET",
|
||||
f"/api/v1/chats/{chat['id']}/messages?limit=100",
|
||||
200,
|
||||
token=token,
|
||||
)
|
||||
assert any(item["id"] == message["id"] for item in history["items"])
|
||||
|
||||
dc("start", "worker")
|
||||
started = time.monotonic()
|
||||
|
||||
deadline = started + 90
|
||||
while time.monotonic() < deadline:
|
||||
current = api("GET", path, 200, token=token)
|
||||
if current["status"] == "ready":
|
||||
break
|
||||
if current["status"] == "failed":
|
||||
raise AssertionError(
|
||||
"Attachment failed: " + str(current["error_code"])
|
||||
)
|
||||
time.sleep(0.5)
|
||||
else:
|
||||
raise AssertionError("Attachment did not become ready")
|
||||
|
||||
status, downloaded = http("GET", path + "/content", token=token)
|
||||
assert status == 200 and downloaded == original
|
||||
|
||||
variants = {item["name"]: item for item in current["variants"]}
|
||||
assert set(variants) == {"thumbnail", "crop", "optimized"}
|
||||
|
||||
for name, fmt in (
|
||||
("thumbnail", "JPEG"),
|
||||
("crop", "JPEG"),
|
||||
("optimized", "WEBP"),
|
||||
):
|
||||
status, body = http(
|
||||
"GET", path + "/content?variant=" + name, token=token,
|
||||
)
|
||||
assert status == 200
|
||||
metadata = variants[name]
|
||||
assert len(body) == metadata["size_bytes"]
|
||||
assert hashlib.sha256(body).hexdigest() == metadata["sha256"]
|
||||
with Image.open(io.BytesIO(body)) as image:
|
||||
image.load()
|
||||
assert image.format == fmt
|
||||
assert image.size == (metadata["width"], metadata["height"])
|
||||
|
||||
evidence["ready_after_worker_start_s"] = round(
|
||||
time.monotonic() - started, 3
|
||||
)
|
||||
evidence["original_sha256"] = hashlib.sha256(original).hexdigest()
|
||||
evidence["variants"] = list(variants)
|
||||
evidence["result"] = "PASS"
|
||||
|
||||
except BaseException as exc:
|
||||
evidence["result"] = "FAIL"
|
||||
evidence["error_type"] = type(exc).__name__
|
||||
raise
|
||||
|
||||
finally:
|
||||
try:
|
||||
if stopped:
|
||||
dc("start", "worker")
|
||||
finally:
|
||||
EVIDENCE.parent.mkdir(exist_ok=True)
|
||||
EVIDENCE.write_text(
|
||||
json.dumps(evidence, ensure_ascii=False, indent=2) + "\n"
|
||||
)
|
||||
|
||||
print("PASS: queued task survived API/broker restart; original and variants verified")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,226 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import secrets
|
||||
import time
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from queued_restart import ROOT, api, dc, wait_ready
|
||||
from before_ack import sql
|
||||
from failed_replay import staging_write
|
||||
|
||||
|
||||
EVIDENCE = ROOT / "evidence" / "retry_exhaustion.json"
|
||||
|
||||
|
||||
def job_details(attachment_id):
|
||||
return json.loads(sql(f"""
|
||||
SELECT json_build_object(
|
||||
'job_id', j.id,
|
||||
'status', j.status,
|
||||
'attempt', j.attempt,
|
||||
'max_attempts', j.max_attempts,
|
||||
'error_code', j.error_code,
|
||||
'lease_cleared',
|
||||
j.lease_token IS NULL AND j.lease_until IS NULL,
|
||||
'outbox', (
|
||||
SELECT json_agg(
|
||||
json_build_object(
|
||||
'attempt', o.attempt,
|
||||
'published', o.published_at IS NOT NULL,
|
||||
'delay_seconds',
|
||||
extract(epoch FROM (o.available_at - o.created_at))
|
||||
)
|
||||
ORDER BY o.attempt
|
||||
)
|
||||
FROM attachment_outbox o
|
||||
WHERE o.job_id = j.id
|
||||
)
|
||||
)
|
||||
FROM attachment_jobs j
|
||||
WHERE j.attachment_id = '{attachment_id}'::uuid
|
||||
AND j.pipeline_version = 1;
|
||||
"""))
|
||||
|
||||
|
||||
def remove_staging(attachment_id):
|
||||
code = """
|
||||
import sys
|
||||
from uuid import UUID
|
||||
from app.job_db import connect_db
|
||||
from app.storage import bucket_name, get_client
|
||||
|
||||
with connect_db() as db:
|
||||
row = db.execute(
|
||||
"SELECT staging_key FROM attachments WHERE id = %s",
|
||||
(UUID(sys.argv[1]),),
|
||||
).fetchone()
|
||||
|
||||
if row is None or not row["staging_key"]:
|
||||
raise RuntimeError("Expected staging reference")
|
||||
|
||||
get_client().delete_object(
|
||||
Bucket=bucket_name(),
|
||||
Key=row["staging_key"],
|
||||
)
|
||||
"""
|
||||
dc("exec", "-T", "api", "python", "-c", code, attachment_id)
|
||||
|
||||
|
||||
def main():
|
||||
evidence = {"scenario": "temporary S3 read failure exhausts bounded retries"}
|
||||
restore_worker = False
|
||||
restore_staging = False
|
||||
attachment_id = None
|
||||
original = None
|
||||
|
||||
try:
|
||||
wait_ready()
|
||||
for service in ("publisher", "worker"):
|
||||
if not dc("ps", "--status", "running", "-q", service):
|
||||
raise AssertionError(f"Start {service} before this test")
|
||||
|
||||
username = "rty_" + uuid4().hex[:16]
|
||||
password = secrets.token_hex(16)
|
||||
user = api(
|
||||
"POST", "/api/v1/users", 201,
|
||||
payload={
|
||||
"username": username,
|
||||
"display_name": "Retry exhaustion test",
|
||||
"password": password,
|
||||
},
|
||||
)
|
||||
basic = base64.b64encode(
|
||||
f"{username}:{password}".encode()
|
||||
).decode()
|
||||
token = api(
|
||||
"POST", "/api/v1/auth/sessions", 201,
|
||||
headers={"Authorization": "Basic " + basic},
|
||||
)["token"]
|
||||
chat = api(
|
||||
"POST", "/api/v1/chats", 201,
|
||||
token=token,
|
||||
payload={
|
||||
"title": "Retry exhaustion",
|
||||
"member_ids": [user["id"]],
|
||||
},
|
||||
)
|
||||
|
||||
restore_worker = True
|
||||
dc("stop", "worker")
|
||||
assert not dc("ps", "--status", "running", "-q", "worker")
|
||||
|
||||
original = (ROOT / "tests/fixtures/sample.png").read_bytes()
|
||||
attachment = api(
|
||||
"POST",
|
||||
f"/api/v1/chats/{chat['id']}/attachments?filename=sample.png",
|
||||
202,
|
||||
token=token,
|
||||
data=original,
|
||||
headers={"Content-Type": "image/png"},
|
||||
)
|
||||
attachment_id = str(UUID(attachment["id"]))
|
||||
path = f"/api/v1/attachments/{attachment_id}"
|
||||
|
||||
initial = job_details(attachment_id)
|
||||
assert initial["attempt"] == 1
|
||||
assert initial["max_attempts"] == 5
|
||||
evidence["attachment_id"] = attachment_id
|
||||
evidence["initial_job"] = initial
|
||||
|
||||
restore_staging = True
|
||||
remove_staging(attachment_id)
|
||||
print("Test staging object removed; configured attempt limit=5")
|
||||
|
||||
dc("start", "worker")
|
||||
started = time.monotonic()
|
||||
deadline = started + 180
|
||||
observed_attempts = []
|
||||
previous_attempt = None
|
||||
|
||||
while time.monotonic() < deadline:
|
||||
current = job_details(attachment_id)
|
||||
if current["attempt"] != previous_attempt:
|
||||
previous_attempt = current["attempt"]
|
||||
observed_attempts.append(previous_attempt)
|
||||
print(
|
||||
f"Observed attempt={previous_attempt}, "
|
||||
f"status={current['status']}",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
if current["status"] == "failed":
|
||||
break
|
||||
if current["status"] == "ready":
|
||||
raise AssertionError("Missing staging unexpectedly reached ready")
|
||||
time.sleep(0.5)
|
||||
else:
|
||||
raise AssertionError("Retry exhaustion timeout")
|
||||
|
||||
assert current["job_id"] == initial["job_id"]
|
||||
assert current["attempt"] == current["max_attempts"] == 5
|
||||
assert current["error_code"] == "retry_exhausted"
|
||||
assert current["lease_cleared"] is True
|
||||
|
||||
outbox = current["outbox"]
|
||||
assert [row["attempt"] for row in outbox] == [1, 2, 3, 4, 5]
|
||||
assert all(row["published"] for row in outbox)
|
||||
|
||||
for row in outbox:
|
||||
if row["attempt"] == 1:
|
||||
continue
|
||||
previous = row["attempt"] - 1
|
||||
upper = min(30.0, 2.0 ** min(previous, 5))
|
||||
assert 0.49 <= row["delay_seconds"] <= upper + 0.1
|
||||
|
||||
failed = api("GET", path, 200, token=token)
|
||||
assert failed["status"] == "failed"
|
||||
assert failed["error_code"] == "retry_exhausted"
|
||||
assert failed["variants"] == []
|
||||
api("GET", path + "/content", 409, token=token)
|
||||
|
||||
count = int(sql(f"""
|
||||
SELECT count(*) FROM attachment_variants
|
||||
WHERE attachment_id = '{attachment_id}'::uuid;
|
||||
"""))
|
||||
assert count == 0
|
||||
|
||||
# Короткое дополнительное наблюдение терминального состояния.
|
||||
time.sleep(5)
|
||||
assert job_details(attachment_id) == current
|
||||
|
||||
evidence["observed_attempts"] = observed_attempts
|
||||
evidence["final_job"] = current
|
||||
evidence["terminal_observation_seconds"] = 5
|
||||
evidence["elapsed_s"] = round(time.monotonic() - started, 3)
|
||||
evidence["result"] = "PASS"
|
||||
|
||||
except BaseException as exc:
|
||||
evidence["result"] = "FAIL"
|
||||
evidence["error_type"] = type(exc).__name__
|
||||
raise
|
||||
|
||||
finally:
|
||||
try:
|
||||
# Возвращаем принятые bytes, но failed автоматически не повторяем.
|
||||
if restore_staging and attachment_id is not None:
|
||||
staging_write(attachment_id, original=original)
|
||||
finally:
|
||||
try:
|
||||
if restore_worker:
|
||||
dc("start", "worker")
|
||||
finally:
|
||||
EVIDENCE.parent.mkdir(exist_ok=True)
|
||||
EVIDENCE.write_text(
|
||||
json.dumps(evidence, ensure_ascii=False, indent=2) + "\n"
|
||||
)
|
||||
|
||||
print(
|
||||
"PASS: five published attempts exhausted retries; "
|
||||
"backoff bounds verified; terminal failed/retry_exhausted, no variants"
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,284 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
|
||||
from client import Client, Failure, arguments, check, run
|
||||
|
||||
|
||||
PROJECT = "mtusi2026lab3"
|
||||
|
||||
|
||||
def timestamp():
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def command(*args):
|
||||
return subprocess.check_output(
|
||||
list(args),
|
||||
cwd=ROOT,
|
||||
text=True,
|
||||
timeout=45,
|
||||
).strip()
|
||||
|
||||
|
||||
def compose(*args):
|
||||
return command(
|
||||
"docker",
|
||||
"compose",
|
||||
"-p",
|
||||
PROJECT,
|
||||
*args,
|
||||
)
|
||||
|
||||
|
||||
def inspect(container, template):
|
||||
return command(
|
||||
"docker",
|
||||
"inspect",
|
||||
"--format",
|
||||
template,
|
||||
container,
|
||||
)
|
||||
|
||||
|
||||
def api_ids():
|
||||
ids = compose("ps", "-q", "api").split()
|
||||
return {
|
||||
inspect(container, "{{.Id}}")
|
||||
for container in ids
|
||||
}
|
||||
|
||||
|
||||
def healthy(container):
|
||||
return inspect(
|
||||
container,
|
||||
"{{.State.Running}} {{.State.Health.Status}}",
|
||||
) == "true healthy"
|
||||
|
||||
|
||||
def instance_id(container):
|
||||
code = (
|
||||
"import urllib.request; "
|
||||
"r=urllib.request.urlopen("
|
||||
"'http://127.0.0.1:8080/health/live', timeout=2); "
|
||||
"print(r.headers.get('X-Instance-Id', '')); "
|
||||
"r.close()"
|
||||
)
|
||||
return command(
|
||||
"docker",
|
||||
"exec",
|
||||
container,
|
||||
"python",
|
||||
"-c",
|
||||
code,
|
||||
)
|
||||
|
||||
|
||||
def main():
|
||||
args = arguments("Sequential API container replacement")
|
||||
|
||||
client = Client(
|
||||
args.base_url,
|
||||
args.ca_file,
|
||||
timeout=2,
|
||||
)
|
||||
alice, bob, eve, chat = client.bootstrap()
|
||||
|
||||
import uuid
|
||||
|
||||
key = uuid.uuid4().hex
|
||||
text = "Сохранить при замене " + key
|
||||
|
||||
original_message = client.message(
|
||||
alice,
|
||||
chat,
|
||||
text,
|
||||
key=key,
|
||||
)["json"]
|
||||
|
||||
original_ids = sorted(api_ids())
|
||||
check(len(original_ids) == 2, "expected exactly two API containers")
|
||||
|
||||
for container in original_ids:
|
||||
check(healthy(container), "both API containers must be healthy")
|
||||
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"event": "replacement_started",
|
||||
"utc": timestamp(),
|
||||
"original_container_ids": original_ids,
|
||||
},
|
||||
ensure_ascii=False,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
|
||||
replacements = []
|
||||
|
||||
for old in original_ids:
|
||||
current = api_ids()
|
||||
check(len(current) == 2, "expected two APIs before replacement")
|
||||
check(old in current, "original target container is missing")
|
||||
|
||||
other = next(container for container in current if container != old)
|
||||
check(healthy(other), "remaining API is not healthy")
|
||||
|
||||
other_started = inspect(other, "{{.State.StartedAt}}")
|
||||
old_instance = instance_id(old)
|
||||
step_started = time.monotonic()
|
||||
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"event": "replacing",
|
||||
"utc": timestamp(),
|
||||
"old_container_id": old,
|
||||
"old_instance": old_instance,
|
||||
"remaining_container_id": other,
|
||||
},
|
||||
ensure_ascii=False,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
|
||||
command("docker", "stop", "--time", "10", old)
|
||||
command("docker", "rm", old)
|
||||
|
||||
compose(
|
||||
"up",
|
||||
"-d",
|
||||
"--no-deps",
|
||||
"--no-recreate",
|
||||
"--no-build",
|
||||
"--scale",
|
||||
"api=2",
|
||||
"api",
|
||||
)
|
||||
|
||||
deadline = time.monotonic() + 30
|
||||
replacement = None
|
||||
|
||||
while time.monotonic() < deadline:
|
||||
current = api_ids()
|
||||
candidates = current - {other}
|
||||
|
||||
if (
|
||||
len(current) == 2
|
||||
and len(candidates) == 1
|
||||
and other in current
|
||||
):
|
||||
candidate = next(iter(candidates))
|
||||
if candidate != old and healthy(candidate):
|
||||
replacement = candidate
|
||||
break
|
||||
|
||||
time.sleep(0.5)
|
||||
|
||||
check(replacement is not None, "replacement did not become healthy")
|
||||
|
||||
new_instance = instance_id(replacement)
|
||||
check(new_instance != old_instance, "instance did not change")
|
||||
|
||||
seen_through_lb = False
|
||||
deadline = time.monotonic() + 20
|
||||
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
response = client.call(
|
||||
"GET",
|
||||
"/api/v1/users/me",
|
||||
actor=alice,
|
||||
schema="User",
|
||||
)
|
||||
except Failure:
|
||||
time.sleep(0.2)
|
||||
continue
|
||||
|
||||
if response["headers"]["X-Instance-Id"] == new_instance:
|
||||
seen_through_lb = True
|
||||
break
|
||||
|
||||
time.sleep(0.2)
|
||||
|
||||
check(
|
||||
seen_through_lb,
|
||||
"new API was not observed through LB with existing session",
|
||||
)
|
||||
check(healthy(other), "remaining API became unhealthy")
|
||||
check(
|
||||
inspect(other, "{{.State.StartedAt}}") == other_started,
|
||||
"the other API was restarted unexpectedly",
|
||||
)
|
||||
|
||||
replay = client.message(
|
||||
alice,
|
||||
chat,
|
||||
text,
|
||||
key=key,
|
||||
)["json"]
|
||||
|
||||
check(
|
||||
replay == original_message,
|
||||
"confirmed message or idempotency result changed",
|
||||
)
|
||||
|
||||
result = {
|
||||
"event": "replacement_ready",
|
||||
"utc": timestamp(),
|
||||
"old_container_id": old,
|
||||
"new_container_id": replacement,
|
||||
"old_instance": old_instance,
|
||||
"new_instance": new_instance,
|
||||
"step_seconds": round(
|
||||
time.monotonic() - step_started, 3
|
||||
),
|
||||
"other_api_not_restarted": True,
|
||||
"existing_session_works": True,
|
||||
"replay_preserved": True,
|
||||
}
|
||||
|
||||
replacements.append(result)
|
||||
print(json.dumps(result, ensure_ascii=False), flush=True)
|
||||
|
||||
time.sleep(3)
|
||||
|
||||
final_ids = api_ids()
|
||||
|
||||
check(len(final_ids) == 2, "expected two APIs after replacement")
|
||||
check(
|
||||
not final_ids.intersection(original_ids),
|
||||
"some original API containers were not replaced",
|
||||
)
|
||||
|
||||
for container in final_ids:
|
||||
check(healthy(container), "final API is not healthy")
|
||||
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"event": "replacement_finished",
|
||||
"utc": timestamp(),
|
||||
"replacements": len(replacements),
|
||||
"final_container_ids": sorted(final_ids),
|
||||
},
|
||||
ensure_ascii=False,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
|
||||
print(
|
||||
"PASS: both API containers replaced sequentially; "
|
||||
"existing session and confirmed message preserved.",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
run(main)
|
||||
@@ -0,0 +1,274 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import secrets
|
||||
import time
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from queued_restart import ROOT, api, dc, http, wait_ready
|
||||
|
||||
|
||||
EVIDENCE = ROOT / "evidence" / "staging_cleanup.json"
|
||||
|
||||
|
||||
def storage_action(attachment_id, action):
|
||||
code = """
|
||||
import json
|
||||
import sys
|
||||
from uuid import UUID
|
||||
|
||||
from app.job_db import connect_db
|
||||
from app.storage import _get, _put, bucket_name, get_client
|
||||
|
||||
attachment_id = UUID(sys.argv[1])
|
||||
action = sys.argv[2]
|
||||
prefix = f"staging/{attachment_id}/"
|
||||
extra = prefix + "v99"
|
||||
client = get_client()
|
||||
bucket = bucket_name()
|
||||
|
||||
if action == "add_orphan":
|
||||
_put(extra, b"synthetic unreferenced staging object", "image/png")
|
||||
|
||||
elif action == "remove_orphan":
|
||||
client.delete_object(Bucket=bucket, Key=extra)
|
||||
|
||||
elif action == "add_ready_leftover":
|
||||
with connect_db() as db:
|
||||
with db.transaction():
|
||||
row = db.execute(
|
||||
'''
|
||||
SELECT object_key, content_type, status
|
||||
FROM attachments
|
||||
WHERE id = %s
|
||||
FOR UPDATE
|
||||
''',
|
||||
(attachment_id,),
|
||||
).fetchone()
|
||||
if row is None or row["status"] != "ready":
|
||||
raise RuntimeError("Expected a genuinely ready attachment")
|
||||
data = _get(row["object_key"])
|
||||
staging_key = prefix + "v1"
|
||||
_put(staging_key, data, row["content_type"])
|
||||
db.execute(
|
||||
"UPDATE attachments SET staging_key = %s WHERE id = %s",
|
||||
(staging_key, attachment_id),
|
||||
)
|
||||
|
||||
elif action != "inspect":
|
||||
raise ValueError("Unknown action")
|
||||
|
||||
keys = []
|
||||
for page in client.get_paginator("list_objects_v2").paginate(
|
||||
Bucket=bucket, Prefix=prefix,
|
||||
):
|
||||
keys.extend(item["Key"] for item in page.get("Contents", []))
|
||||
|
||||
with connect_db() as db:
|
||||
row = db.execute(
|
||||
'''
|
||||
SELECT status, staging_key IS NOT NULL AS staging_reference
|
||||
FROM attachments WHERE id = %s
|
||||
''',
|
||||
(attachment_id,),
|
||||
).fetchone()
|
||||
|
||||
print(json.dumps({"keys": sorted(keys), **row}))
|
||||
"""
|
||||
return json.loads(
|
||||
dc(
|
||||
"exec", "-T", "api",
|
||||
"python", "-c", code, attachment_id, action,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def cleanup(attachment_id, *, apply=False):
|
||||
args = [
|
||||
"exec", "-T", "worker",
|
||||
"python", "-m", "app.cleanup_staging",
|
||||
"--attachment-id", attachment_id,
|
||||
"--grace-seconds", "0",
|
||||
"--allow-short-grace",
|
||||
]
|
||||
if apply:
|
||||
args.append("--apply")
|
||||
return json.loads(dc(*args))
|
||||
|
||||
|
||||
def main():
|
||||
evidence = {"scenario": "scoped staging cleanup safety"}
|
||||
attachment_id = None
|
||||
restore = False
|
||||
|
||||
try:
|
||||
wait_ready()
|
||||
if not dc("ps", "--status", "running", "-q", "worker"):
|
||||
raise AssertionError("Start worker before this test")
|
||||
|
||||
username = "cln_" + uuid4().hex[:16]
|
||||
password = secrets.token_hex(16)
|
||||
user = api(
|
||||
"POST", "/api/v1/users", 201,
|
||||
payload={
|
||||
"username": username,
|
||||
"display_name": "Staging cleanup test",
|
||||
"password": password,
|
||||
},
|
||||
)
|
||||
basic = base64.b64encode(
|
||||
f"{username}:{password}".encode()
|
||||
).decode()
|
||||
token = api(
|
||||
"POST", "/api/v1/auth/sessions", 201,
|
||||
headers={"Authorization": "Basic " + basic},
|
||||
)["token"]
|
||||
|
||||
chat = api(
|
||||
"POST", "/api/v1/chats", 201,
|
||||
token=token,
|
||||
payload={
|
||||
"title": "Staging cleanup",
|
||||
"member_ids": [user["id"]],
|
||||
},
|
||||
)
|
||||
|
||||
restore = True
|
||||
dc("stop", "worker")
|
||||
assert not dc("ps", "--status", "running", "-q", "worker")
|
||||
|
||||
original = (ROOT / "tests/fixtures/sample.png").read_bytes()
|
||||
attachment = api(
|
||||
"POST",
|
||||
f"/api/v1/chats/{chat['id']}/attachments?filename=sample.png",
|
||||
202,
|
||||
token=token,
|
||||
data=original,
|
||||
headers={"Content-Type": "image/png"},
|
||||
)
|
||||
attachment_id = str(UUID(attachment["id"]))
|
||||
path = f"/api/v1/attachments/{attachment_id}"
|
||||
active_key = f"staging/{attachment_id}/v1"
|
||||
orphan_key = f"staging/{attachment_id}/v99"
|
||||
|
||||
state = storage_action(attachment_id, "add_orphan")
|
||||
assert state["status"] == "queued"
|
||||
assert state["keys"] == sorted([active_key, orphan_key])
|
||||
|
||||
# worker остановлен, поэтому уборщик запускаем в API-контейнере.
|
||||
# Модуль и настройки хранилищ у него те же.
|
||||
def scoped_cleanup(apply):
|
||||
args = [
|
||||
"exec", "-T", "api",
|
||||
"python", "-m", "app.cleanup_staging",
|
||||
"--attachment-id", attachment_id,
|
||||
"--grace-seconds", "0",
|
||||
"--allow-short-grace",
|
||||
]
|
||||
if apply:
|
||||
args.append("--apply")
|
||||
return json.loads(dc(*args))
|
||||
|
||||
dry = scoped_cleanup(False)
|
||||
assert dry["active"] == 1
|
||||
assert dry["candidates"] == 1
|
||||
assert dry["deleted"] == 0
|
||||
assert storage_action(attachment_id, "inspect")["keys"] == state["keys"]
|
||||
|
||||
applied = scoped_cleanup(True)
|
||||
assert applied["active"] == 1
|
||||
assert applied["deleted"] == 1
|
||||
|
||||
state = storage_action(attachment_id, "inspect")
|
||||
assert state["keys"] == [active_key]
|
||||
assert state["staging_reference"] is True
|
||||
assert api("GET", path, 200, token=token)["status"] == "queued"
|
||||
|
||||
evidence["queued_dry_run"] = dry
|
||||
evidence["queued_apply"] = applied
|
||||
print("Dry-run preserved both objects; apply removed only the orphan")
|
||||
|
||||
dc("start", "worker")
|
||||
deadline = time.monotonic() + 90
|
||||
while time.monotonic() < deadline:
|
||||
ready = api("GET", path, 200, token=token)
|
||||
if ready["status"] == "ready":
|
||||
break
|
||||
if ready["status"] == "failed":
|
||||
raise AssertionError("Active staging was not processed")
|
||||
time.sleep(0.5)
|
||||
else:
|
||||
raise AssertionError("Ready timeout")
|
||||
|
||||
# Дожидаемся завершения штатной уборки в job_runner.
|
||||
deadline = time.monotonic() + 30
|
||||
while time.monotonic() < deadline:
|
||||
state = storage_action(attachment_id, "inspect")
|
||||
if not state["keys"] and not state["staging_reference"]:
|
||||
break
|
||||
time.sleep(0.5)
|
||||
else:
|
||||
raise AssertionError("Normal staging cleanup did not finish")
|
||||
|
||||
# Имитируем остаток после сбоя удаления, не подделывая ready:
|
||||
# оригинал и варианты уже созданы настоящим worker.
|
||||
storage_action(attachment_id, "add_ready_leftover")
|
||||
|
||||
ready_dry = cleanup(attachment_id)
|
||||
assert ready_dry["candidates"] == 1
|
||||
assert ready_dry["deleted"] == 0
|
||||
|
||||
ready_apply = cleanup(attachment_id, apply=True)
|
||||
assert ready_apply["deleted"] == 1
|
||||
|
||||
state = storage_action(attachment_id, "inspect")
|
||||
assert state["status"] == "ready"
|
||||
assert state["keys"] == []
|
||||
assert state["staging_reference"] is False
|
||||
|
||||
assert api("GET", path, 200, token=token) == ready
|
||||
status, body = http("GET", path + "/content", token=token)
|
||||
assert status == 200 and body == original
|
||||
|
||||
for variant in ready["variants"]:
|
||||
status, body = http(
|
||||
"GET", path + "/content?variant=" + variant["name"],
|
||||
token=token,
|
||||
)
|
||||
assert status == 200
|
||||
assert len(body) == variant["size_bytes"]
|
||||
|
||||
evidence["attachment_id"] = attachment_id
|
||||
evidence["ready_dry_run"] = ready_dry
|
||||
evidence["ready_apply"] = ready_apply
|
||||
evidence["final_storage_state"] = state
|
||||
evidence["result"] = "PASS"
|
||||
|
||||
except BaseException as exc:
|
||||
evidence["result"] = "FAIL"
|
||||
evidence["error_type"] = type(exc).__name__
|
||||
raise
|
||||
|
||||
finally:
|
||||
try:
|
||||
if attachment_id is not None:
|
||||
storage_action(attachment_id, "remove_orphan")
|
||||
finally:
|
||||
try:
|
||||
if restore:
|
||||
dc("start", "worker")
|
||||
finally:
|
||||
EVIDENCE.parent.mkdir(exist_ok=True)
|
||||
EVIDENCE.write_text(
|
||||
json.dumps(evidence, ensure_ascii=False, indent=2) + "\n"
|
||||
)
|
||||
|
||||
print(
|
||||
"PASS: dry-run is non-destructive; queued staging preserved; "
|
||||
"orphan and ready leftover removed; ready content intact"
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,318 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Targeted API failure test. Uses synthetic data; leaves target stopped."""
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import uuid
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
|
||||
from client import Client, Failure, arguments, check, run
|
||||
from resilience import history
|
||||
|
||||
|
||||
PROJECT = "mtusi2026lab3"
|
||||
|
||||
|
||||
def docker_output(*args):
|
||||
return subprocess.check_output(
|
||||
["docker", *args],
|
||||
cwd=ROOT,
|
||||
text=True,
|
||||
).strip()
|
||||
|
||||
|
||||
def target_instance(container):
|
||||
labels = json.loads(
|
||||
docker_output(
|
||||
"inspect",
|
||||
"--format",
|
||||
"{{json .Config.Labels}}",
|
||||
container,
|
||||
)
|
||||
)
|
||||
|
||||
check(
|
||||
labels.get("com.docker.compose.project") == PROJECT,
|
||||
"target belongs to another Compose project",
|
||||
)
|
||||
check(
|
||||
labels.get("com.docker.compose.service") == "api",
|
||||
"target is not an API container",
|
||||
)
|
||||
check(
|
||||
docker_output(
|
||||
"inspect",
|
||||
"--format",
|
||||
"{{.State.Running}}",
|
||||
container,
|
||||
) == "true",
|
||||
"target container is not running",
|
||||
)
|
||||
|
||||
code = (
|
||||
"import urllib.request; "
|
||||
"r=urllib.request.urlopen("
|
||||
"'http://127.0.0.1:8080/health/live', timeout=2); "
|
||||
"print(r.headers.get('X-Instance-Id', '')); "
|
||||
"r.close()"
|
||||
)
|
||||
|
||||
instance = docker_output(
|
||||
"exec", container, "python", "-c", code
|
||||
)
|
||||
check(bool(instance), "target has no X-Instance-Id")
|
||||
return instance
|
||||
|
||||
|
||||
def main():
|
||||
def configure(parser):
|
||||
parser.add_argument(
|
||||
"--target-container",
|
||||
required=True,
|
||||
)
|
||||
|
||||
args = arguments(
|
||||
"Targeted API failover through the load balancer",
|
||||
configure,
|
||||
)
|
||||
|
||||
script = ROOT / "scripts" / "stop-one.sh"
|
||||
check(
|
||||
script.is_file() and os.access(script, os.X_OK),
|
||||
"stop-one.sh must exist and be executable",
|
||||
)
|
||||
|
||||
target = target_instance(args.target_container)
|
||||
client = Client(args.base_url, args.ca_file, timeout=2)
|
||||
|
||||
alice, bob, eve, chat = client.bootstrap()
|
||||
|
||||
original_key = uuid.uuid4().hex
|
||||
original_text = "Подтверждено до отказа " + original_key
|
||||
original = client.message(
|
||||
alice,
|
||||
chat,
|
||||
original_text,
|
||||
key=original_key,
|
||||
)["json"]
|
||||
|
||||
observed = set()
|
||||
for _ in range(40):
|
||||
response = client.call(
|
||||
"GET",
|
||||
"/api/v1/users/me",
|
||||
actor=alice,
|
||||
schema="User",
|
||||
)
|
||||
observed.add(response["headers"]["X-Instance-Id"])
|
||||
|
||||
check(len(observed) >= 2, "less than two API instances")
|
||||
check(target in observed, "target is not observed through LB")
|
||||
|
||||
env = dict(os.environ)
|
||||
env["TARGET_INSTANCE_ID"] = target
|
||||
|
||||
print(
|
||||
"Target container:",
|
||||
args.target_container,
|
||||
"| instance:",
|
||||
target,
|
||||
flush=True,
|
||||
)
|
||||
|
||||
started = time.monotonic()
|
||||
proc = subprocess.Popen(
|
||||
[str(script)],
|
||||
cwd=ROOT,
|
||||
env=env,
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,
|
||||
)
|
||||
|
||||
successes = []
|
||||
write_failures = 0
|
||||
read_successes = 0
|
||||
read_failures = 0
|
||||
after_instances = []
|
||||
first_recovery = None
|
||||
attempts = 0
|
||||
|
||||
key = uuid.uuid4().hex
|
||||
text = "Во время отказа " + key
|
||||
|
||||
try:
|
||||
while time.monotonic() - started < 15:
|
||||
code = proc.poll()
|
||||
elapsed = time.monotonic() - started
|
||||
|
||||
check(
|
||||
code is None or code == 0,
|
||||
"stop-one hook failed",
|
||||
)
|
||||
check(
|
||||
code is not None or elapsed < 10,
|
||||
"stop-one hook exceeded 10 seconds",
|
||||
)
|
||||
|
||||
attempts += 1
|
||||
|
||||
try:
|
||||
response = client.message(
|
||||
alice,
|
||||
chat,
|
||||
text,
|
||||
key=key,
|
||||
)
|
||||
except Failure:
|
||||
write_failures += 1
|
||||
# Keep the same key and text after uncertain POST.
|
||||
else:
|
||||
successes.append(response["json"])
|
||||
|
||||
if proc.poll() == 0:
|
||||
if first_recovery is None:
|
||||
first_recovery = time.monotonic() - started
|
||||
|
||||
after_instances.append(
|
||||
response["headers"]["X-Instance-Id"]
|
||||
)
|
||||
|
||||
key = uuid.uuid4().hex
|
||||
text = "Во время отказа " + key
|
||||
|
||||
if attempts % 5 == 0:
|
||||
try:
|
||||
page = client.call(
|
||||
"GET",
|
||||
(
|
||||
"/api/v1/chats/"
|
||||
+ chat["id"]
|
||||
+ "/messages?limit=100"
|
||||
),
|
||||
actor=bob,
|
||||
schema="MessagePage",
|
||||
)["json"]
|
||||
except Failure:
|
||||
read_failures += 1
|
||||
else:
|
||||
check(
|
||||
any(
|
||||
message == original
|
||||
for message in page["items"]
|
||||
),
|
||||
"confirmed message absent during read",
|
||||
)
|
||||
read_successes += 1
|
||||
|
||||
time.sleep(0.1)
|
||||
|
||||
elapsed = time.monotonic() - started
|
||||
|
||||
check(proc.poll() == 0, "stop-one hook did not finish")
|
||||
check(
|
||||
first_recovery is not None and first_recovery <= 10,
|
||||
"writes did not recover within 10 seconds",
|
||||
)
|
||||
check(
|
||||
len(after_instances) >= 5,
|
||||
"too few writes after stop",
|
||||
)
|
||||
check(
|
||||
target not in after_instances[-5:],
|
||||
"stopped instance still serves requests",
|
||||
)
|
||||
check(read_successes > 0, "no successful history reads")
|
||||
|
||||
check(
|
||||
docker_output(
|
||||
"inspect",
|
||||
"--format",
|
||||
"{{.State.Running}}",
|
||||
args.target_container,
|
||||
) == "false",
|
||||
"target container is still running",
|
||||
)
|
||||
|
||||
replay = client.message(
|
||||
alice,
|
||||
chat,
|
||||
original_text,
|
||||
key=original_key,
|
||||
)["json"]
|
||||
|
||||
check(
|
||||
replay == original,
|
||||
"replay after failure returned another message",
|
||||
)
|
||||
|
||||
messages = history(client, bob, chat)
|
||||
by_id = {message["id"]: message for message in messages}
|
||||
|
||||
check(
|
||||
len(by_id) == len(messages),
|
||||
"duplicate message IDs",
|
||||
)
|
||||
check(
|
||||
by_id.get(original["id"]) == original,
|
||||
"pre-failure message lost or altered",
|
||||
)
|
||||
|
||||
for message in successes:
|
||||
check(
|
||||
by_id.get(message["id"]) == message,
|
||||
"confirmed in-flight message lost or altered",
|
||||
)
|
||||
|
||||
texts = [message["text"] for message in messages]
|
||||
check(
|
||||
len(texts) == len(set(texts)),
|
||||
"retry created duplicate message text",
|
||||
)
|
||||
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"target_container": args.target_container,
|
||||
"target_instance": target,
|
||||
"observed_instances_before": len(observed),
|
||||
"duration_seconds": round(elapsed, 3),
|
||||
"successful_writes": len(successes),
|
||||
"failed_write_attempts": write_failures,
|
||||
"successful_reads": read_successes,
|
||||
"failed_read_attempts": read_failures,
|
||||
"successful_write_rps": round(
|
||||
len(successes) / elapsed, 3
|
||||
),
|
||||
"first_recovery_seconds": round(
|
||||
first_recovery, 3
|
||||
),
|
||||
"replay_preserved": True,
|
||||
"target_stopped": True,
|
||||
},
|
||||
ensure_ascii=False,
|
||||
)
|
||||
)
|
||||
|
||||
print(
|
||||
"PASS: targeted failover; session, messages "
|
||||
"and replay preserved. Target remains stopped."
|
||||
)
|
||||
|
||||
finally:
|
||||
if proc.poll() is None:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
proc.wait()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
run(main)
|
||||
@@ -0,0 +1,147 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
import unittest
|
||||
from unittest.mock import MagicMock, patch
|
||||
from uuid import UUID
|
||||
|
||||
from app.cleanup_staging import sweep
|
||||
|
||||
|
||||
ATTACHMENT_ID = UUID("11111111-1111-4111-8111-111111111111")
|
||||
KEY = f"staging/{ATTACHMENT_ID}/v1"
|
||||
|
||||
|
||||
class CleanupStagingTests(unittest.TestCase):
|
||||
def run_sweep(
|
||||
self,
|
||||
*,
|
||||
status=None,
|
||||
locked=True,
|
||||
recent_failure=False,
|
||||
recent_object=False,
|
||||
apply=True,
|
||||
):
|
||||
now = datetime.now(timezone.utc)
|
||||
old = now - timedelta(days=2)
|
||||
modified = now + timedelta(seconds=1) if recent_object else old
|
||||
|
||||
row = None
|
||||
if status is not None:
|
||||
row = {
|
||||
"id": ATTACHMENT_ID,
|
||||
"status": status,
|
||||
"updated_at": (
|
||||
now + timedelta(seconds=1)
|
||||
if recent_failure else old
|
||||
),
|
||||
}
|
||||
|
||||
db = MagicMock()
|
||||
db.__enter__.return_value = db
|
||||
|
||||
def execute(query, params=None):
|
||||
cursor = MagicMock()
|
||||
if "pg_try_advisory_xact_lock" in query:
|
||||
cursor.fetchone.return_value = {"locked": locked}
|
||||
elif "SELECT id, status, updated_at" in query:
|
||||
cursor.fetchone.return_value = row
|
||||
return cursor
|
||||
|
||||
db.execute.side_effect = execute
|
||||
|
||||
client = MagicMock()
|
||||
client.get_paginator.return_value.paginate.return_value = [
|
||||
{"Contents": [{"Key": KEY, "LastModified": modified}]}
|
||||
]
|
||||
client.head_object.return_value = {"LastModified": modified}
|
||||
|
||||
with (
|
||||
patch("app.cleanup_staging.connect_db", return_value=db),
|
||||
patch("app.cleanup_staging.get_client", return_value=client),
|
||||
patch("app.cleanup_staging.bucket_name", return_value="test-bucket"),
|
||||
):
|
||||
result = sweep(
|
||||
apply=apply,
|
||||
grace_seconds=86400,
|
||||
attachment_id=ATTACHMENT_ID,
|
||||
)
|
||||
|
||||
return result, db, client
|
||||
|
||||
def assert_no_delete(self, client):
|
||||
client.delete_object.assert_not_called()
|
||||
|
||||
def test_queued_is_preserved_even_when_old(self):
|
||||
result, _, client = self.run_sweep(status="queued")
|
||||
self.assertEqual(result["active"], 1)
|
||||
self.assertEqual(result["deleted"], 0)
|
||||
self.assert_no_delete(client)
|
||||
|
||||
def test_processing_is_preserved_even_when_old(self):
|
||||
result, _, client = self.run_sweep(status="processing")
|
||||
self.assertEqual(result["active"], 1)
|
||||
self.assertEqual(result["deleted"], 0)
|
||||
self.assert_no_delete(client)
|
||||
|
||||
def test_busy_upload_lock_prevents_deletion(self):
|
||||
result, _, client = self.run_sweep(locked=False)
|
||||
self.assertEqual(result["busy"], 1)
|
||||
self.assert_no_delete(client)
|
||||
client.head_object.assert_not_called()
|
||||
|
||||
def test_recent_failed_state_preserves_old_object(self):
|
||||
result, _, client = self.run_sweep(
|
||||
status="failed",
|
||||
recent_failure=True,
|
||||
)
|
||||
self.assertEqual(result["fresh"], 1)
|
||||
self.assert_no_delete(client)
|
||||
|
||||
def test_recent_object_is_preserved(self):
|
||||
result, _, client = self.run_sweep(
|
||||
status="failed",
|
||||
recent_object=True,
|
||||
)
|
||||
self.assertEqual(result["fresh"], 1)
|
||||
self.assert_no_delete(client)
|
||||
|
||||
def test_expired_failed_staging_is_deleted_and_reference_cleared(self):
|
||||
result, db, client = self.run_sweep(status="failed")
|
||||
self.assertEqual(result["deleted"], 1)
|
||||
client.delete_object.assert_called_once_with(
|
||||
Bucket="test-bucket",
|
||||
Key=KEY,
|
||||
)
|
||||
updates = [
|
||||
call for call in db.execute.call_args_list
|
||||
if "SET staging_key = NULL" in call.args[0]
|
||||
]
|
||||
self.assertEqual(len(updates), 1)
|
||||
self.assertEqual(updates[0].args[1], (ATTACHMENT_ID, KEY))
|
||||
|
||||
def test_expired_failed_dry_run_is_non_destructive(self):
|
||||
result, db, client = self.run_sweep(
|
||||
status="failed",
|
||||
apply=False,
|
||||
)
|
||||
self.assertEqual(result["candidates"], 1)
|
||||
self.assertEqual(result["deleted"], 0)
|
||||
self.assert_no_delete(client)
|
||||
self.assertFalse(any(
|
||||
"SET staging_key = NULL" in call.args[0]
|
||||
for call in db.execute.call_args_list
|
||||
))
|
||||
|
||||
def test_old_orphan_is_deleted_without_attachment_update(self):
|
||||
result, db, client = self.run_sweep()
|
||||
self.assertEqual(result["deleted"], 1)
|
||||
client.delete_object.assert_called_once()
|
||||
self.assertFalse(any(
|
||||
"SET staging_key = NULL" in call.args[0]
|
||||
for call in db.execute.call_args_list
|
||||
))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
@@ -0,0 +1,94 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import io
|
||||
import unittest
|
||||
|
||||
from PIL import Image
|
||||
|
||||
from app.image_pipeline import InvalidImage, render_variants
|
||||
|
||||
|
||||
def make_image(size, *, mode="RGB", color="red", orientation=None):
|
||||
with Image.new(mode, size, color) as image:
|
||||
output = io.BytesIO()
|
||||
if orientation is None:
|
||||
image.save(output, format="PNG")
|
||||
else:
|
||||
exif = Image.Exif()
|
||||
exif[274] = orientation
|
||||
image.save(output, format="JPEG", exif=exif)
|
||||
return output.getvalue()
|
||||
|
||||
|
||||
def variants_by_name(data):
|
||||
return {
|
||||
variant["name"]: variant
|
||||
for variant in render_variants(data)
|
||||
}
|
||||
|
||||
|
||||
class ImagePipelineTests(unittest.TestCase):
|
||||
def assert_variant(self, variant, fmt, size):
|
||||
data = variant["data"]
|
||||
self.assertEqual(variant["size_bytes"], len(data))
|
||||
self.assertEqual(
|
||||
variant["sha256"],
|
||||
hashlib.sha256(data).hexdigest(),
|
||||
)
|
||||
self.assertEqual(
|
||||
(variant["width"], variant["height"]),
|
||||
size,
|
||||
)
|
||||
with Image.open(io.BytesIO(data)) as image:
|
||||
image.load()
|
||||
self.assertEqual(image.format, fmt)
|
||||
self.assertEqual(image.size, size)
|
||||
self.assertFalse(dict(image.getexif()))
|
||||
self.assertFalse(image.info.get("icc_profile"))
|
||||
self.assertFalse(image.info.get("xmp"))
|
||||
|
||||
def test_contract_dimensions_and_original(self):
|
||||
data = make_image((320, 200))
|
||||
before = hashlib.sha256(data).hexdigest()
|
||||
variants = variants_by_name(data)
|
||||
|
||||
self.assertEqual(set(variants), {"thumbnail", "crop", "optimized"})
|
||||
self.assert_variant(variants["thumbnail"], "JPEG", (256, 160))
|
||||
self.assert_variant(variants["crop"], "JPEG", (128, 128))
|
||||
self.assert_variant(variants["optimized"], "WEBP", (320, 200))
|
||||
self.assertEqual(hashlib.sha256(data).hexdigest(), before)
|
||||
|
||||
def test_rounding_down(self):
|
||||
variants = variants_by_name(make_image((321, 200)))
|
||||
self.assert_variant(variants["thumbnail"], "JPEG", (256, 159))
|
||||
|
||||
def test_small_image_not_upscaled_except_crop(self):
|
||||
variants = variants_by_name(make_image((64, 32)))
|
||||
self.assert_variant(variants["thumbnail"], "JPEG", (64, 32))
|
||||
self.assert_variant(variants["optimized"], "WEBP", (64, 32))
|
||||
self.assert_variant(variants["crop"], "JPEG", (128, 128))
|
||||
|
||||
def test_transparency_becomes_white(self):
|
||||
data = make_image((64, 32), mode="RGBA", color=(255, 0, 0, 0))
|
||||
variants = variants_by_name(data)
|
||||
for variant in variants.values():
|
||||
with Image.open(io.BytesIO(variant["data"])) as image:
|
||||
pixel = image.convert("RGB").getpixel((0, 0))
|
||||
self.assertTrue(all(channel >= 245 for channel in pixel))
|
||||
|
||||
def test_exif_orientation(self):
|
||||
data = make_image((320, 200), orientation=6)
|
||||
variants = variants_by_name(data)
|
||||
self.assert_variant(variants["thumbnail"], "JPEG", (160, 256))
|
||||
self.assert_variant(variants["optimized"], "WEBP", (200, 320))
|
||||
|
||||
def test_invalid_input(self):
|
||||
for data in (b"", b"not an image"):
|
||||
with self.subTest(data=data):
|
||||
with self.assertRaises(InvalidImage):
|
||||
render_variants(data)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user