ЛР5 — БПИ2302 — Перевалова — на 5 #44

Closed
perevalova wants to merge 1 commits from perevalova/mtusi2026lab5:solution/bpi2302-perevalova into main
69 changed files with 7706 additions and 31 deletions
+23
View File
@@ -0,0 +1,23 @@
FROM python:3.11.9-slim-bookworm
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PIP_NO_CACHE_DIR=1
WORKDIR /app
RUN groupadd --system app && useradd --system --gid app --home-dir /app app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app ./app
COPY migrations ./migrations
COPY scripts ./scripts
COPY tests ./tests
COPY contracts ./contracts
COPY lab.json .
COPY alembic.ini .
RUN chown -R app:app /app
USER app
EXPOSE 8080
HEALTHCHECK --interval=5s --timeout=3s --start-period=20s --retries=3 \
CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=2)"
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8080", "--timeout-keep-alive", "5", "--timeout-graceful-shutdown", "20"]
+316 -24
View File
@@ -1,37 +1,329 @@
# Отчёт — лабораторная 5
# Отчёт — лабораторная работа № 5
## Очереди и асинхронная обработка изображений
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
- **Автор, группа:** Перевалова Виктория, БПИ2302
- **Целевая оценка:** 5
- **Выбранный трек:** Трек 1 — очередь под перегрузкой
- **Commit/PR:** заполнить после публикации
- **Использованные источники, библиотеки и помощь ИИ:** исходный репозиторий лабораторной работы и его контракты; FastAPI, SQLAlchemy, Alembic, aio-pika, Pillow, boto3, Redis; Docker Compose. При подготовке и отладке использовалась помощь ChatGPT.
## Запуск из чистого clone
---
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
## 1. Запуск из чистого clone
## Архитектура и решения
### Требования
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab4, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
- Docker Engine и Docker Compose plugin.
- Git.
- Для локальных тестовых скриптов: Python 3.11.9 и `pip`.
- Для локального окружения рекомендуется виртуальное окружение Python.
## Доказательства критериев
Приложение собирается в Docker-образе на основе `python:3.11.9-slim-bookworm`. Зависимости приложения зафиксированы в `requirements.txt`.
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
### Подготовка
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
```powershell
git clone <URL_репозитория>
cd mtusi2026lab5
## Эксперименты и отказы
Copy-Item .env.example .env
```
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
Перед запуском необходимо заполнить локальный `.env` значениями для PostgreSQL, RabbitMQ, Redis и S3/SeaweedFS. Секреты нельзя добавлять в Git или в отчёт.
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
### Сборка и запуск
## Ограничения и следующие шаги
```powershell
docker compose --env-file .env config --quiet
docker compose --env-file .env up -d --build
docker compose --env-file .env ps
```
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
Сервис `migrate` выполняет `alembic upgrade head`; API и worker зависят от успешного завершения миграций. PostgreSQL и RabbitMQ проходят healthcheck перед запуском зависимых сервисов.
В используемой конфигурации API доступен через Nginx:
```text
http://localhost:18081
```
Проверка готовности:
```powershell
Invoke-RestMethod http://localhost:18081/health/ready
```
Публичный smoke-тест:
```powershell
python tests/smoke.py --base-url http://localhost:18081
```
Локальная проверка OpenAPI:
```powershell
python scripts/validate_openapi.py
```
Проверка структуры контракта:
```powershell
python scripts/check_contract.py
```
Для локального запуска тестовых скриптов:
```powershell
py -3.11 -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -r requirements-dev.txt
```
### Остановка
Остановить контейнеры, сохранив данные в volumes:
```powershell
docker compose --env-file .env down
```
Удалять учебные данные вместе с volumes следует только намеренно:
```powershell
docker compose --env-file .env down -v
```
Команда `down -v` необратимо удаляет данные PostgreSQL, RabbitMQ, Redis и SeaweedFS.
### Основные версии
| Компонент | Версия / образ |
|---|---|
| Python | 3.11.9 |
| FastAPI | 0.115.6 |
| Uvicorn | 0.34.0 |
| Pydantic | 2.10.4 |
| SQLAlchemy | 2.0.36 |
| psycopg | 3.2.3 |
| Alembic | 1.14.0 |
| Redis client | 5.2.1 |
| boto3 | 1.35.99 |
| Pillow | 11.0.0 |
| aio-pika | 9.5.5 |
| PostgreSQL | 16.6-bookworm |
| Redis | 7.4.2-alpine |
| RabbitMQ | 4.0.5-management-alpine |
| SeaweedFS | 4.46 |
| Nginx | 1.27.4-alpine |
---
## 2. Архитектура и принятые решения
### Общая схема
```text
Клиент
|
v
Nginx :18081
|
v
FastAPI
|------------------------------|
| |
v v
SeaweedFS staging PostgreSQL
(исходный файл) (attachment/job/outbox)
|
v
Outbox Publisher
|
v
RabbitMQ
|
v
Worker
|
|--------------|--------------|
v v v
SeaweedFS PostgreSQL RabbitMQ
(результаты) (статус/lease) (retry/DLQ)
```
### Компоненты
- **Nginx** — единая HTTP-точка входа; проксирует запросы к API.
- **FastAPI** — принимает загрузку, проверяет входные данные, создаёт вложение и задание, предоставляет API для получения состояния и содержимого.
- **PostgreSQL** — источник истины для метаданных вложений, заданий и transactional outbox.
- **SeaweedFS** — S3-совместимое объектное хранилище исходных файлов на staging-этапе и готовых вариантов.
- **RabbitMQ** — устойчивый брокер заданий с подтверждениями доставки.
- **Outbox Publisher** — публикует задания, сохранённые в outbox, чтобы закрыть разрыв между транзакцией БД и отправкой сообщения в брокер.
- **Worker** — отдельный процесс асинхронной обработки изображений.
- **Redis** — дополнительный инфраструктурный компонент, используемый приложением согласно конфигурации.
### Жизненный цикл вложения
1. Клиент отправляет файл в API.
2. API проверяет допустимость файла и сохраняет исходные байты в устойчивое staging-хранилище.
3. В транзакции PostgreSQL сохраняются метаданные вложения, задание и запись outbox.
4. API возвращает `202 Accepted` после устойчивого приёма файла и фиксации задания.
5. Outbox Publisher публикует сообщение в RabbitMQ.
6. Worker получает задание, захватывает его с lease и обрабатывает изображение.
7. Worker сохраняет оригинал и производные варианты в S3-хранилище, затем фиксирует итоговое состояние и метаданные в PostgreSQL.
8. После безопасной фиксации результата staging-объект может быть удалён.
9. Клиент получает актуальное состояние через API.
В сообщение брокера не помещаются байты изображения и credentials: передаются идентификаторы задания/вложения и необходимые метаданные.
### Варианты изображения
Реализованы:
- оригинал;
- `thumbnail` размером 256×256;
- `crop` размером 128×128;
- `optimized` в формате WebP.
Для WebP используется логическое имя варианта `optimized`; формат и content type соответствуют WebP. Оригинал сохраняется отдельно от производных вариантов.
### Надёжность и обработка ошибок
- Состояния вложения: `queued`, `processing`, `ready`, `failed`.
- Доставка задания — at-least-once; повторная доставка считается штатной ситуацией.
- Повторная обработка использует стабильные ключи объектов и идемпотентную запись результатов.
- Transactional outbox позволяет восстановить публикацию задания после сбоя между фиксацией транзакции PostgreSQL и публикацией в RabbitMQ.
- Повторы ограничены настройкой `WORKER_MAX_ATTEMPTS` (в текущем стенде — 5).
- Для временных ошибок используется backoff с jitter; постоянные ошибки завершают задание с безопасным `error_code`.
- Для неуспешных заданий предусмотрен реестр failed/DLQ и отдельный явный replay через `scripts/replay_failed_job.py`.
- Worker использует lease/heartbeat и fencing для контроля владения заданием.
- Обработка изображения выполняется в отдельном процессе с жёстким timeout. При превышении времени дочерний процесс завершается.
- Ограничения стенда: `MAX_UPLOAD_BYTES=10485760` (10 MiB), `MAX_IMAGE_PIXELS=40000000`, `IMAGE_PROCESSING_TIMEOUT_SECONDS=60`.
- Для worker в Compose заданы ограничения `cpus: 1.0` и `mem_limit: 1g`.
### Переменные окружения
Фактическая конфигурация приложения задаётся локальным `.env`. Используются настройки подключения к PostgreSQL, RabbitMQ, Redis и S3, ограничения загрузки и обработки, число попыток, lease, retry backoff, retention staging, concurrency и prefetch. Файл `.env` содержит секреты и не должен публиковаться. Имена переменных в `.env.example` являются шаблоном skeleton; при запуске следует ориентироваться на фактические настройки приложения и локальный `.env`.
---
## 3. Доказательства критериев
| Критерий | Команда / сценарий | Наблюдаемый результат | Доказательство |
|---|---|---|---|
| Публичный API smoke | `python tests/smoke.py --base-url http://localhost:18081` | `PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.` | `evidence/smoke.txt` |
| OpenAPI runtime | Получение runtime OpenAPI | JSON runtime-контракта сохранён | `evidence/openapi-runtime.json` |
| Проверка контракта | `python scripts/check_contract.py` | `PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.` | `evidence/check-contract.txt` |
| Валидация OpenAPI | `python scripts/validate_openapi.py` | `PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.` | `evidence/validate-openapi.txt` |
| Ограничение приёма / backpressure | `python -m tests.test_backpressure --base-url http://localhost:18081` | При остановленном worker приняты 20 заданий; 21-я загрузка получила HTTP 429; после запуска worker задания обработаны | `evidence/backpressure.txt` |
| Жёсткий timeout обработки | `python -m tests.test_image_processing_timeout` | Нормальная обработка создала варианты; принудительный timeout завершил дочерний процесс; `ALL TESTS PASSED` | `evidence/image-processing-timeout.txt` |
| Replay failed job | `python scripts/replay_failed_job.py --help` | CLI-справка получена. В текущей БД записей `attachments.status='failed'` нет, поэтому отдельный replay-сценарий в рамках данного запуска не выполнялся и не заявляется как подтверждённый evidence | Скрипт: `scripts/replay_failed_job.py`; отдельного `evidence/replay.txt` нет |
| Ограничения worker | `docker inspect` контейнера worker | `NanoCpus=1000000000 Memory=1073741824` — 1 CPU и 1 GiB памяти | `evidence/worker-limits.txt` |
| Конфигурация worker B | `docker compose --env-file .env logs --tail=20 worker` | Worker возвращён к конфигурации B: `concurrency=2 prefetch=4` | `evidence/worker-config.txt` |
| Track 1 — A | `python -m tests.track1_benchmark --config A --base-url http://localhost:18081` | 18/18; throughput 0,312 img/s; p95 14,844 s | `evidence/track1_A.csv` |
| Track 1 — B | `python -m tests.track1_benchmark --config B --base-url http://localhost:18081` | 18/18; throughput 0,374 img/s; p95 11,797 s | `evidence/track1_B.csv` |
| Track 1 — C | `python -m tests.track1_benchmark --config C --base-url http://localhost:18081` | 18/18; throughput 0,298 img/s; p95 16,797 s | `evidence/track1_C.csv` |
| Восстановление после эксперимента | Переключение worker обратно на B и smoke | Worker запущен с `concurrency=2/prefetch=4`; smoke PASS | `evidence/recovery-after-track1.txt` |
**Примечание:** в таблице перечислены фактически сохранённые результаты из каталога `evidence/`. Для replay отдельно зафиксировано ограничение: в текущей базе нет failed-вложений, поэтому запускать скрипт было не на чем. Это не подменяется искусственно созданным логом.
---
## 4. Эксперимент Track 1 — очередь под перегрузкой
### 4.1. Цель и гипотеза
Цель эксперимента — определить влияние concurrency и prefetch на пропускную способность, ожидание в очереди, время готовности изображений и потребление памяти worker.
**Гипотеза:** увеличение concurrency с 1 до 2 уменьшит ожидание в очереди и p95 time-to-ready. Дальнейшее увеличение до 4 при ограничении worker одним CPU может привести к конкуренции за ресурсы и ухудшить суммарную производительность.
### 4.2. Методика
Для каждого прогона использовался одинаковый набор из 18 синтетических изображений:
- 6 small — 640×480, около 91 KB;
- 6 medium — 1600×1200, около 556 KB;
- 6 large — 3000×2000, около 1,72 MB.
Все изображения укладываются в лимит 10 MiB и ограничение 40 млн пикселей.
Параметры стенда:
- worker: 1 CPU, 1 GiB RAM;
- API: `http://localhost:18081`;
- 18 заданий на конфигурацию, по 6 изображений каждого размера;
- параллельность загрузки клиента: 4;
- измерение времени от начала загрузки до состояния `ready`;
- queue lag и processing elapsed извлекались из структурированных логов worker;
- память измерялась через `docker stats` с периодом опроса 0,5 секунды.
### 4.3. Конфигурации
| Конфигурация | WORKER_CONCURRENCY | RABBITMQ_PREFETCH |
|---|---:|---:|
| A | 1 | 1 |
| B | 2 | 4 |
| C | 4 | 8 |
### 4.4. Результаты
| Метрика | A (1/1) | B (2/4) | C (4/8) |
|---|---:|---:|---:|
| Завершено заданий | 18/18 | 18/18 | 18/18 |
| Длительность, с | 57,72 | 48,17 | 60,33 |
| Throughput, изображений/с | 0,312 | **0,374** | 0,298 |
| p95 time-to-ready, с | 14,844 | **11,797** | 16,797 |
| Средний queue lag, мс | 8003,64 | 4226,12 | **496,06** |
| Средний processing elapsed, мс | **3053,64** | 5139,11 | 11542,28 |
| Пиковая память контейнера, MiB | 240,40 | 326,90 | 495,90 |
| Повторные загрузки | 0 | 0 | 0 |
### 4.5. Анализ
Конфигурация A показала минимальное потребление памяти и минимальное среднее processing elapsed, но при ней наблюдалось самое большое ожидание в очереди. Это увеличило общее время готовности.
Конфигурация B показала лучший общий результат: throughput увеличился примерно на 19,9% относительно A, p95 time-to-ready снизился примерно на 20,5%, а средний queue lag уменьшился примерно на 47,2%. Пиковая память контейнера составила 326,9 MiB.
В конфигурации C средний queue lag стал минимальным, однако processing elapsed вырос до 11,54 секунды, throughput снизился до 0,298 изображения/с, а p95 time-to-ready увеличился до 16,797 секунды. При лимите в один CPU четыре параллельных обработчика конкурируют за вычислительные ресурсы. Поэтому уменьшение ожидания в очереди не компенсировало рост времени выполнения заданий.
**Вывод:** для исследованной нагрузки оптимальна конфигурация B: `concurrency=2`, `prefetch=4`. Она обеспечивает лучший баланс throughput, p95 latency и потребления памяти.
### 4.6. Backpressure и восстановление
Отдельный тест backpressure выполнялся при остановленном worker. API принял 20 заданий — значение соответствует `MAX_ACTIVE_PROCESSING_JOBS=20`. Следующая загрузка была отклонена с HTTP 429. После запуска worker задания были обработаны; в проверке в БД зафиксировано 20 принятых тестовых заданий, а итоговое число ready включало также ранее существовавшие записи.
После экспериментов worker был возвращён к конфигурации B. В логах подтверждено `concurrency=2 prefetch=4`, повторный публичный smoke завершился PASS.
### 4.7. Ограничения измерений
- Пиковая память получена через `docker stats` и относится ко всему контейнеру worker, а не к RSS отдельного процесса Python. Поэтому в таблице она названа «пиковая память контейнера», а не RSS.
- Период опроса памяти — 0,5 секунды; кратковременные пики между измерениями могли быть пропущены.
- Каждый вариант конфигурации измерялся одним прогоном на 18 изображениях. Для строгого статистического вывода нужны повторные прогоны и анализ разброса.
- Результаты относятся к конкретному стенду с лимитом 1 CPU и 1 GiB RAM; на другом оборудовании оптимальная concurrency может отличаться.
---
## 5. Эксперименты и отказы
| Воздействие | Ожидаемое поведение | Наблюдение | Восстановление / потеря данных |
|---|---|---|---|
| Worker остановлен, выполняется загрузка | API сохраняет staging и задание, возвращает 202; ограничение приёма не позволяет бесконечно накапливать работу | В backpressure-тесте приняты 20 заданий, 21-е отклонено с 429 | После запуска worker все 20 тестовых заданий обработаны |
| Принудительный timeout обработки | Дочерний процесс обработки должен быть остановлен; ошибка должна попасть в retry/failure pipeline | Тест подтвердил нормальную обработку и завершение дочернего процесса при принудительном timeout | Проверка завершилась PASS |
| Replay failed job | Явный replay должен вернуть допустимое failed-задание в очередь, не создавая дубль успешного результата | Скрипт и CLI доступны; при проверке текущей БД failed-вложений не обнаружено, поэтому повторный запуск в этом прогоне не выполнялся | Не включено в подтверждённые экспериментальные результаты |
| Повышение concurrency до 4 | Ожидалось уменьшение ожидания и рост throughput | Queue lag уменьшился, но processing elapsed и p95 выросли; throughput снизился | Worker возвращён к B, smoke PASS |
| Повторная доставка после side effect до ack | Идемпотентные ключи должны не допустить дублирования конечных объектов | В реализации предусмотрены стабильные ключи, lease и fencing; отдельный fault-injection в рамках данного прогона не фиксировался | Описано как архитектурная гарантия, не как отдельный эксперимент |
---
## 6. Ограничения и следующие шаги
1. Экспериментальные значения памяти отражают память контейнера, а не RSS процесса. Для полного соответствия формулировке трека требуется отдельное измерение RSS.
2. Три конфигурации сравнивались по одному прогону. Для повышения достоверности следует выполнить несколько повторов каждого варианта.
3. Результаты производительности зависят от лимита CPU, доступной памяти, хранилища и состояния Docker-хоста.
4. At-least-once доставка не является exactly-once. Повторная доставка возможна, а корректность обеспечивается идемпотентной обработкой и устойчивой фиксацией результата.
5. Отдельный сценарий replay failed job и fault-injection повторной доставки не подтверждены журналами текущего прогона; они не включены в число выполненных экспериментов.
## 7. Итог
В лабораторной работе реализован асинхронный pipeline обработки изображений с отдельными API, outbox publisher и worker, устойчивым staging, RabbitMQ, PostgreSQL и S3-совместимым хранилищем. Реализованы производные изображения, ограниченные повторы, transactional outbox, lease/heartbeat, timeout обработки, backpressure и явный replay failed jobs.
В рамках Track 1 проведено сравнение трёх конфигураций concurrency/prefetch на трёх размерах изображений. На исследованном стенде лучший баланс показала конфигурация B (`2/4`). Увеличение параллелизма до C (`4/8`) снизило queue lag, но ухудшило время обработки и итоговую p95 latency из-за конкуренции за ограниченный CPU.
Цель работы — отделить приём файла от тяжёлой обработки и обеспечить устойчивое выполнение заданий при повторной доставке и остановке worker — реализована в рамках описанных ограничений.
+117
View File
@@ -0,0 +1,117 @@
# A generic, single database configuration.
[alembic]
# path to migration scripts
# Use forward slashes (/) also on windows to provide an os agnostic path
script_location = migrations
# template used to generate migration file names; The default value is %%(rev)s_%%(slug)s
# Uncomment the line below if you want the files to be prepended with date and time
# see https://alembic.sqlalchemy.org/en/latest/tutorial.html#editing-the-ini-file
# for all available tokens
# file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s
# sys.path path, will be prepended to sys.path if present.
# defaults to the current working directory.
prepend_sys_path = .
# timezone to use when rendering the date within the migration file
# as well as the filename.
# If specified, requires the python>=3.9 or backports.zoneinfo library.
# Any required deps can installed by adding `alembic[tz]` to the pip requirements
# string value is passed to ZoneInfo()
# leave blank for localtime
# timezone =
# max length of characters to apply to the "slug" field
# truncate_slug_length = 40
# set to 'true' to run the environment during
# the 'revision' command, regardless of autogenerate
# revision_environment = false
# set to 'true' to allow .pyc and .pyo files without
# a source .py file to be detected as revisions in the
# versions/ directory
# sourceless = false
# version location specification; This defaults
# to migrations/versions. When using multiple version
# directories, initial revisions must be specified with --version-path.
# The path separator used here should be the separator specified by "version_path_separator" below.
# version_locations = %(here)s/bar:%(here)s/bat:migrations/versions
# version path separator; As mentioned above, this is the character used to split
# version_locations. The default within new alembic.ini files is "os", which uses os.pathsep.
# If this key is omitted entirely, it falls back to the legacy behavior of splitting on spaces and/or commas.
# Valid values for version_path_separator are:
#
# version_path_separator = :
# version_path_separator = ;
# version_path_separator = space
# version_path_separator = newline
version_path_separator = os # Use os.pathsep. Default configuration used for new projects.
# set to 'true' to search source files recursively
# in each "version_locations" directory
# new in Alembic version 1.10
# recursive_version_locations = false
# the output encoding used when revision files
# are written from script.py.mako
# output_encoding = utf-8
sqlalchemy.url = driver://user:pass@localhost/dbname
[post_write_hooks]
# post_write_hooks defines scripts or Python functions that are run
# on newly generated revision scripts. See the documentation for further
# detail and examples
# format using "black" - use the console_scripts runner, against the "black" entrypoint
# hooks = black
# black.type = console_scripts
# black.entrypoint = black
# black.options = -l 79 REVISION_SCRIPT_FILENAME
# lint with attempts to fix using "ruff" - use the exec runner, execute a binary
# hooks = ruff
# ruff.type = exec
# ruff.executable = %(here)s/.venv/bin/ruff
# ruff.options = --fix REVISION_SCRIPT_FILENAME
# Logging configuration
[loggers]
keys = root,sqlalchemy,alembic
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = WARNING
handlers = console
qualname =
[logger_sqlalchemy]
level = WARNING
handlers =
qualname = sqlalchemy.engine
[logger_alembic]
level = INFO
handlers =
qualname = alembic
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S
View File
View File
+634
View File
@@ -0,0 +1,634 @@
import hashlib
import tempfile
import uuid
from datetime import datetime, timezone, timedelta
from urllib.parse import quote
from fastapi import APIRouter, Depends, Header, HTTPException, Query, Request, Response
from fastapi.responses import StreamingResponse
from sqlalchemy import select, func, text
from sqlalchemy.orm import Session
from botocore.exceptions import BotoCoreError, ClientError
from app.api.deps import get_db, get_current_user
from app.core.config import settings
from app.models import CreateAttachmentUpload
from app.db.models import (
Attachment,
AttachmentUpload,
AttachmentVariant,
ProcessingJob,
OutboxEvent,
User,
)
from app.services.object_storage import client, create_presigned_upload_url
from app.services.image_validation import MAX_BYTES, validate_image_file
from app.repositories.chat_repository import ChatRepository
router = APIRouter(prefix="/api/v1", tags=["Attachments"])
def iso_utc(value):
return value.astimezone(timezone.utc).isoformat(timespec="microseconds").replace("+00:00", "Z")
def attachment_json(item, db):
variants = db.scalars(
select(AttachmentVariant)
.where(AttachmentVariant.attachment_id == item.id)
.order_by(AttachmentVariant.variant)
).all()
return {
"id": str(item.id),
"chat_id": str(item.chat_id),
"owner_id": str(item.owner_id),
"filename": item.filename,
"content_type": item.content_type,
"size_bytes": item.size_bytes,
"sha256": item.sha256,
"status": item.status,
"variants": [
{
"name": variant.variant,
"content_type": variant.content_type,
"size_bytes": variant.size_bytes,
"width": variant.width,
"height": variant.height,
"sha256": variant.sha256,
}
for variant in variants
],
"error_code": item.error_code,
"created_at": iso_utc(item.created_at),
"updated_at": iso_utc(item.updated_at),
}
def member_attachment(db, attachment_id, user_id):
item = db.get(Attachment, attachment_id)
if item is None or not ChatRepository(db).is_member(item.chat_id, user_id):
raise HTTPException(404, "Attachment not found")
return item
async def read_limited(request: Request):
declared = request.headers.get("content-length")
if declared and declared.isdigit() and int(declared) > MAX_BYTES:
raise HTTPException(413, "Image exceeds 10 MiB")
size = 0
# Keep at most 1 MiB in RAM; larger bodies spill to a temporary file.
spool = tempfile.SpooledTemporaryFile(max_size=1024 * 1024, mode="w+b")
try:
async for chunk in request.stream():
size += len(chunk)
if size > MAX_BYTES:
raise HTTPException(413, "Image exceeds 10 MiB")
spool.write(chunk)
spool.seek(0)
return spool, size
except Exception:
spool.close()
raise
def sha256_file(fileobj):
digest = hashlib.sha256()
fileobj.seek(0)
while True:
chunk = fileobj.read(1024 * 1024)
if not chunk: break
digest.update(chunk)
fileobj.seek(0)
return digest.hexdigest()
@router.post("/chats/{chat_id}/attachments/presigned", status_code=201)
def create_attachment_upload(
chat_id: str,
payload: CreateAttachmentUpload,
db: Session = Depends(get_db),
user: User = Depends(get_current_user),
):
try:
chat_uuid = uuid.UUID(chat_id)
except ValueError:
raise HTTPException(400, "Invalid chat UUID") from None
if not ChatRepository(db).is_member(chat_uuid, user.id):
raise HTTPException(404, "Chat not found")
if payload.size_bytes > settings.max_upload_bytes:
raise HTTPException(413, "Image exceeds configured upload limit")
upload_id = uuid.uuid4()
object_key = f"uploads/{chat_uuid}/{upload_id}"
expires_at = datetime.now(timezone.utc) + timedelta(
seconds=settings.presigned_upload_ttl_seconds
)
try:
upload_url = create_presigned_upload_url(
object_key=object_key,
content_type=payload.content_type,
)
except (BotoCoreError, ClientError):
raise HTTPException(503, "Object storage unavailable") from None
upload = AttachmentUpload(
id=upload_id,
chat_id=chat_uuid,
owner_id=user.id,
filename=payload.filename,
content_type=payload.content_type,
expected_size_bytes=payload.size_bytes,
expected_sha256=payload.sha256.lower(),
object_key=object_key,
status="pending",
expires_at=expires_at,
)
try:
db.add(upload)
db.commit()
except Exception:
db.rollback()
raise HTTPException(503, "Could not save upload metadata") from None
return {
"upload_id": str(upload_id),
"upload_url": upload_url,
"method": "PUT",
"headers": {
"Content-Type": payload.content_type,
},
"expires_in": settings.presigned_upload_ttl_seconds,
"expires_at": iso_utc(expires_at),
}
@router.post("/attachments/uploads/{upload_id}/complete", status_code=200)
def complete_attachment_upload(
upload_id: str,
db: Session = Depends(get_db),
user: User = Depends(get_current_user),
):
try:
parsed_id = uuid.UUID(upload_id)
except ValueError:
raise HTTPException(400, "Invalid upload UUID") from None
# Lock the upload row to prevent concurrent completion.
upload = db.scalar(
select(AttachmentUpload)
.where(AttachmentUpload.id == parsed_id)
.with_for_update()
)
if upload is None:
raise HTTPException(404, "Upload not found")
if (
upload.owner_id != user.id
or not ChatRepository(db).is_member(upload.chat_id, user.id)
):
raise HTTPException(404, "Upload not found")
# Idempotent completion: return the existing attachment.
if upload.status == "completed":
item = db.scalar(
select(Attachment).where(
Attachment.object_key == upload.object_key
)
)
if item is None:
raise HTTPException(409, "Completed attachment not found")
return attachment_json(item, db)
if upload.status != "pending":
raise HTTPException(409, "Upload is not pending")
now = datetime.now(timezone.utc)
if upload.expires_at <= now:
upload.status = "expired"
db.commit()
try:
client().delete_object(
Bucket=settings.s3_bucket,
Key=upload.object_key,
)
except Exception:
pass
raise HTTPException(409, "Upload has expired")
s3 = client()
# Read the object from S3.
try:
result = s3.get_object(
Bucket=settings.s3_bucket,
Key=upload.object_key,
)
except ClientError as exc:
code = exc.response.get("Error", {}).get("Code", "")
if code in {"NoSuchKey", "404", "NotFound"}:
raise HTTPException(409, "Uploaded object not found") from None
raise HTTPException(503, "Object storage unavailable") from None
except BotoCoreError:
raise HTTPException(503, "Object storage unavailable") from None
body = result["Body"]
spool = tempfile.SpooledTemporaryFile(
max_size=1024 * 1024,
mode="w+b",
)
def reject_upload(status_code: int, detail: str):
upload.status = "failed"
db.commit()
try:
s3.delete_object(
Bucket=settings.s3_bucket,
Key=upload.object_key,
)
except Exception:
pass
raise HTTPException(status_code=status_code, detail=detail)
try:
actual_size = result.get("ContentLength")
if actual_size != upload.expected_size_bytes:
reject_upload(409, "Uploaded file size mismatch")
if actual_size > settings.max_upload_bytes:
reject_upload(413, "Image exceeds configured upload limit")
total = 0
digest = hashlib.sha256()
try:
while True:
chunk = body.read(64 * 1024)
if not chunk:
break
total += len(chunk)
if total > settings.max_upload_bytes:
reject_upload(413, "Image exceeds configured upload limit")
if total > upload.expected_size_bytes:
reject_upload(409, "Uploaded file size mismatch")
digest.update(chunk)
spool.write(chunk)
finally:
body.close()
if total != upload.expected_size_bytes:
reject_upload(409, "Uploaded file size mismatch")
if digest.hexdigest() != upload.expected_sha256:
reject_upload(409, "Uploaded file checksum mismatch")
# Validate the actual image content, not only S3 metadata.
try:
actual_type, _, _ = validate_image_file(
spool,
upload.content_type,
)
except HTTPException as exc:
reject_upload(exc.status_code, str(exc.detail))
attachment = Attachment(
id=uuid.uuid4(),
chat_id=upload.chat_id,
owner_id=upload.owner_id,
filename=upload.filename,
content_type=actual_type,
size_bytes=total,
sha256=digest.hexdigest(),
object_key=upload.object_key,
status="ready",
error_code=None,
)
upload.status = "completed"
upload.completed_at = now
db.add(attachment)
db.commit()
db.refresh(attachment)
return attachment_json(attachment, db)
except HTTPException:
raise
except (BotoCoreError, ClientError):
db.rollback()
raise HTTPException(503, "Object storage unavailable") from None
except Exception:
db.rollback()
raise
finally:
spool.close()
@router.post("/chats/{chat_id}/attachments", status_code=202)
async def upload_attachment(
chat_id: str,
request: Request,
filename: str = Query(..., min_length=1, max_length=255),
db: Session = Depends(get_db),
user: User = Depends(get_current_user),
):
try:
chat_uuid = uuid.UUID(chat_id)
except ValueError:
raise HTTPException(400, "Invalid chat UUID") from None
if not ChatRepository(db).is_member(chat_uuid, user.id):
raise HTTPException(404, "Chat not found")
if (
not filename.strip()
or any(c in filename for c in ("/", "\\", "\x00"))
or any(ord(c) < 32 or ord(c) == 127 for c in filename)
):
raise HTTPException(400, "Unsafe filename")
content_type = (
request.headers.get("content-type", "")
.split(";", 1)[0]
.strip()
.lower()
)
if content_type not in {"image/png", "image/jpeg"}:
raise HTTPException(415, "Only PNG and JPEG are supported")
spool, size = await read_limited(request)
attachment_id = uuid.uuid4()
job_id = uuid.uuid4()
staging_key = f"staging/{chat_uuid}/{attachment_id}"
object_key = f"original/{chat_uuid}/{attachment_id}"
staged = False
try:
actual_type, _, _ = validate_image_file(spool, content_type)
digest = sha256_file(spool)
# Serialize admission checks across all API instances.
db.execute(
text("SELECT pg_advisory_xact_lock(:lock_id)"),
{"lock_id": 72420261110},
)
active_jobs = db.scalar(
select(func.count())
.select_from(ProcessingJob)
.where(ProcessingJob.status.in_(("queued", "processing")))
) or 0
if active_jobs >= settings.max_active_processing_jobs:
raise HTTPException(
status_code=429,
detail="Processing queue is full; retry later",
headers={"Retry-After": "5"},
)
s3 = client()
try:
spool.seek(0)
s3.put_object(
Bucket=settings.s3_bucket,
Key=staging_key,
Body=spool,
ContentType=actual_type,
)
staged = True
except (BotoCoreError, ClientError):
raise HTTPException(
503, "Object storage unavailable"
) from None
now = datetime.now(timezone.utc)
attachment = Attachment(
id=attachment_id,
chat_id=chat_uuid,
owner_id=user.id,
filename=filename,
content_type=actual_type,
size_bytes=size,
sha256=digest,
object_key=object_key,
staging_key=staging_key,
status="queued",
error_code=None,
pipeline_version=1,
)
job = ProcessingJob(
id=job_id,
attachment_id=attachment_id,
pipeline_version=1,
attempt=0,
status="queued",
available_at=now,
)
event = OutboxEvent(
event_type="attachment.process.v1",
aggregate_id=attachment_id,
payload={
"schema_version": 1,
"job_id": str(job_id),
"type": "attachment.process.v1",
"attachment_id": str(attachment_id),
"pipeline_version": 1,
"attempt": 1,
"created_at": now.isoformat(),
},
status="pending",
)
db.add_all([attachment, job, event])
db.commit()
db.refresh(attachment)
return attachment_json(attachment, db)
except HTTPException:
db.rollback()
if staged:
try:
client().delete_object(
Bucket=settings.s3_bucket,
Key=staging_key,
)
except Exception:
pass
raise
except Exception:
db.rollback()
if staged:
try:
client().delete_object(
Bucket=settings.s3_bucket,
Key=staging_key,
)
except Exception:
pass
raise HTTPException(
503, "Could not save attachment processing job"
) from None
finally:
spool.close()
@router.get("/attachments/{attachment_id}")
def get_attachment(attachment_id: str, db: Session = Depends(get_db), user: User = Depends(get_current_user)):
try: parsed = uuid.UUID(attachment_id)
except ValueError: raise HTTPException(400, "Invalid attachment UUID") from None
item = member_attachment(db, parsed, user.id)
return attachment_json(item, db)
@router.get("/attachments/{attachment_id}/content")
def download_attachment(
attachment_id: str,
variant: str = "original",
db: Session = Depends(get_db),
user: User = Depends(get_current_user),
):
try:
parsed = uuid.UUID(attachment_id)
except ValueError:
raise HTTPException(400, "Invalid attachment UUID") from None
item = member_attachment(db, parsed, user.id)
if item.status != "ready":
raise HTTPException(409, "Attachment is not ready")
# Select the original file or a generated variant.
if variant == "original":
object_key = item.object_key
expected_size = item.size_bytes
expected_sha256 = item.sha256
content_type = item.content_type
else:
variant_item = db.scalar(
select(AttachmentVariant).where(
AttachmentVariant.attachment_id == item.id,
AttachmentVariant.variant == variant,
)
)
if variant_item is None:
raise HTTPException(404, "Variant not found")
object_key = variant_item.object_key
expected_size = variant_item.size_bytes
expected_sha256 = variant_item.sha256
content_type = variant_item.content_type
spool = tempfile.SpooledTemporaryFile(
max_size=1024 * 1024,
mode="w+b",
)
try:
result = client().get_object(
Bucket=settings.s3_bucket,
Key=object_key,
)
body = result["Body"]
total = 0
digest = hashlib.sha256()
try:
while True:
chunk = body.read(64 * 1024)
if not chunk:
break
total += len(chunk)
if total > MAX_BYTES:
raise HTTPException(
503,
"Stored object exceeds configured limit",
)
digest.update(chunk)
spool.write(chunk)
finally:
body.close()
# Verify stored object integrity.
if (
total != expected_size
or digest.hexdigest() != expected_sha256
):
raise HTTPException(
503,
"Stored object checksum mismatch",
)
spool.seek(0)
except (BotoCoreError, ClientError):
spool.close()
raise HTTPException(
503,
"Object storage unavailable",
) from None
except Exception:
spool.close()
raise
def stream_verified_file():
try:
while True:
chunk = spool.read(64 * 1024)
if not chunk:
break
yield chunk
finally:
spool.close()
return StreamingResponse(
stream_verified_file(),
media_type=content_type,
headers={
"Content-Disposition": (
f"inline; filename*=UTF-8''{quote(item.filename)}"
)
},
)
+96
View File
@@ -0,0 +1,96 @@
from datetime import timezone
from fastapi import APIRouter, Depends, Header, HTTPException, status
from fastapi.security import HTTPBasic, HTTPBasicCredentials
from sqlalchemy.orm import Session
from app.api.deps import get_db
from app.api.users import user_to_response
from app.services.auth_service import (
AuthService,
InvalidCredentialsError,
InvalidSessionError,
)
router = APIRouter(prefix="/api/v1/auth", tags=["Authentication"])
basic_security = HTTPBasic(auto_error=False)
@router.post("/sessions", status_code=status.HTTP_201_CREATED)
def create_session(
credentials: HTTPBasicCredentials = Depends(basic_security),
db: Session = Depends(get_db),
):
if credentials is None:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Basic authentication required",
headers={
"WWW-Authenticate": 'Basic realm="messenger", charset="UTF-8"'
},
)
try:
result = AuthService(db).create_session(
username=credentials.username,
password=credentials.password,
)
expires_at = result["expires_at"]
if expires_at.tzinfo is None:
expires_at = expires_at.replace(tzinfo=timezone.utc)
else:
expires_at = expires_at.astimezone(timezone.utc)
return {
"token": result["token"],
"token_type": result["token_type"],
"expires_at": expires_at.isoformat(
timespec="microseconds"
).replace("+00:00", "Z"),
"user": user_to_response(result["user"]),
}
except InvalidCredentialsError:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid username or password",
headers={
"WWW-Authenticate": 'Basic realm="messenger", charset="UTF-8"'
},
) from None
@router.delete("/sessions/current", status_code=status.HTTP_204_NO_CONTENT)
def delete_current_session(
authorization: str | None = Header(default=None),
db: Session = Depends(get_db),
):
if not authorization or not authorization.startswith("Bearer "):
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Bearer token required",
headers={"WWW-Authenticate": 'Bearer realm="messenger"'},
)
token = authorization[7:].strip()
if not token:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Bearer token required",
headers={"WWW-Authenticate": 'Bearer realm="messenger"'},
)
try:
AuthService(db).delete_session(token)
except InvalidSessionError:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid or expired session",
headers={"WWW-Authenticate": 'Bearer realm="messenger"'},
) from None
+90
View File
@@ -0,0 +1,90 @@
import uuid
from fastapi import APIRouter, Depends, HTTPException, Query, status
from sqlalchemy.orm import Session
from app.api.deps import get_db, get_current_user
from app.db.models import User as UserModel
from app.models import Chat as ChatResponse, CreateChat
from app.services.chat_service import (
ChatService,
ChatNotFoundError,
InvalidChatDataError,
)
router = APIRouter(prefix="/api/v1/chats", tags=["Chats"])
def chat_to_response(chat) -> ChatResponse:
return ChatResponse(
id=str(chat.id),
title=chat.title,
member_ids=[str(member.user_id) for member in chat.members],
created_at=chat.created_at,
)
@router.post("", response_model=ChatResponse, status_code=status.HTTP_201_CREATED)
def create_chat(
body: CreateChat,
db: Session = Depends(get_db),
user: UserModel = Depends(get_current_user),
):
try:
chat = ChatService(db).create_chat(
title=body.title,
member_ids=body.member_ids,
creator_id=user.id,
)
return chat_to_response(chat)
except InvalidChatDataError as exc:
raise HTTPException(
status_code=400,
detail=str(exc),
) from None
@router.get("")
def list_chats(
limit: int = Query(default=50, ge=1, le=100),
cursor: str | None = None,
db: Session = Depends(get_db),
user: UserModel = Depends(get_current_user),
):
if cursor is not None:
raise HTTPException(
status_code=400,
detail="Chat cursor pagination is not implemented yet",
)
chats = ChatService(db).list_chats(user.id)
chats = chats[:limit]
return {
"items": [chat_to_response(chat) for chat in chats],
"next_cursor": None,
}
@router.get("/{chat_id}", response_model=ChatResponse)
def get_chat(
chat_id: str,
db: Session = Depends(get_db),
user: UserModel = Depends(get_current_user),
):
try:
parsed_id = uuid.UUID(chat_id)
except ValueError:
raise HTTPException(status_code=400, detail="Invalid chat UUID") from None
try:
chat = ChatService(db).get_chat(parsed_id, user.id)
return chat_to_response(chat)
except ChatNotFoundError:
raise HTTPException(
status_code=404,
detail="Chat not found",
) from None
+40
View File
@@ -0,0 +1,40 @@
from fastapi import Depends, HTTPException, status
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
from sqlalchemy.orm import Session
from app.db.session import SessionLocal
from app.services.auth_service import AuthService, InvalidSessionError
bearer_scheme = HTTPBearer(auto_error=False)
def get_db():
db = SessionLocal()
try:
yield db
finally:
db.close()
def get_current_user(
credentials: HTTPAuthorizationCredentials | None = Depends(bearer_scheme),
db: Session = Depends(get_db),
):
if credentials is None or credentials.scheme.lower() != "bearer":
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Bearer token required",
headers={"WWW-Authenticate": "Bearer"},
)
try:
user = AuthService(db).authenticate(credentials.credentials)
return user
except InvalidSessionError:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid or expired token",
headers={"WWW-Authenticate": "Bearer"},
) from None
+73
View File
@@ -0,0 +1,73 @@
import re
import uuid
from fastapi import APIRouter, Depends, Header, HTTPException, Query, status
from sqlalchemy.orm import Session
from app.api.deps import get_db, get_current_user
from app.db.models import User as UserModel
from app.models import CreateMessage, Message as MessageResponse
from app.services.message_service import (
MessageService, MessageNotFoundError, InvalidMessageDataError,
IdempotencyConflictError,
)
router = APIRouter(prefix="/api/v1/chats/{chat_id}/messages", tags=["Messages"])
IDEMPOTENCY_KEY_RE = re.compile(r"^[A-Za-z0-9._:-]{1,128}$")
def message_to_response(message) -> MessageResponse:
return MessageResponse(
id=str(message.id), chat_id=str(message.chat_id),
sender_id=str(message.sender_id), text=message.text,
created_at=message.created_at, attachment_ids=message.attachment_ids or [],
)
def parse_chat_id(chat_id: str) -> uuid.UUID:
try:
return uuid.UUID(chat_id)
except ValueError:
raise HTTPException(status_code=400, detail="Invalid chat UUID") from None
@router.post("", response_model=MessageResponse, status_code=status.HTTP_201_CREATED)
def create_message(
chat_id: str, body: CreateMessage,
idempotency_key: str = Header(..., alias="Idempotency-Key", min_length=1, max_length=128),
db: Session = Depends(get_db),
user: UserModel = Depends(get_current_user),
):
if not IDEMPOTENCY_KEY_RE.fullmatch(idempotency_key):
raise HTTPException(status_code=400, detail="Invalid Idempotency-Key")
parsed_id = parse_chat_id(chat_id)
try:
message = MessageService(db).create_message(
chat_id=parsed_id, sender_id=user.id, text=body.text,
idempotency_key=idempotency_key, attachment_ids=body.attachment_ids,
)
return message_to_response(message)
except MessageNotFoundError:
raise HTTPException(status_code=404, detail="Chat not found") from None
except InvalidMessageDataError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from None
except IdempotencyConflictError:
raise HTTPException(status_code=409, detail="Idempotency-Key was already used with a different request") from None
@router.get("")
def list_messages(
chat_id: str, limit: int = Query(default=50, ge=1, le=100),
cursor: str | None = None, db: Session = Depends(get_db),
user: UserModel = Depends(get_current_user),
):
parsed_id = parse_chat_id(chat_id)
try:
messages, next_cursor = MessageService(db).get_history(
chat_id=parsed_id, user_id=user.id, limit=limit, cursor=cursor
)
return {"items": [message_to_response(item) for item in messages], "next_cursor": next_cursor}
except MessageNotFoundError:
raise HTTPException(status_code=404, detail="Chat not found") from None
except InvalidMessageDataError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from None
+55
View File
@@ -0,0 +1,55 @@
from fastapi import APIRouter, Depends, HTTPException, status
from sqlalchemy.orm import Session
from app.api.deps import get_db, get_current_user
from app.db.models import User as UserModel
from app.models import CreateUser, User
from app.services.user_service import (
UserService,
UsernameAlreadyExistsError,
)
router = APIRouter(prefix="/api/v1/users", tags=["Users"])
def user_to_response(user: UserModel) -> User:
return User(
id=str(user.id),
username=user.username,
display_name=user.display_name,
created_at=user.created_at,
)
@router.post("", response_model=User, status_code=status.HTTP_201_CREATED)
def register_user(
body: CreateUser,
db: Session = Depends(get_db),
):
try:
user = UserService(db).register(
username=body.username,
display_name=body.display_name,
password=body.password,
)
return user_to_response(user)
except UsernameAlreadyExistsError:
raise HTTPException(
status_code=status.HTTP_409_CONFLICT,
detail="Username already exists",
) from None
except ValueError as exc:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=str(exc),
) from None
@router.get("/me", response_model=User)
def get_me(
current_user: UserModel = Depends(get_current_user),
):
return user_to_response(current_user)
View File
+73
View File
@@ -0,0 +1,73 @@
import os
from pydantic import AliasChoices, Field
from pydantic_settings import BaseSettings, SettingsConfigDict
class Settings(BaseSettings):
app_name: str = "Messenger API"
app_env: str = "lab"
database_url: str = "postgresql+psycopg://messenger:messenger@postgres:5432/messenger"
db_pool_size: int = 5
db_max_overflow: int = 5
db_pool_timeout: int = 10
session_ttl_seconds: int = 3600
cache_url: str = "redis://redis:6379/0"
message_cache_ttl_seconds: int = 60
idempotency_ttl_seconds: int = 86400
instance_id: str = os.getenv("INSTANCE_ID") or os.getenv("HOSTNAME") or "api-local"
shutdown_grace_seconds: int = 20
max_active_processing_jobs: int = Field(default=20, ge=1, le=10000)
s3_endpoint_url: str = Field(
default="http://minio:9000",
validation_alias=AliasChoices("S3_ENDPOINT_URL", "S3_ENDPOINT"),
)
s3_public_endpoint_url: str = Field(
default="http://localhost:9000",
validation_alias=AliasChoices("S3_PUBLIC_ENDPOINT_URL"),
)
presigned_upload_ttl_seconds: int = 300
attachment_cleanup_interval_seconds: int = Field(default=60, ge=5, le=3600)
attachment_cleanup_grace_seconds: int = Field(default=600, ge=0, le=86400)
attachment_cleanup_batch_size: int = Field(default=100, ge=1, le=1000)
s3_region: str = Field(default="us-east-1", validation_alias=AliasChoices("S3_REGION"))
s3_bucket: str = Field(default="messenger-attachments", validation_alias=AliasChoices("S3_BUCKET"))
s3_access_key: str = Field(
default="messenger-api",
validation_alias=AliasChoices("S3_ACCESS_KEY", "S3_ACCESS_KEY_ID"),
)
s3_secret_key: str = Field(
default="change-me-api-secret",
validation_alias=AliasChoices("S3_SECRET_KEY", "S3_SECRET_ACCESS_KEY"),
)
max_upload_bytes: int = 10 * 1024 * 1024
max_image_pixels: int = 40_000_000
# Durable asynchronous image processing.
rabbitmq_url: str = Field(
default="amqp://guest:guest@rabbitmq:5672/",
validation_alias=AliasChoices("RABBITMQ_URL"),
)
rabbitmq_exchange: str = Field(default="attachments", validation_alias=AliasChoices("RABBITMQ_EXCHANGE"))
rabbitmq_queue: str = Field(default="attachment.process.v1", validation_alias=AliasChoices("RABBITMQ_QUEUE"))
rabbitmq_prefetch: int = Field(default=4, ge=1, le=64)
outbox_poll_interval_seconds: float = Field(default=1.0, ge=0.1, le=30)
outbox_batch_size: int = Field(default=50, ge=1, le=500)
worker_concurrency: int = Field(default=2, ge=1, le=16)
worker_max_attempts: int = Field(default=5, ge=1, le=20)
worker_lease_seconds: int = Field(default=180, ge=30, le=3600)
worker_retry_base_seconds: float = Field(default=2.0, ge=0.1, le=300)
worker_retry_max_seconds: float = Field(default=120.0, ge=1, le=3600)
worker_staging_retention_seconds: int = Field(default=86400, ge=3600)
image_processing_timeout_seconds: int = Field(default=60, ge=1, le=600)
model_config = SettingsConfigDict(
env_file=".env",
env_file_encoding="utf-8",
extra="ignore",
)
settings = Settings()
View File
+5
View File
@@ -0,0 +1,5 @@
from sqlalchemy.orm import DeclarativeBase
class Base(DeclarativeBase):
pass
+488
View File
@@ -0,0 +1,488 @@
import uuid
from datetime import datetime, timezone
from sqlalchemy import (
DateTime,
JSON,
ForeignKey,
Index,
String,
Text,
UniqueConstraint,
)
from sqlalchemy.dialects.postgresql import UUID
from sqlalchemy.orm import Mapped, mapped_column, relationship
from app.db.base import Base
def utc_now() -> datetime:
return datetime.now(timezone.utc)
class User(Base):
__tablename__ = "users"
id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
primary_key=True,
default=uuid.uuid4,
)
username: Mapped[str] = mapped_column(
String(32),
unique=True,
nullable=False,
index=True,
)
display_name: Mapped[str] = mapped_column(
String(100),
nullable=False,
)
password_hash: Mapped[str] = mapped_column(
String(255),
nullable=False,
)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
nullable=False,
)
memberships: Mapped[list["ChatMember"]] = relationship(
back_populates="user",
cascade="all, delete-orphan",
)
messages: Mapped[list["Message"]] = relationship(
back_populates="sender",
)
class Chat(Base):
__tablename__ = "chats"
id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
primary_key=True,
default=uuid.uuid4,
)
title: Mapped[str] = mapped_column(
String(100),
nullable=False,
)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
nullable=False,
)
members: Mapped[list["ChatMember"]] = relationship(
back_populates="chat",
cascade="all, delete-orphan",
)
messages: Mapped[list["Message"]] = relationship(
back_populates="chat",
cascade="all, delete-orphan",
)
class ChatMember(Base):
__tablename__ = "chat_members"
chat_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("chats.id", ondelete="CASCADE"),
primary_key=True,
)
user_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("users.id", ondelete="CASCADE"),
primary_key=True,
)
chat: Mapped["Chat"] = relationship(
back_populates="members",
)
user: Mapped["User"] = relationship(
back_populates="memberships",
)
class Message(Base):
__tablename__ = "messages"
id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
primary_key=True,
default=uuid.uuid4,
)
chat_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("chats.id", ondelete="CASCADE"),
nullable=False,
)
sender_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("users.id"),
nullable=False,
)
text: Mapped[str] = mapped_column(
Text,
nullable=False,
)
attachment_ids: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
nullable=False,
)
chat: Mapped["Chat"] = relationship(
back_populates="messages",
)
sender: Mapped["User"] = relationship(
back_populates="messages",
)
__table_args__ = (
Index(
"ix_messages_chat_created_id",
"chat_id",
"created_at",
"id",
),
)
class Session(Base):
__tablename__ = "sessions"
id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
primary_key=True,
default=uuid.uuid4,
)
token_hash: Mapped[str] = mapped_column(
String(64),
unique=True,
nullable=False,
index=True,
)
user_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
)
expires_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
nullable=False,
)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
nullable=False,
)
user: Mapped["User"] = relationship()
class IdempotencyRecord(Base):
__tablename__ = "idempotency_records"
id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True), primary_key=True, default=uuid.uuid4
)
user_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True), ForeignKey("users.id", ondelete="CASCADE"), nullable=False
)
method: Mapped[str] = mapped_column(String(10), nullable=False)
path: Mapped[str] = mapped_column(String(255), nullable=False)
idempotency_key: Mapped[str] = mapped_column(String(128), nullable=False)
request_hash: Mapped[str] = mapped_column(String(64), nullable=False)
# NULL is visible only inside the creating transaction; the message and
# idempotency row are committed atomically.
message_id: Mapped[uuid.UUID | None] = mapped_column(
UUID(as_uuid=True), ForeignKey("messages.id", ondelete="CASCADE"), nullable=True
)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), default=utc_now, nullable=False
)
expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
__table_args__ = (
UniqueConstraint(
"user_id", "method", "path", "idempotency_key",
name="uq_idempotency_scope_key",
),
Index("ix_idempotency_records_expires_at", "expires_at"),
)
class Attachment(Base):
__tablename__ = "attachments"
id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
primary_key=True,
default=uuid.uuid4,
)
chat_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("chats.id", ondelete="CASCADE"),
nullable=False,
index=True,
)
owner_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
)
filename: Mapped[str] = mapped_column(String(255), nullable=False)
content_type: Mapped[str] = mapped_column(String(32), nullable=False)
size_bytes: Mapped[int] = mapped_column(nullable=False)
sha256: Mapped[str] = mapped_column(String(64), nullable=False)
# Ключ оригинала в конечном S3-prefix.
object_key: Mapped[str] = mapped_column(
String(512),
unique=True,
nullable=False,
)
# Ключ исходного файла в durable staging.
staging_key: Mapped[str | None] = mapped_column(
String(512),
nullable=True,
)
status: Mapped[str] = mapped_column(
String(16),
nullable=False,
default="ready",
)
error_code: Mapped[str | None] = mapped_column(
String(64),
nullable=True,
)
pipeline_version: Mapped[int] = mapped_column(
nullable=False,
default=1,
server_default="1", )
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
nullable=False,
)
updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
onupdate=utc_now,
nullable=False,
)
__table_args__ = (
UniqueConstraint(
"staging_key",
name="uq_attachments_staging_key",
),
)
class AttachmentUpload(Base):
__tablename__ = "attachment_uploads"
id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
primary_key=True,
default=uuid.uuid4,
)
chat_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("chats.id", ondelete="CASCADE"),
nullable=False,
)
owner_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
)
filename: Mapped[str] = mapped_column(
String(255),
nullable=False,
)
content_type: Mapped[str] = mapped_column(
String(32),
nullable=False,
)
expected_size_bytes: Mapped[int] = mapped_column(
nullable=False,
)
expected_sha256: Mapped[str] = mapped_column(
String(64),
nullable=False,
)
object_key: Mapped[str] = mapped_column(
String(512),
unique=True,
nullable=False,
)
status: Mapped[str] = mapped_column(
String(16),
nullable=False,
default="pending",
)
expires_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
nullable=False,
)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
nullable=False,
)
completed_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True),
nullable=True,
)
__table_args__ = (
Index("ix_attachment_uploads_chat_id", "chat_id"),
Index("ix_attachment_uploads_owner_id", "owner_id"),
Index("ix_attachment_uploads_expires_at", "expires_at"),
)
class ProcessingJob(Base):
__tablename__ = "processing_jobs"
id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
primary_key=True,
default=uuid.uuid4,
)
attachment_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("attachments.id", ondelete="CASCADE"),
nullable=False,
)
pipeline_version: Mapped[int] = mapped_column(
nullable=False,
default=1,
server_default="1", )
attempt: Mapped[int] = mapped_column(
nullable=False,
default=1,
server_default="1", )
status: Mapped[str] = mapped_column(
String(16),
nullable=False,
default="queued",
server_default="queued", )
available_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
nullable=False,
)
lease_until: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True),
nullable=True,
)
worker_id: Mapped[str | None] = mapped_column(
String(128),
nullable=True,
)
last_error_code: Mapped[str | None] = mapped_column(
String(64),
nullable=True,
)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
nullable=False,
)
updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
onupdate=utc_now,
nullable=False,
)
__table_args__ = (
UniqueConstraint(
"attachment_id",
"pipeline_version",
name="uq_processing_job_attachment_pipeline",
),
Index(
"ix_processing_jobs_status_available",
"status",
"available_at",
),
Index(
"ix_processing_jobs_lease_until",
"lease_until",
),
)
class AttachmentVariant(Base):
__tablename__ = "attachment_variants"
id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
primary_key=True,
default=uuid.uuid4,
)
attachment_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("attachments.id", ondelete="CASCADE"),
nullable=False,
)
variant: Mapped[str] = mapped_column(
String(32),
nullable=False,
)
object_key: Mapped[str] = mapped_column(
String(512),
unique=True,
nullable=False,
)
content_type: Mapped[str] = mapped_column(
String(32),
nullable=False,
)
width: Mapped[int] = mapped_column(nullable=False)
height: Mapped[int] = mapped_column(nullable=False)
size_bytes: Mapped[int] = mapped_column(nullable=False)
sha256: Mapped[str] = mapped_column(
String(64),
nullable=False,
)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
nullable=False,
)
__table_args__ = (
UniqueConstraint(
"attachment_id",
"variant",
name="uq_attachment_variant",
),
Index(
"ix_attachment_variants_attachment_id",
"attachment_id",
),
)
class OutboxEvent(Base):
__tablename__ = "outbox_events"
id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
primary_key=True,
default=uuid.uuid4,
)
event_type: Mapped[str] = mapped_column(
String(128),
nullable=False,
)
aggregate_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
nullable=False,
)
payload: Mapped[dict] = mapped_column(
JSON,
nullable=False,
)
status: Mapped[str] = mapped_column(
String(16),
nullable=False,
default="pending",
server_default="pending", )
attempts: Mapped[int] = mapped_column(
nullable=False,
default=0,
server_default="0", )
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
nullable=False,
)
published_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True),
nullable=True,
)
last_error: Mapped[str | None] = mapped_column(
String(256),
nullable=True,
)
__table_args__ = (
Index(
"ix_outbox_events_status_created",
"status",
"created_at",
),
)
+15
View File
@@ -0,0 +1,15 @@
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from app.core.config import settings
engine = create_engine(
settings.database_url,
pool_pre_ping=True,
pool_size=settings.db_pool_size,
max_overflow=settings.db_max_overflow,
pool_timeout=settings.db_pool_timeout,
pool_recycle=1800,
)
SessionLocal = sessionmaker(bind=engine, autoflush=False, autocommit=False)
+106
View File
@@ -0,0 +1,106 @@
import asyncio
import logging
import uuid
from contextlib import asynccontextmanager, suppress
from fastapi import FastAPI, HTTPException, Request
from fastapi.exceptions import RequestValidationError
from fastapi.responses import JSONResponse
from pathlib import Path
from sqlalchemy import text
from sqlalchemy.exc import SQLAlchemyError
from app.api.auth import router as auth_router
from app.api.users import router as users_router
from app.api.chats import router as chats_router
from app.api.messages import router as messages_router
from app.api.attachments import router as attachments_router
from app.core.config import settings
from app.db.session import engine
from app.services.object_storage import client as s3_client
from app.services.attachment_cleanup import cleanup_expired_uploads
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s %(message)s")
logger = logging.getLogger(__name__)
async def attachment_cleanup_loop():
while True:
try:
# Synchronous boto3/SQLAlchemy I/O must not block the event loop.
await asyncio.to_thread(cleanup_expired_uploads)
except Exception:
logger.exception("Attachment cleanup worker iteration failed")
await asyncio.sleep(settings.attachment_cleanup_interval_seconds)
@asynccontextmanager
async def lifespan(app: FastAPI):
logger.info("Application startup instance_id=%s", settings.instance_id)
cleanup_task = asyncio.create_task(attachment_cleanup_loop())
try:
yield
finally:
cleanup_task.cancel()
with suppress(asyncio.CancelledError):
await cleanup_task
engine.dispose()
logger.info("Application shutdown instance_id=%s", settings.instance_id)
app = FastAPI(title=settings.app_name, version="1.4.0", lifespan=lifespan)
app.include_router(users_router)
app.include_router(auth_router)
app.include_router(chats_router)
app.include_router(messages_router)
app.include_router(attachments_router)
@app.middleware("http")
async def response_headers(request: Request, call_next):
request_id = str(uuid.uuid4())
request.state.request_id = request_id
try:
response = await call_next(request)
except Exception:
logger.exception("Unhandled request error; request_id=%s", request_id)
response = JSONResponse(status_code=500, content={"error": {"code": "internal_error", "message": "Unexpected error", "request_id": request_id}})
response.headers["X-Request-Id"] = request_id
response.headers["X-Instance-Id"] = settings.instance_id
response.headers["X-Content-Type-Options"] = "nosniff"
return response
@app.exception_handler(HTTPException)
async def http_exception_handler(request: Request, exc: HTTPException):
request_id = getattr(request.state, "request_id", str(uuid.uuid4()))
code = {400: "invalid_request", 401: "unauthorized", 404: "not_found", 409: "conflict", 413: "payload_too_large", 415: "unsupported_media_type", 429: "rate_limited", 500: "internal_error", 503: "unavailable"}.get(exc.status_code, "internal_error")
headers = dict(exc.headers or {})
headers["X-Request-Id"] = request_id
headers["X-Instance-Id"] = settings.instance_id
return JSONResponse(status_code=exc.status_code, content={"error": {"code": code, "message": exc.detail if isinstance(exc.detail, str) else "Request failed", "request_id": request_id}}, headers=headers)
@app.exception_handler(RequestValidationError)
async def validation_exception_handler(request: Request, exc: RequestValidationError):
request_id = getattr(request.state, "request_id", str(uuid.uuid4()))
return JSONResponse(status_code=400, content={"error": {"code": "invalid_request", "message": "Validation error", "request_id": request_id}}, headers={"X-Request-Id": request_id, "X-Instance-Id": settings.instance_id})
@app.get("/health/live")
def liveness():
return {"status": "ok"}
DRAIN_FILE = Path("/tmp/api-draining")
@app.get("/health/ready")
def readiness():
if DRAIN_FILE.exists():
raise HTTPException(
status_code=503,
detail="Instance is draining",
)
try:
with engine.connect() as connection:
connection.execute(text("SELECT 1"))
s3_client().head_bucket(Bucket=settings.s3_bucket)
return {"status": "ok"}
except Exception:
logger.exception("PostgreSQL or S3 readiness check failed")
raise HTTPException(
status_code=503,
detail="Required storage unavailable",
) from None
+138
View File
@@ -0,0 +1,138 @@
from datetime import datetime, timezone
from typing import List, Optional, Literal
from pydantic import BaseModel, ConfigDict, Field, field_validator
class BaseConfig(BaseModel):
model_config = ConfigDict(
from_attributes=True,
json_encoders={
datetime: lambda value: (
value.astimezone(timezone.utc)
.isoformat(timespec="microseconds")
.replace("+00:00", "Z")
)
},
)
class CreateUser(BaseConfig):
username: str = Field(..., pattern=r"^[a-z0-9_]{3,32}$")
display_name: str = Field(..., min_length=1, max_length=100)
password: str = Field(..., min_length=12, max_length=128)
@field_validator("display_name")
@classmethod
def validate_display_name(cls, value: str) -> str:
value = value.strip()
if not value:
raise ValueError("display_name must not be blank")
return value
class User(BaseConfig):
id: str
username: str
display_name: str
created_at: datetime
class CreateChat(BaseConfig):
title: str = Field(..., min_length=1, max_length=100)
member_ids: List[str] = Field(..., min_length=1, max_length=100)
@field_validator("title")
@classmethod
def validate_title(cls, value: str) -> str:
value = value.strip()
if not value:
raise ValueError("title must not be blank")
return value
@field_validator("member_ids")
@classmethod
def validate_unique_members(cls, value: List[str]) -> List[str]:
if len(value) != len(set(value)):
raise ValueError("member_ids must be unique")
return value
class Chat(BaseConfig):
id: str
title: str
member_ids: List[str]
created_at: datetime
class CreateMessage(BaseConfig):
text: str = Field(..., max_length=4000)
attachment_ids: List[str] = Field(default_factory=list, max_length=10)
@field_validator("attachment_ids")
@classmethod
def validate_attachment_ids(cls, value):
if len(value) != len(set(value)):
raise ValueError("attachment_ids must be unique")
return value
@field_validator("text")
@classmethod
def validate_text(cls, value: str) -> str:
value = value.strip()
return value
class Message(BaseConfig):
id: str
chat_id: str
sender_id: str
text: str
created_at: datetime
attachment_ids: List[str] = Field(default_factory=list)
class Health(BaseConfig):
status: str = "ok"
class ErrorDetail(BaseConfig):
code: str
message: str
request_id: str
class ErrorResponse(BaseConfig):
error: ErrorDetail
class ChatPage(BaseConfig):
items: List[Chat]
next_cursor: Optional[str] = None
class MessagePage(BaseConfig):
items: List[Message]
next_cursor: Optional[str] = None
class CreateAttachmentUpload(BaseConfig):
filename: str = Field(..., min_length=1, max_length=255)
content_type: Literal["image/png", "image/jpeg"]
size_bytes: int = Field(..., gt=0, le=10 * 1024 * 1024)
sha256: str = Field(..., pattern=r"^[a-fA-F0-9]{64}$")
@field_validator("filename")
@classmethod
def validate_filename(cls, value: str) -> str:
value = value.strip()
if (
not value
or any(c in value for c in ("/", "\\", "\x00"))
or any(ord(c) < 32 or ord(c) == 127 for c in value)
):
raise ValueError("Unsafe filename")
return value
View File
+71
View File
@@ -0,0 +1,71 @@
import uuid
from sqlalchemy import select
from sqlalchemy.orm import Session, selectinload
from app.db.models import Chat, ChatMember
class ChatRepository:
def __init__(self, db: Session):
self.db = db
def get_by_id(self, chat_id: uuid.UUID) -> Chat | None:
statement = (
select(Chat)
.options(selectinload(Chat.members))
.where(Chat.id == chat_id)
)
return self.db.scalar(statement)
def get_by_user(
self,
user_id: uuid.UUID,
limit: int = 50,
) -> list[Chat]:
statement = (
select(Chat)
.join(ChatMember, ChatMember.chat_id == Chat.id)
.where(ChatMember.user_id == user_id)
.order_by(Chat.created_at.asc(), Chat.id.asc())
.limit(limit)
)
return list(self.db.scalars(statement).all())
def is_member(
self,
chat_id: uuid.UUID,
user_id: uuid.UUID,
) -> bool:
statement = select(ChatMember).where(
ChatMember.chat_id == chat_id,
ChatMember.user_id == user_id,
)
return self.db.scalar(statement) is not None
def create(
self,
title: str,
member_ids: list[uuid.UUID],
) -> Chat:
chat = Chat(title=title)
self.db.add(chat)
self.db.flush()
memberships = [
ChatMember(
chat_id=chat.id,
user_id=user_id,
)
for user_id in member_ids
]
self.db.add_all(memberships)
self.db.flush()
return chat
+73
View File
@@ -0,0 +1,73 @@
import uuid
from datetime import datetime
from sqlalchemy import and_, or_, select
from sqlalchemy.orm import Session
from app.db.models import Message
class MessageRepository:
def __init__(self, db: Session):
self.db = db
def get_by_id(
self,
message_id: uuid.UUID,
) -> Message | None:
statement = select(Message).where(
Message.id == message_id
)
return self.db.scalar(statement)
def create(
self,
chat_id: uuid.UUID,
sender_id: uuid.UUID,
text: str,
attachment_ids: list[str] | None = None,
) -> Message:
message = Message(
chat_id=chat_id,
sender_id=sender_id,
text=text,
attachment_ids=attachment_ids or [],
)
self.db.add(message)
self.db.flush()
return message
def get_history(
self,
chat_id: uuid.UUID,
limit: int = 50,
cursor_created_at: datetime | None = None,
cursor_id: uuid.UUID | None = None,
) -> list[Message]:
statement = select(Message).where(
Message.chat_id == chat_id
)
if cursor_created_at is not None and cursor_id is not None:
statement = statement.where(
or_(
Message.created_at > cursor_created_at,
and_(
Message.created_at == cursor_created_at,
Message.id > cursor_id,
),
)
)
statement = (
statement
.order_by(Message.created_at.asc(), Message.id.asc())
.limit(limit)
)
return list(self.db.scalars(statement).all())
+78
View File
@@ -0,0 +1,78 @@
import uuid
from datetime import datetime, timezone
from sqlalchemy import delete, select
from sqlalchemy.orm import Session
from app.db.models import Session as UserSession
class SessionRepository:
def __init__(self, db: Session):
self.db = db
def create(
self,
user_id: uuid.UUID,
token_hash: str,
expires_at: datetime,
) -> UserSession:
session = UserSession(
user_id=user_id,
token_hash=token_hash,
expires_at=expires_at,
)
self.db.add(session)
self.db.flush()
return session
def get_by_token_hash(
self,
token_hash: str,
) -> UserSession | None:
statement = select(UserSession).where(
UserSession.token_hash == token_hash
)
return self.db.scalar(statement)
def get_valid_session(
self,
token_hash: str,
) -> UserSession | None:
now = datetime.now(timezone.utc)
statement = select(UserSession).where(
UserSession.token_hash == token_hash,
UserSession.expires_at > now,
)
return self.db.scalar(statement)
def delete_by_token_hash(
self,
token_hash: str,
) -> bool:
statement = (
delete(UserSession)
.where(UserSession.token_hash == token_hash)
.returning(UserSession.id)
)
return self.db.scalar(statement) is not None
def delete_expired(self) -> int:
now = datetime.now(timezone.utc)
statement = (
delete(UserSession)
.where(UserSession.expires_at <= now)
)
result = self.db.execute(statement)
return result.rowcount
+41
View File
@@ -0,0 +1,41 @@
import uuid
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.db.models import User
class UserRepository:
def __init__(self, db: Session):
self.db = db
def get_by_id(self, user_id: uuid.UUID) -> User | None:
statement = select(User).where(User.id == user_id)
return self.db.scalar(statement)
def get_by_username(self, username: str) -> User | None:
statement = select(User).where(
User.username == username
)
return self.db.scalar(statement)
def create(
self,
username: str,
display_name: str,
password_hash: str,
) -> User:
user = User(
username=username,
display_name=display_name,
password_hash=password_hash,
)
self.db.add(user)
self.db.flush()
return user
View File
+66
View File
@@ -0,0 +1,66 @@
"""Best-effort cleanup for abandoned presigned uploads."""
import logging
from datetime import datetime, timedelta, timezone
from sqlalchemy import select
from app.core.config import settings
from app.db.models import AttachmentUpload
from app.db.session import SessionLocal
from app.services.object_storage import client
logger = logging.getLogger(__name__)
def cleanup_expired_uploads() -> int:
"""Delete old pending objects and mark their upload records expired.
A grace period protects clients completing an upload near URL expiry.
PostgreSQL SKIP LOCKED makes this safe across multiple API replicas.
"""
cutoff = datetime.now(timezone.utc) - timedelta(
seconds=settings.attachment_cleanup_grace_seconds
)
deleted_count = 0
with SessionLocal() as db:
uploads = db.scalars(
select(AttachmentUpload)
.where(
AttachmentUpload.status == "pending",
AttachmentUpload.expires_at <= cutoff,
)
.order_by(AttachmentUpload.expires_at.asc())
.limit(settings.attachment_cleanup_batch_size)
.with_for_update(skip_locked=True)
).all()
if not uploads:
return 0
s3 = client()
try:
for upload in uploads:
# DeleteObject is idempotent: a missing object is also safe.
s3.delete_object(
Bucket=settings.s3_bucket,
Key=upload.object_key,
)
upload.status = "expired"
deleted_count += 1
db.commit()
except Exception:
db.rollback()
logger.exception(
"Attachment cleanup batch failed; pending records will be retried"
)
return 0
if deleted_count:
logger.info(
"Expired abandoned attachment uploads cleaned: count=%d",
deleted_count,
)
return deleted_count
+159
View File
@@ -0,0 +1,159 @@
import hashlib
import secrets
from datetime import datetime, timedelta, timezone
from redis import Redis
from redis.backoff import NoBackoff
from redis.exceptions import RedisError
from redis.retry import Retry
from sqlalchemy.orm import Session
from app.core.config import settings
from app.db.models import User
from app.repositories.session_repository import SessionRepository
from app.repositories.user_repository import UserRepository
from app.services.user_service import password_hasher
class InvalidCredentialsError(Exception):
pass
class InvalidSessionError(Exception):
pass
class AuthService:
def __init__(self, db: Session):
self.db = db
self.sessions = SessionRepository(db)
self.users = UserRepository(db)
self._redis_available = True
self.redis = Redis.from_url(
settings.cache_url,
decode_responses=True,
socket_connect_timeout=0.5,
socket_timeout=0.5,
retry_on_timeout=False,
retry=Retry(NoBackoff(), 0),
)
@staticmethod
def _hash_token(token: str) -> str:
return hashlib.sha256(token.encode("utf-8")).hexdigest()
def create_session(self, username: str, password: str) -> dict:
user = self.users.get_by_username(username)
if user is None or not password_hasher.verify(
password, user.password_hash
):
raise InvalidCredentialsError("Invalid username or password")
token = secrets.token_urlsafe(32)
token_hash = self._hash_token(token)
expires_at = datetime.now(timezone.utc) + timedelta(
seconds=settings.session_ttl_seconds
)
try:
self.sessions.create(
user_id=user.id,
token_hash=token_hash,
expires_at=expires_at,
)
self.db.commit()
except Exception:
self.db.rollback()
raise
self._cache_session(token_hash, user.id, expires_at)
return {
"token": token,
"token_type": "Bearer",
"expires_at": expires_at,
"user": user,
}
def _cache_session(
self,
token_hash: str,
user_id,
expires_at,
) -> None:
if not self._redis_available:
return
ttl = max(
1,
int((expires_at - datetime.now(timezone.utc)).total_seconds()),
)
try:
self.redis.setex(
f"session:{token_hash}",
ttl,
str(user_id),
)
except RedisError:
self._redis_available = False
def authenticate(self, token: str) -> User:
token_hash = self._hash_token(token)
cached_user_id = None
if self._redis_available:
try:
cached_user_id = self.redis.get(
f"session:{token_hash}"
)
except RedisError:
self._redis_available = False
session = self.sessions.get_valid_session(token_hash)
if session is None:
if self._redis_available:
try:
self.redis.delete(f"session:{token_hash}")
except RedisError:
self._redis_available = False
raise InvalidSessionError(
"Session is invalid or expired"
)
if cached_user_id is None:
self._cache_session(
token_hash,
session.user_id,
session.expires_at,
)
return session.user
def delete_session(self, token: str) -> None:
token_hash = self._hash_token(token)
try:
deleted = self.sessions.delete_by_token_hash(token_hash)
if not deleted:
raise InvalidSessionError(
"Session is invalid or expired"
)
self.db.commit()
except Exception:
self.db.rollback()
raise
if self._redis_available:
try:
self.redis.delete(f"session:{token_hash}")
except RedisError:
self._redis_available = False
+94
View File
@@ -0,0 +1,94 @@
import uuid
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from app.db.models import Chat
from app.repositories.chat_repository import ChatRepository
from app.repositories.user_repository import UserRepository
class ChatNotFoundError(Exception):
pass
class ChatAccessError(Exception):
pass
class InvalidChatDataError(Exception):
pass
class ChatService:
def __init__(self, db: Session):
self.db = db
self.chats = ChatRepository(db)
self.users = UserRepository(db)
def create_chat(
self,
title: str,
member_ids: list[str],
creator_id: uuid.UUID,
) -> Chat:
title = title.strip()
if not title or len(title) > 100:
raise InvalidChatDataError("Invalid chat title")
if not member_ids or len(member_ids) > 100:
raise InvalidChatDataError("Invalid members")
try:
parsed_ids = [uuid.UUID(str(value)) for value in member_ids]
except (ValueError, TypeError, AttributeError):
raise InvalidChatDataError("Invalid member UUID") from None
if len(parsed_ids) != len(set(parsed_ids)):
raise InvalidChatDataError("Duplicate members")
if creator_id not in parsed_ids:
raise InvalidChatDataError("Creator must be a member")
for user_id in parsed_ids:
if self.users.get_by_id(user_id) is None:
raise InvalidChatDataError("Unknown member")
try:
chat = self.chats.create(
title=title,
member_ids=parsed_ids,
)
self.db.commit()
self.db.refresh(chat)
return chat
except IntegrityError as exc:
self.db.rollback()
raise InvalidChatDataError("Unable to create chat") from exc
except Exception:
self.db.rollback()
raise
def get_chat(self, chat_id: uuid.UUID, user_id: uuid.UUID) -> Chat:
chat = self.chats.get_by_id(chat_id)
if chat is None:
raise ChatNotFoundError()
if not self.chats.is_member(chat_id, user_id):
raise ChatNotFoundError()
return chat
def list_chats(self, user_id: uuid.UUID) -> list[Chat]:
return self.chats.get_by_user(user_id)
def check_membership(
self,
chat_id: uuid.UUID,
user_id: uuid.UUID,
) -> bool:
return self.chats.is_member(chat_id, user_id)
+281
View File
@@ -0,0 +1,281 @@
"""Deterministic image variants for the attachment pipeline."""
import hashlib
import io
import multiprocessing
import time
import warnings
from dataclasses import dataclass
from PIL import Image, ImageOps, UnidentifiedImageError
from app.core.config import settings
@dataclass(frozen=True)
class ProcessedVariant:
name: str
content_type: str
extension: str
width: int
height: int
data: bytes
sha256: str
class ImageProcessingTimeout(TimeoutError):
"""Raised when image processing exceeds its configured deadline."""
def _load_oriented(data: bytes) -> Image.Image:
if len(data) > settings.max_upload_bytes:
raise ValueError("image_too_large")
Image.MAX_IMAGE_PIXELS = settings.max_image_pixels
with warnings.catch_warnings():
warnings.simplefilter("error", Image.DecompressionBombWarning)
try:
source = Image.open(io.BytesIO(data))
if source.format not in {"JPEG", "PNG"}:
raise ValueError("unsupported_image")
source.verify()
source = Image.open(io.BytesIO(data))
image = ImageOps.exif_transpose(source)
image.load()
except (
UnidentifiedImageError,
OSError,
Image.DecompressionBombError,
Image.DecompressionBombWarning,
):
raise ValueError("invalid_image") from None
if image.width * image.height > settings.max_image_pixels:
raise ValueError("image_too_many_pixels")
return image
def _rgb_white(image: Image.Image) -> Image.Image:
if image.mode in ("RGBA", "LA") or (
image.mode == "P" and "transparency" in image.info
):
rgba = image.convert("RGBA")
background = Image.new(
"RGBA",
rgba.size,
(255, 255, 255, 255),
)
return Image.alpha_composite(background, rgba).convert("RGB")
return image.convert("RGB")
def _fit(
image: Image.Image,
box: tuple[int, int],
allow_upscale: bool = False,
) -> Image.Image:
image = image.copy()
image.thumbnail(box, Image.Resampling.LANCZOS)
if allow_upscale:
scale = min(
box[0] / image.width,
box[1] / image.height,
)
if scale > 1:
image = image.resize(
(
max(1, int(image.width * scale)),
max(1, int(image.height * scale)),
),
Image.Resampling.LANCZOS,
)
return image
def _encode(
name: str,
image: Image.Image,
fmt: str,
content_type: str,
extension: str,
**kwargs,
) -> ProcessedVariant:
output = io.BytesIO()
image.save(output, format=fmt, **kwargs)
data = output.getvalue()
return ProcessedVariant(
name=name,
content_type=content_type,
extension=extension,
width=image.width,
height=image.height,
data=data,
sha256=hashlib.sha256(data).hexdigest(),
)
def create_variants(original: bytes) -> list[ProcessedVariant]:
source = _load_oriented(original)
rgb = _rgb_white(source)
# 256x256 fit; never upscale; dimensions retain aspect ratio.
thumb = _fit(rgb, (256, 256))
thumbnail = _encode(
"thumbnail",
thumb,
"JPEG",
"image/jpeg",
"jpg",
quality=85,
optimize=True,
progressive=True,
)
# Center crop to square, 128x128. Upscaling is allowed by contract.
side = min(rgb.width, rgb.height)
left = (rgb.width - side) // 2
top = (rgb.height - side) // 2
crop = rgb.crop((left, top, left + side, top + side))
crop = crop.resize((128, 128), Image.Resampling.LANCZOS)
cropped = _encode(
"crop",
crop,
"JPEG",
"image/jpeg",
"jpg",
quality=85,
optimize=True,
progressive=True,
)
# Optimized WebP fit 1280x1280; never upscale.
webp_image = _fit(rgb, (1280, 1280))
webp = _encode(
"optimized",
webp_image,
"WEBP",
"image/webp",
"webp",
quality=82,
method=6,
)
return [thumbnail, cropped, webp]
def _variant_process_entry(connection, original: bytes) -> None:
"""Run Pillow processing in an isolated child process."""
try:
result = create_variants(original)
connection.send(("ok", result))
except ValueError as exc:
connection.send(("validation_error", str(exc)[:100]))
except Exception:
connection.send(("processing_error", "processing_failed"))
finally:
connection.close()
def create_variants_with_timeout(
original: bytes,
timeout_seconds: float,
) -> list[ProcessedVariant]:
"""Run image processing in a child process with a hard deadline."""
if timeout_seconds <= 0:
raise ValueError("invalid_processing_timeout")
context = multiprocessing.get_context("spawn")
receive_connection, send_connection = context.Pipe(duplex=False)
process = context.Process(
target=_variant_process_entry,
args=(send_connection, original),
name="image-processor",
)
try:
process.start()
send_connection.close()
deadline = time.monotonic() + timeout_seconds
message = None
while time.monotonic() < deadline:
if receive_connection.poll(0.1):
try:
message = receive_connection.recv()
except EOFError:
message = None
break
if not process.is_alive():
if receive_connection.poll():
try:
message = receive_connection.recv()
except EOFError:
message = None
break
if message is None and process.is_alive():
process.terminate()
process.join(timeout=2)
if process.is_alive():
process.kill()
process.join()
raise ImageProcessingTimeout(
"image_processing_timeout"
)
process.join(timeout=2)
if process.is_alive():
process.terminate()
process.join(timeout=2)
if message is None:
raise RuntimeError("image_processing_failed")
status, payload = message
if status == "ok":
return payload
if status == "validation_error":
raise ValueError(payload)
raise RuntimeError("image_processing_failed")
finally:
if process.is_alive():
process.terminate()
process.join(timeout=2)
if process.is_alive():
process.kill()
process.join()
receive_connection.close()
try:
send_connection.close()
except OSError:
pass
+41
View File
@@ -0,0 +1,41 @@
from io import BytesIO
from PIL import Image, UnidentifiedImageError
from fastapi import HTTPException
MAX_BYTES = 10 * 1024 * 1024
MAX_PIXELS = 40_000_000
ALLOWED = {"PNG": "image/png", "JPEG": "image/jpeg"}
def validate_image_file(fileobj, declared_type: str):
"""Validate a seekable, size-limited file without loading its full encoded bytes."""
try:
fileobj.seek(0)
with Image.open(fileobj) as image:
fmt = image.format
width, height = image.size
if fmt not in ALLOWED:
raise HTTPException(415, "Only PNG and JPEG are supported")
if width * height > MAX_PIXELS:
raise HTTPException(413, "Image exceeds 40 million pixels")
image.verify()
fileobj.seek(0)
with Image.open(fileobj) as image:
image.load()
except HTTPException:
raise
except (UnidentifiedImageError, OSError, ValueError, SyntaxError, Image.DecompressionBombError):
raise HTTPException(415, "Invalid or corrupted image") from None
actual_type = ALLOWED[fmt]
if declared_type != actual_type:
raise HTTPException(415, "Content-Type does not match image content")
fileobj.seek(0)
return actual_type, width, height
def validate_image(data: bytes, declared_type: str):
if not data:
raise HTTPException(415, "Empty image")
if len(data) > MAX_BYTES:
raise HTTPException(413, "Image exceeds 10 MiB")
return validate_image_file(BytesIO(data), declared_type)
+444
View File
@@ -0,0 +1,444 @@
import base64
import hashlib
import json
import time
import uuid
from datetime import datetime, timedelta, timezone
from types import SimpleNamespace
from redis import Redis
from redis.backoff import NoBackoff
from redis.exceptions import RedisError
from redis.retry import Retry
from sqlalchemy import delete, select, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.orm import Session
from app.core.config import settings
from app.db.models import Attachment, IdempotencyRecord, Message
from app.repositories.chat_repository import ChatRepository
from app.repositories.message_repository import MessageRepository
class MessageNotFoundError(Exception):
pass
class MessageAccessError(Exception):
pass
class InvalidMessageDataError(Exception):
pass
class IdempotencyConflictError(Exception):
pass
class MessageService:
def __init__(self, db: Session):
self.db = db
self.messages = MessageRepository(db)
self.chats = ChatRepository(db)
self._redis_available = True
self.redis = Redis.from_url(
settings.cache_url,
decode_responses=True,
socket_connect_timeout=0.5,
socket_timeout=0.5,
retry_on_timeout=False,
retry=Retry(NoBackoff(), 0),
)
@staticmethod
def encode_cursor(message: Message) -> str:
payload = {
"created_at": message.created_at.isoformat(),
"id": str(message.id),
}
raw = json.dumps(payload).encode("utf-8")
return (
base64.urlsafe_b64encode(raw)
.decode("ascii")
.rstrip("=")
)
@staticmethod
def decode_cursor(cursor: str | None):
if not cursor:
return None, None
try:
padded = cursor + "=" * (-len(cursor) % 4)
raw = base64.urlsafe_b64decode(padded.encode("ascii"))
payload = json.loads(raw.decode("utf-8"))
created_at = datetime.fromisoformat(payload["created_at"])
message_id = uuid.UUID(payload["id"])
if created_at.tzinfo is None:
raise ValueError("Cursor timestamp must include timezone")
return created_at, message_id
except (ValueError, TypeError, KeyError, UnicodeError):
raise InvalidMessageDataError("Invalid cursor") from None
def _version_key(self, chat_id: uuid.UUID) -> str:
return f"messages:version:{chat_id}"
def _cache_key(
self,
chat_id: uuid.UUID,
version: int,
limit: int,
cursor: str | None,
) -> str:
cursor_hash = hashlib.sha256(
(cursor or "first-page").encode("utf-8")
).hexdigest()[:16]
return (
f"messages:{chat_id}:v{version}:"
f"limit:{limit}:cursor:{cursor_hash}"
)
def _get_version(self, chat_id: uuid.UUID) -> int:
if not self._redis_available:
return 0
try:
value = self.redis.get(self._version_key(chat_id))
return int(value or 0)
except (RedisError, ValueError):
self._redis_available = False
return 0
def _bump_version(self, chat_id: uuid.UUID) -> None:
if not self._redis_available:
return
try:
self.redis.incr(self._version_key(chat_id))
except RedisError:
self._redis_available = False
@staticmethod
def _serialize(messages, next_cursor):
return json.dumps({
"items": [
{
"id": str(message.id),
"chat_id": str(message.chat_id),
"sender_id": str(message.sender_id),
"text": message.text,
"attachment_ids": message.attachment_ids or [],
"created_at": message.created_at.isoformat(),
}
for message in messages
],
"next_cursor": next_cursor,
})
@staticmethod
def _deserialize(payload: str):
data = json.loads(payload)
messages = [
SimpleNamespace(
id=uuid.UUID(item["id"]),
chat_id=uuid.UUID(item["chat_id"]),
sender_id=uuid.UUID(item["sender_id"]),
text=item["text"],
attachment_ids=item.get("attachment_ids", []),
created_at=datetime.fromisoformat(item["created_at"]),
)
for item in data["items"]
]
return messages, data["next_cursor"]
def _read_cache(self, key: str):
if not self._redis_available:
return None
try:
payload = self.redis.get(key)
if payload is None:
return None
return self._deserialize(payload)
except RedisError:
self._redis_available = False
return None
except (ValueError, TypeError, KeyError):
return None
def _load_from_database(
self,
chat_id,
limit,
cursor_created_at,
cursor_id,
):
rows = self.messages.get_history(
chat_id=chat_id,
limit=limit + 1,
cursor_created_at=cursor_created_at,
cursor_id=cursor_id,
)
has_more = len(rows) > limit
items = rows[:limit]
next_cursor = (
self.encode_cursor(items[-1])
if has_more and items
else None
)
return items, next_cursor
def create_message(
self,
chat_id: uuid.UUID,
sender_id: uuid.UUID,
text: str,
idempotency_key: str,
attachment_ids: list[str] | None = None,
) -> Message:
text = text.strip()
attachment_ids = attachment_ids or []
if (not text and not attachment_ids) or len(text) > 4000 or len(attachment_ids) > 10:
raise InvalidMessageDataError("Invalid message text")
# Access is checked for every retry, before consulting the key.
if not self.chats.is_member(chat_id, sender_id):
raise MessageNotFoundError()
if len(attachment_ids) != len(set(attachment_ids)):
raise InvalidMessageDataError("Duplicate attachment IDs")
if attachment_ids:
try:
parsed_attachment_ids = [uuid.UUID(value) for value in attachment_ids]
except (ValueError, TypeError):
raise InvalidMessageDataError("Invalid attachment UUID") from None
attachments = list(
self.db.scalars(
select(Attachment).where(
Attachment.id.in_(parsed_attachment_ids)
)
).all()
)
if (
len(attachments) != len(parsed_attachment_ids)
or any(attachment.chat_id != chat_id for attachment in attachments)
):
raise MessageNotFoundError()
method = "POST"
path = f"/api/v1/chats/{chat_id}/messages"
request_hash = hashlib.sha256(
json.dumps({"text": text, "attachment_ids": attachment_ids}, sort_keys=True, separators=(",", ":")).encode("utf-8")
).hexdigest()
now = datetime.now(timezone.utc)
expires_at = now + timedelta(seconds=max(settings.idempotency_ttl_seconds, 86400))
scope = (
IdempotencyRecord.user_id == sender_id,
IdempotencyRecord.method == method,
IdempotencyRecord.path == path,
IdempotencyRecord.idempotency_key == idempotency_key,
)
try:
# An expired key may be reused. Deletion and claim are serialized by
# PostgreSQL's unique constraint for the scope.
self.db.execute(
delete(IdempotencyRecord).where(*scope, IdempotencyRecord.expires_at <= now)
)
claim = (
pg_insert(IdempotencyRecord)
.values(
id=uuid.uuid4(), user_id=sender_id, method=method, path=path,
idempotency_key=idempotency_key, request_hash=request_hash,
message_id=None, created_at=now, expires_at=expires_at,
)
.on_conflict_do_nothing(constraint="uq_idempotency_scope_key")
.returning(IdempotencyRecord.id)
)
claimed_id = self.db.execute(claim).scalar_one_or_none()
if claimed_id is None:
existing = self.db.execute(
select(IdempotencyRecord).where(*scope).with_for_update()
).scalar_one_or_none()
if existing is None:
# Rare expiration race: retry once in a fresh transaction.
self.db.rollback()
return self.create_message(chat_id, sender_id, text, idempotency_key, attachment_ids)
if existing.request_hash != request_hash:
self.db.rollback()
raise IdempotencyConflictError()
if existing.message_id is None:
raise RuntimeError("Committed idempotency record has no message")
message = self.db.get(Message, existing.message_id)
if message is None:
raise RuntimeError("Idempotency record references a missing message")
self.db.commit()
return message
message = self.messages.create(
chat_id=chat_id, sender_id=sender_id, text=text, attachment_ids=attachment_ids
)
self.db.execute(
update(IdempotencyRecord)
.where(IdempotencyRecord.id == claimed_id)
.values(message_id=message.id)
)
self.db.commit()
self.db.refresh(message)
self._bump_version(chat_id)
return message
except IdempotencyConflictError:
raise
except Exception:
self.db.rollback()
raise
def get_history(
self,
chat_id: uuid.UUID,
user_id: uuid.UUID,
limit: int = 50,
cursor: str | None = None,
):
if not 1 <= limit <= 100:
raise InvalidMessageDataError(
"Limit must be between 1 and 100"
)
if not self.chats.is_member(chat_id, user_id):
raise MessageNotFoundError()
cursor_created_at, cursor_id = self.decode_cursor(cursor)
version = self._get_version(chat_id)
cache_key = self._cache_key(
chat_id,
version,
limit,
cursor,
)
cached = self._read_cache(cache_key)
if cached is not None:
return cached
if not self._redis_available:
return self._load_from_database(
chat_id,
limit,
cursor_created_at,
cursor_id,
)
lock_key = f"lock:{cache_key}"
lock_token = str(uuid.uuid4())
try:
acquired = self.redis.set(
lock_key,
lock_token,
nx=True,
ex=10,
)
except RedisError:
self._redis_available = False
return self._load_from_database(
chat_id,
limit,
cursor_created_at,
cursor_id,
)
if not acquired:
for _ in range(10):
time.sleep(0.05)
cached = self._read_cache(cache_key)
if cached is not None:
return cached
if not self._redis_available:
break
return self._load_from_database(
chat_id,
limit,
cursor_created_at,
cursor_id,
)
try:
cached = self._read_cache(cache_key)
if cached is not None:
return cached
if not self._redis_available:
return self._load_from_database(
chat_id,
limit,
cursor_created_at,
cursor_id,
)
items, next_cursor = self._load_from_database(
chat_id,
limit,
cursor_created_at,
cursor_id,
)
if self._redis_available and self._get_version(chat_id) == version:
try:
self.redis.setex(
cache_key,
settings.message_cache_ttl_seconds,
self._serialize(items, next_cursor),
)
except RedisError:
self._redis_available = False
return items, next_cursor
finally:
if self._redis_available:
try:
self.redis.eval(
"""
if redis.call('get', KEYS[1]) == ARGV[1]
then
return redis.call('del', KEYS[1])
else
return 0
end
""",
1,
lock_key,
lock_token,
)
except RedisError:
self._redis_available = False
+39
View File
@@ -0,0 +1,39 @@
import boto3
from botocore.config import Config
from app.core.config import settings
def client(endpoint_url=None):
return boto3.client(
"s3", endpoint_url=endpoint_url or settings.s3_endpoint_url,
aws_access_key_id=settings.s3_access_key,
aws_secret_access_key=settings.s3_secret_key,
region_name=settings.s3_region,
config=Config(signature_version="s3v4", connect_timeout=3, read_timeout=15, retries={"max_attempts": 2}, s3={"addressing_style": "path"}),
)
def ensure_bucket() -> None:
s3 = client()
try:
s3.head_bucket(Bucket=settings.s3_bucket)
except Exception:
s3.create_bucket(Bucket=settings.s3_bucket)
def presigned_client():
return client(endpoint_url=settings.s3_public_endpoint_url)
def create_presigned_upload_url(object_key: str, content_type: str) -> str:
s3 = presigned_client()
return s3.generate_presigned_url(
ClientMethod="put_object",
Params={
"Bucket": settings.s3_bucket,
"Key": object_key,
"ContentType": content_type,
},
ExpiresIn=settings.presigned_upload_ttl_seconds,
)
+102
View File
@@ -0,0 +1,102 @@
"""Publish durable PostgreSQL outbox events to RabbitMQ.
The database row is marked published only after RabbitMQ publisher confirmation.
If the process crashes after broker confirmation but before the DB commit, the
same event is published again; consumers must therefore be idempotent.
"""
import asyncio
import json
import logging
import random
from datetime import datetime, timezone
from typing import Any
import aio_pika
from aio_pika import DeliveryMode, Message
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.core.config import settings
from app.db.models import OutboxEvent
from app.db.session import SessionLocal
logger = logging.getLogger(__name__)
def utc_now() -> datetime:
return datetime.now(timezone.utc)
async def publish_once() -> int:
"""Publish one batch; return number of confirmed events."""
connection = await aio_pika.connect_robust(settings.rabbitmq_url)
try:
channel = await connection.channel(publisher_confirms=True)
exchange = await channel.declare_exchange(
settings.rabbitmq_exchange,
aio_pika.ExchangeType.DIRECT,
durable=True,
)
queue = await channel.declare_queue(settings.rabbitmq_queue, durable=True)
await queue.bind(exchange, routing_key=settings.rabbitmq_queue)
published = 0
db: Session = SessionLocal()
try:
# Lock a bounded batch. SKIP LOCKED allows multiple publisher replicas.
events = db.scalars(
select(OutboxEvent)
.where(OutboxEvent.status == "pending")
.order_by(OutboxEvent.created_at)
.with_for_update(skip_locked=True)
.limit(settings.outbox_batch_size)
).all()
for event in events:
body = json.dumps(event.payload, separators=(",", ":")).encode("utf-8")
message = Message(
body=body,
content_type="application/json",
delivery_mode=DeliveryMode.PERSISTENT,
message_id=str(event.id),
type=event.event_type,
timestamp=utc_now(),
)
await exchange.publish(message, routing_key=settings.rabbitmq_queue, mandatory=True)
event.status = "published"
event.published_at = utc_now()
event.attempts += 1
published += 1
db.commit()
return published
except Exception:
db.rollback()
raise
finally:
db.close()
finally:
await connection.close()
async def run_publisher() -> None:
delay = settings.outbox_poll_interval_seconds
while True:
try:
count = await publish_once()
if count:
logger.info("Published %s outbox events", count)
delay = settings.outbox_poll_interval_seconds
except asyncio.CancelledError:
raise
except Exception:
logger.exception("Outbox publisher iteration failed")
delay = min(max(delay * 2, settings.outbox_poll_interval_seconds), 15.0)
delay += random.uniform(0, delay * 0.2)
await asyncio.sleep(delay)
if __name__ == "__main__":
logging.basicConfig(level=logging.INFO)
asyncio.run(run_publisher())
+81
View File
@@ -0,0 +1,81 @@
import re
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from pwdlib import PasswordHash
from app.db.models import User
from app.repositories.user_repository import UserRepository
password_hasher = PasswordHash.recommended()
class UsernameAlreadyExistsError(Exception):
pass
class UserService:
def __init__(self, db: Session):
self.db = db
self.users = UserRepository(db)
def register(
self,
username: str,
display_name: str,
password: str,
) -> User:
if not re.fullmatch(r"[a-z0-9_]{3,32}", username):
raise ValueError("Invalid username")
if not display_name or not display_name.strip():
raise ValueError("Display name cannot be empty")
if len(display_name) > 100:
raise ValueError("Display name is too long")
if not 12 <= len(password) <= 128:
raise ValueError("Invalid password length")
if self.users.get_by_username(username) is not None:
raise UsernameAlreadyExistsError(
"Username already exists"
)
password_hash = password_hasher.hash(password)
try:
user = self.users.create(
username=username,
display_name=display_name,
password_hash=password_hash,
)
self.db.commit()
self.db.refresh(user)
return user
except IntegrityError as exc:
self.db.rollback()
raise UsernameAlreadyExistsError(
"Username already exists"
) from exc
except Exception:
self.db.rollback()
raise
def verify_password(
self,
user: User,
password: str,
) -> bool:
return password_hasher.verify(
password,
user.password_hash,
)
+193
View File
@@ -0,0 +1,193 @@
import base64
import bisect
import binascii
import json
import os
import threading
import uuid
from datetime import datetime, timezone
from typing import Dict, List, Optional, Tuple
from .models import Chat, Message, User
Key = Tuple[datetime, str]
def _encode_cursor(kind: str, scope: str, item) -> str:
payload = {
"v": 1,
"kind": kind,
"scope": scope,
"created_at": item.created_at.isoformat(),
"id": item.id,
}
raw = json.dumps(payload, separators=(",", ":")).encode("utf-8")
return base64.urlsafe_b64encode(raw).decode("ascii").rstrip("=")
def _decode_cursor(cursor: str, kind: str, scope: str) -> Key:
try:
if not cursor or len(cursor) > 4096:
raise ValueError
padded = cursor + "=" * (-len(cursor) % 4)
raw = base64.b64decode(padded, altchars=b"-_", validate=True)
payload = json.loads(raw.decode("utf-8"))
if not isinstance(payload, dict) or payload.get("v") != 1:
raise ValueError
if payload.get("kind") != kind or payload.get("scope") != scope:
raise ValueError
item_id = payload.get("id")
timestamp = payload.get("created_at")
if not isinstance(item_id, str) or not isinstance(timestamp, str):
raise ValueError
created_at = datetime.fromisoformat(timestamp)
if created_at.tzinfo is None:
raise ValueError
created_at = created_at.astimezone(timezone.utc)
return created_at, item_id
except (
ValueError,
TypeError,
KeyError,
UnicodeDecodeError,
json.JSONDecodeError,
binascii.Error,
):
raise ValueError("invalid_request") from None
class InMemoryStore:
def __init__(self):
self._lock = threading.RLock()
self.users: Dict[str, User] = {}
self.chats: Dict[str, Chat] = {}
self.messages: Dict[str, List[Message]] = {}
self.usernames: Dict[str, str] = {}
self.message_keys: Dict[str, List[Key]] = {}
self.message_index_enabled = os.getenv("MESSAGE_INDEX_ENABLED", "1") != "0"
def create_user(self, username: str, display_name: str) -> User:
with self._lock:
if username in self.usernames:
raise ValueError("conflict")
user_id = str(uuid.uuid4())
user = User(
id=user_id,
username=username,
display_name=display_name,
created_at=datetime.now(timezone.utc),
)
self.users[user_id] = user
self.usernames[username] = user_id
return user
def get_user(self, user_id: str) -> Optional[User]:
with self._lock:
return self.users.get(user_id)
def create_chat(self, title: str, member_ids: List[str], creator_id: str) -> Chat:
with self._lock:
if creator_id not in member_ids or len(member_ids) != len(set(member_ids)):
raise ValueError("invalid_request")
if any(member_id not in self.users for member_id in member_ids):
raise ValueError("invalid_request")
chat_id = str(uuid.uuid4())
chat = Chat(
id=chat_id,
title=title,
member_ids=list(member_ids),
created_at=datetime.now(timezone.utc),
)
self.chats[chat_id] = chat
self.messages[chat_id] = []
self.message_keys[chat_id] = []
return chat
def get_chat(self, chat_id: str, user_id: str) -> Optional[Chat]:
with self._lock:
chat = self.chats.get(chat_id)
if chat is None or user_id not in chat.member_ids:
return None
return chat
def list_chats(
self, user_id: str, limit: int, cursor: Optional[str]
) -> Tuple[List[Chat], Optional[str]]:
with self._lock:
user_chats = [chat for chat in self.chats.values() if user_id in chat.member_ids]
user_chats.sort(key=lambda chat: (chat.created_at, chat.id))
keys = [(chat.created_at, chat.id) for chat in user_chats]
start_index = 0
if cursor:
target_key = _decode_cursor(cursor, kind="chats", scope=user_id)
start_index = bisect.bisect_right(keys, target_key)
page_with_extra = user_chats[start_index : start_index + limit + 1]
has_next = len(page_with_extra) > limit
page = page_with_extra[:limit]
next_cursor = None
if has_next and page:
next_cursor = _encode_cursor("chats", user_id, page[-1])
return page, next_cursor
def create_message(self, chat_id: str, sender_id: str, text: str) -> Message:
with self._lock:
chat = self.chats.get(chat_id)
if chat is None or sender_id not in chat.member_ids:
raise ValueError("not_found")
message = Message(
id=str(uuid.uuid4()),
chat_id=chat_id,
sender_id=sender_id,
text=text,
created_at=datetime.now(timezone.utc),
)
messages = self.messages[chat_id]
keys = self.message_keys[chat_id]
key = (message.created_at, message.id)
if self.message_index_enabled:
index = bisect.bisect_right(keys, key)
keys.insert(index, key)
messages.insert(index, message)
else:
current_keys = [(item.created_at, item.id) for item in messages]
index = bisect.bisect_right(current_keys, key)
messages.insert(index, message)
keys.insert(index, key)
return message
def list_messages(
self,
chat_id: str,
user_id: str,
limit: int,
cursor: Optional[str],
) -> Tuple[List[Message], Optional[str]]:
with self._lock:
chat = self.chats.get(chat_id)
if chat is None or user_id not in chat.member_ids:
raise ValueError("not_found")
messages = self.messages[chat_id]
if self.message_index_enabled:
keys = self.message_keys[chat_id]
else:
keys = [(item.created_at, item.id) for item in messages]
start_index = 0
if cursor:
target_key = _decode_cursor(cursor, kind="messages", scope=chat_id)
start_index = bisect.bisect_right(keys, target_key)
page_with_extra = messages[start_index : start_index + limit + 1]
has_next = len(page_with_extra) > limit
page = page_with_extra[:limit]
next_cursor = None
if has_next and page:
next_cursor = _encode_cursor("messages", chat_id, page[-1])
return page, next_cursor
+982
View File
@@ -0,0 +1,982 @@
"""RabbitMQ consumer process for durable image processing."""
import asyncio
import json
import logging
import random
import socket
import threading
import uuid
import time
from datetime import datetime, timedelta, timezone
import aio_pika
from aio_pika import DeliveryMode, Message
from botocore.exceptions import BotoCoreError, ClientError
from sqlalchemy import select, update
from sqlalchemy.exc import SQLAlchemyError
from app.core.config import settings
from app.db.models import Attachment, AttachmentVariant, ProcessingJob
from app.db.session import SessionLocal
from app.services.image_processor import (
ImageProcessingTimeout,
create_variants_with_timeout,
)
from app.services.object_storage import client as s3_client
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)s %(name)s %(message)s",
)
logger = logging.getLogger("attachment-worker")
WORKER_ID = (
f"{socket.gethostname()[:40]}-{uuid.uuid4().hex[:16]}"
)
DEAD_ROUTING_KEY = f"{settings.rabbitmq_queue}.dead"
PROCESSING_SEMAPHORE = None
def utc_now() -> datetime:
return datetime.now(timezone.utc)
def _safe_uuid(value) -> uuid.UUID:
return uuid.UUID(str(value))
def _claim_job(job_id, attachment_id, pipeline_version) -> dict:
"""Atomically claim a job or return the reason it cannot be claimed."""
job_id = _safe_uuid(job_id)
attachment_id = _safe_uuid(attachment_id)
pipeline_version = int(pipeline_version)
db = SessionLocal()
try:
job = db.scalar(
select(ProcessingJob)
.where(
ProcessingJob.id == job_id,
ProcessingJob.attachment_id == attachment_id,
ProcessingJob.pipeline_version == pipeline_version,
)
.with_for_update()
)
if job is None:
db.rollback()
return {"state": "missing"}
attachment = db.get(Attachment, attachment_id)
if attachment is None:
db.rollback()
return {"state": "missing"}
now = utc_now()
if attachment.status == "ready":
db.commit()
return {"state": "ready"}
if job.status == "failed":
db.commit()
return {"state": "failed"}
if (
job.status == "processing"
and job.lease_until is not None
and job.lease_until > now
):
delay = (job.lease_until - now).total_seconds() + 1
db.commit()
return {
"state": "busy",
"delay": max(1.0, delay),
}
if job.available_at > now:
delay = (job.available_at - now).total_seconds() + 0.5
db.commit()
return {
"state": "deferred",
"delay": max(1.0, delay),
}
if job.attempt >= settings.worker_max_attempts:
job.status = "failed"
job.worker_id = None
job.lease_until = None
job.last_error_code = "processing_attempts_exhausted"
attachment.status = "failed"
attachment.error_code = "processing_attempts_exhausted"
db.commit()
return {"state": "failed"}
claim_worker_id = f"{WORKER_ID}:{uuid.uuid4().hex[:16]}"
job.status = "processing"
job.worker_id = claim_worker_id
job.lease_until = now + timedelta(
seconds=settings.worker_lease_seconds
)
job.attempt += 1
attachment.status = "processing"
attachment.error_code = None
claimed_attempt = job.attempt
queue_lag_ms = max(
0.0,
(now - job.available_at).total_seconds() * 1000,
)
db.commit()
logger.info(
"Claimed job=%s attachment=%s attempt=%s queue_lag_ms=%.2f",
job_id,
attachment_id,
claimed_attempt,
queue_lag_ms,
)
return {
"state": "claimed",
"worker_id": claim_worker_id,
"attempt": claimed_attempt,
"queue_lag_ms": queue_lag_ms,
}
except Exception:
db.rollback()
raise
finally:
db.close()
def _renew_lease(
job_id: uuid.UUID,
worker_id: str,
attempt: int,
stop_event: threading.Event,
lost_event: threading.Event,
) -> None:
"""Renew the lease while image processing is running."""
interval = max(
5,
settings.worker_lease_seconds // 3,
)
while not stop_event.wait(interval):
db = SessionLocal()
try:
result = db.execute(
update(ProcessingJob)
.where(
ProcessingJob.id == job_id,
ProcessingJob.worker_id == worker_id,
ProcessingJob.attempt == attempt,
ProcessingJob.status == "processing",
)
.values(
lease_until=utc_now()
+ timedelta(seconds=settings.worker_lease_seconds)
)
)
db.commit()
if result.rowcount != 1:
lost_event.set()
logger.warning(
"Lease ownership lost: job=%s worker=%s",
job_id,
worker_id,
)
return
except Exception:
db.rollback()
lost_event.set()
logger.exception(
"Could not renew lease for job=%s",
job_id,
)
return
finally:
db.close()
def _fail_job(
job_id,
attachment_id,
worker_id,
attempt,
error_code,
permanent=False,
) -> dict:
"""Record a failure only if this worker still owns the job."""
db = SessionLocal()
try:
job = db.scalar(
select(ProcessingJob)
.where(ProcessingJob.id == job_id)
.with_for_update()
)
attachment = db.scalar(
select(Attachment)
.where(Attachment.id == attachment_id)
.with_for_update()
)
if job is None or attachment is None:
db.rollback()
return {"state": "missing"}
if (
job.status != "processing"
or job.worker_id != worker_id
or job.attempt != attempt
):
db.rollback()
logger.warning(
"Ignoring failure from stale worker: job=%s worker=%s",
job_id,
worker_id,
)
return {
"state": "stale",
"delay": float(settings.worker_lease_seconds),
}
exhausted = job.attempt >= settings.worker_max_attempts
job.last_error_code = error_code
if permanent or exhausted:
job.status = "failed"
job.worker_id = None
job.lease_until = None
attachment.status = "failed"
attachment.error_code = error_code
db.commit()
logger.error(
"Job permanently failed: job=%s error=%s",
job_id,
error_code,
)
return {"state": "failed"}
delay = min(
settings.worker_retry_base_seconds
* (2 ** max(job.attempt - 1, 0)),
settings.worker_retry_max_seconds,
)
delay += random.uniform(0, delay * 0.25)
job.status = "queued"
job.available_at = utc_now() + timedelta(seconds=delay)
job.lease_until = None
job.worker_id = None
attachment.status = "queued"
attachment.error_code = error_code
db.commit()
logger.warning(
"Job scheduled for retry: job=%s delay=%.2fs error=%s",
job_id,
delay,
error_code,
)
return {
"state": "retry",
"delay": delay,
}
except Exception:
db.rollback()
raise
finally:
db.close()
def _process(payload: dict) -> dict:
"""Process one image using a fenced, renewable job lease."""
job_id = _safe_uuid(payload["job_id"])
attachment_id = _safe_uuid(payload["attachment_id"])
pipeline_version = int(payload["pipeline_version"])
claim = _claim_job(
job_id,
attachment_id,
pipeline_version,
)
if claim["state"] != "claimed":
return claim
worker_id = claim["worker_id"]
attempt = claim["attempt"]
queue_lag_ms = claim["queue_lag_ms"]
processing_started_at = time.perf_counter()
stop_event = threading.Event()
lost_event = threading.Event()
heartbeat = threading.Thread(
target=_renew_lease,
args=(
job_id,
worker_id,
attempt,
stop_event,
lost_event,
),
daemon=True,
)
heartbeat.start()
staging_key = None
db = None
try:
db = SessionLocal()
attachment = db.get(Attachment, attachment_id)
if attachment is None:
raise ValueError("attachment_missing")
staging_key = attachment.staging_key
object_key = attachment.object_key
content_type = attachment.content_type
if not staging_key:
raise ValueError("staging_missing")
db.close()
db = None
if lost_event.is_set():
return {
"state": "retry",
"delay": float(settings.worker_lease_seconds),
}
s3 = s3_client()
response = s3.get_object(
Bucket=settings.s3_bucket,
Key=staging_key,
)
body = response["Body"]
try:
attachment_data = body.read(
settings.max_upload_bytes + 1
)
finally:
body.close()
if len(attachment_data) > settings.max_upload_bytes:
raise ValueError("image_too_large")
if lost_event.is_set():
return {
"state": "retry",
"delay": float(settings.worker_lease_seconds),
}
variants = create_variants_with_timeout(
attachment_data,
settings.image_processing_timeout_seconds,
)
# The original object uses a deterministic key.
s3.put_object(
Bucket=settings.s3_bucket,
Key=object_key,
Body=attachment_data,
ContentType=content_type,
)
variant_records = []
for variant in variants:
key = (
f"variants/{attachment_id}/"
f"v{pipeline_version}/"
f"{variant.name}.{variant.extension}"
)
s3.put_object(
Bucket=settings.s3_bucket,
Key=key,
Body=variant.data,
ContentType=variant.content_type,
)
variant_records.append(
{
"name": variant.name,
"key": key,
"content_type": variant.content_type,
"width": variant.width,
"height": variant.height,
"size_bytes": len(variant.data),
"sha256": variant.sha256,
}
)
if lost_event.is_set():
logger.warning(
"Worker lost lease before final commit: job=%s",
job_id,
)
return {
"state": "retry",
"delay": float(settings.worker_lease_seconds),
}
# Use a fresh transaction and verify the fencing token.
db = SessionLocal()
job = db.scalar(
select(ProcessingJob)
.where(ProcessingJob.id == job_id)
.with_for_update()
)
attachment = db.scalar(
select(Attachment)
.where(Attachment.id == attachment_id)
.with_for_update()
)
if job is None or attachment is None:
db.rollback()
return {"state": "missing"}
if (
job.status != "processing"
or job.worker_id != worker_id
or job.attempt != attempt
or job.lease_until is None
or job.lease_until <= utc_now()
or lost_event.is_set()
):
db.rollback()
logger.warning(
"Fencing check rejected stale worker: job=%s",
job_id,
)
return {
"state": "retry",
"delay": float(settings.worker_lease_seconds),
}
for item in variant_records:
existing = db.scalar(
select(AttachmentVariant)
.where(
AttachmentVariant.attachment_id == attachment_id,
AttachmentVariant.variant == item["name"],
)
)
if existing is None:
existing = AttachmentVariant(
attachment_id=attachment_id,
variant=item["name"],
object_key=item["key"],
content_type=item["content_type"],
width=item["width"],
height=item["height"],
size_bytes=item["size_bytes"],
sha256=item["sha256"],
)
db.add(existing)
else:
existing.object_key = item["key"]
existing.content_type = item["content_type"]
existing.width = item["width"]
existing.height = item["height"]
existing.size_bytes = item["size_bytes"]
existing.sha256 = item["sha256"]
job.status = "ready"
job.lease_until = None
job.worker_id = None
job.last_error_code = None
attachment.status = "ready"
attachment.error_code = None
attachment.updated_at = utc_now()
db.commit()
processing_elapsed_ms = (
time.perf_counter() - processing_started_at
) * 1000
logger.info(
(
"Image processing completed: attachment=%s "
"attempt=%s queue_lag_ms=%.2f processing_elapsed_ms=%.2f"
),
attachment_id,
attempt,
queue_lag_ms,
processing_elapsed_ms,
)
db.close()
db = None
# Staging cleanup is performed only after the final DB commit.
try:
s3.delete_object(
Bucket=settings.s3_bucket,
Key=staging_key,
)
cleanup_db = SessionLocal()
try:
cleanup_attachment = cleanup_db.scalar(
select(Attachment)
.where(Attachment.id == attachment_id)
.with_for_update()
)
if (
cleanup_attachment is not None
and cleanup_attachment.status == "ready"
and cleanup_attachment.staging_key == staging_key
):
cleanup_attachment.staging_key = None
cleanup_db.commit()
else:
cleanup_db.rollback()
finally:
cleanup_db.close()
except Exception:
logger.warning(
"Could not delete staging object for attachment=%s",
attachment_id,
exc_info=True,
)
return {"state": "ready"}
except ImageProcessingTimeout:
if db is not None:
db.rollback()
stop_event.set()
heartbeat.join(timeout=2)
logger.warning(
"Image processing timed out: job=%s attachment=%s timeout=%ss",
job_id,
attachment_id,
settings.image_processing_timeout_seconds,
)
return _fail_job(
job_id,
attachment_id,
worker_id,
attempt,
"image_processing_timeout",
permanent=False,
)
except ValueError as exc:
if db is not None:
db.rollback()
stop_event.set()
heartbeat.join(timeout=2)
code = str(exc)[:64] or "invalid_image"
return _fail_job(
job_id,
attachment_id,
worker_id,
attempt,
code,
permanent=True,
)
except ClientError as exc:
if db is not None:
db.rollback()
stop_event.set()
heartbeat.join(timeout=2)
error = exc.response.get("Error", {})
error_code = error.get("Code", "")
if error_code in {
"NoSuchKey",
"NoSuchBucket",
"404",
"NotFound",
}:
return _fail_job(
job_id,
attachment_id,
worker_id,
attempt,
"staging_missing",
permanent=True,
)
return _fail_job(
job_id,
attachment_id,
worker_id,
attempt,
"storage_unavailable",
permanent=False,
)
except (BotoCoreError, SQLAlchemyError):
if db is not None:
db.rollback()
stop_event.set()
heartbeat.join(timeout=2)
return _fail_job(
job_id,
attachment_id,
worker_id,
attempt,
"processing_unavailable",
permanent=False,
)
except Exception:
if db is not None:
db.rollback()
logger.exception(
"Unexpected processing error: job=%s",
job_id,
)
stop_event.set()
heartbeat.join(timeout=2)
return _fail_job(
job_id,
attachment_id,
worker_id,
attempt,
"processing_failed",
permanent=False,
)
finally:
stop_event.set()
heartbeat.join(timeout=2)
if db is not None:
db.close()
async def _publish_delayed(
exchange,
payload: dict,
delay_seconds: float,
) -> None:
"""Publish a persistent message to the TTL retry queue."""
delay_ms = max(
1000,
int(delay_seconds * 1000),
)
message = Message(
body=json.dumps(
payload,
separators=(",", ":"),
).encode("utf-8"),
content_type="application/json",
delivery_mode=DeliveryMode.PERSISTENT,
message_id=str(uuid.uuid4()),
type="attachment.process.retry",
expiration=timedelta(milliseconds=delay_ms),
)
await exchange.publish(
message,
routing_key=f"{settings.rabbitmq_queue}.retry",
mandatory=True,
)
logger.info(
"Scheduled delayed delivery in %.2fs",
delay_ms / 1000,
)
async def _quarantine_message(exchange, message) -> None:
"""Move an invalid message to the durable dead-letter queue."""
dead_message = Message(
body=message.body,
content_type=message.content_type or "application/octet-stream",
delivery_mode=DeliveryMode.PERSISTENT,
message_id=message.message_id or str(uuid.uuid4()),
type="attachment.process.invalid",
)
await exchange.publish(
dead_message,
routing_key=DEAD_ROUTING_KEY,
mandatory=True,
)
await message.ack()
logger.error(
"Invalid message moved to dead-letter queue: message_id=%s",
message.message_id,
)
async def handle_message(
message: aio_pika.IncomingMessage,
retry_exchange,
dead_exchange,
semaphore: asyncio.Semaphore,
) -> None:
"""Consume, process, retry or quarantine one RabbitMQ message."""
async with semaphore:
try:
payload = json.loads(message.body.decode("utf-8"))
if not isinstance(payload, dict):
raise ValueError("invalid_payload")
for field in (
"job_id",
"attachment_id",
"pipeline_version",
):
if field not in payload:
raise ValueError("missing_payload_field")
_safe_uuid(payload["job_id"])
_safe_uuid(payload["attachment_id"])
int(payload["pipeline_version"])
except (ValueError, TypeError, KeyError, UnicodeDecodeError):
try:
await _quarantine_message(
dead_exchange,
message,
)
except Exception:
logger.exception(
"Could not quarantine invalid message"
)
await message.nack(requeue=True)
return
try:
result = await asyncio.to_thread(
_process,
payload,
)
state = result["state"]
if state in {
"busy",
"deferred",
"retry",
}:
await _publish_delayed(
retry_exchange,
payload,
result.get("delay", 1.0),
)
await message.ack()
logger.info(
"Message handled: job=%s state=%s",
payload["job_id"],
state,
)
except asyncio.CancelledError:
raise
except Exception:
logger.exception(
"Message processing failed: message_id=%s",
message.message_id,
)
# If delayed publication or DB processing fails,
# preserve the original message.
await message.nack(requeue=True)
async def main() -> None:
"""Start the durable RabbitMQ consumer."""
connection = await aio_pika.connect_robust(
settings.rabbitmq_url
)
async with connection:
channel = await connection.channel(
publisher_confirms=True
)
await channel.set_qos(
prefetch_count=settings.rabbitmq_prefetch
)
exchange = await channel.declare_exchange(
settings.rabbitmq_exchange,
aio_pika.ExchangeType.DIRECT,
durable=True,
)
queue = await channel.declare_queue(
settings.rabbitmq_queue,
durable=True,
)
await queue.bind(
exchange,
routing_key=settings.rabbitmq_queue,
)
# Delayed retry exchange and queue.
retry_exchange = await channel.declare_exchange(
f"{settings.rabbitmq_exchange}.retry",
aio_pika.ExchangeType.DIRECT,
durable=True,
)
retry_queue = await channel.declare_queue(
f"{settings.rabbitmq_queue}.retry",
durable=True,
arguments={
"x-dead-letter-exchange": settings.rabbitmq_exchange,
"x-dead-letter-routing-key": settings.rabbitmq_queue,
},
)
await retry_queue.bind(
retry_exchange,
routing_key=f"{settings.rabbitmq_queue}.retry",
)
# Durable quarantine queue for invalid messages.
dead_exchange = await channel.declare_exchange(
f"{settings.rabbitmq_exchange}.dead",
aio_pika.ExchangeType.DIRECT,
durable=True,
)
dead_queue = await channel.declare_queue(
f"{settings.rabbitmq_queue}.dead",
durable=True,
)
await dead_queue.bind(
dead_exchange,
routing_key=DEAD_ROUTING_KEY,
)
semaphore = asyncio.Semaphore(
settings.worker_concurrency
)
async def consumer(message):
await handle_message(
message,
retry_exchange,
dead_exchange,
semaphore,
)
logger.info(
"Worker %s consuming queue=%s concurrency=%s prefetch=%s",
WORKER_ID,
settings.rabbitmq_queue,
settings.worker_concurrency,
settings.rabbitmq_prefetch,
)
await queue.consume(consumer)
await asyncio.Future()
if __name__ == "__main__":
asyncio.run(main())
+161
View File
@@ -0,0 +1,161 @@
services:
postgres:
image: postgres:16.6-bookworm
environment:
POSTGRES_DB: ${POSTGRES_DB}
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
interval: 5s
timeout: 3s
retries: 15
restart: unless-stopped
networks:
- backend
redis:
image: redis:7.4.2-alpine
command: ["redis-server", "--appendonly", "yes", "--save", "60", "1000"]
volumes:
- redis_data:/data
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 3s
retries: 10
restart: unless-stopped
networks:
- backend
rabbitmq:
image: rabbitmq:4.0.5-management-alpine
environment:
RABBITMQ_DEFAULT_USER: ${RABBITMQ_USER}
RABBITMQ_DEFAULT_PASS: ${RABBITMQ_PASSWORD}
volumes:
- rabbitmq_data:/var/lib/rabbitmq
healthcheck:
test: ["CMD", "rabbitmq-diagnostics", "-q", "ping"]
interval: 10s
timeout: 5s
retries: 10
restart: unless-stopped
networks:
- backend
seaweedfs:
image: chrislusf/seaweedfs:4.46
command: ["mini", "-dir=/data"]
environment:
AWS_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID}
AWS_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY}
S3_BUCKET: ${S3_BUCKET}
volumes:
- seaweedfs_data:/data
ports:
- "127.0.0.1:8333:8333"
restart: unless-stopped
networks:
- backend
migrate:
build:
context: .
dockerfile: Dockerfile
env_file: .env
command: ["alembic", "upgrade", "head"]
depends_on:
postgres:
condition: service_healthy
healthcheck:
disable: true
restart: "no"
networks:
- backend
api:
build:
context: .
dockerfile: Dockerfile
env_file: .env
expose:
- "8080"
depends_on:
migrate:
condition: service_completed_successfully
seaweedfs:
condition: service_started
redis:
condition: service_healthy
restart: unless-stopped
networks:
- backend
outbox-publisher:
build:
context: .
dockerfile: Dockerfile
env_file: .env
command: ["python", "-m", "app.services.outbox_publisher"]
depends_on:
migrate:
condition: service_completed_successfully
rabbitmq:
condition: service_healthy
healthcheck:
disable: true
restart: unless-stopped
networks:
- backend
worker:
build:
context: .
dockerfile: Dockerfile
env_file: .env
cpus: 1.0
mem_limit: 1g
command: ["python", "-m", "app.worker"]
depends_on:
migrate:
condition: service_completed_successfully
seaweedfs:
condition: service_started
rabbitmq:
condition: service_healthy
healthcheck:
disable: true
restart: unless-stopped
networks:
- backend
nginx:
image: nginx:1.27.4-alpine
ports:
- "127.0.0.1:18081:80"
volumes:
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
depends_on:
api:
condition: service_healthy
healthcheck:
test: ["CMD-SHELL", "wget -q -O /dev/null http://127.0.0.1/health/live || exit 1"]
interval: 10s
timeout: 3s
retries: 10
restart: unless-stopped
networks:
- backend
volumes:
postgres_data:
redis_data:
rabbitmq_data:
seaweedfs_data:
networks:
backend:
driver: bridge
+44
View File
@@ -0,0 +1,44 @@
Uploading 20 attachments...
Accepted 1/20: ce69f0fd-c66e-4e78-b691-36d5eefc3cc4
Accepted 2/20: 1f00d79b-2a0c-43a6-94d8-cc1532f05720
Accepted 3/20: 6d63d5e7-eda5-486f-a3c5-7a261f312bef
Accepted 4/20: 5a7136d2-b247-431b-8127-78919f597a58
Accepted 5/20: ec1bdefe-9703-4e88-870d-8f4fb09bf444
Accepted 6/20: 36321d6e-3fcf-4159-bd6c-ac7e3d45e7be
Accepted 7/20: 7faddedb-11e3-42c8-857a-627efb730eb8
Accepted 8/20: e8a7b11c-d389-48d8-a29e-0db92a716ffe
Accepted 9/20: ecd33521-d324-4b25-8168-7d21d317212b
Accepted 10/20: 1fcfe9d1-3dea-42ea-80e7-ba3910ee8d33
Accepted 11/20: 6220da5e-7dfd-4ee8-a60f-81a41f099cb9
Accepted 12/20: db2f236b-ec3a-4d8d-a7a3-9cd1ecaecf41
Accepted 13/20: 05124e2a-8b32-4162-9525-cb8acdeede2c
Accepted 14/20: d034b728-28ce-4ea1-8015-d8958a242c38
Accepted 15/20: 2408e0b6-f14d-4fef-b678-97ea0c680c00
Accepted 16/20: 55ebb6cf-5592-4610-9c8c-261fbfd5629f
Accepted 17/20: 0490610f-8a47-40e3-97eb-fb16eb323f53
Accepted 18/20: 524bd16a-6691-466e-939b-edd57dbdd654
Accepted 19/20: a0a1cc11-ee6a-4cfe-bf6b-e2763b75d1fd
Accepted 20/20: 233605c5-4267-4640-bd68-861da6bbda05
Sending the 21st attachment...
PASS: 20 attachments accepted; the 21st was rejected with HTTP 429.
Accepted attachment IDs:
ce69f0fd-c66e-4e78-b691-36d5eefc3cc4
1f00d79b-2a0c-43a6-94d8-cc1532f05720
6d63d5e7-eda5-486f-a3c5-7a261f312bef
5a7136d2-b247-431b-8127-78919f597a58
ec1bdefe-9703-4e88-870d-8f4fb09bf444
36321d6e-3fcf-4159-bd6c-ac7e3d45e7be
7faddedb-11e3-42c8-857a-627efb730eb8
e8a7b11c-d389-48d8-a29e-0db92a716ffe
ecd33521-d324-4b25-8168-7d21d317212b
1fcfe9d1-3dea-42ea-80e7-ba3910ee8d33
6220da5e-7dfd-4ee8-a60f-81a41f099cb9
db2f236b-ec3a-4d8d-a7a3-9cd1ecaecf41
05124e2a-8b32-4162-9525-cb8acdeede2c
d034b728-28ce-4ea1-8015-d8958a242c38
2408e0b6-f14d-4fef-b678-97ea0c680c00
55ebb6cf-5592-4610-9c8c-261fbfd5629f
0490610f-8a47-40e3-97eb-fb16eb323f53
524bd16a-6691-466e-939b-edd57dbdd654
a0a1cc11-ee6a-4cfe-bf6b-e2763b75d1fd
233605c5-4267-4640-bd68-861da6bbda05
+1
View File
@@ -0,0 +1 @@
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
+5
View File
@@ -0,0 +1,5 @@
Test 1: normal image processing
PASS: created variants: ['crop', 'optimized', 'thumbnail']
Test 2: forced processing timeout
PASS: child process was terminated after timeout
ALL TESTS PASSED
File diff suppressed because it is too large Load Diff
+1
View File
@@ -0,0 +1 @@
PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.
+1
View File
@@ -0,0 +1 @@
PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.
+19
View File
@@ -0,0 +1,19 @@
config,size,attachment_id,time_to_ready_s,queue_lag_ms,processing_elapsed_ms,upload_attempts,throughput_rps,p95_time_to_ready_s,peak_container_memory_mib
A,small,a754c13a-d6ef-4783-8e30-9775068bfe78,4.39,1231.09,2887.88,1,0.3119,14.844,240.4
A,small,00cdbf09-9898-474d-a5cc-89dc7ba1fb1e,7.312,4081.19,2642.3,1,0.3119,14.844,240.4
A,small,a8511db6-4216-4261-8fc2-b2896573dfc9,9.984,6660.28,2534.62,1,0.3119,14.844,240.4
A,small,90f2c328-55fa-40c1-8fdf-553ba2972673,11.719,9166.32,1866.52,1,0.3119,14.844,240.4
A,small,d481b5c3-4e29-437e-9565-932574fc154f,10.531,6851.14,3308.64,1,0.3119,14.844,240.4
A,small,a9ebc53d-0ab1-47b1-afad-0658fa4a06d4,11.157,7285.03,3500.07,1,0.3119,14.844,240.4
A,medium,53408ad3-e1c9-4775-afa6-baf6127c8889,11.906,8199.29,3409.63,1,0.3119,14.844,240.4
A,medium,b4bc59a2-67d0-46ba-b6e5-ba959b27ab67,13.593,9831.98,3352.79,1,0.3119,14.844,240.4
A,medium,727ee422-ac2b-4347-aefa-27381ca01366,12.484,9752.64,2386.9,1,0.3119,14.844,240.4
A,medium,118a85de-f488-48ea-b9c4-d6e5fac7e654,11.828,8608.78,2705.93,1,0.3119,14.844,240.4
A,medium,1083095c-f7ae-4982-a4ef-fdd0c0a0696b,11.156,8023.31,2894.3,1,0.3119,14.844,240.4
A,medium,39f992e8-d086-4afc-b2b7-bffa1a1c4f67,12.094,7597.54,4028.67,1,0.3119,14.844,240.4
A,large,8481829b-b77a-4e43-8c35-103ae1cd37a3,13.516,9145.3,3711.28,1,0.3119,14.844,240.4
A,large,e22ed476-bbc8-4f68-8df5-b9cfb13d73ce,14.5,10030.24,3963.45,1,0.3119,14.844,240.4
A,large,8d37dca8-f4ae-4ac5-927e-cb938ca19fe8,14.844,11460.5,2799.74,1,0.3119,14.844,240.4
A,large,058c578a-787f-4850-bfa7-960bd613ae4a,12.765,9794.21,2397.55,1,0.3119,14.844,240.4
A,large,8875aadf-440e-41dc-96e4-3fde98f5c5a1,12.265,8464.58,3138.52,1,0.3119,14.844,240.4
A,large,25d64251-4202-445b-ad3f-70432421fa55,11.578,7882.16,3436.82,1,0.3119,14.844,240.4
1 config size attachment_id time_to_ready_s queue_lag_ms processing_elapsed_ms upload_attempts throughput_rps p95_time_to_ready_s peak_container_memory_mib
2 A small a754c13a-d6ef-4783-8e30-9775068bfe78 4.39 1231.09 2887.88 1 0.3119 14.844 240.4
3 A small 00cdbf09-9898-474d-a5cc-89dc7ba1fb1e 7.312 4081.19 2642.3 1 0.3119 14.844 240.4
4 A small a8511db6-4216-4261-8fc2-b2896573dfc9 9.984 6660.28 2534.62 1 0.3119 14.844 240.4
5 A small 90f2c328-55fa-40c1-8fdf-553ba2972673 11.719 9166.32 1866.52 1 0.3119 14.844 240.4
6 A small d481b5c3-4e29-437e-9565-932574fc154f 10.531 6851.14 3308.64 1 0.3119 14.844 240.4
7 A small a9ebc53d-0ab1-47b1-afad-0658fa4a06d4 11.157 7285.03 3500.07 1 0.3119 14.844 240.4
8 A medium 53408ad3-e1c9-4775-afa6-baf6127c8889 11.906 8199.29 3409.63 1 0.3119 14.844 240.4
9 A medium b4bc59a2-67d0-46ba-b6e5-ba959b27ab67 13.593 9831.98 3352.79 1 0.3119 14.844 240.4
10 A medium 727ee422-ac2b-4347-aefa-27381ca01366 12.484 9752.64 2386.9 1 0.3119 14.844 240.4
11 A medium 118a85de-f488-48ea-b9c4-d6e5fac7e654 11.828 8608.78 2705.93 1 0.3119 14.844 240.4
12 A medium 1083095c-f7ae-4982-a4ef-fdd0c0a0696b 11.156 8023.31 2894.3 1 0.3119 14.844 240.4
13 A medium 39f992e8-d086-4afc-b2b7-bffa1a1c4f67 12.094 7597.54 4028.67 1 0.3119 14.844 240.4
14 A large 8481829b-b77a-4e43-8c35-103ae1cd37a3 13.516 9145.3 3711.28 1 0.3119 14.844 240.4
15 A large e22ed476-bbc8-4f68-8df5-b9cfb13d73ce 14.5 10030.24 3963.45 1 0.3119 14.844 240.4
16 A large 8d37dca8-f4ae-4ac5-927e-cb938ca19fe8 14.844 11460.5 2799.74 1 0.3119 14.844 240.4
17 A large 058c578a-787f-4850-bfa7-960bd613ae4a 12.765 9794.21 2397.55 1 0.3119 14.844 240.4
18 A large 8875aadf-440e-41dc-96e4-3fde98f5c5a1 12.265 8464.58 3138.52 1 0.3119 14.844 240.4
19 A large 25d64251-4202-445b-ad3f-70432421fa55 11.578 7882.16 3436.82 1 0.3119 14.844 240.4
+19
View File
@@ -0,0 +1,19 @@
config,size,attachment_id,time_to_ready_s,queue_lag_ms,processing_elapsed_ms,upload_attempts,throughput_rps,p95_time_to_ready_s,peak_container_memory_mib
B,small,eaacd3f3-c1a8-4598-9391-b545c7e36186,6.343,319.61,5649.45,1,0.3737,11.797,326.9
B,small,52c48a19-eb6b-4b51-b591-4b377bfc19fe,6.969,360.85,5710.47,1,0.3737,11.797,326.9
B,small,79301675-dcfc-40f5-8e67-b1f09da42890,10.953,5975.58,4542.37,1,0.3737,11.797,326.9
B,small,de17b5db-17e3-4ec4-b9d2-f7202c9701d1,10.938,5969.01,4532.82,1,0.3737,11.797,326.9
B,small,0cad49c6-f462-4cb8-bad3-de84b7610610,9.891,4064.02,5748.62,1,0.3737,11.797,326.9
B,small,9d092109-3acf-40aa-bb72-f4814ed87786,10.422,4244.16,5813.27,1,0.3737,11.797,326.9
B,medium,9699f964-e70e-4318-aed4-35db6bbc6905,10.547,5303.49,4741.75,1,0.3737,11.797,326.9
B,medium,99d0562a-e3d8-419c-abb8-3da27e638485,10.532,5262.62,4746.26,1,0.3737,11.797,326.9
B,medium,98e04c6e-4715-4946-a342-af904bdb921d,8.657,4490.85,3803.61,1,0.3737,11.797,326.9
B,medium,76d46d50-e0fa-4d6f-aac2-6bb0ccd4c2d1,8.984,4346.04,4219.95,1,0.3737,11.797,326.9
B,medium,9b975eb1-225f-4104-82e5-76d319716ec0,9.11,3392.85,5261.69,1,0.3737,11.797,326.9
B,medium,582363a0-ef83-495f-96db-e368921e56b0,9.172,3803.65,5216.11,1,0.3737,11.797,326.9
B,large,cf88aa06-14b6-4263-98ed-fa839824350c,11.406,4631.93,6073.93,1,0.3737,11.797,326.9
B,large,abc42f51-1757-42c7-83ad-c74be4199bf9,11.797,4695.17,6656.51,1,0.3737,11.797,326.9
B,large,a65e0d27-181b-4fdb-bb6f-1e964e332c39,10.219,5410.6,4322.83,1,0.3737,11.797,326.9
B,large,0373095f-05d3-4d9e-9027-e11649bdf3c4,10.891,6251.35,4276.78,1,0.3737,11.797,326.9
B,large,07365ec2-4591-46eb-aef6-35ae7e418b7a,9.656,3774.33,5528.66,1,0.3737,11.797,326.9
B,large,24c799f7-442d-42c6-ba26-e5b5cc4fa76c,9.64,3774.12,5658.87,1,0.3737,11.797,326.9
1 config size attachment_id time_to_ready_s queue_lag_ms processing_elapsed_ms upload_attempts throughput_rps p95_time_to_ready_s peak_container_memory_mib
2 B small eaacd3f3-c1a8-4598-9391-b545c7e36186 6.343 319.61 5649.45 1 0.3737 11.797 326.9
3 B small 52c48a19-eb6b-4b51-b591-4b377bfc19fe 6.969 360.85 5710.47 1 0.3737 11.797 326.9
4 B small 79301675-dcfc-40f5-8e67-b1f09da42890 10.953 5975.58 4542.37 1 0.3737 11.797 326.9
5 B small de17b5db-17e3-4ec4-b9d2-f7202c9701d1 10.938 5969.01 4532.82 1 0.3737 11.797 326.9
6 B small 0cad49c6-f462-4cb8-bad3-de84b7610610 9.891 4064.02 5748.62 1 0.3737 11.797 326.9
7 B small 9d092109-3acf-40aa-bb72-f4814ed87786 10.422 4244.16 5813.27 1 0.3737 11.797 326.9
8 B medium 9699f964-e70e-4318-aed4-35db6bbc6905 10.547 5303.49 4741.75 1 0.3737 11.797 326.9
9 B medium 99d0562a-e3d8-419c-abb8-3da27e638485 10.532 5262.62 4746.26 1 0.3737 11.797 326.9
10 B medium 98e04c6e-4715-4946-a342-af904bdb921d 8.657 4490.85 3803.61 1 0.3737 11.797 326.9
11 B medium 76d46d50-e0fa-4d6f-aac2-6bb0ccd4c2d1 8.984 4346.04 4219.95 1 0.3737 11.797 326.9
12 B medium 9b975eb1-225f-4104-82e5-76d319716ec0 9.11 3392.85 5261.69 1 0.3737 11.797 326.9
13 B medium 582363a0-ef83-495f-96db-e368921e56b0 9.172 3803.65 5216.11 1 0.3737 11.797 326.9
14 B large cf88aa06-14b6-4263-98ed-fa839824350c 11.406 4631.93 6073.93 1 0.3737 11.797 326.9
15 B large abc42f51-1757-42c7-83ad-c74be4199bf9 11.797 4695.17 6656.51 1 0.3737 11.797 326.9
16 B large a65e0d27-181b-4fdb-bb6f-1e964e332c39 10.219 5410.6 4322.83 1 0.3737 11.797 326.9
17 B large 0373095f-05d3-4d9e-9027-e11649bdf3c4 10.891 6251.35 4276.78 1 0.3737 11.797 326.9
18 B large 07365ec2-4591-46eb-aef6-35ae7e418b7a 9.656 3774.33 5528.66 1 0.3737 11.797 326.9
19 B large 24c799f7-442d-42c6-ba26-e5b5cc4fa76c 9.64 3774.12 5658.87 1 0.3737 11.797 326.9
+19
View File
@@ -0,0 +1,19 @@
config,size,attachment_id,time_to_ready_s,queue_lag_ms,processing_elapsed_ms,upload_attempts,throughput_rps,p95_time_to_ready_s,peak_container_memory_mib
C,small,3db43294-7bce-4c10-a06c-979cfc05621f,15.875,316.38,14327.13,1,0.2984,16.797,495.9
C,small,729f217e-8e55-4f8d-a654-f996857d0376,15.828,279.21,14417.89,1,0.2984,16.797,495.9
C,small,74eb43f6-2970-422c-90b0-96ae84e413c8,15.828,430.62,14346.88,1,0.2984,16.797,495.9
C,small,02ee0440-932a-4c8d-879f-933cd1cb9ac9,16.797,373.54,14489.64,1,0.2984,16.797,495.9
C,small,13b07bb5-ea3b-4b5a-9ce2-45f8a61802ac,10.031,321.9,9888.86,1,0.2984,16.797,495.9
C,small,abf62445-c5a7-401f-94ff-c7d1071ef9d5,9.86,285.36,9912.19,1,0.2984,16.797,495.9
C,medium,f78e266a-847b-4859-b060-b9f1d9d4da36,12.188,147.91,12338.41,1,0.2984,16.797,495.9
C,medium,53fa8203-d82c-4d8b-97a6-96cd7dd16249,11.875,634.91,11700.91,1,0.2984,16.797,495.9
C,medium,903f91c5-05b4-49eb-b132-2fc961423f6a,12.609,934.07,10895.75,1,0.2984,16.797,495.9
C,medium,e5b9af8c-b115-42aa-a710-d2073fc00240,13.062,862.15,11410.17,1,0.2984,16.797,495.9
C,medium,fd5f2b7f-031f-4b54-917b-6b7dcf1d7920,12.875,652.8,11293.01,1,0.2984,16.797,495.9
C,medium,d44970ea-f99a-4ecb-ba2e-7fa99f7e76cf,13.078,319.57,11698.85,1,0.2984,16.797,495.9
C,large,3380f070-8f44-4227-a497-b2fd9e04c70b,11.703,497.37,10215.07,1,0.2984,16.797,495.9
C,large,5e408b89-9d30-4317-a78c-73e3da3a3b2d,13.656,832.42,11908.02,1,0.2984,16.797,495.9
C,large,9bac4a07-451b-4f47-b611-fe6882bba7ff,13.281,636.45,12313.27,1,0.2984,16.797,495.9
C,large,8ee0106d-a2e3-4036-aebc-02e7f1866fe8,13.094,97.13,12621.62,1,0.2984,16.797,495.9
C,large,afe17baf-8cca-436f-8ddf-87a4989dfbb0,7.969,350.67,7731.19,1,0.2984,16.797,495.9
C,large,d9f99004-bca4-40fe-afae-f40a6683ec21,7.203,956.68,6252.16,1,0.2984,16.797,495.9
1 config size attachment_id time_to_ready_s queue_lag_ms processing_elapsed_ms upload_attempts throughput_rps p95_time_to_ready_s peak_container_memory_mib
2 C small 3db43294-7bce-4c10-a06c-979cfc05621f 15.875 316.38 14327.13 1 0.2984 16.797 495.9
3 C small 729f217e-8e55-4f8d-a654-f996857d0376 15.828 279.21 14417.89 1 0.2984 16.797 495.9
4 C small 74eb43f6-2970-422c-90b0-96ae84e413c8 15.828 430.62 14346.88 1 0.2984 16.797 495.9
5 C small 02ee0440-932a-4c8d-879f-933cd1cb9ac9 16.797 373.54 14489.64 1 0.2984 16.797 495.9
6 C small 13b07bb5-ea3b-4b5a-9ce2-45f8a61802ac 10.031 321.9 9888.86 1 0.2984 16.797 495.9
7 C small abf62445-c5a7-401f-94ff-c7d1071ef9d5 9.86 285.36 9912.19 1 0.2984 16.797 495.9
8 C medium f78e266a-847b-4859-b060-b9f1d9d4da36 12.188 147.91 12338.41 1 0.2984 16.797 495.9
9 C medium 53fa8203-d82c-4d8b-97a6-96cd7dd16249 11.875 634.91 11700.91 1 0.2984 16.797 495.9
10 C medium 903f91c5-05b4-49eb-b132-2fc961423f6a 12.609 934.07 10895.75 1 0.2984 16.797 495.9
11 C medium e5b9af8c-b115-42aa-a710-d2073fc00240 13.062 862.15 11410.17 1 0.2984 16.797 495.9
12 C medium fd5f2b7f-031f-4b54-917b-6b7dcf1d7920 12.875 652.8 11293.01 1 0.2984 16.797 495.9
13 C medium d44970ea-f99a-4ecb-ba2e-7fa99f7e76cf 13.078 319.57 11698.85 1 0.2984 16.797 495.9
14 C large 3380f070-8f44-4227-a497-b2fd9e04c70b 11.703 497.37 10215.07 1 0.2984 16.797 495.9
15 C large 5e408b89-9d30-4317-a78c-73e3da3a3b2d 13.656 832.42 11908.02 1 0.2984 16.797 495.9
16 C large 9bac4a07-451b-4f47-b611-fe6882bba7ff 13.281 636.45 12313.27 1 0.2984 16.797 495.9
17 C large 8ee0106d-a2e3-4036-aebc-02e7f1866fe8 13.094 97.13 12621.62 1 0.2984 16.797 495.9
18 C large afe17baf-8cca-436f-8ddf-87a4989dfbb0 7.969 350.67 7731.19 1 0.2984 16.797 495.9
19 C large d9f99004-bca4-40fe-afae-f40a6683ec21 7.203 956.68 6252.16 1 0.2984 16.797 495.9
+1
View File
@@ -0,0 +1 @@
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
+30
View File
@@ -0,0 +1,30 @@
worker-1 | 2026-11-10 15:42:52,266 INFO attachment-worker Image processing completed: attachment=36321d6e-3fcf-4159-bd6c-ac7e3d45e7be attempt=1 queue_lag_ms=11716.23 processing_elapsed_ms=4906.37
worker-1 | 2026-11-10 15:42:52,277 INFO attachment-worker Message handled: job=ed8c9986-28e4-4c41-aecb-0fe2648a7a44 state=ready
worker-1 | 2026-11-10 15:42:52,286 INFO attachment-worker Claimed job=dd6120d9-e5e1-4496-b993-900cb9e546d2 attachment=e8a7b11c-d389-48d8-a29e-0db92a716ffe attempt=1 queue_lag_ms=16347.88
worker-1 | 2026-11-10 15:42:54,208 INFO attachment-worker Image processing completed: attachment=7faddedb-11e3-42c8-857a-627efb730eb8 attempt=1 queue_lag_ms=14074.69 processing_elapsed_ms=4334.03
worker-1 | 2026-11-10 15:42:54,263 INFO attachment-worker Message handled: job=75932af6-fb9e-43bd-972a-da2535953d73 state=ready
worker-1 | 2026-11-10 15:42:54,278 INFO attachment-worker Claimed job=dd834e4a-d0c3-4e14-a68f-ebc4608278c2 attachment=ecd33521-d324-4b25-8168-7d21d317212b attempt=1 queue_lag_ms=18254.34
worker-1 | 2026-11-10 15:42:57,308 INFO attachment-worker Image processing completed: attachment=e8a7b11c-d389-48d8-a29e-0db92a716ffe attempt=1 queue_lag_ms=16347.88 processing_elapsed_ms=5021.46
worker-1 | 2026-11-10 15:42:57,359 INFO attachment-worker Message handled: job=dd6120d9-e5e1-4496-b993-900cb9e546d2 state=ready
worker-1 | 2026-11-10 15:42:57,375 INFO attachment-worker Claimed job=71833773-37b2-44bd-9b50-17248160f4bd attachment=1fcfe9d1-3dea-42ea-80e7-ba3910ee8d33 attempt=1 queue_lag_ms=21213.35
worker-1 | 2026-11-10 15:42:59,382 INFO attachment-worker Image processing completed: attachment=ecd33521-d324-4b25-8168-7d21d317212b attempt=1 queue_lag_ms=18254.34 processing_elapsed_ms=5104.10
worker-1 | 2026-11-10 15:42:59,398 INFO attachment-worker Message handled: job=dd834e4a-d0c3-4e14-a68f-ebc4608278c2 state=ready
worker-1 | 2026-11-10 15:42:59,411 INFO attachment-worker Claimed job=cbc64b30-bb64-4e3c-b356-ebef99e33a95 attachment=6220da5e-7dfd-4ee8-a60f-81a41f099cb9 attempt=1 queue_lag_ms=23075.62
worker-1 | 2026-11-10 15:43:01,996 INFO attachment-worker Image processing completed: attachment=1fcfe9d1-3dea-42ea-80e7-ba3910ee8d33 attempt=1 queue_lag_ms=21213.35 processing_elapsed_ms=4620.93
worker-1 | 2026-11-10 15:43:02,016 INFO attachment-worker Message handled: job=71833773-37b2-44bd-9b50-17248160f4bd state=ready
worker-1 | 2026-11-10 15:43:02,069 INFO attachment-worker Claimed job=594cb3d0-f71f-4acc-a9cd-5e87e553279e attachment=db2f236b-ec3a-4d8d-a7a3-9cd1ecaecf41 attempt=1 queue_lag_ms=25598.65
worker-1 | 2026-11-10 15:43:04,010 INFO attachment-worker Image processing completed: attachment=6220da5e-7dfd-4ee8-a60f-81a41f099cb9 attempt=1 queue_lag_ms=23075.62 processing_elapsed_ms=4598.93
worker-1 | 2026-11-10 15:43:04,062 INFO attachment-worker Message handled: job=cbc64b30-bb64-4e3c-b356-ebef99e33a95 state=ready
worker-1 | 2026-11-10 15:43:04,072 INFO attachment-worker Claimed job=b4e6a7b3-f99c-4a9b-84bc-56de2b2325dd attachment=05124e2a-8b32-4162-9525-cb8acdeede2c attempt=1 queue_lag_ms=27478.55
worker-1 | 2026-11-10 15:43:06,359 INFO attachment-worker Image processing completed: attachment=db2f236b-ec3a-4d8d-a7a3-9cd1ecaecf41 attempt=1 queue_lag_ms=25598.65 processing_elapsed_ms=4289.04
worker-1 | 2026-11-10 15:43:06,371 INFO attachment-worker Message handled: job=594cb3d0-f71f-4acc-a9cd-5e87e553279e state=ready
worker-1 | 2026-11-10 15:43:06,383 INFO attachment-worker Claimed job=a470fd31-27cb-48d2-95ec-f1e9257b136b attachment=d034b728-28ce-4ea1-8015-d8958a242c38 attempt=1 queue_lag_ms=29643.24
worker-1 | 2026-11-10 15:43:09,074 INFO attachment-worker Image processing completed: attachment=05124e2a-8b32-4162-9525-cb8acdeede2c attempt=1 queue_lag_ms=27478.55 processing_elapsed_ms=5002.09
worker-1 | 2026-11-10 15:43:09,094 INFO attachment-worker Message handled: job=b4e6a7b3-f99c-4a9b-84bc-56de2b2325dd state=ready
worker-1 | 2026-11-10 15:43:09,110 INFO attachment-worker Claimed job=b73e112c-90be-442d-9207-fd1efd77a119 attachment=2408e0b6-f14d-4fef-b678-97ea0c680c00 attempt=1 queue_lag_ms=32187.94
worker-1 | 2026-11-10 15:43:12,107 INFO attachment-worker Image processing completed: attachment=d034b728-28ce-4ea1-8015-d8958a242c38 attempt=1 queue_lag_ms=29643.24 processing_elapsed_ms=5731.99
worker-1 | 2026-11-10 15:43:12,166 INFO attachment-worker Message handled: job=a470fd31-27cb-48d2-95ec-f1e9257b136b state=ready
worker-1 | 2026-11-10 15:43:12,190 INFO attachment-worker Claimed job=e25d63ed-bdb7-440f-a078-bd663d76a3cd attachment=55ebb6cf-5592-4610-9c8c-261fbfd5629f attempt=1 queue_lag_ms=35092.89
worker-1 | 2026-11-10 15:43:15,680 INFO attachment-worker Image processing completed: attachment=2408e0b6-f14d-4fef-b678-97ea0c680c00 attempt=1 queue_lag_ms=32187.94 processing_elapsed_ms=6578.19
worker-1 | 2026-11-10 15:43:15,711 INFO attachment-worker Message handled: job=b73e112c-90be-442d-9207-fd1efd77a119 state=ready
worker-1 | 2026-11-10 15:43:15,753 INFO attachment-worker Claimed job=e6de0002-5615-4e2d-95fc-4f679dc02d86 attachment=0490610f-8a47-40e3-97eb-fb16eb323f53 attempt=1 queue_lag_ms=38546.91
+1
View File
@@ -0,0 +1 @@
NanoCpus=1000000000 Memory=1073741824
+1
View File
@@ -0,0 +1 @@
Generic single-database configuration.
+60
View File
@@ -0,0 +1,60 @@
from logging.config import fileConfig
from alembic import context
from sqlalchemy import engine_from_config, pool
from app.core.config import settings
from app.db.base import Base
from app.db import models
config = context.config
if config.config_file_name is not None:
fileConfig(config.config_file_name)
config.set_main_option(
"sqlalchemy.url",
settings.database_url.replace("%", "%%"),
)
target_metadata = Base.metadata
def run_migrations_offline() -> None:
url = settings.database_url
context.configure(
url=url,
target_metadata=target_metadata,
literal_binds=True,
dialect_opts={"paramstyle": "named"},
compare_type=True,
)
with context.begin_transaction():
context.run_migrations()
def run_migrations_online() -> None:
connectable = engine_from_config(
config.get_section(config.config_ini_section, {}),
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
with connectable.connect() as connection:
context.configure(
connection=connection,
target_metadata=target_metadata,
compare_type=True,
)
with context.begin_transaction():
context.run_migrations()
if context.is_offline_mode():
run_migrations_offline()
else:
run_migrations_online()
+26
View File
@@ -0,0 +1,26 @@
"""${message}
Revision ID: ${up_revision}
Revises: ${down_revision | comma,n}
Create Date: ${create_date}
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
${imports if imports else ""}
# revision identifiers, used by Alembic.
revision: str = ${repr(up_revision)}
down_revision: Union[str, None] = ${repr(down_revision)}
branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)}
depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)}
def upgrade() -> None:
${upgrades if upgrades else "pass"}
def downgrade() -> None:
${downgrades if downgrades else "pass"}
@@ -0,0 +1,79 @@
"""create initial database schema
Revision ID: 21ad89656327
Revises:
Create Date: 2026-11-04 17:31:39.336368
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = '21ad89656327'
down_revision: Union[str, None] = None
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('chats',
sa.Column('id', sa.UUID(), nullable=False),
sa.Column('title', sa.String(length=100), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.PrimaryKeyConstraint('id')
)
op.create_table('users',
sa.Column('id', sa.UUID(), nullable=False),
sa.Column('username', sa.String(length=32), nullable=False),
sa.Column('display_name', sa.String(length=100), nullable=False),
sa.Column('password_hash', sa.String(length=255), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.PrimaryKeyConstraint('id')
)
op.create_index(op.f('ix_users_username'), 'users', ['username'], unique=True)
op.create_table('chat_members',
sa.Column('chat_id', sa.UUID(), nullable=False),
sa.Column('user_id', sa.UUID(), nullable=False),
sa.ForeignKeyConstraint(['chat_id'], ['chats.id'], ondelete='CASCADE'),
sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('chat_id', 'user_id')
)
op.create_table('messages',
sa.Column('id', sa.UUID(), nullable=False),
sa.Column('chat_id', sa.UUID(), nullable=False),
sa.Column('sender_id', sa.UUID(), nullable=False),
sa.Column('text', sa.Text(), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(['chat_id'], ['chats.id'], ondelete='CASCADE'),
sa.ForeignKeyConstraint(['sender_id'], ['users.id'], ),
sa.PrimaryKeyConstraint('id')
)
op.create_index('ix_messages_chat_created_id', 'messages', ['chat_id', 'created_at', 'id'], unique=False)
op.create_table('sessions',
sa.Column('id', sa.UUID(), nullable=False),
sa.Column('token_hash', sa.String(length=64), nullable=False),
sa.Column('user_id', sa.UUID(), nullable=False),
sa.Column('expires_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id')
)
op.create_index(op.f('ix_sessions_token_hash'), 'sessions', ['token_hash'], unique=True)
# ### end Alembic commands ###
def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.drop_index(op.f('ix_sessions_token_hash'), table_name='sessions')
op.drop_table('sessions')
op.drop_index('ix_messages_chat_created_id', table_name='messages')
op.drop_table('messages')
op.drop_table('chat_members')
op.drop_index(op.f('ix_users_username'), table_name='users')
op.drop_table('users')
op.drop_table('chats')
# ### end Alembic commands ###
@@ -0,0 +1,37 @@
"""add attachment metadata"""
from alembic import op
import sqlalchemy as sa
from typing import Sequence, Union
revision: str = "4d9a1f2c7b10"
down_revision: Union[str, None] = "8c3d9f4a21b7"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.add_column("messages", sa.Column("attachment_ids", sa.JSON(), nullable=False, server_default="[]"))
op.create_table(
"attachments",
sa.Column("id", sa.UUID(), nullable=False),
sa.Column("chat_id", sa.UUID(), nullable=False),
sa.Column("owner_id", sa.UUID(), nullable=False),
sa.Column("filename", sa.String(255), nullable=False),
sa.Column("content_type", sa.String(32), nullable=False),
sa.Column("size_bytes", sa.Integer(), nullable=False),
sa.Column("sha256", sa.String(64), nullable=False),
sa.Column("object_key", sa.String(512), nullable=False),
sa.Column("status", sa.String(16), nullable=False),
sa.Column("error_code", sa.String(64), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(["chat_id"], ["chats.id"], ondelete="CASCADE"),
sa.ForeignKeyConstraint(["owner_id"], ["users.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("object_key"),
)
op.create_index("ix_attachments_chat_id", "attachments", ["chat_id"])
def downgrade() -> None:
op.drop_index("ix_attachments_chat_id", table_name="attachments")
op.drop_table("attachments")
op.drop_column("messages", "attachment_ids")
@@ -0,0 +1,39 @@
"""add shared PostgreSQL idempotency records
Revision ID: 8c3d9f4a21b7
Revises: 21ad89656327
Create Date: 2026-11-05
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
revision: str = "8c3d9f4a21b7"
down_revision: Union[str, None] = "21ad89656327"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"idempotency_records",
sa.Column("id", sa.UUID(), nullable=False),
sa.Column("user_id", sa.UUID(), nullable=False),
sa.Column("method", sa.String(length=10), nullable=False),
sa.Column("path", sa.String(length=255), nullable=False),
sa.Column("idempotency_key", sa.String(length=128), nullable=False),
sa.Column("request_hash", sa.String(length=64), nullable=False),
sa.Column("message_id", sa.UUID(), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(["message_id"], ["messages.id"], ondelete="CASCADE"),
sa.ForeignKeyConstraint(["user_id"], ["users.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("user_id", "method", "path", "idempotency_key", name="uq_idempotency_scope_key"),
)
op.create_index("ix_idempotency_records_expires_at", "idempotency_records", ["expires_at"], unique=False)
def downgrade() -> None:
op.drop_index("ix_idempotency_records_expires_at", table_name="idempotency_records")
op.drop_table("idempotency_records")
@@ -0,0 +1,78 @@
"""add attachment uploads
Revision ID: a7c2e91d5f40
Revises: 4d9a1f2c7b10
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
revision: str = "a7c2e91d5f40"
down_revision: Union[str, None] = "4d9a1f2c7b10"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"attachment_uploads",
sa.Column("id", sa.UUID(), nullable=False),
sa.Column("chat_id", sa.UUID(), nullable=False),
sa.Column("owner_id", sa.UUID(), nullable=False),
sa.Column("filename", sa.String(length=255), nullable=False),
sa.Column("content_type", sa.String(length=32), nullable=False),
sa.Column("expected_size_bytes", sa.Integer(), nullable=False),
sa.Column("expected_sha256", sa.String(length=64), nullable=False),
sa.Column("object_key", sa.String(length=512), nullable=False),
sa.Column("status", sa.String(length=16), nullable=False),
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("completed_at", sa.DateTime(timezone=True), nullable=True),
sa.ForeignKeyConstraint(
["chat_id"],
["chats.id"],
ondelete="CASCADE",
),
sa.ForeignKeyConstraint(
["owner_id"],
["users.id"],
ondelete="CASCADE",
),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("object_key"),
)
op.create_index(
"ix_attachment_uploads_chat_id",
"attachment_uploads",
["chat_id"],
)
op.create_index(
"ix_attachment_uploads_owner_id",
"attachment_uploads",
["owner_id"],
)
op.create_index(
"ix_attachment_uploads_expires_at",
"attachment_uploads",
["expires_at"],
)
def downgrade() -> None:
op.drop_index(
"ix_attachment_uploads_expires_at",
table_name="attachment_uploads",
)
op.drop_index(
"ix_attachment_uploads_owner_id",
table_name="attachment_uploads",
)
op.drop_index(
"ix_attachment_uploads_chat_id",
table_name="attachment_uploads",
)
op.drop_table("attachment_uploads")
@@ -0,0 +1,210 @@
"""add durable image processing jobs"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
revision: str = "a9d670bf1c16"
down_revision: Union[str, None] = "a7c2e91d5f40"
branch_labels = None
depends_on = None
def upgrade() -> None:
# Дополнительные поля для хранения исходника и версии конвейера.
op.add_column(
"attachments",
sa.Column("staging_key", sa.String(length=512), nullable=True),
)
op.create_unique_constraint(
"uq_attachments_staging_key",
"attachments",
["staging_key"],
)
op.add_column(
"attachments",
sa.Column(
"pipeline_version",
sa.Integer(),
nullable=False,
server_default="1",
),
)
# Задания на асинхронную обработку.
op.create_table(
"processing_jobs",
sa.Column("id", sa.UUID(), nullable=False),
sa.Column("attachment_id", sa.UUID(), nullable=False),
sa.Column(
"pipeline_version",
sa.Integer(),
nullable=False,
server_default="1",
),
sa.Column(
"attempt",
sa.Integer(),
nullable=False,
server_default="1",
),
sa.Column(
"status",
sa.String(length=16),
nullable=False,
server_default="queued",
),
sa.Column(
"available_at",
sa.DateTime(timezone=True),
nullable=False,
),
sa.Column("lease_until", sa.DateTime(timezone=True), nullable=True),
sa.Column("worker_id", sa.String(length=128), nullable=True),
sa.Column(
"last_error_code",
sa.String(length=64),
nullable=True,
),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
nullable=False,
),
sa.Column(
"updated_at",
sa.DateTime(timezone=True),
nullable=False,
),
sa.ForeignKeyConstraint(
["attachment_id"],
["attachments.id"],
ondelete="CASCADE",
),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint(
"attachment_id",
"pipeline_version",
name="uq_processing_job_attachment_pipeline",
),
)
op.create_index(
"ix_processing_jobs_status_available",
"processing_jobs",
["status", "available_at"],
)
op.create_index(
"ix_processing_jobs_lease_until",
"processing_jobs",
["lease_until"],
)
# Созданные варианты изображений.
op.create_table(
"attachment_variants",
sa.Column("id", sa.UUID(), nullable=False),
sa.Column("attachment_id", sa.UUID(), nullable=False),
sa.Column("variant", sa.String(length=32), nullable=False),
sa.Column("object_key", sa.String(length=512), nullable=False),
sa.Column("content_type", sa.String(length=32), nullable=False),
sa.Column("width", sa.Integer(), nullable=False),
sa.Column("height", sa.Integer(), nullable=False),
sa.Column("size_bytes", sa.Integer(), nullable=False),
sa.Column("sha256", sa.String(length=64), nullable=False),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
nullable=False,
),
sa.ForeignKeyConstraint(
["attachment_id"],
["attachments.id"],
ondelete="CASCADE",
),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("object_key"),
sa.UniqueConstraint(
"attachment_id",
"variant",
name="uq_attachment_variant",
),
)
op.create_index(
"ix_attachment_variants_attachment_id",
"attachment_variants",
["attachment_id"],
)
# Transactional outbox: события публикуются в брокер
# отдельным процессом после фиксации транзакции.
op.create_table(
"outbox_events",
sa.Column("id", sa.UUID(), nullable=False),
sa.Column("event_type", sa.String(length=128), nullable=False),
sa.Column("aggregate_id", sa.UUID(), nullable=False),
sa.Column("payload", sa.JSON(), nullable=False),
sa.Column(
"status",
sa.String(length=16),
nullable=False,
server_default="pending",
),
sa.Column(
"attempts",
sa.Integer(),
nullable=False,
server_default="0",
),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
nullable=False,
),
sa.Column(
"published_at",
sa.DateTime(timezone=True),
nullable=True,
),
sa.Column("last_error", sa.String(length=256), nullable=True),
sa.PrimaryKeyConstraint("id"),
)
op.create_index(
"ix_outbox_events_status_created",
"outbox_events",
["status", "created_at"],
)
def downgrade() -> None:
op.drop_index(
"ix_outbox_events_status_created",
table_name="outbox_events",
)
op.drop_table("outbox_events")
op.drop_index(
"ix_attachment_variants_attachment_id",
table_name="attachment_variants",
)
op.drop_table("attachment_variants")
op.drop_index(
"ix_processing_jobs_lease_until",
table_name="processing_jobs",
)
op.drop_index(
"ix_processing_jobs_status_available",
table_name="processing_jobs",
)
op.drop_table("processing_jobs")
op.drop_column("attachments", "pipeline_version")
op.drop_constraint(
"uq_attachments_staging_key",
"attachments",
type_="unique",
)
op.drop_column("attachments", "staging_key")
+37
View File
@@ -0,0 +1,37 @@
worker_processes auto;
events {
worker_connections 1024;
}
http {
resolver 127.0.0.11 valid=10s ipv6=off;
upstream api_backend {
zone api_backend 64k;
server api:8080 resolve;
}
server {
listen 80;
server_name _;
client_max_body_size 10m;
location / {
proxy_pass http://api_backend;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 5s;
proxy_send_timeout 120s;
proxy_read_timeout 120s;
proxy_next_upstream error timeout http_502 http_503 http_504;
}
}
}
+12
View File
@@ -0,0 +1,12 @@
fastapi==0.115.6
uvicorn[standard]==0.34.0
pydantic==2.10.4
pydantic-settings==2.7.1
SQLAlchemy==2.0.36
psycopg[binary]==3.2.3
alembic==1.14.0
redis==5.2.1
pwdlib[argon2]==0.2.1
boto3==1.35.99
Pillow==11.0.0
aio-pika==9.5.5
+8 -4
View File
@@ -8,8 +8,12 @@ import re
ROOT = Path(__file__).resolve().parents[1]
def main():
lab = json.loads((ROOT/'lab.json').read_text())['number']
spec = json.loads((ROOT/'contracts/openapi.json').read_text())
lab = json.loads(
(ROOT / 'lab.json').read_text(encoding='utf-8')
)['number']
spec = json.loads(
(ROOT / 'contracts/openapi.json').read_text(encoding='utf-8')
)
assert spec['openapi']=='3.1.0'
assert spec['info']['version']=='1.%d.0'%lab
operation_ids = set()
@@ -41,11 +45,11 @@ def main():
assert ('/api/v1/auth/sessions' in spec['paths']) == (lab>=2)
assert ('/api/v1/attachments/{attachment_id}' in spec['paths']) == (lab>=4)
for file in list((ROOT/'tests').glob('*.py'))+list((ROOT/'scripts').glob('*.py')):
ast.parse(file.read_text(),filename=str(file))
ast.parse(file.read_text(encoding='utf-8'), filename=str(file))
for file in ROOT.rglob('*.json'):
if any(part in {'.git','.venv','node_modules','vendor','output'} for part in file.parts):
continue
json.loads(file.read_text())
json.loads(file.read_text(encoding='utf-8'))
for required in ('README.md','COURSE.md','CONTRIBUTING.md','REPORT.md','contracts/README.md','tests/smoke.py','tests/README.md'):
assert (ROOT/required).is_file(), 'missing '+required
print('PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.')
+158
View File
@@ -0,0 +1,158 @@
"""Safely replay a failed attachment processing job.
Usage:
python scripts/replay_failed_job.py ATTACHMENT_UUID
"""
import argparse
import logging
import sys
import uuid
from datetime import datetime, timezone
from pathlib import Path
# Add project root to Python import path.
PROJECT_ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(PROJECT_ROOT))
from sqlalchemy import select
from app.db.models import Attachment, OutboxEvent, ProcessingJob
from app.db.session import SessionLocal
logger = logging.getLogger("replay_failed_job")
EVENT_TYPE = "attachment.process.v1"
def utc_now() -> datetime:
return datetime.now(timezone.utc)
def replay_failed_job(attachment_id: uuid.UUID) -> None:
db = SessionLocal()
try:
# Lock the attachment first to serialize concurrent replay requests.
attachment = db.scalar(
select(Attachment)
.where(Attachment.id == attachment_id)
.with_for_update()
)
if attachment is None:
raise ValueError(f"Attachment not found: {attachment_id}")
job = db.scalar(
select(ProcessingJob)
.where(
ProcessingJob.attachment_id == attachment_id,
ProcessingJob.pipeline_version == attachment.pipeline_version,
)
.with_for_update()
)
if job is None:
raise ValueError(
f"Processing job not found for attachment: {attachment_id}"
)
# Only terminally failed jobs may be replayed.
if job.status != "failed":
raise ValueError(
f"Job cannot be replayed: current job status is '{job.status}'"
)
if attachment.status != "failed":
raise ValueError(
"Attachment and job states are inconsistent: "
f"attachment status is '{attachment.status}'"
)
# The worker needs the original staged file.
if not attachment.staging_key:
raise ValueError(
"Cannot replay: staging_key is empty. "
"The original file is not available for processing."
)
now = utc_now()
# Reset the existing job. Keep its ID and pipeline version.
job.attempt = 0
job.status = "queued"
job.available_at = now
job.lease_until = None
job.worker_id = None
job.last_error_code = None
# Return attachment metadata to the queued state.
attachment.status = "queued"
attachment.error_code = None
# Create a new durable outbox event in the same transaction.
event = OutboxEvent(
event_type=EVENT_TYPE,
aggregate_id=attachment.id,
payload={
"schema_version": 1,
"job_id": str(job.id),
"type": EVENT_TYPE,
"attachment_id": str(attachment.id),
"pipeline_version": job.pipeline_version,
"attempt": 1,
"created_at": now.isoformat(),
},
status="pending",
)
db.add(event)
db.commit()
print("Replay request successfully committed.")
print(f"Attachment ID: {attachment.id}")
print(f"Job ID: {job.id}")
print(f"Job status: {job.status}")
print(f"Attempt: {job.attempt}")
print(f"Outbox event: {event.id}")
print("Outbox status: pending")
print()
print("The existing outbox-publisher will publish the event.")
except Exception:
db.rollback()
raise
finally:
db.close()
def main() -> int:
parser = argparse.ArgumentParser(
description="Replay a terminally failed attachment processing job."
)
parser.add_argument(
"attachment_id",
help="UUID of the attachment to replay",
)
args = parser.parse_args()
try:
attachment_id = uuid.UUID(args.attachment_id)
replay_failed_job(attachment_id)
return 0
except ValueError as exc:
print(f"ERROR: {exc}", file=sys.stderr)
return 1
except Exception:
logger.exception("Could not replay failed processing job")
return 1
if __name__ == "__main__":
logging.basicConfig(level=logging.INFO)
raise SystemExit(main())
+7 -3
View File
@@ -6,17 +6,21 @@ from openapi_spec_validator import validate
from jsonschema import Draft202012Validator
root = Path(__file__).resolve().parents[1]
spec = json.loads((root/'contracts/openapi.json').read_text())
spec = json.loads(
(root / 'contracts/openapi.json').read_text(encoding='utf-8')
)
validate(spec)
from referencing import Registry, Resource
# The base URI resolves local #/components references inside examples.
resource = dict(spec, **{'$schema': 'https://json-schema.org/draft/2020-12/schema'})
registry = Registry().with_resource('urn:mtusi:openapi', Resource.from_contents(resource))
for name, example in json.loads((root/'contracts/examples.json').read_text()).items():
for name, example in json.loads(
(root / 'contracts/examples.json').read_text(encoding='utf-8')
).items():
schema = {'$ref': 'urn:mtusi:openapi#/components/schemas/' + name}
Draft202012Validator(schema, registry=registry).validate(example)
for path in (root/'contracts').glob('*.schema.json'):
schema = json.loads(path.read_text())
schema = json.loads(path.read_text(encoding='utf-8'))
Draft202012Validator.check_schema(schema)
for example in schema.get('examples',[]):
Draft202012Validator(schema).validate(example)
BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 1.6 MiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 543 KiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 89 KiB

+55
View File
@@ -0,0 +1,55 @@
"""Generate deterministic image fixtures for Track 1 benchmarks."""
import random
from pathlib import Path
from PIL import Image
ROOT = Path(__file__).resolve().parent / "fixtures"
SIZES = {
"small": (640, 480),
"medium": (1600, 1200),
"large": (3000, 2000),
}
def generate_image(name: str, width: int, height: int) -> None:
rng = random.Random(42)
image = Image.new("RGB", (width, height))
pixels = image.load()
for y in range(height):
for x in range(width):
noise = rng.randrange(0, 24)
pixels[x, y] = (
(x * 255 // width + noise) % 256,
(y * 255 // height + noise) % 256,
((x + y) * 255 // (width + height) + noise) % 256,
)
path = ROOT / f"{name}.jpg"
image.save(
path,
format="JPEG",
quality=85,
optimize=True,
)
print(
f"{name}: {width}x{height}, "
f"{path.stat().st_size} bytes"
)
def main() -> None:
ROOT.mkdir(parents=True, exist_ok=True)
for name, (width, height) in SIZES.items():
generate_image(name, width, height)
if __name__ == "__main__":
main()
+69
View File
@@ -0,0 +1,69 @@
#!/usr/bin/env python3
"""Verify the API admission limit while the worker is stopped."""
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / "tests"))
from client import Client, arguments, check, run
def main():
args = arguments("Backpressure admission-limit test")
client = Client(args.base_url, args.ca_file)
alice, bob, eve, chat = client.bootstrap()
content = (ROOT / "tests" / "fixtures" / "sample.png").read_bytes()
upload_path = (
"/api/v1/chats/" + chat["id"]
+ "/attachments?filename=backpressure.png"
)
headers = {"Content-Type": "image/png"}
accepted = []
print("Uploading 20 attachments...")
for number in range(1, 21):
result = client.call(
"POST",
upload_path,
expected=202,
actor=alice,
raw=content,
headers=headers,
schema="Attachment",
)
attachment = result["json"]
check(
attachment["status"] == "queued",
"accepted attachment is not queued",
)
accepted.append(attachment["id"])
print(f"Accepted {number}/20: {attachment['id']}")
print("Sending the 21st attachment...")
rejected = client.call(
"POST",
upload_path,
expected=429,
actor=alice,
raw=content,
headers=headers,
)
check(rejected["status"] == 429, "expected HTTP 429")
print("PASS: 20 attachments accepted; the 21st was rejected with HTTP 429.")
print("Accepted attachment IDs:")
for attachment_id in accepted:
print(attachment_id)
if __name__ == "__main__":
run(main)
+52
View File
@@ -0,0 +1,52 @@
import io
from PIL import Image
from app.services.image_processor import (
ImageProcessingTimeout,
create_variants_with_timeout,
)
def make_test_image() -> bytes:
output = io.BytesIO()
image = Image.new("RGB", (640, 400), (120, 160, 200))
image.save(output, format="PNG")
return output.getvalue()
def main() -> None:
image_data = make_test_image()
print("Test 1: normal image processing")
variants = create_variants_with_timeout(
image_data,
timeout_seconds=30,
)
names = sorted(variant.name for variant in variants)
assert names == ["crop", "optimized", "thumbnail"], names
print(f"PASS: created variants: {names}")
print("Test 2: forced processing timeout")
try:
create_variants_with_timeout(
image_data,
timeout_seconds=0.001,
)
except ImageProcessingTimeout:
print("PASS: child process was terminated after timeout")
else:
raise AssertionError("Expected ImageProcessingTimeout")
print("ALL TESTS PASSED")
if __name__ == "__main__":
main()
+420
View File
@@ -0,0 +1,420 @@
"""Track 1 benchmark: image processing throughput, latency and memory."""
import csv
from concurrent.futures import ThreadPoolExecutor, as_completed
import datetime as dt
import json
import re
import subprocess
import threading
import time
from pathlib import Path
from tests.client import Client, Failure, arguments, check, run
ROOT = Path(__file__).resolve().parents[1]
FIXTURES = ROOT / "tests" / "fixtures"
IMAGE_SIZES = ("small", "medium", "large")
COMPLETION_RE = re.compile(
r"Image processing completed: attachment=(\S+) "
r"attempt=(\d+) queue_lag_ms=([\d.]+) "
r"processing_elapsed_ms=([\d.]+)"
)
def percentile(values, percentile_value):
"""Nearest-rank percentile."""
if not values:
return None
ordered = sorted(values)
index = max(
0,
min(
len(ordered) - 1,
int((percentile_value / 100) * len(ordered) + 0.999999) - 1,
),
)
return ordered[index]
def memory_to_mib(value):
match = re.match(r"([\d.]+)\s*(B|KiB|MiB|GiB|KB|MB|GB)", value)
if not match:
return None
number = float(match.group(1))
unit = match.group(2)
factors = {
"B": 1 / (1024 * 1024),
"KiB": 1 / 1024,
"MiB": 1,
"GiB": 1024,
"KB": 1000 / (1024 * 1024),
"MB": 1000 * 1000 / (1024 * 1024),
"GB": 1000 * 1000 * 1000 / (1024 * 1024),
}
return number * factors[unit]
class MemorySampler:
"""Samples aggregate worker container memory using docker stats."""
def __init__(self, container_name):
self.container_name = container_name
self.peak_mib = 0.0
self.error = None
self.stop_event = threading.Event()
self.thread = threading.Thread(target=self._sample, daemon=True)
def start(self):
self.thread.start()
def _sample(self):
while not self.stop_event.is_set():
try:
result = subprocess.run(
[
"docker",
"stats",
"--no-stream",
"--format",
"{{.MemUsage}}",
self.container_name,
],
capture_output=True,
text=True,
timeout=10,
check=True,
)
usage = result.stdout.strip().split("/")[0].strip()
value = memory_to_mib(usage)
if value is not None:
self.peak_mib = max(self.peak_mib, value)
except Exception as exc:
self.error = type(exc).__name__
self.stop_event.wait(0.5)
def stop(self):
self.stop_event.set()
self.thread.join(timeout=12)
def read_worker_metrics(since):
"""Read queue lag and processing duration from worker logs."""
result = subprocess.run(
[
"docker",
"compose",
"--env-file",
".env",
"logs",
"--since",
since,
"worker",
],
capture_output=True,
text=True,
timeout=30,
)
if result.returncode != 0:
raise Failure("could not read worker logs")
metrics = {}
for line in result.stdout.splitlines():
match = COMPLETION_RE.search(line)
if not match:
continue
attachment_id, attempt, queue_lag, processing_elapsed = (
match.groups()
)
metrics[attachment_id] = {
"attempt": int(attempt),
"queue_lag_ms": float(queue_lag),
"processing_elapsed_ms": float(processing_elapsed),
}
return metrics
def upload_and_wait(
size_name,
image_bytes,
actor,
chat,
base_url,
ca_file,
wait_seconds,
):
client = Client(base_url=base_url, ca_file=ca_file)
filename = f"{size_name}.jpg"
path = (
f"/api/v1/chats/{chat['id']}/attachments"
f"?filename={filename}"
)
started = time.monotonic()
upload_attempts = 0
while True:
upload_attempts += 1
response = client.call(
"POST",
path,
expected=[202, 429],
actor=actor,
raw=image_bytes,
headers={"Content-Type": "image/jpeg"},
schema="Attachment",
)
if response["status"] == 202:
attachment = response["json"]
break
retry_after = response["headers"].get("Retry-After", "1")
try:
delay = max(1.0, float(retry_after))
except ValueError:
delay = 1.0
time.sleep(min(delay, 10.0))
check(
attachment["status"] == "queued",
"upload did not return queued attachment",
)
ready = client.wait_ready(
actor,
attachment,
wait_seconds=wait_seconds,
)
elapsed = time.monotonic() - started
return {
"size": size_name,
"attachment_id": attachment["id"],
"time_to_ready_s": elapsed,
"upload_attempts": upload_attempts,
"final_status": ready["status"],
}
def main():
def configure(parser):
parser.add_argument("--config", required=True, choices=["A", "B", "C"])
parser.add_argument("--per-size", type=int, default=6)
parser.add_argument("--upload-concurrency", type=int, default=4)
parser.add_argument("--wait-seconds", type=int, default=180)
parser.add_argument(
"--worker-container",
default="mtusi2026lab5-worker-1",
)
parser.add_argument("--output", default=None)
args = arguments(
"Track 1 image processing benchmark",
configure,
)
check(args.per_size >= 1, "--per-size must be positive")
check(
args.upload_concurrency >= 1,
"--upload-concurrency must be positive",
)
images = {
name: (FIXTURES / f"{name}.jpg").read_bytes()
for name in IMAGE_SIZES
}
for name, content in images.items():
check(bool(content), f"{name}.jpg is empty")
bootstrap_client = Client(
base_url=args.base_url,
ca_file=args.ca_file,
)
alice, _, _, chat = bootstrap_client.bootstrap()
tasks = [
(name, images[name])
for name in IMAGE_SIZES
for _ in range(args.per_size)
]
log_since = (
dt.datetime.now(dt.timezone.utc)
- dt.timedelta(seconds=2)
).isoformat()
sampler = MemorySampler(args.worker_container)
sampler.start()
benchmark_started = time.monotonic()
results = []
try:
with ThreadPoolExecutor(
max_workers=args.upload_concurrency
) as executor:
futures = [
executor.submit(
upload_and_wait,
name,
content,
alice,
chat,
args.base_url,
args.ca_file,
args.wait_seconds,
)
for name, content in tasks
]
for future in as_completed(futures):
results.append(future.result())
finally:
benchmark_elapsed = time.monotonic() - benchmark_started
sampler.stop()
worker_metrics = read_worker_metrics(log_since)
for result in results:
result.update(
worker_metrics.get(
result["attachment_id"],
{},
)
)
completed = [
item for item in results
if item["final_status"] == "ready"
]
check(
len(completed) == len(tasks),
"not all benchmark images reached ready",
)
ready_times = [
item["time_to_ready_s"] for item in completed
]
throughput = len(completed) / benchmark_elapsed
p95_ready = percentile(ready_times, 95)
queue_lags = [
item["queue_lag_ms"]
for item in completed
if "queue_lag_ms" in item
]
processing_times = [
item["processing_elapsed_ms"]
for item in completed
if "processing_elapsed_ms" in item
]
output = args.output or f"evidence/track1_{args.config}.csv"
output_path = ROOT / output
output_path.parent.mkdir(parents=True, exist_ok=True)
fields = [
"config",
"size",
"attachment_id",
"time_to_ready_s",
"queue_lag_ms",
"processing_elapsed_ms",
"upload_attempts",
"throughput_rps",
"p95_time_to_ready_s",
"peak_container_memory_mib",
]
with output_path.open(
"w",
newline="",
encoding="utf-8-sig",
) as file:
writer = csv.DictWriter(file, fieldnames=fields)
writer.writeheader()
for item in results:
writer.writerow(
{
"config": args.config,
"size": item["size"],
"attachment_id": item["attachment_id"],
"time_to_ready_s": round(
item["time_to_ready_s"], 4
),
"queue_lag_ms": item.get("queue_lag_ms", ""),
"processing_elapsed_ms": item.get(
"processing_elapsed_ms", ""
),
"upload_attempts": item["upload_attempts"],
"throughput_rps": round(throughput, 4),
"p95_time_to_ready_s": round(p95_ready, 4),
"peak_container_memory_mib": round(
sampler.peak_mib, 2
),
}
)
print()
print(f"Configuration: {args.config}")
print(f"Completed: {len(completed)}/{len(tasks)}")
print(f"Duration: {benchmark_elapsed:.2f} s")
print(f"Throughput: {throughput:.3f} images/s")
print(f"p95 time-to-ready: {p95_ready:.3f} s")
print(
"Average queue lag: "
+ (
f"{sum(queue_lags) / len(queue_lags):.2f} ms"
if queue_lags else "not available"
)
)
print(
"Average processing time: "
+ (
f"{sum(processing_times) / len(processing_times):.2f} ms"
if processing_times else "not available"
)
)
print(
f"Peak worker container memory: {sampler.peak_mib:.2f} MiB"
)
if sampler.error:
print(f"Memory sampler warning: {sampler.error}")
missing_metrics = len(completed) - len(queue_lags)
if missing_metrics:
print(f"WARNING: metrics missing for {missing_metrics} jobs")
print(f"CSV: {output_path}")
if __name__ == "__main__":
run(main)