ЛР 5 - БВТ2304 - Травкин - на 5 #57
@@ -30,3 +30,5 @@ coverage/
|
||||
.vscode/
|
||||
# Keep small, sanitized evidence summaries in evidence/ under version control.
|
||||
# Inspect git diff --cached: ignore rules are not a secret scanner.
|
||||
|
||||
infra/rabbitmq/rabbitmq.conf
|
||||
|
||||
+27
@@ -0,0 +1,27 @@
|
||||
FROM python:3.13.15-slim
|
||||
|
||||
ENV PYTHONDONTWRITEBYTECODE=1
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY requirements.txt .
|
||||
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
COPY alembic.ini .
|
||||
COPY migrations ./migrations
|
||||
COPY src ./src
|
||||
COPY scripts ./scripts
|
||||
COPY tests ./tests
|
||||
|
||||
RUN useradd --create-home --uid 10001 appuser
|
||||
|
||||
USER appuser
|
||||
|
||||
EXPOSE 8080
|
||||
|
||||
HEALTHCHECK --interval=10s --timeout=3s --start-period=5s --retries=3 \
|
||||
CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=2)" || exit 1
|
||||
|
||||
CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8080"]
|
||||
@@ -1,37 +1,454 @@
|
||||
# Отчёт — лабораторная 5
|
||||
# Отчёт — лабораторная работа №5
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
- **Автор, группа:** Станислав Травкин, БВТ2304
|
||||
- **Целевая оценка:** 5
|
||||
- **Выбранный трек оценки 5:** Трек 2 — Доказанная идемпотентность pipeline
|
||||
- **Ветка:** `solution/bvt2304-travkin`
|
||||
- **Commit/PR:** заполняется после публикации
|
||||
- **Технологии:** Python 3.13, FastAPI, PostgreSQL, Redis, RabbitMQ, MinIO (S3), Docker Compose, Nginx, Pillow, SQLAlchemy, Alembic
|
||||
- **Источники и помощь ИИ:** методические материалы лабораторной работы, документация используемых технологий; ChatGPT применялся для обсуждения архитектуры, написания и проверки кода, сценариев отказов и подготовки отчёта. Фактические результаты тестов получены при выполнении команд в локальном окружении.
|
||||
|
||||
## Запуск из чистого clone
|
||||
## 1. Запуск из чистого clone
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
### 1.1. Требования
|
||||
|
||||
## Архитектура и решения
|
||||
Для запуска необходимы Docker Engine, Docker Compose, Bash, Python 3.13 и доступ к реестрам контейнерных образов. Проверки выполнялись в Ubuntu 22.04 под WSL. PostgreSQL, Redis, RabbitMQ и MinIO локально устанавливать не нужно: они запускаются в контейнерах.
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab4, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
```bash
|
||||
cd mtusi2026lab5
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
bash scripts/bootstrap.sh
|
||||
docker compose config --quiet
|
||||
docker compose up -d --build
|
||||
docker compose ps
|
||||
```
|
||||
|
||||
## Доказательства критериев
|
||||
`bootstrap.sh` создаёт `.env`, каталог `secrets` и необходимые файлы секретов, включая `rabbitmq_password`, если они ещё не существуют. Повторный запуск не должен менять уже созданные секреты. Контейнер `migrate` применяет миграции Alembic, а инициализация S3 подготавливает хранилище. Данные PostgreSQL, Redis, RabbitMQ и MinIO размещаются в именованных томах Docker Compose.
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
Для проверки:
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
```bash
|
||||
make check
|
||||
make validate
|
||||
make test
|
||||
```
|
||||
|
||||
## Эксперименты и отказы
|
||||
`make validate` требует установленных зависимостей проверки OpenAPI. `make check` и `make validate` проверяют структуру и контракт, но не заменяют интеграционный тест работающего приложения.
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
Результаты последней проверки:
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
```text
|
||||
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
|
||||
PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.
|
||||
```
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
Синтаксис `scripts/bootstrap.sh` проверен через `bash -n`, конфигурация — через `docker compose config --quiet`. На существующем стенде оба действия завершились без ошибок. Полный запуск на отдельном чистом хосте с пустыми томами не проводился; поэтому воспроизводимость чистого развёртывания подтверждена проверкой конфигурации, но не независимым испытанием.
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
## 2. Архитектура и решения
|
||||
|
||||
### 2.1. Компоненты
|
||||
|
||||
| Компонент | Назначение |
|
||||
|---|---|
|
||||
| Nginx | Входная HTTP-точка, распределение запросов между API-репликами |
|
||||
| `api-1`, `api-2` (FastAPI) | Приём запросов, работа с пользователями, чатами, сообщениями и вложениями |
|
||||
| PostgreSQL | Основные данные, состояния вложений, долговременные задания обработки |
|
||||
| Redis | Кеш и вспомогательное хранилище |
|
||||
| RabbitMQ | Доставка асинхронных заданий с возможными повторами |
|
||||
| Dispatcher | Восстановимая публикация заданий из PostgreSQL в RabbitMQ |
|
||||
| Worker | Проверка оригинала, создание изображений, запись результатов |
|
||||
| MinIO | S3-совместимое хранилище оригиналов и вариантов |
|
||||
|
||||
API и worker выполняются отдельно. HTTP-запрос не ожидает создания производных изображений.
|
||||
|
||||
### 2.2. Путь задания
|
||||
|
||||
1. Клиент загружает PNG или JPEG через API.
|
||||
2. API ограничивает размер и проверяет изображение; исходный объект записывается в `staging/` MinIO.
|
||||
3. В одной транзакции PostgreSQL создаются запись вложения и задание обработки.
|
||||
4. После успешного сохранения оригинала и COMMIT клиент получает HTTP 202.
|
||||
5. Dispatcher находит неопубликованное задание, отправляет сообщение в RabbitMQ с publisher confirms и отмечает публикацию.
|
||||
6. Worker принимает сообщение, проверяет актуальность задания и переводит его в `processing`.
|
||||
7. Worker читает оригинал, создаёт thumbnail, crop и optimized, сохраняет объекты в MinIO.
|
||||
8. Перед COMMIT результата worker повторно проверяет `generation`, `attempt` и `execution_id`.
|
||||
9. Актуальное выполнение фиксирует `ready`; сообщение подтверждается RabbitMQ после обработки.
|
||||
10. При временном сбое выполняется повторная попытка, при окончательном отказе — `failed` и публикация в DLQ.
|
||||
|
||||
Основные состояния: `queued`, `processing`, `ready`, `failed`. HTTP 202 означает принятие задания, а не завершение обработки. До готовности производные объекты не должны выдаваться как доступные.
|
||||
|
||||
### 2.3. Версионирование объектов
|
||||
|
||||
Для текущего pipeline используются ключи:
|
||||
|
||||
```text
|
||||
attachments/{chat_id}/{attachment_id}/g{generation}/original
|
||||
variants/{chat_id}/{attachment_id}/g{generation}/thumbnail.jpg
|
||||
variants/{chat_id}/{attachment_id}/g{generation}/crop.jpg
|
||||
variants/{chat_id}/{attachment_id}/g{generation}/optimized.webp
|
||||
```
|
||||
|
||||
Разные поколения не используют один и тот же ключ результата. Устаревшие сообщения не могут легитимно заменить запись актуального поколения в PostgreSQL. В базе могут сохраняться старые вложения с ключами до введения `g{generation}`; это не означает, что они обработаны новым pipeline.
|
||||
|
||||
## 3. Критерии на оценку 3
|
||||
|
||||
### 3.1. Асинхронный приём и сохранность
|
||||
|
||||
Реализация: `src/main.py`, `src/dispatcher.py`, `src/worker.py`.
|
||||
|
||||
При загрузке API записывает исходный файл в MinIO и создаёт задание в PostgreSQL. Только после успешной фиксации возвращается HTTP 202. Даже при остановленном worker задание остаётся в долговременном хранилище. Сообщение с вложением можно создать до готовности производных изображений; это не блокирует остальной API.
|
||||
|
||||
### 3.2. Отдельная очередь и подтверждение обработки
|
||||
|
||||
RabbitMQ использует очередь `attachment.process.v1`. Сообщения доставляются по модели **at-least-once**: после сбоя возможна повторная доставка. Worker подтверждает сообщение после выполнения обработки, а не сразу после получения. Поэтому обработчик обязан проверять состояние и избегать повторного изменения готового результата.
|
||||
|
||||
### 3.3. Thumbnail, статусы и ошибки
|
||||
|
||||
Создаётся JPEG-thumbnail с сохранением пропорций и ограничением 256 × 256 пикселей. Оригинал остаётся доступным. Статусы обработки представлены как `queued`, `processing`, `ready`, `failed`. При временных ошибках предусмотрены ограниченные повторы; при исчерпании попыток сохраняется безопасный код ошибки.
|
||||
|
||||
### 3.4. Демонстрация остановленного worker
|
||||
|
||||
Сценарий реализован в `tests/queue_grade3.py`. При остановленном worker команда `prepare` загружает изображение и привязывает его к сообщению. После восстановления worker команда `verify` проверяет готовность, оригинал и thumbnail.
|
||||
|
||||
Полученные ранее результаты:
|
||||
|
||||
```text
|
||||
PASS: HTTP 202, queued, message linked
|
||||
PASS: restart recovery, ready, original, JPEG thumbnail 256x160
|
||||
```
|
||||
|
||||
Перезапуск RabbitMQ следует проводить с сохранением тома; `docker compose down -v` в сценарии сохранности данных недопустим.
|
||||
|
||||
## 4. Критерии на оценку 4
|
||||
|
||||
### 4.1. Три варианта изображений
|
||||
|
||||
В `src/worker.py` реализованы:
|
||||
|
||||
| Вариант | Формат | Назначение |
|
||||
|---|---|---|
|
||||
| `thumbnail` | JPEG | Уменьшенное изображение с сохранением пропорций |
|
||||
| `crop` | JPEG | Центральное кадрирование до 128 × 128 пикселей |
|
||||
| `optimized` | WebP | Сжатый вариант с параметром качества 80 |
|
||||
|
||||
Варианты создаются заново, без намеренного переноса EXIF-метаданных исходного файла. Метаданные вариантов включают тип, размеры, размер в байтах, S3-ключ и контрольную сумму.
|
||||
|
||||
### 4.2. Ограничения обработки
|
||||
|
||||
Входные изображения ограничены 10 МиБ и 40 млн пикселей. Допускаются PNG и JPEG, формат проверяется по содержимому. Worker повторно валидирует полученный из MinIO оригинал. Число неподтверждённых сообщений ограничивается `prefetch_count=1` на consumer.
|
||||
|
||||
В worker применяется `asyncio.wait_for(..., timeout=MAX_PROCESSING_SECONDS)`; значение по умолчанию — 30 секунд. Это ограничивает ожидание асинхронной задачи, но **не гарантирует остановку** уже запущенного потока Pillow в `asyncio.to_thread()`. Для строгого принудительного ограничения CPU-времени потребовалась бы изоляция в отдельном процессе.
|
||||
|
||||
### 4.3. Разрыв между PostgreSQL и RabbitMQ
|
||||
|
||||
Вместо непосредственной публикации только из HTTP-запроса используется восстановимая схема с долговременным заданием в PostgreSQL. Dispatcher периодически выбирает неопубликованные задания, использует блокировку строк `FOR UPDATE SKIP LOCKED` и publisher confirms RabbitMQ. При сбое между публикацией и фиксацией отметки о публикации сообщение может появиться повторно, поэтому идемпотентность worker является обязательной частью решения.
|
||||
|
||||
Для зависших заданий `processing` dispatcher выполняет восстановление после истечения интервала ожидания (90 секунд), сбрасывая сведения об устаревшем выполнении. Это позволяет повторно выдать работу, но не делает внешние S3-операции частью транзакции PostgreSQL.
|
||||
|
||||
### 4.4. Retry, DLQ и replay
|
||||
|
||||
Временные ошибки обрабатываются повторными попытками с экспоненциальной задержкой и jitter; число попыток ограничено пятью. Постоянные ошибки не должны бесконечно повторяться. После окончательного отказа состояние становится `failed`, а сведения о задании публикуются в `attachment.failed.v1`.
|
||||
|
||||
Скрипт `scripts/replay_job.py` допускает повторный запуск, только если и задание, и вложение находятся в `failed`. При replay `generation` увеличивается, попытки и сведения о предыдущем выполнении сбрасываются, новое поколение возвращается в очередь.
|
||||
|
||||
Пример запуска для существующего неуспешного задания:
|
||||
|
||||
```bash
|
||||
python -m scripts.replay_job <job_id>
|
||||
```
|
||||
|
||||
### 4.5. Staging и его очистка
|
||||
|
||||
Временный оригинал записывается в `staging/{chat_id}/{attachment_id}.{extension}`. Для удаления временных копий успешно обработанных вложений реализован `scripts/cleanup_staging.py`. По умолчанию это dry-run; `--apply` выполняет удаление. Учитывается срок хранения `STAGING_RETENTION_SECONDS=86400` (24 часа) и фактическое наличие объекта в MinIO.
|
||||
|
||||
```bash
|
||||
docker compose run --rm --no-deps worker python -m scripts.cleanup_staging
|
||||
docker compose run --rm --no-deps worker python -m scripts.cleanup_staging --apply
|
||||
```
|
||||
|
||||
В выполненном эксперименте обнаружены и удалены три временных объекта. Повторная проверка вернула `Candidates: 0`. Сохранность постоянного оригинала для проверенного старого вложения подтверждена отдельной проверкой `ORIGINAL EXISTS: True`.
|
||||
|
||||
Очистка запускается вручную, а не автоматически по расписанию. Скрипт не является универсальным сборщиком всех возможных S3-«сирот», в частности оставшихся после прерванных внешних операций.
|
||||
|
||||
### 4.6. Проверка метаданных и реальных S3-объектов
|
||||
|
||||
Последний интеграционный тест завершился:
|
||||
|
||||
```text
|
||||
PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.
|
||||
```
|
||||
|
||||
SQL-проверка пяти последних вложений показала `ready` для вложения и задания, `generation=1`, ключи `/g1/original` и `variants_count=3`. Для одного из этих вложений выполнена независимая проверка наличия объектов в MinIO:
|
||||
|
||||
```text
|
||||
original: True
|
||||
thumbnail: True
|
||||
crop: True
|
||||
optimized: True
|
||||
PASS: original and all 3 variants exist in MinIO
|
||||
```
|
||||
|
||||
Это проверяет как метаданные PostgreSQL, так и существование всех четырёх объектов.
|
||||
|
||||
## 5. Критерии на оценку 5 — Трек 2: Доказанная идемпотентность pipeline
|
||||
|
||||
### 5.1. Исследовательский вопрос и гипотеза
|
||||
|
||||
**Исследовательский вопрос:** можно ли сохранить согласованность результата при дублировании, нарушении порядка доставки, падении worker до/после побочного эффекта и конкурентном выполнении старой и новой попыток?
|
||||
|
||||
**Гипотеза:** сочетание `generation`, `attempt`, уникального `execution_id`, транзакционных проверок состояния и версионированных S3-ключей предотвращает фиксацию устаревшего результата в исследуемых сценариях, несмотря на повторную доставку сообщений.
|
||||
|
||||
### 5.2. Правила выполнения
|
||||
|
||||
- `job_id` и `attachment_id` идентифицируют работу и вложение.
|
||||
- `generation` различает поколения после replay.
|
||||
- `attempt` различает попытки внутри поколения.
|
||||
- `execution_id` различает конкретные выполнения, в том числе конкурирующие.
|
||||
- Worker проверяет актуальность сообщения до работы и непосредственно перед фиксацией результата.
|
||||
- Старое сообщение не может обновить завершённое или уже переизданное задание.
|
||||
- S3-ключи результатов включают поколение, поэтому разные поколения не пишут в один и тот же ключ.
|
||||
|
||||
Миграция `migrations/versions/c9b12026a005_execution_id.py` добавляет `execution_id` для fencing устаревших исполнителей.
|
||||
|
||||
### 5.3. Проверяемые инварианты
|
||||
|
||||
1. Повторная доставка уже завершённого задания не меняет его результат.
|
||||
2. Устаревшая попытка не изменяет состояние более новой попытки.
|
||||
3. Сообщение с неверным поколением не изменяет актуальное поколение.
|
||||
4. Старое выполнение не может зафиксировать результат после завершения нового выполнения.
|
||||
5. В исследуемых аварийных сценариях принятое задание восстанавливается и не теряется.
|
||||
6. После успешного завершения оригинал и все три варианта существуют в MinIO.
|
||||
|
||||
### 5.4. Автоматическое внедрение отказов
|
||||
|
||||
Используются три самостоятельных набора тестов:
|
||||
|
||||
```bash
|
||||
python tests/fault_injection_grade5.py
|
||||
python tests/crash_grade5.py
|
||||
python tests/concurrent_grade5.py
|
||||
```
|
||||
|
||||
`fault_injection_grade5.py` проверяет дублирование, нарушение порядка, неверные поколения и replay. `crash_grade5.py` намеренно завершает worker в точках `before_side_effect`, `after_side_effect`, `before_ack`, затем проверяет восстановление. `concurrent_grade5.py` запускает конкурирующие выполнения, задерживает старое, передаёт работу новому и проверяет запрет устаревшего COMMIT.
|
||||
|
||||
### 5.5. Результаты
|
||||
|
||||
| Сценарий | Проверяемое свойство | Результат |
|
||||
|---|---|---|
|
||||
| Повторная доставка | Неизменность готового результата | PASS |
|
||||
| Нарушение порядка попыток | Отклонение старой попытки | PASS |
|
||||
| Старое и будущее поколение | Проверка `generation` | PASS |
|
||||
| Replay `g1 → g2` | Старое сообщение не меняет новое поколение | PASS |
|
||||
| Crash before side effect | Восстановление до записи в S3 | PASS |
|
||||
| Crash after side effect | Восстановление после записи в S3 | PASS |
|
||||
| Crash before ACK | Повторная доставка после завершения работы | PASS |
|
||||
| Два конкурирующих worker | Отклонение устаревшего `execution_id` | PASS |
|
||||
|
||||
Фактический вывод финального запуска:
|
||||
|
||||
```text
|
||||
PASS: duplicate delivery, state unchanged, original intact
|
||||
PASS: reordered attempts, state unchanged, original intact
|
||||
PASS: stale and future generations rejected, result unchanged
|
||||
PASS: replay generation 1 -> 2, stale message rejected
|
||||
RUN: before_side_effect
|
||||
PASS: before_side_effect, recovered, original and variants intact
|
||||
RUN: after_side_effect
|
||||
PASS: after_side_effect, recovered, original and variants intact
|
||||
RUN: before_ack
|
||||
PASS: before_ack, recovered, original and variants intact
|
||||
Restoring normal worker configuration...
|
||||
FAULT INJECTION DISABLED
|
||||
PASS: stale execution rejected after newer execution completed
|
||||
Normal worker configuration restored
|
||||
```
|
||||
|
||||
После испытаний дополнительно запущен `tests/smoke.py`, получен `PASS`. Команда `docker compose ps` показала девять работающих сервисов: две API-реплики, dispatcher, Nginx, PostgreSQL, RabbitMQ, Redis, MinIO и worker. Инфраструктурные сервисы с healthcheck имели статус `healthy`.
|
||||
|
||||
### 5.6. Почему это не exactly-once delivery
|
||||
|
||||
RabbitMQ может доставить сообщение повторно, если worker завершился после побочного эффекта, но до ACK. Между PostgreSQL, RabbitMQ и MinIO нет единой распределённой транзакции. Поэтому утверждать **exactly-once delivery** нельзя.
|
||||
|
||||
Реализована схема **at-least-once delivery + идемпотентные проверки + fencing старых исполнений**. Она предотвращает некорректное изменение итогового состояния в проверенных сценариях, но не исключает повторные физические вычисления и временные объекты в S3.
|
||||
|
||||
### 5.7. Интерпретация результатов
|
||||
|
||||
Все восемь автоматических сценариев завершились успешно. Результаты согласуются с гипотезой: в проверенных условиях повторная доставка и устаревшие выполнения не изменили актуальный результат, а задания восстановились после внедрённых отказов.
|
||||
|
||||
Это экспериментальное подтверждение перечисленных инвариантов для реализованных сценариев, а не математическое доказательство корректности при любой последовательности отказов.
|
||||
|
||||
## 6. Доказательства критериев README
|
||||
|
||||
| Критерий | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
|---|---|---|---|
|
||||
| HTTP 202 и задание при остановленном worker | `queue_grade3.py prepare` | Принято, `queued`, вложение привязано к сообщению | `tests/queue_grade3.py` |
|
||||
| Восстановление и thumbnail | `queue_grade3.py verify` | `ready`, оригинал сохранён, JPEG 256 × 160 | `tests/queue_grade3.py` |
|
||||
| Публичный API | `make test` | PASS | `tests/smoke.py` |
|
||||
| Контракт | `make check`, `make validate` | PASS, PASS | `scripts/check_contract.py`, `scripts/validate_openapi.py` |
|
||||
| Три варианта | SQL и `object_exists` в MinIO | 3 метаданных варианта, 4 объекта существуют | `src/worker.py`, `src/storage.py` |
|
||||
| Ограничения входа | Валидация загрузки | 10 МиБ, 40 млн пикселей, PNG/JPEG | `src/images.py`, `src/main.py` |
|
||||
| Восстановимая публикация | Dispatcher и publisher confirms | Долговременное задание и повторная публикация | `src/dispatcher.py` |
|
||||
| Retry, DLQ | Ограниченные попытки и очередь отказов | `failed`, `attachment.failed.v1` | `src/worker.py`, `src/dispatcher.py` |
|
||||
| Replay | Повторный запуск неуспешного задания | `generation` увеличивается | `scripts/replay_job.py` |
|
||||
| Очистка staging | Dry-run и `--apply` | Удалены 3 объекта, затем 0 кандидатов | `scripts/cleanup_staging.py` |
|
||||
| Дубликаты и reorder | `fault_injection_grade5.py` | 4 PASS | `tests/fault_injection_grade5.py` |
|
||||
| Падение worker | `crash_grade5.py` | 3 PASS и восстановление | `tests/crash_grade5.py` |
|
||||
| Устаревший worker | `concurrent_grade5.py` | PASS, старый COMMIT отклонён | `tests/concurrent_grade5.py` |
|
||||
|
||||
Для критериев, не имеющих отдельного сохранённого протокола в этом отчёте (например, независимый чистый запуск и количественный нагрузочный эксперимент), наличие реализации не следует путать с доказательством завершённого эксперимента.
|
||||
|
||||
## 7. Экспериментальные условия, метрики и ограничения
|
||||
|
||||
### 7.1. Окружение
|
||||
|
||||
| Компонент | Версия / окружение |
|
||||
|---|---|
|
||||
| ОС | Ubuntu 22.04 (WSL) |
|
||||
| Python | 3.13.15 |
|
||||
| PostgreSQL | 17.6 |
|
||||
| Redis | 8.2.2 |
|
||||
| RabbitMQ | 4.1.4 |
|
||||
| Nginx | 1.29 |
|
||||
| MinIO | RELEASE.2025-09-07T16-13-09Z |
|
||||
|
||||
Испытания выполнялись на локальном Docker Compose-стенде с реальными PostgreSQL, RabbitMQ и MinIO. Для fault injection использовались управляемые точки отказа worker. В конкурентном тесте использовались два worker и искусственно устаревшее выполнение.
|
||||
|
||||
### 7.2. Нагрузка и измерения
|
||||
|
||||
В проекте есть `make load`, `make test-persistence ACTION_SCRIPT=...` и `make test-failover ACTION_SCRIPT=...`, но в этой серии для оценки 5 проверялась прежде всего корректность состояний при отказах.
|
||||
|
||||
Не зафиксированы отдельные воспроизводимые измерения RPS, доли HTTP-ошибок, p50/p95/p99, CPU и RAM. Не зафиксированы также единая интенсивность поступления запросов, длительность прогрева, число повторений нагрузочного теста и набор сырых метрик. Поэтому численные результаты производительности в отчёте **не заявляются**.
|
||||
|
||||
### 7.3. Границы применимости
|
||||
|
||||
- Тесты не перебирают все возможные межпоточные расписания и комбинации отказов.
|
||||
- Не исследованы повреждение диска, потеря постоянного тома, длительное сетевое разделение и одновременный отказ всех хранилищ.
|
||||
- `asyncio.wait_for()` не прерывает уже выполняющуюся CPU-операцию Pillow в потоке.
|
||||
- Между записью в S3 и COMMIT PostgreSQL могут оставаться неиспользуемые объекты; нужна отдельная периодическая сборка мусора.
|
||||
- Очистка staging сейчас ручная и ограничена выбранными успешно завершёнными заданиями.
|
||||
- Наличие версионированных ключей и fencing защищает логическое состояние, но не гарантирует отсутствие повторных физических операций.
|
||||
- Независимый запуск на полностью чистой машине с пустыми томами в рамках итоговой проверки не выполнялся.
|
||||
|
||||
## 8. Демонстрация
|
||||
|
||||
Команды выполнять из корня проекта с активированным `.venv`. При демонстрации отказов дождаться завершения каждого сценария перед следующим.
|
||||
|
||||
### 8.1. Запуск и состояние инфраструктуры
|
||||
|
||||
```bash
|
||||
docker compose up -d --build
|
||||
docker compose ps
|
||||
make check
|
||||
make validate
|
||||
make test
|
||||
```
|
||||
|
||||
Показать две API-реплики, отдельные dispatcher и worker, PostgreSQL, Redis, RabbitMQ и MinIO. Объяснить, что `make check` и `make validate` не заменяют `make test`.
|
||||
|
||||
### 8.2. Очередь и обработка при остановленном worker
|
||||
|
||||
```bash
|
||||
docker compose stop worker
|
||||
python tests/queue_grade3.py prepare
|
||||
docker compose exec -T rabbitmq rabbitmqctl list_queues name messages_ready messages_unacknowledged
|
||||
docker compose start worker
|
||||
python tests/queue_grade3.py verify
|
||||
```
|
||||
|
||||
Показать HTTP 202, `queued`, связанное сообщение и последующее `ready`. Объяснить, почему обработка не блокирует API.
|
||||
|
||||
### 8.3. Сохранность после перезапуска брокера
|
||||
|
||||
```bash
|
||||
docker compose stop worker
|
||||
python tests/queue_grade3.py prepare
|
||||
docker compose restart rabbitmq
|
||||
docker compose start worker
|
||||
python tests/queue_grade3.py verify
|
||||
```
|
||||
|
||||
Проверить восстановление RabbitMQ перед запуском worker. Не удалять тома Docker.
|
||||
|
||||
### 8.4. Метаданные трёх вариантов
|
||||
|
||||
```bash
|
||||
docker compose exec -T postgres psql -U lab2 -d messenger -c "
|
||||
SELECT a.id, a.status, j.status AS job_status, j.generation,
|
||||
a.s3_key, json_array_length(a.variants::json) AS variants_count
|
||||
FROM attachments a
|
||||
JOIN attachment_jobs j ON j.attachment_id = a.id
|
||||
ORDER BY a.created_at DESC
|
||||
LIMIT 5;
|
||||
"
|
||||
```
|
||||
|
||||
Показать `ready`, `/g1/original` и `variants_count=3` для свежих вложений. При необходимости дополнительно проверить существование ключей в MinIO через `src.storage.object_exists`.
|
||||
|
||||
### 8.5. Очередь ошибок и replay
|
||||
|
||||
```bash
|
||||
docker compose exec -T rabbitmq rabbitmqctl list_queues name messages_ready messages_unacknowledged
|
||||
docker compose logs --tail=100 worker
|
||||
```
|
||||
|
||||
Объяснить ограничение числа попыток, backoff, jitter, DLQ `attachment.failed.v1`. Для существующего окончательно неуспешного задания:
|
||||
|
||||
```bash
|
||||
python -m scripts.replay_job <job_id>
|
||||
```
|
||||
|
||||
Replay нельзя применять к произвольному заданию в `ready` или `processing`.
|
||||
|
||||
### 8.6. Очистка временных файлов
|
||||
|
||||
```bash
|
||||
docker compose run --rm --no-deps worker python -m scripts.cleanup_staging
|
||||
```
|
||||
|
||||
При наличии подходящих кандидатов и после объяснения dry-run:
|
||||
|
||||
```bash
|
||||
docker compose run --rm --no-deps worker python -m scripts.cleanup_staging --apply
|
||||
```
|
||||
|
||||
### 8.7. Исследовательский трек 2: дубликаты и версии
|
||||
|
||||
```bash
|
||||
python tests/fault_injection_grade5.py
|
||||
```
|
||||
|
||||
Показать четыре PASS: duplicate delivery, reordered attempts, stale/future generations, replay `g1 → g2`.
|
||||
|
||||
### 8.8. Падения до и после побочного эффекта
|
||||
|
||||
```bash
|
||||
python tests/crash_grade5.py
|
||||
```
|
||||
|
||||
Показать три PASS и строку `FAULT INJECTION DISABLED`. Объяснить различие между записью в MinIO, COMMIT PostgreSQL и ACK RabbitMQ.
|
||||
|
||||
### 8.9. Конкурентные worker
|
||||
|
||||
```bash
|
||||
python tests/concurrent_grade5.py
|
||||
```
|
||||
|
||||
Показать `PASS: stale execution rejected after newer execution completed`. Объяснить `execution_id` и проверку перед COMMIT.
|
||||
|
||||
### 8.10. Финальная проверка после экспериментов
|
||||
|
||||
```bash
|
||||
python tests/smoke.py --base-url http://localhost:8080
|
||||
docker compose ps
|
||||
git status --short
|
||||
```
|
||||
|
||||
Проверить работоспособность API и восстановление штатного worker. `git status --short` после коммита должен быть пустым, если нет дополнительных локальных изменений.
|
||||
|
||||
## 9. Ограничения и следующие шаги
|
||||
|
||||
Для эксплуатации за пределами лабораторного стенда полезно добавить периодическую очистку staging и неиспользуемых версионированных объектов, изоляцию CPU-обработки в отдельные процессы, метрики времени обработки и очередей, алерты по DLQ и зависшим заданиям, полноценные нагрузочные серии с сохранением исходных результатов, а также тесты длительных сетевых разделений и одновременных отказов.
|
||||
|
||||
Текущая реализация подтверждена автоматическими испытаниями в заявленных сценариях; выводы не следует распространять на непроверенные режимы без дополнительных экспериментов.
|
||||
|
||||
+149
@@ -0,0 +1,149 @@
|
||||
# A generic, single database configuration.
|
||||
|
||||
[alembic]
|
||||
# path to migration scripts.
|
||||
# this is typically a path given in POSIX (e.g. forward slashes)
|
||||
# format, relative to the token %(here)s which refers to the location of this
|
||||
# ini file
|
||||
script_location = %(here)s/migrations
|
||||
|
||||
# template used to generate migration file names; The default value is %%(rev)s_%%(slug)s
|
||||
# Uncomment the line below if you want the files to be prepended with date and time
|
||||
# see https://alembic.sqlalchemy.org/en/latest/tutorial.html#editing-the-ini-file
|
||||
# for all available tokens
|
||||
# file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s
|
||||
# Or organize into date-based subdirectories (requires recursive_version_locations = true)
|
||||
# file_template = %%(year)d/%%(month).2d/%%(day).2d_%%(hour).2d%%(minute).2d_%%(second).2d_%%(rev)s_%%(slug)s
|
||||
|
||||
# sys.path path, will be prepended to sys.path if present.
|
||||
# defaults to the current working directory. for multiple paths, the path separator
|
||||
# is defined by "path_separator" below.
|
||||
prepend_sys_path = .
|
||||
|
||||
# timezone to use when rendering the date within the migration file
|
||||
# as well as the filename.
|
||||
# If specified, requires the tzdata library which can be installed by adding
|
||||
# `alembic[tz]` to the pip requirements.
|
||||
# string value is passed to ZoneInfo()
|
||||
# leave blank for localtime
|
||||
# timezone =
|
||||
|
||||
# max length of characters to apply to the "slug" field
|
||||
# truncate_slug_length = 40
|
||||
|
||||
# set to 'true' to run the environment during
|
||||
# the 'revision' command, regardless of autogenerate
|
||||
# revision_environment = false
|
||||
|
||||
# set to 'true' to allow .pyc and .pyo files without
|
||||
# a source .py file to be detected as revisions in the
|
||||
# versions/ directory
|
||||
# sourceless = false
|
||||
|
||||
# version location specification; This defaults
|
||||
# to <script_location>/versions. When using multiple version
|
||||
# directories, initial revisions must be specified with --version-path.
|
||||
# The path separator used here should be the separator specified by "path_separator"
|
||||
# below.
|
||||
# version_locations = %(here)s/bar:%(here)s/bat:%(here)s/alembic/versions
|
||||
|
||||
# path_separator; This indicates what character is used to split lists of file
|
||||
# paths, including version_locations and prepend_sys_path within configparser
|
||||
# files such as alembic.ini.
|
||||
# The default rendered in new alembic.ini files is "os", which uses os.pathsep
|
||||
# to provide os-dependent path splitting.
|
||||
#
|
||||
# Note that in order to support legacy alembic.ini files, this default does NOT
|
||||
# take place if path_separator is not present in alembic.ini. If this
|
||||
# option is omitted entirely, fallback logic is as follows:
|
||||
#
|
||||
# 1. Parsing of the version_locations option falls back to using the legacy
|
||||
# "version_path_separator" key, which if absent then falls back to the legacy
|
||||
# behavior of splitting on spaces and/or commas.
|
||||
# 2. Parsing of the prepend_sys_path option falls back to the legacy
|
||||
# behavior of splitting on spaces, commas, or colons.
|
||||
#
|
||||
# Valid values for path_separator are:
|
||||
#
|
||||
# path_separator = :
|
||||
# path_separator = ;
|
||||
# path_separator = space
|
||||
# path_separator = newline
|
||||
#
|
||||
# Use os.pathsep. Default configuration used for new projects.
|
||||
path_separator = os
|
||||
|
||||
|
||||
# set to 'true' to search source files recursively
|
||||
# in each "version_locations" directory
|
||||
# new in Alembic version 1.10
|
||||
# recursive_version_locations = false
|
||||
|
||||
# the output encoding used when revision files
|
||||
# are written from script.py.mako
|
||||
# output_encoding = utf-8
|
||||
|
||||
# database URL. This is consumed by the user-maintained env.py script only.
|
||||
# other means of configuring database URLs may be customized within the env.py
|
||||
# file.
|
||||
sqlalchemy.url =
|
||||
|
||||
|
||||
[post_write_hooks]
|
||||
# post_write_hooks defines scripts or Python functions that are run
|
||||
# on newly generated revision scripts. See the documentation for further
|
||||
# detail and examples
|
||||
|
||||
# format using "black" - use the console_scripts runner, against the "black" entrypoint
|
||||
# hooks = black
|
||||
# black.type = console_scripts
|
||||
# black.entrypoint = black
|
||||
# black.options = -l 79 REVISION_SCRIPT_FILENAME
|
||||
|
||||
# lint with attempts to fix using "ruff" - use the module runner, against the "ruff" module
|
||||
# hooks = ruff
|
||||
# ruff.type = module
|
||||
# ruff.module = ruff
|
||||
# ruff.options = check --fix REVISION_SCRIPT_FILENAME
|
||||
|
||||
# Alternatively, use the exec runner to execute a binary found on your PATH
|
||||
# hooks = ruff
|
||||
# ruff.type = exec
|
||||
# ruff.executable = ruff
|
||||
# ruff.options = check --fix REVISION_SCRIPT_FILENAME
|
||||
|
||||
# Logging configuration. This is also consumed by the user-maintained
|
||||
# env.py script only.
|
||||
[loggers]
|
||||
keys = root,sqlalchemy,alembic
|
||||
|
||||
[handlers]
|
||||
keys = console
|
||||
|
||||
[formatters]
|
||||
keys = generic
|
||||
|
||||
[logger_root]
|
||||
level = WARNING
|
||||
handlers = console
|
||||
qualname =
|
||||
|
||||
[logger_sqlalchemy]
|
||||
level = WARNING
|
||||
handlers =
|
||||
qualname = sqlalchemy.engine
|
||||
|
||||
[logger_alembic]
|
||||
level = INFO
|
||||
handlers =
|
||||
qualname = alembic
|
||||
|
||||
[handler_console]
|
||||
class = StreamHandler
|
||||
args = (sys.stderr,)
|
||||
level = NOTSET
|
||||
formatter = generic
|
||||
|
||||
[formatter_generic]
|
||||
format = %(levelname)-5.5s [%(name)s] %(message)s
|
||||
datefmt = %H:%M:%S
|
||||
@@ -0,0 +1,14 @@
|
||||
services:
|
||||
worker:
|
||||
environment:
|
||||
FAULT_INJECTION_ENABLED: "1"
|
||||
FAULT_INJECTION_POINT: pause_before_commit
|
||||
|
||||
worker-second:
|
||||
extends:
|
||||
service: worker
|
||||
file: compose.yaml
|
||||
environment:
|
||||
FAULT_INJECTION_ENABLED: "0"
|
||||
FAULT_INJECTION_POINT: ""
|
||||
restart: "no"
|
||||
@@ -0,0 +1,5 @@
|
||||
services:
|
||||
worker:
|
||||
environment:
|
||||
FAULT_INJECTION_ENABLED: "1"
|
||||
FAULT_INJECTION_POINT: before_ack
|
||||
+228
@@ -0,0 +1,228 @@
|
||||
x-api: &api
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
environment: &api-environment
|
||||
APP_ENV: lab
|
||||
SESSION_TTL_SECONDS: 3600
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
CACHE_URL_FILE: /run/secrets/cache_url
|
||||
S3_ENDPOINT: http://s3:9000
|
||||
S3_REGION: us-east-1
|
||||
S3_BUCKET: messenger-attachments
|
||||
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
|
||||
MAX_UPLOAD_BYTES: 10485760
|
||||
MAX_IMAGE_PIXELS: 40000000
|
||||
RABBITMQ_HOST: rabbitmq
|
||||
RABBITMQ_PASSWORD_FILE: /run/secrets/rabbitmq_password
|
||||
secrets:
|
||||
- database_url
|
||||
- cache_url
|
||||
- s3_access_key_id
|
||||
- s3_secret_access_key
|
||||
- rabbitmq_password
|
||||
depends_on:
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
redis:
|
||||
condition: service_healthy
|
||||
s3-init:
|
||||
condition: service_completed_successfully
|
||||
expose:
|
||||
- "8080"
|
||||
restart: unless-stopped
|
||||
stop_grace_period: 15s
|
||||
mem_limit: 512m
|
||||
cpus: 1.0
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:17.6
|
||||
environment:
|
||||
POSTGRES_DB: messenger
|
||||
POSTGRES_USER: lab2
|
||||
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
|
||||
secrets:
|
||||
- postgres_password
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U lab2 -d messenger"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
restart: unless-stopped
|
||||
cpus: 1.0
|
||||
mem_limit: 768m
|
||||
|
||||
redis:
|
||||
image: redis:8.2.2
|
||||
command:
|
||||
- redis-server
|
||||
- --appendonly
|
||||
- "yes"
|
||||
- --appendfsync
|
||||
- everysec
|
||||
volumes:
|
||||
- redis_data:/data
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
restart: unless-stopped
|
||||
cpus: 0.5
|
||||
mem_limit: 256m
|
||||
|
||||
s3:
|
||||
image: dappros/minio:RELEASE.2025-09-07T16-13-09Z
|
||||
command: server /data --console-address ":9001"
|
||||
environment:
|
||||
MINIO_ROOT_USER_FILE: /run/secrets/minio_root_user
|
||||
MINIO_ROOT_PASSWORD_FILE: /run/secrets/minio_root_password
|
||||
secrets:
|
||||
- minio_root_user
|
||||
- minio_root_password
|
||||
- s3_access_key_id
|
||||
- s3_secret_access_key
|
||||
volumes:
|
||||
- s3_data:/data
|
||||
ports:
|
||||
- "127.0.0.1:9001:9001"
|
||||
healthcheck:
|
||||
test: [ "CMD", "mc", "ready", "local" ]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 10
|
||||
start_period: 15s
|
||||
restart: unless-stopped
|
||||
cpus: 1.0
|
||||
mem_limit: 512m
|
||||
|
||||
s3-init:
|
||||
image: dappros/minio:RELEASE.2025-09-07T16-13-09Z
|
||||
entrypoint: [ "/bin/sh", "/config/init.sh" ]
|
||||
volumes:
|
||||
- ./infra/minio/init.sh:/config/init.sh:ro
|
||||
- ./infra/minio/attachments-policy.json:/config/attachments-policy.json:ro
|
||||
secrets:
|
||||
- minio_root_user
|
||||
- minio_root_password
|
||||
- s3_access_key_id
|
||||
- s3_secret_access_key
|
||||
depends_on:
|
||||
s3:
|
||||
condition: service_healthy
|
||||
restart: "no"
|
||||
cpus: 0.5
|
||||
mem_limit: 256m
|
||||
|
||||
migrate:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
command: ["alembic", "upgrade", "head"]
|
||||
secrets:
|
||||
- database_url
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
restart: "no"
|
||||
cpus: 1.0
|
||||
mem_limit: 512m
|
||||
|
||||
api-1:
|
||||
<<: *api
|
||||
environment:
|
||||
<<: *api-environment
|
||||
INSTANCE_ID: api-1
|
||||
|
||||
api-2:
|
||||
<<: *api
|
||||
environment:
|
||||
<<: *api-environment
|
||||
INSTANCE_ID: api-2
|
||||
|
||||
nginx:
|
||||
image: nginx:1.29-alpine
|
||||
ports:
|
||||
- "8080:8080"
|
||||
volumes:
|
||||
- ./infra/nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
depends_on:
|
||||
- api-1
|
||||
- api-2
|
||||
restart: unless-stopped
|
||||
cpus: 0.5
|
||||
mem_limit: 256m
|
||||
|
||||
rabbitmq:
|
||||
image: rabbitmq:4.1.4-management-alpine
|
||||
hostname: rabbitmq
|
||||
environment:
|
||||
RABBITMQ_CONFIG_FILE: /etc/rabbitmq/rabbitmq
|
||||
secrets:
|
||||
- rabbitmq_password
|
||||
volumes:
|
||||
- rabbitmq_data:/var/lib/rabbitmq
|
||||
- ./infra/rabbitmq/rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf:ro
|
||||
healthcheck:
|
||||
test: ["CMD", "rabbitmq-diagnostics", "-q", "check_running"]
|
||||
interval: 10s
|
||||
timeout: 15s
|
||||
retries: 10
|
||||
start_period: 40s
|
||||
restart: unless-stopped
|
||||
mem_limit: 512m
|
||||
cpus: 0.5
|
||||
|
||||
dispatcher:
|
||||
<<: *api
|
||||
command: [ "python", "-m", "src.dispatcher" ]
|
||||
depends_on:
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
rabbitmq:
|
||||
condition: service_healthy
|
||||
s3-init:
|
||||
condition: service_completed_successfully
|
||||
healthcheck:
|
||||
disable: true
|
||||
|
||||
worker:
|
||||
<<: *api
|
||||
command: [ "python", "-m", "src.worker" ]
|
||||
depends_on:
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
rabbitmq:
|
||||
condition: service_healthy
|
||||
s3-init:
|
||||
condition: service_completed_successfully
|
||||
healthcheck:
|
||||
disable: true
|
||||
|
||||
secrets:
|
||||
postgres_password:
|
||||
file: ./secrets/postgres_password
|
||||
database_url:
|
||||
file: ./secrets/database_url
|
||||
cache_url:
|
||||
file: ./secrets/cache_url
|
||||
minio_root_user:
|
||||
file: ./secrets/minio_root_user
|
||||
minio_root_password:
|
||||
file: ./secrets/minio_root_password
|
||||
s3_access_key_id:
|
||||
file: ./secrets/s3_access_key_id
|
||||
s3_secret_access_key:
|
||||
file: ./secrets/s3_secret_access_key
|
||||
rabbitmq_password:
|
||||
file: ./secrets/rabbitmq_password
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
redis_data:
|
||||
s3_data:
|
||||
rabbitmq_data:
|
||||
@@ -0,0 +1,26 @@
|
||||
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"s3:ListBucket"
|
||||
],
|
||||
"Resource": [
|
||||
"arn:aws:s3:::messenger-attachments"
|
||||
]
|
||||
},
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"s3:GetObject",
|
||||
"s3:PutObject",
|
||||
"s3:DeleteObject"
|
||||
],
|
||||
"Resource": [
|
||||
"arn:aws:s3:::messenger-attachments/*"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
Executable
+19
@@ -0,0 +1,19 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
|
||||
ROOT_USER="$(cat /run/secrets/minio_root_user)"
|
||||
ROOT_PASSWORD="$(cat /run/secrets/minio_root_password)"
|
||||
APP_USER="$(cat /run/secrets/s3_access_key_id)"
|
||||
APP_PASSWORD="$(cat /run/secrets/s3_secret_access_key)"
|
||||
|
||||
mc alias set local http://s3:9000 "$ROOT_USER" "$ROOT_PASSWORD"
|
||||
|
||||
mc mb --ignore-existing local/messenger-attachments
|
||||
mc anonymous set private local/messenger-attachments
|
||||
|
||||
mc admin policy create local messenger-attachments-rw /config/attachments-policy.json
|
||||
mc admin user add local "$APP_USER" "$APP_PASSWORD"
|
||||
mc admin policy attach local messenger-attachments-rw --user "$APP_USER"
|
||||
|
||||
mc admin user info local "$APP_USER"
|
||||
mc ls local/messenger-attachments
|
||||
@@ -0,0 +1,37 @@
|
||||
events {
|
||||
worker_connections 1024;
|
||||
}
|
||||
|
||||
http {
|
||||
resolver 127.0.0.11 valid=5s ipv6=off;
|
||||
resolver_timeout 2s;
|
||||
|
||||
upstream api_backend {
|
||||
zone api_backend 64k;
|
||||
|
||||
server api-1:8080 resolve max_fails=1 fail_timeout=5s;
|
||||
server api-2:8080 resolve max_fails=1 fail_timeout=5s;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 8080;
|
||||
client_max_body_size 12m;
|
||||
|
||||
location / {
|
||||
proxy_pass http://api_backend;
|
||||
proxy_http_version 1.1;
|
||||
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
|
||||
proxy_connect_timeout 2s;
|
||||
proxy_read_timeout 10s;
|
||||
proxy_send_timeout 10s;
|
||||
|
||||
proxy_next_upstream error timeout http_502 http_503 http_504;
|
||||
proxy_next_upstream_tries 2;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
Generic single-database configuration with an async dbapi.
|
||||
@@ -0,0 +1,50 @@
|
||||
import asyncio
|
||||
|
||||
from alembic import context
|
||||
from sqlalchemy import pool
|
||||
from sqlalchemy.ext.asyncio import async_engine_from_config
|
||||
|
||||
|
||||
config = context.config
|
||||
|
||||
DATABASE_URL_FILE = "/run/secrets/database_url"
|
||||
|
||||
|
||||
def read_database_url() -> str:
|
||||
with open(DATABASE_URL_FILE, encoding="utf-8") as file:
|
||||
return file.read().strip()
|
||||
|
||||
|
||||
config.set_main_option("sqlalchemy.url", read_database_url())
|
||||
|
||||
target_metadata = None
|
||||
|
||||
|
||||
def do_run_migrations(connection) -> None:
|
||||
context.configure(
|
||||
connection=connection,
|
||||
target_metadata=target_metadata,
|
||||
)
|
||||
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
|
||||
|
||||
async def run_async_migrations() -> None:
|
||||
connectable = async_engine_from_config(
|
||||
config.get_section(config.config_ini_section, {}),
|
||||
prefix="sqlalchemy.",
|
||||
poolclass=pool.NullPool,
|
||||
)
|
||||
|
||||
async with connectable.connect() as connection:
|
||||
await connection.run_sync(do_run_migrations)
|
||||
|
||||
await connectable.dispose()
|
||||
|
||||
|
||||
def run_migrations_online() -> None:
|
||||
asyncio.run(run_async_migrations())
|
||||
|
||||
|
||||
run_migrations_online()
|
||||
@@ -0,0 +1,28 @@
|
||||
"""${message}
|
||||
|
||||
Revision ID: ${up_revision}
|
||||
Revises: ${down_revision | comma,n}
|
||||
Create Date: ${create_date}
|
||||
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
${imports if imports else ""}
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = ${repr(up_revision)}
|
||||
down_revision: Union[str, Sequence[str], None] = ${repr(down_revision)}
|
||||
branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)}
|
||||
depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)}
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
"""Upgrade schema."""
|
||||
${upgrades if upgrades else "pass"}
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
"""Downgrade schema."""
|
||||
${downgrades if downgrades else "pass"}
|
||||
@@ -0,0 +1,42 @@
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = "a6b12026a002"
|
||||
down_revision = "e5a12026a001"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.add_column(
|
||||
"attachment_jobs",
|
||||
sa.Column("next_attempt_at", sa.DateTime(timezone=True)),
|
||||
)
|
||||
|
||||
op.add_column(
|
||||
"attachment_jobs",
|
||||
sa.Column("last_error_code", sa.String(64)),
|
||||
)
|
||||
|
||||
op.add_column(
|
||||
"attachment_jobs",
|
||||
sa.Column("finished_at", sa.DateTime(timezone=True)),
|
||||
)
|
||||
|
||||
op.create_index(
|
||||
"ix_attachment_jobs_retry_schedule",
|
||||
"attachment_jobs",
|
||||
["status", "next_attempt_at"],
|
||||
)
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_index(
|
||||
"ix_attachment_jobs_retry_schedule",
|
||||
table_name="attachment_jobs",
|
||||
)
|
||||
|
||||
op.drop_column("attachment_jobs", "finished_at")
|
||||
op.drop_column("attachment_jobs", "last_error_code")
|
||||
op.drop_column("attachment_jobs", "next_attempt_at")
|
||||
@@ -0,0 +1,33 @@
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = "a7b12026a003"
|
||||
down_revision = "a6b12026a002"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.add_column(
|
||||
"attachment_jobs",
|
||||
sa.Column("failed_published_at", sa.DateTime(timezone=True)),
|
||||
)
|
||||
|
||||
op.create_index(
|
||||
"ix_attachment_jobs_failed_unpublished",
|
||||
"attachment_jobs",
|
||||
["created_at"],
|
||||
postgresql_where=sa.text(
|
||||
"status = 'failed' AND failed_published_at IS NULL"
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_index(
|
||||
"ix_attachment_jobs_failed_unpublished",
|
||||
table_name="attachment_jobs",
|
||||
)
|
||||
|
||||
op.drop_column("attachment_jobs", "failed_published_at")
|
||||
@@ -0,0 +1,61 @@
|
||||
"""Message idempotency for lab 3.
|
||||
|
||||
Revision ID: b3a91c0e7d42
|
||||
Revises: f7c51e1e945b
|
||||
"""
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = "b3a91c0e7d42"
|
||||
down_revision = "f7c51e1e945b"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"message_idempotency",
|
||||
sa.Column("user_id", sa.Uuid(), nullable=False),
|
||||
sa.Column("chat_id", sa.Uuid(), nullable=False),
|
||||
sa.Column("key", sa.String(length=128), nullable=False),
|
||||
sa.Column("request_hash", sa.String(length=64), nullable=False),
|
||||
sa.Column("message_id", sa.Uuid(), nullable=False),
|
||||
sa.Column(
|
||||
"created_at",
|
||||
sa.DateTime(timezone=True),
|
||||
server_default=sa.text("CURRENT_TIMESTAMP"),
|
||||
nullable=False,
|
||||
),
|
||||
sa.ForeignKeyConstraint(
|
||||
["user_id"],
|
||||
["users.id"],
|
||||
ondelete="CASCADE",
|
||||
),
|
||||
sa.ForeignKeyConstraint(
|
||||
["chat_id"],
|
||||
["chats.id"],
|
||||
ondelete="CASCADE",
|
||||
),
|
||||
sa.ForeignKeyConstraint(
|
||||
["message_id"],
|
||||
["messages.id"],
|
||||
ondelete="CASCADE",
|
||||
),
|
||||
sa.PrimaryKeyConstraint("user_id", "chat_id", "key"),
|
||||
)
|
||||
|
||||
op.create_index(
|
||||
"ix_message_idempotency_created_at",
|
||||
"message_idempotency",
|
||||
["created_at"],
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index(
|
||||
"ix_message_idempotency_created_at",
|
||||
table_name="message_idempotency",
|
||||
)
|
||||
op.drop_table("message_idempotency")
|
||||
@@ -0,0 +1,24 @@
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = "b8b12026a004"
|
||||
down_revision = "a7b12026a003"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.add_column(
|
||||
"attachment_jobs",
|
||||
sa.Column(
|
||||
"generation",
|
||||
sa.Integer(),
|
||||
nullable=False,
|
||||
server_default="1",
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_column("attachment_jobs", "generation")
|
||||
@@ -0,0 +1,78 @@
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = "c8f42a19e6b0"
|
||||
down_revision = "b3a91c0e7d42"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"attachments",
|
||||
sa.Column("id", sa.Uuid(), nullable=False),
|
||||
sa.Column("chat_id", sa.Uuid(), nullable=False),
|
||||
sa.Column("uploader_id", sa.Uuid(), nullable=False),
|
||||
sa.Column("filename", sa.String(length=255), nullable=False),
|
||||
sa.Column("content_type", sa.String(length=100), nullable=False),
|
||||
sa.Column("size_bytes", sa.BigInteger(), nullable=False),
|
||||
sa.Column("s3_key", sa.String(length=512), nullable=False),
|
||||
sa.Column("status", sa.String(length=20), nullable=False),
|
||||
sa.Column(
|
||||
"created_at",
|
||||
sa.DateTime(timezone=True),
|
||||
server_default=sa.text("CURRENT_TIMESTAMP"),
|
||||
nullable=False,
|
||||
),
|
||||
sa.ForeignKeyConstraint(
|
||||
["chat_id"],
|
||||
["chats.id"],
|
||||
ondelete="CASCADE",
|
||||
),
|
||||
sa.ForeignKeyConstraint(
|
||||
["uploader_id"],
|
||||
["users.id"],
|
||||
ondelete="RESTRICT",
|
||||
),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint("s3_key", name="uq_attachments_s3_key"),
|
||||
sa.CheckConstraint(
|
||||
"size_bytes > 0 AND size_bytes <= 10485760",
|
||||
name="ck_attachments_size",
|
||||
),
|
||||
sa.CheckConstraint(
|
||||
"status IN ('pending', 'ready')",
|
||||
name="ck_attachments_status",
|
||||
),
|
||||
)
|
||||
|
||||
op.create_index(
|
||||
"ix_attachments_chat_id",
|
||||
"attachments",
|
||||
["chat_id"],
|
||||
)
|
||||
|
||||
op.create_table(
|
||||
"message_attachments",
|
||||
sa.Column("message_id", sa.Uuid(), nullable=False),
|
||||
sa.Column("attachment_id", sa.Uuid(), nullable=False),
|
||||
sa.ForeignKeyConstraint(
|
||||
["message_id"],
|
||||
["messages.id"],
|
||||
ondelete="CASCADE",
|
||||
),
|
||||
sa.ForeignKeyConstraint(
|
||||
["attachment_id"],
|
||||
["attachments.id"],
|
||||
ondelete="RESTRICT",
|
||||
),
|
||||
sa.PrimaryKeyConstraint("message_id", "attachment_id"),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_table("message_attachments")
|
||||
op.drop_index("ix_attachments_chat_id", table_name="attachments")
|
||||
op.drop_table("attachments")
|
||||
@@ -0,0 +1,19 @@
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = "c9b12026a005"
|
||||
down_revision = "b8b12026a004"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.add_column(
|
||||
"attachment_jobs",
|
||||
sa.Column("execution_id", sa.UUID(), nullable=True),
|
||||
)
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_column("attachment_jobs", "execution_id")
|
||||
@@ -0,0 +1,80 @@
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = "d9e53b20f7c1"
|
||||
down_revision = "c8f42a19e6b0"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.alter_column(
|
||||
"attachments",
|
||||
"uploader_id",
|
||||
new_column_name="owner_id",
|
||||
)
|
||||
|
||||
op.add_column(
|
||||
"attachments",
|
||||
sa.Column(
|
||||
"sha256",
|
||||
sa.String(length=64),
|
||||
nullable=False,
|
||||
),
|
||||
)
|
||||
|
||||
op.add_column(
|
||||
"attachments",
|
||||
sa.Column(
|
||||
"variants",
|
||||
sa.JSON(),
|
||||
nullable=False,
|
||||
server_default=sa.text("'[]'::json"),
|
||||
),
|
||||
)
|
||||
|
||||
op.add_column(
|
||||
"attachments",
|
||||
sa.Column(
|
||||
"error_code",
|
||||
sa.String(length=100),
|
||||
nullable=True,
|
||||
),
|
||||
)
|
||||
|
||||
op.add_column(
|
||||
"attachments",
|
||||
sa.Column(
|
||||
"updated_at",
|
||||
sa.DateTime(timezone=True),
|
||||
server_default=sa.text("CURRENT_TIMESTAMP"),
|
||||
nullable=False,
|
||||
),
|
||||
)
|
||||
|
||||
op.create_check_constraint(
|
||||
"ck_attachments_sha256",
|
||||
"attachments",
|
||||
"sha256 ~ '^[a-f0-9]{64}$'",
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_constraint(
|
||||
"ck_attachments_sha256",
|
||||
"attachments",
|
||||
type_="check",
|
||||
)
|
||||
|
||||
op.drop_column("attachments", "updated_at")
|
||||
op.drop_column("attachments", "error_code")
|
||||
op.drop_column("attachments", "variants")
|
||||
op.drop_column("attachments", "sha256")
|
||||
|
||||
op.alter_column(
|
||||
"attachments",
|
||||
"owner_id",
|
||||
new_column_name="uploader_id",
|
||||
)
|
||||
@@ -0,0 +1,77 @@
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision = "e5a12026a001"
|
||||
down_revision = "d9e53b20f7c1"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.drop_constraint(
|
||||
"ck_attachments_status",
|
||||
"attachments",
|
||||
type_="check",
|
||||
)
|
||||
op.create_check_constraint(
|
||||
"ck_attachments_status",
|
||||
"attachments",
|
||||
"status IN ('queued', 'processing', 'ready', 'failed')",
|
||||
)
|
||||
|
||||
op.create_table(
|
||||
"attachment_jobs",
|
||||
sa.Column("id", sa.Uuid(), primary_key=True),
|
||||
sa.Column(
|
||||
"attachment_id",
|
||||
sa.Uuid(),
|
||||
sa.ForeignKey("attachments.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
unique=True,
|
||||
),
|
||||
sa.Column(
|
||||
"attempt",
|
||||
sa.Integer(),
|
||||
nullable=False,
|
||||
server_default="1",
|
||||
),
|
||||
sa.Column(
|
||||
"status",
|
||||
sa.String(20),
|
||||
nullable=False,
|
||||
server_default="queued",
|
||||
),
|
||||
sa.Column("published_at", sa.DateTime(timezone=True)),
|
||||
sa.Column("started_at", sa.DateTime(timezone=True)),
|
||||
sa.Column(
|
||||
"created_at",
|
||||
sa.DateTime(timezone=True),
|
||||
nullable=False,
|
||||
server_default=sa.text("CURRENT_TIMESTAMP"),
|
||||
),
|
||||
)
|
||||
|
||||
op.create_index(
|
||||
"ix_attachment_jobs_unpublished",
|
||||
"attachment_jobs",
|
||||
["published_at"],
|
||||
)
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_index(
|
||||
"ix_attachment_jobs_unpublished",
|
||||
table_name="attachment_jobs",
|
||||
)
|
||||
op.drop_table("attachment_jobs")
|
||||
|
||||
op.drop_constraint(
|
||||
"ck_attachments_status",
|
||||
"attachments",
|
||||
type_="check",
|
||||
)
|
||||
op.create_check_constraint(
|
||||
"ck_attachments_status",
|
||||
"attachments",
|
||||
"status IN ('pending', 'ready')",
|
||||
)
|
||||
@@ -0,0 +1,110 @@
|
||||
"""initial schema
|
||||
|
||||
Revision ID: f7c51e1e945b
|
||||
Revises:
|
||||
Create Date: 2026-10-07 18:33:37.771593
|
||||
|
||||
"""
|
||||
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "f7c51e1e945b"
|
||||
down_revision: Union[str, Sequence[str], None] = None
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"users",
|
||||
sa.Column("id", sa.Uuid(), nullable=False),
|
||||
sa.Column("username", sa.String(length=32), nullable=False),
|
||||
sa.Column("display_name", sa.String(length=100), nullable=False),
|
||||
sa.Column("password_hash", sa.Text(), nullable=False),
|
||||
sa.Column(
|
||||
"created_at",
|
||||
sa.DateTime(timezone=True),
|
||||
server_default=sa.text("CURRENT_TIMESTAMP"),
|
||||
nullable=False,
|
||||
),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint("username", name="uq_users_username"),
|
||||
)
|
||||
|
||||
op.create_table(
|
||||
"chats",
|
||||
sa.Column("id", sa.Uuid(), nullable=False),
|
||||
sa.Column("title", sa.String(length=100), nullable=False),
|
||||
sa.Column(
|
||||
"created_at",
|
||||
sa.DateTime(timezone=True),
|
||||
server_default=sa.text("CURRENT_TIMESTAMP"),
|
||||
nullable=False,
|
||||
),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
)
|
||||
|
||||
op.create_table(
|
||||
"chat_members",
|
||||
sa.Column("chat_id", sa.Uuid(), nullable=False),
|
||||
sa.Column("user_id", sa.Uuid(), nullable=False),
|
||||
sa.ForeignKeyConstraint(
|
||||
["chat_id"],
|
||||
["chats.id"],
|
||||
ondelete="CASCADE",
|
||||
),
|
||||
sa.ForeignKeyConstraint(
|
||||
["user_id"],
|
||||
["users.id"],
|
||||
ondelete="CASCADE",
|
||||
),
|
||||
sa.PrimaryKeyConstraint("chat_id", "user_id"),
|
||||
)
|
||||
|
||||
op.create_table(
|
||||
"messages",
|
||||
sa.Column("id", sa.Uuid(), nullable=False),
|
||||
sa.Column("chat_id", sa.Uuid(), nullable=False),
|
||||
sa.Column("sender_id", sa.Uuid(), nullable=False),
|
||||
sa.Column("text", sa.String(length=4000), nullable=False),
|
||||
sa.Column(
|
||||
"created_at",
|
||||
sa.DateTime(timezone=True),
|
||||
server_default=sa.text("CURRENT_TIMESTAMP"),
|
||||
nullable=False,
|
||||
),
|
||||
sa.ForeignKeyConstraint(
|
||||
["chat_id"],
|
||||
["chats.id"],
|
||||
ondelete="CASCADE",
|
||||
),
|
||||
sa.ForeignKeyConstraint(
|
||||
["sender_id"],
|
||||
["users.id"],
|
||||
ondelete="RESTRICT",
|
||||
),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
)
|
||||
|
||||
op.create_index(
|
||||
"ix_messages_chat_created_id",
|
||||
"messages",
|
||||
["chat_id", "created_at", "id"],
|
||||
unique=False,
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index(
|
||||
"ix_messages_chat_created_id",
|
||||
table_name="messages",
|
||||
)
|
||||
op.drop_table("messages")
|
||||
op.drop_table("chat_members")
|
||||
op.drop_table("chats")
|
||||
op.drop_table("users")
|
||||
@@ -0,0 +1,23 @@
|
||||
annotated-doc==0.0.5
|
||||
annotated-types==0.8.0
|
||||
anyio==4.15.1
|
||||
click==8.5.0
|
||||
fastapi==0.141.1
|
||||
h11==0.16.0
|
||||
idna==3.20
|
||||
pydantic==2.13.5
|
||||
pydantic_core==2.46.5
|
||||
starlette==1.7.0
|
||||
typing-inspection==0.4.4
|
||||
typing_extensions==4.16.0
|
||||
uvicorn==0.53.0
|
||||
SQLAlchemy==2.1.4
|
||||
asyncpg==0.32.0
|
||||
alembic==1.20.0
|
||||
greenlet==3.5.6
|
||||
argon2-cffi==25.1.0
|
||||
redis==7.0.1
|
||||
boto3==1.43.120
|
||||
Pillow==13.0.0
|
||||
python-multipart==0.0.32
|
||||
pika==1.3.2
|
||||
Executable
+41
@@ -0,0 +1,41 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
mkdir -p secrets
|
||||
chmod 700 secrets
|
||||
umask 077
|
||||
|
||||
create_random_secret() {
|
||||
local name="$1"
|
||||
local value="$2"
|
||||
|
||||
if [[ ! -s "secrets/$name" ]]; then
|
||||
printf '%s\n' "$value" > "secrets/$name"
|
||||
fi
|
||||
|
||||
chmod 644 "secrets/$name"
|
||||
}
|
||||
|
||||
create_random_secret postgres_password "$(python3 -c 'import secrets; print(secrets.token_hex(24))')"
|
||||
create_random_secret rabbitmq_password "$(python3 -c 'import secrets; print(secrets.token_hex(24))')"
|
||||
|
||||
if [[ ! -s secrets/database_url ]]; then
|
||||
PASSWORD="$(cat secrets/postgres_password)"
|
||||
printf 'postgresql+asyncpg://lab2:%s@postgres:5432/messenger\n' "$PASSWORD" > secrets/database_url
|
||||
fi
|
||||
|
||||
create_random_secret cache_url 'redis://redis:6379/0'
|
||||
create_random_secret minio_root_user "minio-$(python3 -c 'import secrets; print(secrets.token_hex(12))')"
|
||||
create_random_secret minio_root_password "$(python3 -c 'import secrets; print(secrets.token_hex(32))')"
|
||||
create_random_secret s3_access_key_id "$(python3 -c 'import secrets; print(secrets.token_hex(20))')"
|
||||
create_random_secret s3_secret_access_key "$(python3 -c 'import secrets; print(secrets.token_hex(32))')"
|
||||
|
||||
chmod 644 secrets/*
|
||||
|
||||
if [[ ! -f .env ]]; then
|
||||
cp .env.example .env
|
||||
fi
|
||||
|
||||
docker compose config --quiet
|
||||
@@ -0,0 +1,68 @@
|
||||
import argparse
|
||||
import asyncio
|
||||
import os
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import text
|
||||
|
||||
from src.database import SessionFactory, engine
|
||||
from src.storage import delete_object, object_exists
|
||||
|
||||
|
||||
RETENTION_SECONDS = int(os.getenv("STAGING_RETENTION_SECONDS", "86400"))
|
||||
|
||||
|
||||
async def cleanup(apply: bool):
|
||||
cutoff = datetime.now(timezone.utc) - timedelta(seconds=RETENTION_SECONDS)
|
||||
|
||||
async with SessionFactory() as session:
|
||||
result = await session.execute(
|
||||
text("""
|
||||
SELECT a.id, a.chat_id, a.content_type, a.s3_key,
|
||||
a.created_at
|
||||
FROM attachments a
|
||||
JOIN attachment_jobs j ON j.attachment_id = a.id
|
||||
WHERE a.status = 'ready'
|
||||
AND j.status = 'ready'
|
||||
AND a.s3_key LIKE 'attachments/%'
|
||||
AND a.created_at < :cutoff
|
||||
"""),
|
||||
{"cutoff": cutoff},
|
||||
)
|
||||
|
||||
rows = result.all()
|
||||
|
||||
candidates = 0
|
||||
|
||||
for row in rows:
|
||||
extension = "png" if row.content_type == "image/png" else "jpg"
|
||||
staging_key = f"staging/{row.chat_id}/{row.id}.{extension}"
|
||||
|
||||
if not await object_exists(staging_key):
|
||||
continue
|
||||
|
||||
candidates += 1
|
||||
|
||||
if not apply:
|
||||
print(f"DRY RUN: {staging_key}")
|
||||
continue
|
||||
|
||||
await delete_object(staging_key)
|
||||
print(f"DELETED: {staging_key}")
|
||||
|
||||
print(f"Candidates: {candidates}")
|
||||
|
||||
|
||||
async def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--apply", action="store_true")
|
||||
args = parser.parse_args()
|
||||
|
||||
try:
|
||||
await cleanup(args.apply)
|
||||
finally:
|
||||
await engine.dispose()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,83 @@
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
from uuid import UUID
|
||||
|
||||
from sqlalchemy import text
|
||||
|
||||
from src.database import SessionFactory, engine
|
||||
|
||||
|
||||
async def replay(job_id: UUID):
|
||||
async with SessionFactory() as session:
|
||||
async with session.begin():
|
||||
result = await session.execute(
|
||||
text("""
|
||||
SELECT j.id, j.attachment_id, j.status,
|
||||
j.generation, a.status AS attachment_status
|
||||
FROM attachment_jobs j
|
||||
JOIN attachments a ON a.id = j.attachment_id
|
||||
WHERE j.id = :job_id
|
||||
FOR UPDATE OF j, a
|
||||
"""),
|
||||
{"job_id": job_id},
|
||||
)
|
||||
|
||||
row = result.one_or_none()
|
||||
|
||||
if row is None:
|
||||
print("REJECTED: job not found")
|
||||
return
|
||||
|
||||
if row.status != "failed" or row.attachment_status != "failed":
|
||||
print("REJECTED: job and attachment must be failed")
|
||||
return
|
||||
|
||||
await session.execute(
|
||||
text("""
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'queued',
|
||||
generation = generation + 1,
|
||||
attempt = 1,
|
||||
published_at = NULL,
|
||||
started_at = NULL,
|
||||
execution_id = NULL,
|
||||
next_attempt_at = NULL,
|
||||
last_error_code = NULL,
|
||||
finished_at = NULL,
|
||||
failed_published_at = NULL
|
||||
WHERE id = :job_id
|
||||
"""),
|
||||
{"job_id": job_id},
|
||||
)
|
||||
|
||||
await session.execute(
|
||||
text("""
|
||||
UPDATE attachments
|
||||
SET status = 'queued',
|
||||
error_code = NULL,
|
||||
updated_at = now()
|
||||
WHERE id = :attachment_id
|
||||
"""),
|
||||
{"attachment_id": row.attachment_id},
|
||||
)
|
||||
|
||||
print(
|
||||
f"REPLAY ACCEPTED: job={job_id}, "
|
||||
f"generation={row.generation + 1}"
|
||||
)
|
||||
|
||||
|
||||
async def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("job_id", type=UUID)
|
||||
args = parser.parse_args()
|
||||
|
||||
try:
|
||||
await replay(args.job_id)
|
||||
finally:
|
||||
await engine.dispose()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
Executable
+4
@@ -0,0 +1,4 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
docker compose restart postgres redis api-1 api-2
|
||||
Executable
+14
@@ -0,0 +1,14 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
: "${TARGET_INSTANCE_ID:?}"
|
||||
|
||||
case "$TARGET_INSTANCE_ID" in
|
||||
api-1|api-2)
|
||||
docker compose stop "$TARGET_INSTANCE_ID"
|
||||
;;
|
||||
*)
|
||||
echo "Unknown instance: $TARGET_INSTANCE_ID" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
@@ -0,0 +1,86 @@
|
||||
import base64
|
||||
import binascii
|
||||
import json
|
||||
from datetime import datetime
|
||||
from uuid import UUID
|
||||
|
||||
|
||||
def encode_cursor(
|
||||
chat_id: UUID,
|
||||
created_at: datetime,
|
||||
message_id: UUID,
|
||||
) -> str:
|
||||
data = {
|
||||
"chat_id": str(chat_id),
|
||||
"created_at": created_at.isoformat(),
|
||||
"message_id": str(message_id),
|
||||
}
|
||||
|
||||
raw = json.dumps(
|
||||
data,
|
||||
separators=(",", ":"),
|
||||
).encode("utf-8")
|
||||
|
||||
return base64.urlsafe_b64encode(raw).decode("ascii")
|
||||
|
||||
|
||||
def decode_cursor(cursor: str) -> tuple[UUID, datetime, UUID]:
|
||||
try:
|
||||
raw = base64.urlsafe_b64decode(cursor.encode("ascii"))
|
||||
|
||||
data = json.loads(raw.decode("utf-8"))
|
||||
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError("Invalid cursor")
|
||||
|
||||
if set(data.keys()) != {
|
||||
"chat_id",
|
||||
"created_at",
|
||||
"message_id",
|
||||
}:
|
||||
raise ValueError("Invalid cursor")
|
||||
|
||||
chat_id = UUID(data["chat_id"])
|
||||
created_at = datetime.fromisoformat(data["created_at"])
|
||||
message_id = UUID(data["message_id"])
|
||||
|
||||
if created_at.tzinfo is None or created_at.utcoffset() is None:
|
||||
raise ValueError("Invalid cursor")
|
||||
|
||||
return chat_id, created_at, message_id
|
||||
|
||||
except (
|
||||
ValueError,
|
||||
KeyError,
|
||||
TypeError,
|
||||
UnicodeError,
|
||||
binascii.Error,
|
||||
json.JSONDecodeError,
|
||||
) as exc:
|
||||
raise ValueError("Invalid cursor") from exc
|
||||
|
||||
|
||||
def encode_chat_cursor(user_id: UUID, created_at: datetime, chat_id: UUID) -> str:
|
||||
data = {
|
||||
"user_id": str(user_id),
|
||||
"created_at": created_at.isoformat(),
|
||||
"chat_id": str(chat_id),
|
||||
}
|
||||
raw = json.dumps(data, separators=(",", ":")).encode("utf-8")
|
||||
return base64.urlsafe_b64encode(raw).decode("ascii")
|
||||
|
||||
|
||||
def decode_chat_cursor(cursor: str) -> tuple[UUID, datetime, UUID]:
|
||||
try:
|
||||
raw = base64.b64decode(cursor.encode("ascii"), altchars=b"-_", validate=True)
|
||||
data = json.loads(raw.decode("utf-8"))
|
||||
if not isinstance(data, dict) or set(data) != {"user_id", "created_at", "chat_id"}:
|
||||
raise ValueError("Invalid cursor")
|
||||
user_id = UUID(data["user_id"])
|
||||
created_at = datetime.fromisoformat(data["created_at"])
|
||||
chat_id = UUID(data["chat_id"])
|
||||
if created_at.tzinfo is None or created_at.utcoffset() is None:
|
||||
raise ValueError("Invalid cursor")
|
||||
return user_id, created_at, chat_id
|
||||
except (ValueError, TypeError, KeyError, UnicodeError, binascii.Error, json.JSONDecodeError) as exc:
|
||||
raise ValueError("Invalid cursor") from exc
|
||||
@@ -0,0 +1,57 @@
|
||||
from collections.abc import AsyncIterator
|
||||
|
||||
import socket
|
||||
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.exc import SQLAlchemyError
|
||||
from sqlalchemy.ext.asyncio import (
|
||||
AsyncSession,
|
||||
async_sessionmaker,
|
||||
create_async_engine,
|
||||
)
|
||||
|
||||
|
||||
DATABASE_URL_FILE = "/run/secrets/database_url"
|
||||
|
||||
POOL_SIZE = 5
|
||||
MAX_OVERFLOW = 5
|
||||
POOL_TIMEOUT_SECONDS = 5
|
||||
DB_CONNECT_TIMEOUT_SECONDS = 5
|
||||
|
||||
|
||||
def read_database_url() -> str:
|
||||
with open(DATABASE_URL_FILE, encoding="utf-8") as file:
|
||||
return file.read().strip()
|
||||
|
||||
|
||||
database_url = read_database_url()
|
||||
|
||||
engine = create_async_engine(
|
||||
database_url,
|
||||
pool_size=POOL_SIZE,
|
||||
max_overflow=MAX_OVERFLOW,
|
||||
pool_timeout=POOL_TIMEOUT_SECONDS,
|
||||
connect_args={
|
||||
"timeout": DB_CONNECT_TIMEOUT_SECONDS,
|
||||
},
|
||||
pool_pre_ping=True,
|
||||
)
|
||||
|
||||
SessionFactory = async_sessionmaker(
|
||||
bind=engine,
|
||||
expire_on_commit=False,
|
||||
)
|
||||
|
||||
|
||||
async def get_db_session() -> AsyncIterator[AsyncSession]:
|
||||
async with SessionFactory() as session:
|
||||
yield session
|
||||
|
||||
|
||||
async def ping_database() -> bool:
|
||||
try:
|
||||
async with engine.connect() as connection:
|
||||
await connection.execute(text("SELECT 1"))
|
||||
return True
|
||||
except (SQLAlchemyError, OSError):
|
||||
return False
|
||||
@@ -0,0 +1,149 @@
|
||||
import asyncio
|
||||
import logging
|
||||
|
||||
from sqlalchemy import text
|
||||
|
||||
from src.database import SessionFactory, engine
|
||||
from src.queue import connect, publish, publish_failed
|
||||
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
|
||||
|
||||
async def dispatch(channel):
|
||||
async with SessionFactory() as session:
|
||||
async with session.begin():
|
||||
await session.execute(
|
||||
text("""
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'queued',
|
||||
published_at = NULL,
|
||||
started_at = NULL,
|
||||
execution_id = NULL
|
||||
WHERE status = 'processing'
|
||||
AND started_at < now() - interval '90 seconds'
|
||||
""")
|
||||
)
|
||||
|
||||
await session.execute(
|
||||
text("""
|
||||
UPDATE attachments
|
||||
SET status = 'queued',
|
||||
updated_at = now()
|
||||
WHERE status = 'processing'
|
||||
AND id IN (
|
||||
SELECT attachment_id
|
||||
FROM attachment_jobs
|
||||
WHERE status = 'queued'
|
||||
)
|
||||
""")
|
||||
)
|
||||
|
||||
result = await session.execute(
|
||||
text("""
|
||||
SELECT id, attachment_id, attempt, generation, created_at
|
||||
FROM attachment_jobs
|
||||
WHERE published_at IS NULL
|
||||
AND status = 'queued'
|
||||
AND (
|
||||
next_attempt_at IS NULL
|
||||
OR next_attempt_at <= now()
|
||||
)
|
||||
ORDER BY created_at LIMIT 25
|
||||
FOR
|
||||
UPDATE SKIP LOCKED
|
||||
""")
|
||||
)
|
||||
|
||||
rows = result.all()
|
||||
|
||||
for row in rows:
|
||||
publish(channel, row)
|
||||
|
||||
await session.execute(
|
||||
text("""
|
||||
UPDATE attachment_jobs
|
||||
SET published_at = now()
|
||||
WHERE id = :id
|
||||
"""),
|
||||
{"id": row.id},
|
||||
)
|
||||
|
||||
return len(rows)
|
||||
|
||||
|
||||
async def dispatch_failed(channel):
|
||||
async with SessionFactory() as session:
|
||||
async with session.begin():
|
||||
result = await session.execute(
|
||||
text("""
|
||||
SELECT id, attachment_id, attempt, generation,
|
||||
last_error_code, created_at, finished_at
|
||||
FROM attachment_jobs
|
||||
WHERE status = 'failed'
|
||||
AND failed_published_at IS NULL
|
||||
ORDER BY created_at
|
||||
LIMIT 25
|
||||
FOR UPDATE SKIP LOCKED
|
||||
""")
|
||||
)
|
||||
|
||||
rows = result.all()
|
||||
|
||||
for row in rows:
|
||||
payload = {
|
||||
"schema_version": 1,
|
||||
"type": "attachment.failed.v1",
|
||||
"job_id": str(row.id),
|
||||
"attachment_id": str(row.attachment_id),
|
||||
"attempt": row.attempt,
|
||||
"generation": row.generation,
|
||||
"error_code": row.last_error_code,
|
||||
"created_at": row.created_at.isoformat(),
|
||||
"finished_at": (
|
||||
row.finished_at.isoformat()
|
||||
if row.finished_at
|
||||
else None
|
||||
),
|
||||
}
|
||||
|
||||
publish_failed(channel, payload)
|
||||
|
||||
await session.execute(
|
||||
text("""
|
||||
UPDATE attachment_jobs
|
||||
SET failed_published_at = now()
|
||||
WHERE id = :id
|
||||
"""),
|
||||
{"id": row.id},
|
||||
)
|
||||
|
||||
return len(rows)
|
||||
|
||||
|
||||
async def main():
|
||||
while True:
|
||||
connection = None
|
||||
|
||||
try:
|
||||
connection, channel = connect()
|
||||
channel.confirm_delivery()
|
||||
|
||||
while True:
|
||||
count = await dispatch(channel)
|
||||
failed_count = await dispatch_failed(channel)
|
||||
await asyncio.sleep(0.2 if count or failed_count else 1)
|
||||
|
||||
except Exception:
|
||||
logging.exception("Dispatcher reconnect")
|
||||
await asyncio.sleep(3)
|
||||
|
||||
finally:
|
||||
if connection and connection.is_open:
|
||||
connection.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
asyncio.run(main())
|
||||
finally:
|
||||
asyncio.run(engine.dispose())
|
||||
@@ -0,0 +1,12 @@
|
||||
class APIError(Exception):
|
||||
def __init__(
|
||||
self,
|
||||
status_code: int,
|
||||
code: str,
|
||||
message: str,
|
||||
headers: dict[str, str] | None = None,
|
||||
):
|
||||
self.status_code = status_code
|
||||
self.code = code
|
||||
self.message = message
|
||||
self.headers = headers
|
||||
@@ -0,0 +1,71 @@
|
||||
import hashlib
|
||||
import io
|
||||
import os
|
||||
|
||||
from PIL import Image, UnidentifiedImageError
|
||||
|
||||
from src.errors import APIError
|
||||
|
||||
|
||||
MAX_UPLOAD_BYTES = int(os.getenv("MAX_UPLOAD_BYTES", "10485760"))
|
||||
MAX_IMAGE_PIXELS = int(os.getenv("MAX_IMAGE_PIXELS", "40000000"))
|
||||
|
||||
ALLOWED_FORMATS = {
|
||||
"PNG": "image/png",
|
||||
"JPEG": "image/jpeg",
|
||||
}
|
||||
|
||||
|
||||
def validate_image(body: bytes, content_type: str) -> tuple[int, str]:
|
||||
if content_type not in ALLOWED_FORMATS.values():
|
||||
raise APIError(
|
||||
status_code=415,
|
||||
code="unsupported_media_type",
|
||||
message="Only PNG and JPEG images are supported",
|
||||
)
|
||||
|
||||
size_bytes = len(body)
|
||||
|
||||
if size_bytes > MAX_UPLOAD_BYTES:
|
||||
raise APIError(
|
||||
status_code=413,
|
||||
code="payload_too_large",
|
||||
message="Image exceeds maximum upload size",
|
||||
)
|
||||
|
||||
if size_bytes == 0:
|
||||
raise APIError(
|
||||
status_code=400,
|
||||
code="invalid_request",
|
||||
message="Image must not be empty",
|
||||
)
|
||||
|
||||
try:
|
||||
with Image.open(io.BytesIO(body)) as image:
|
||||
if image.format not in ALLOWED_FORMATS:
|
||||
raise ValueError("Unsupported image format")
|
||||
|
||||
if ALLOWED_FORMATS[image.format] != content_type:
|
||||
raise ValueError("Content-Type does not match image format")
|
||||
|
||||
width, height = image.size
|
||||
|
||||
if width * height > MAX_IMAGE_PIXELS:
|
||||
raise APIError(
|
||||
status_code=413,
|
||||
code="payload_too_large",
|
||||
message="Image exceeds maximum pixel count",
|
||||
)
|
||||
|
||||
image.load()
|
||||
|
||||
except APIError:
|
||||
raise
|
||||
except (UnidentifiedImageError, OSError, ValueError, Image.DecompressionBombError) as exc:
|
||||
raise APIError(
|
||||
status_code=400,
|
||||
code="invalid_request",
|
||||
message="Invalid image data",
|
||||
) from exc
|
||||
|
||||
return size_bytes, hashlib.sha256(body).hexdigest()
|
||||
+779
@@ -0,0 +1,779 @@
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from uuid import UUID, uuid4
|
||||
import os
|
||||
import re
|
||||
import base64
|
||||
import binascii
|
||||
import asyncio
|
||||
|
||||
from fastapi import Depends, FastAPI, Request, Header, Query
|
||||
from fastapi.exceptions import RequestValidationError
|
||||
from fastapi.responses import JSONResponse, Response
|
||||
from fastapi.security import (
|
||||
HTTPAuthorizationCredentials,
|
||||
HTTPBearer,
|
||||
)
|
||||
from botocore.exceptions import BotoCoreError, ClientError
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.exc import IntegrityError, SQLAlchemyError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from starlette.exceptions import HTTPException as StarletteHTTPException
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
from src.database import engine, get_db_session, ping_database, SessionFactory
|
||||
from src.errors import APIError
|
||||
from src.models import CreateUser, SessionResponse, User, CreateChat, Chat, ChatPage, CreateMessage, Message, MessagePage, Attachment
|
||||
from src.repositories import (
|
||||
create_chat,
|
||||
list_chats,
|
||||
create_user as create_user_in_db,
|
||||
get_user_by_id,
|
||||
get_chat_by_id,
|
||||
get_user_credentials_by_username,
|
||||
create_message,
|
||||
get_messages,
|
||||
create_attachment,
|
||||
get_attachment,
|
||||
attachment_key_exists,
|
||||
)
|
||||
from src.security import verify_password
|
||||
from src.sessions import (
|
||||
create_session,
|
||||
delete_session,
|
||||
get_session_user_id,
|
||||
ping_session_store,
|
||||
redis_client,
|
||||
)
|
||||
from src.images import MAX_UPLOAD_BYTES, validate_image
|
||||
from src.storage import delete_object, put_object, get_object
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
yield
|
||||
await redis_client.aclose()
|
||||
await engine.dispose()
|
||||
|
||||
|
||||
app = FastAPI(lifespan=lifespan)
|
||||
INSTANCE_ID = os.getenv("INSTANCE_ID") or str(uuid4())
|
||||
|
||||
bearer_security = HTTPBearer(auto_error=False)
|
||||
|
||||
|
||||
async def get_current_user(
|
||||
credentials: HTTPAuthorizationCredentials | None = Depends(bearer_security),
|
||||
session: AsyncSession = Depends(get_db_session),
|
||||
) -> User:
|
||||
if credentials is None or credentials.scheme.lower() != "bearer":
|
||||
raise APIError(
|
||||
status_code=401,
|
||||
code="unauthorized",
|
||||
message="Unauthorized",
|
||||
headers={
|
||||
"WWW-Authenticate": "Bearer",
|
||||
},
|
||||
)
|
||||
|
||||
user_id = await get_session_user_id(credentials.credentials)
|
||||
|
||||
if user_id is None:
|
||||
raise APIError(
|
||||
status_code=401,
|
||||
code="unauthorized",
|
||||
message="Unauthorized",
|
||||
headers={
|
||||
"WWW-Authenticate": "Bearer",
|
||||
},
|
||||
)
|
||||
|
||||
user = await get_user_by_id(session, user_id)
|
||||
|
||||
if user is None:
|
||||
raise APIError(
|
||||
status_code=401,
|
||||
code="unauthorized",
|
||||
message="Unauthorized",
|
||||
headers={
|
||||
"WWW-Authenticate": "Bearer",
|
||||
},
|
||||
)
|
||||
|
||||
return user
|
||||
|
||||
|
||||
@app.middleware("http")
|
||||
async def request_middleware(request: Request, call_next):
|
||||
request_id = str(uuid4())
|
||||
request.state.request_id = request_id
|
||||
|
||||
if request.method in {"POST", "PUT", "PATCH"}:
|
||||
content_type = request.headers.get("content-type")
|
||||
|
||||
if content_type is not None:
|
||||
media_type = content_type.split(";", 1)[0].strip().lower()
|
||||
|
||||
is_attachment_upload = (
|
||||
request.method == "POST"
|
||||
and re.fullmatch(
|
||||
r"/api/v1/chats/[^/]+/attachments",
|
||||
request.url.path,
|
||||
) is not None
|
||||
)
|
||||
|
||||
allowed_media_types = (
|
||||
{"image/png", "image/jpeg"}
|
||||
if is_attachment_upload
|
||||
else {"application/json"}
|
||||
)
|
||||
|
||||
if media_type not in allowed_media_types:
|
||||
return JSONResponse(
|
||||
status_code=415,
|
||||
content={
|
||||
"error": {
|
||||
"code": "unsupported_media_type",
|
||||
"message": "Unsupported media type",
|
||||
"request_id": request_id,
|
||||
}
|
||||
},
|
||||
headers={
|
||||
"X-Request-Id": request_id,
|
||||
"X-Instance-Id": INSTANCE_ID,
|
||||
},
|
||||
)
|
||||
|
||||
response = await call_next(request)
|
||||
|
||||
response.headers["X-Request-Id"] = request_id
|
||||
response.headers["X-Instance-Id"] = INSTANCE_ID
|
||||
|
||||
return response
|
||||
|
||||
|
||||
@app.exception_handler(RequestValidationError)
|
||||
async def validation_exception_handler(
|
||||
request: Request,
|
||||
exc: RequestValidationError,
|
||||
):
|
||||
request_id = request.state.request_id
|
||||
|
||||
return JSONResponse(
|
||||
status_code=400,
|
||||
content={
|
||||
"error": {
|
||||
"code": "invalid_request",
|
||||
"message": "Invalid request",
|
||||
"request_id": request_id,
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@app.exception_handler(APIError)
|
||||
async def api_error_handler(request: Request, exc: APIError):
|
||||
request_id = request.state.request_id
|
||||
|
||||
return JSONResponse(
|
||||
status_code=exc.status_code,
|
||||
content={
|
||||
"error": {
|
||||
"code": exc.code,
|
||||
"message": exc.message,
|
||||
"request_id": request_id,
|
||||
}
|
||||
},
|
||||
headers=exc.headers,
|
||||
)
|
||||
|
||||
|
||||
@app.exception_handler(SQLAlchemyError)
|
||||
async def database_exception_handler(
|
||||
request: Request,
|
||||
exc: SQLAlchemyError,
|
||||
):
|
||||
request_id = request.state.request_id
|
||||
|
||||
return JSONResponse(
|
||||
status_code=503,
|
||||
content={
|
||||
"error": {
|
||||
"code": "unavailable",
|
||||
"message": "Database unavailable",
|
||||
"request_id": request_id,
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@app.exception_handler(OSError)
|
||||
async def database_connection_exception_handler(
|
||||
request: Request,
|
||||
exc: OSError,
|
||||
):
|
||||
request_id = request.state.request_id
|
||||
|
||||
return JSONResponse(
|
||||
status_code=503,
|
||||
content={
|
||||
"error": {
|
||||
"code": "unavailable",
|
||||
"message": "Database unavailable",
|
||||
"request_id": request_id,
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@app.exception_handler(StarletteHTTPException)
|
||||
async def http_exception_handler(
|
||||
request: Request,
|
||||
exc: StarletteHTTPException,
|
||||
):
|
||||
request_id = request.state.request_id
|
||||
|
||||
if exc.status_code == 404:
|
||||
code = "not_found"
|
||||
message = "Not found"
|
||||
else:
|
||||
code = "invalid_request"
|
||||
message = "Invalid request"
|
||||
|
||||
return JSONResponse(
|
||||
status_code=exc.status_code,
|
||||
content={
|
||||
"error": {
|
||||
"code": code,
|
||||
"message": message,
|
||||
"request_id": request_id,
|
||||
}
|
||||
},
|
||||
headers=exc.headers,
|
||||
)
|
||||
|
||||
|
||||
@app.exception_handler(Exception)
|
||||
async def unexpected_exception_handler(
|
||||
request: Request,
|
||||
exc: Exception,
|
||||
):
|
||||
request_id = request.state.request_id
|
||||
|
||||
return JSONResponse(
|
||||
status_code=500,
|
||||
content={
|
||||
"error": {
|
||||
"code": "internal_error",
|
||||
"message": "Internal server error",
|
||||
"request_id": request_id,
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@app.get("/health/live")
|
||||
async def health_live():
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
@app.get("/health/ready")
|
||||
async def health_ready():
|
||||
database_ready = await ping_database()
|
||||
session_store_ready = await ping_session_store()
|
||||
|
||||
if not database_ready or not session_store_ready:
|
||||
raise APIError(
|
||||
status_code=503,
|
||||
code="unavailable",
|
||||
message="Service dependencies unavailable",
|
||||
)
|
||||
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
@app.post("/api/v1/users", response_model=User, status_code=201)
|
||||
async def create_user(
|
||||
data: CreateUser,
|
||||
session: AsyncSession = Depends(get_db_session),
|
||||
):
|
||||
try:
|
||||
async with session.begin():
|
||||
return await create_user_in_db(session, data)
|
||||
|
||||
except IntegrityError as exc:
|
||||
raise APIError(
|
||||
status_code=409,
|
||||
code="conflict",
|
||||
message="Username already exists",
|
||||
) from exc
|
||||
|
||||
|
||||
@app.get("/api/v1/users/me", response_model=User)
|
||||
async def get_me(
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
return current_user
|
||||
|
||||
|
||||
@app.post(
|
||||
"/api/v1/auth/sessions",
|
||||
response_model=SessionResponse,
|
||||
status_code=201,
|
||||
)
|
||||
async def create_auth_session(
|
||||
authorization: str | None = Header(default=None),
|
||||
session: AsyncSession = Depends(get_db_session),
|
||||
):
|
||||
credentials = None
|
||||
if authorization is not None:
|
||||
scheme, separator, encoded = authorization.partition(" ")
|
||||
if separator and scheme.lower() == "basic":
|
||||
try:
|
||||
decoded = base64.b64decode(encoded.strip(), validate=True).decode("utf-8")
|
||||
if ":" in decoded:
|
||||
credentials = decoded.split(":", 1)
|
||||
except (ValueError, UnicodeError, binascii.Error):
|
||||
pass
|
||||
if credentials is None:
|
||||
raise APIError(
|
||||
status_code=401,
|
||||
code="unauthorized",
|
||||
message="Unauthorized",
|
||||
headers={
|
||||
"WWW-Authenticate": 'Basic realm="messenger", charset="UTF-8"',
|
||||
},
|
||||
)
|
||||
|
||||
user_credentials = await get_user_credentials_by_username(
|
||||
session,
|
||||
credentials[0],
|
||||
)
|
||||
|
||||
if user_credentials is None:
|
||||
raise APIError(
|
||||
status_code=401,
|
||||
code="unauthorized",
|
||||
message="Unauthorized",
|
||||
headers={
|
||||
"WWW-Authenticate": 'Basic realm="messenger", charset="UTF-8"',
|
||||
},
|
||||
)
|
||||
|
||||
if not verify_password(
|
||||
user_credentials.password_hash,
|
||||
credentials[1],
|
||||
):
|
||||
raise APIError(
|
||||
status_code=401,
|
||||
code="unauthorized",
|
||||
message="Unauthorized",
|
||||
headers={
|
||||
"WWW-Authenticate": 'Basic realm="messenger", charset="UTF-8"',
|
||||
},
|
||||
)
|
||||
|
||||
user = User(
|
||||
id=user_credentials.id,
|
||||
username=user_credentials.username,
|
||||
display_name=user_credentials.display_name,
|
||||
created_at=user_credentials.created_at,
|
||||
)
|
||||
|
||||
token, ttl = await create_session(user.id)
|
||||
|
||||
expires_at = datetime.now(timezone.utc) + timedelta(seconds=ttl)
|
||||
|
||||
return SessionResponse(
|
||||
token=token,
|
||||
token_type="Bearer",
|
||||
expires_at=expires_at,
|
||||
user=user,
|
||||
)
|
||||
|
||||
|
||||
@app.delete(
|
||||
"/api/v1/auth/sessions/current",
|
||||
status_code=204,
|
||||
)
|
||||
async def delete_current_session(
|
||||
credentials: HTTPAuthorizationCredentials | None = Depends(bearer_security),
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
await delete_session(credentials.credentials)
|
||||
|
||||
|
||||
@app.post(
|
||||
"/api/v1/chats",
|
||||
response_model=Chat,
|
||||
status_code=201,
|
||||
)
|
||||
async def create_chat_endpoint(
|
||||
data: CreateChat,
|
||||
current_user: User = Depends(get_current_user),
|
||||
session: AsyncSession = Depends(get_db_session),
|
||||
) -> Chat:
|
||||
try:
|
||||
chat = await create_chat(
|
||||
session=session,
|
||||
data=data,
|
||||
current_user_id=current_user.id,
|
||||
)
|
||||
await session.commit()
|
||||
except Exception:
|
||||
await session.rollback()
|
||||
raise
|
||||
|
||||
return chat
|
||||
|
||||
|
||||
@app.get(
|
||||
"/api/v1/chats",
|
||||
response_model=ChatPage,
|
||||
)
|
||||
async def get_chats(
|
||||
limit: int = 50,
|
||||
cursor: str | None = None,
|
||||
current_user: User = Depends(get_current_user),
|
||||
session: AsyncSession = Depends(get_db_session),
|
||||
) -> ChatPage:
|
||||
if limit < 1 or limit > 100:
|
||||
raise APIError(
|
||||
status_code=400,
|
||||
code="invalid_request",
|
||||
message="Limit must be between 1 and 100",
|
||||
)
|
||||
|
||||
return await list_chats(
|
||||
session=session,
|
||||
current_user_id=current_user.id,
|
||||
limit=limit,
|
||||
cursor=cursor,
|
||||
)
|
||||
|
||||
|
||||
@app.get(
|
||||
"/api/v1/chats/{chat_id}",
|
||||
response_model=Chat,
|
||||
)
|
||||
async def get_chat_endpoint(
|
||||
chat_id: UUID,
|
||||
current_user: User = Depends(get_current_user),
|
||||
session: AsyncSession = Depends(get_db_session),
|
||||
) -> Chat:
|
||||
chat = await get_chat_by_id(
|
||||
session=session,
|
||||
chat_id=chat_id,
|
||||
current_user_id=current_user.id,
|
||||
)
|
||||
|
||||
if chat is None:
|
||||
raise APIError(
|
||||
status_code=404,
|
||||
code="not_found",
|
||||
message="Chat not found",
|
||||
)
|
||||
|
||||
return chat
|
||||
|
||||
|
||||
@app.post(
|
||||
"/api/v1/chats/{chat_id}/messages",
|
||||
response_model=Message,
|
||||
status_code=201,
|
||||
)
|
||||
async def create_message_endpoint(
|
||||
chat_id: UUID,
|
||||
data: CreateMessage,
|
||||
idempotency_key: str | None = Header(
|
||||
default=None,
|
||||
alias="Idempotency-Key",
|
||||
),
|
||||
current_user: User = Depends(get_current_user),
|
||||
session: AsyncSession = Depends(get_db_session),
|
||||
) -> Message:
|
||||
|
||||
if (
|
||||
idempotency_key is None
|
||||
or re.fullmatch(
|
||||
r"[A-Za-z0-9._:-]{1,128}",
|
||||
idempotency_key,
|
||||
) is None
|
||||
):
|
||||
raise APIError(
|
||||
status_code=400,
|
||||
code="invalid_request",
|
||||
message="Invalid or missing Idempotency-Key",
|
||||
)
|
||||
|
||||
try:
|
||||
message = await create_message(
|
||||
session=session,
|
||||
chat_id=chat_id,
|
||||
current_user_id=current_user.id,
|
||||
data=data,
|
||||
idempotency_key=idempotency_key,
|
||||
)
|
||||
|
||||
await session.commit()
|
||||
|
||||
except Exception:
|
||||
await session.rollback()
|
||||
raise
|
||||
|
||||
return message
|
||||
|
||||
|
||||
@app.get(
|
||||
"/api/v1/chats/{chat_id}/messages",
|
||||
response_model=MessagePage,
|
||||
)
|
||||
async def get_messages_endpoint(
|
||||
chat_id: UUID,
|
||||
limit: int = 50,
|
||||
cursor: str | None = None,
|
||||
current_user: User = Depends(get_current_user),
|
||||
session: AsyncSession = Depends(get_db_session),
|
||||
) -> MessagePage:
|
||||
if limit < 1 or limit > 100:
|
||||
raise APIError(
|
||||
status_code=400,
|
||||
code="invalid_request",
|
||||
message="Limit must be between 1 and 100",
|
||||
)
|
||||
|
||||
return await get_messages(
|
||||
session=session,
|
||||
chat_id=chat_id,
|
||||
current_user_id=current_user.id,
|
||||
limit=limit,
|
||||
cursor=cursor,
|
||||
)
|
||||
|
||||
|
||||
@app.post(
|
||||
"/api/v1/chats/{chat_id}/attachments",
|
||||
response_model=Attachment,
|
||||
status_code=202,
|
||||
)
|
||||
async def upload_attachment_endpoint(
|
||||
chat_id: UUID,
|
||||
request: Request,
|
||||
filename: str = Query(min_length=1, max_length=255),
|
||||
current_user: User = Depends(get_current_user),
|
||||
session: AsyncSession = Depends(get_db_session),
|
||||
) -> Attachment:
|
||||
chat = await get_chat_by_id(
|
||||
session=session,
|
||||
chat_id=chat_id,
|
||||
current_user_id=current_user.id,
|
||||
)
|
||||
|
||||
if chat is None:
|
||||
raise APIError(
|
||||
status_code=404,
|
||||
code="not_found",
|
||||
message="Chat not found",
|
||||
)
|
||||
|
||||
content_type = request.headers.get("content-type", "").split(";", 1)[0].strip().lower()
|
||||
|
||||
if content_type not in {"image/png", "image/jpeg"}:
|
||||
raise APIError(
|
||||
status_code=415,
|
||||
code="unsupported_media_type",
|
||||
message="Only PNG and JPEG images are supported",
|
||||
)
|
||||
|
||||
body = bytearray()
|
||||
|
||||
async for chunk in request.stream():
|
||||
if len(body) + len(chunk) > MAX_UPLOAD_BYTES:
|
||||
raise APIError(
|
||||
status_code=413,
|
||||
code="payload_too_large",
|
||||
message="Image exceeds maximum upload size",
|
||||
)
|
||||
|
||||
body.extend(chunk)
|
||||
|
||||
image_bytes = bytes(body)
|
||||
size_bytes, sha256 = validate_image(image_bytes, content_type)
|
||||
|
||||
attachment_id = uuid4()
|
||||
extension = "png" if content_type == "image/png" else "jpg"
|
||||
s3_key = f"staging/{chat_id}/{attachment_id}.{extension}"
|
||||
|
||||
try:
|
||||
await put_object(
|
||||
key=s3_key,
|
||||
body=image_bytes,
|
||||
content_type=content_type,
|
||||
)
|
||||
except (BotoCoreError, ClientError, OSError, TimeoutError) as exc:
|
||||
raise APIError(
|
||||
status_code=503,
|
||||
code="unavailable",
|
||||
message="Object storage unavailable",
|
||||
) from exc
|
||||
|
||||
try:
|
||||
attachment = await create_attachment(
|
||||
session=session,
|
||||
attachment_id=attachment_id,
|
||||
chat_id=chat_id,
|
||||
owner_id=current_user.id,
|
||||
filename=filename,
|
||||
content_type=content_type,
|
||||
size_bytes=size_bytes,
|
||||
sha256=sha256,
|
||||
s3_key=s3_key,
|
||||
)
|
||||
|
||||
await session.execute(
|
||||
text("""
|
||||
INSERT INTO attachment_jobs (id, attachment_id)
|
||||
VALUES (:job_id, :attachment_id)
|
||||
"""),
|
||||
{
|
||||
"job_id": uuid4(),
|
||||
"attachment_id": attachment_id,
|
||||
},
|
||||
)
|
||||
|
||||
await session.commit()
|
||||
|
||||
except BaseException:
|
||||
try:
|
||||
await session.rollback()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
try:
|
||||
async with SessionFactory() as verification_session:
|
||||
exists = await attachment_key_exists(
|
||||
verification_session,
|
||||
s3_key,
|
||||
)
|
||||
except Exception:
|
||||
exists = True
|
||||
|
||||
if not exists:
|
||||
try:
|
||||
await asyncio.shield(delete_object(s3_key))
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
raise
|
||||
|
||||
return attachment
|
||||
|
||||
|
||||
|
||||
@app.get(
|
||||
"/api/v1/attachments/{attachment_id}",
|
||||
response_model=Attachment,
|
||||
)
|
||||
async def get_attachment_endpoint(
|
||||
attachment_id: UUID,
|
||||
current_user: User = Depends(get_current_user),
|
||||
session: AsyncSession = Depends(get_db_session),
|
||||
) -> Attachment:
|
||||
result = await get_attachment(
|
||||
session=session,
|
||||
attachment_id=attachment_id,
|
||||
current_user_id=current_user.id,
|
||||
)
|
||||
|
||||
if result is None:
|
||||
raise APIError(
|
||||
status_code=404,
|
||||
code="not_found",
|
||||
message="Attachment not found",
|
||||
)
|
||||
|
||||
attachment, s3_key = result
|
||||
return attachment
|
||||
|
||||
|
||||
@app.get("/api/v1/attachments/{attachment_id}/content")
|
||||
async def get_attachment_content_endpoint(
|
||||
attachment_id: UUID,
|
||||
current_user: User = Depends(get_current_user),
|
||||
session: AsyncSession = Depends(get_db_session),
|
||||
variant: str = Query(default="original"),
|
||||
):
|
||||
result = await get_attachment(
|
||||
session=session,
|
||||
attachment_id=attachment_id,
|
||||
current_user_id=current_user.id,
|
||||
)
|
||||
|
||||
if result is None:
|
||||
raise APIError(
|
||||
status_code=404,
|
||||
code="not_found",
|
||||
message="Attachment not found",
|
||||
)
|
||||
|
||||
attachment, s3_key = result
|
||||
|
||||
if attachment.status != "ready":
|
||||
raise APIError(
|
||||
status_code=409,
|
||||
code="conflict",
|
||||
message="Attachment is not ready",
|
||||
)
|
||||
|
||||
if variant == "original":
|
||||
media_type = attachment.content_type
|
||||
|
||||
elif variant in {"thumbnail", "crop", "optimized"}:
|
||||
selected = next(
|
||||
(item for item in attachment.variants if item.name == variant),
|
||||
None,
|
||||
)
|
||||
|
||||
if selected is None:
|
||||
raise APIError(
|
||||
status_code=404,
|
||||
code="not_found",
|
||||
message="Variant not found",
|
||||
)
|
||||
|
||||
extension = {
|
||||
"thumbnail": "jpg",
|
||||
"crop": "jpg",
|
||||
"optimized": "webp",
|
||||
}[variant]
|
||||
|
||||
s3_key = selected.s3_key or (
|
||||
f"variants/{attachment.chat_id}/{attachment_id}/"
|
||||
f"{variant}.{extension}"
|
||||
)
|
||||
media_type = selected.content_type
|
||||
|
||||
else:
|
||||
raise APIError(
|
||||
status_code=404,
|
||||
code="not_found",
|
||||
message="Variant not found",
|
||||
)
|
||||
|
||||
try:
|
||||
body = await get_object(s3_key)
|
||||
except (BotoCoreError, ClientError, OSError, TimeoutError) as exc:
|
||||
raise APIError(
|
||||
status_code=503,
|
||||
code="unavailable",
|
||||
message="Object storage unavailable",
|
||||
) from exc
|
||||
|
||||
return Response(
|
||||
content=body,
|
||||
media_type=media_type,
|
||||
headers={
|
||||
"Cache-Control": "private, no-store",
|
||||
"X-Content-Type-Options": "nosniff",
|
||||
},
|
||||
)
|
||||
|
||||
+153
@@ -0,0 +1,153 @@
|
||||
from datetime import datetime
|
||||
from uuid import UUID
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
|
||||
|
||||
|
||||
class CreateUser(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
username: str = Field(
|
||||
min_length=3,
|
||||
max_length=32,
|
||||
pattern=r"^[a-z0-9_]+$",
|
||||
)
|
||||
|
||||
display_name: str = Field(
|
||||
min_length=1,
|
||||
max_length=100,
|
||||
)
|
||||
|
||||
password: str = Field(
|
||||
min_length=12,
|
||||
max_length=128,
|
||||
)
|
||||
|
||||
@field_validator("display_name")
|
||||
@classmethod
|
||||
def validate_display_name(cls, value: str) -> str:
|
||||
if not value.strip():
|
||||
raise ValueError("Display name must not be blank")
|
||||
|
||||
return value
|
||||
|
||||
|
||||
class User(BaseModel):
|
||||
id: UUID
|
||||
username: str
|
||||
display_name: str
|
||||
created_at: datetime
|
||||
|
||||
|
||||
class SessionResponse(BaseModel):
|
||||
token: str
|
||||
token_type: str
|
||||
expires_at: datetime
|
||||
user: User
|
||||
|
||||
|
||||
class CreateChat(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
title: str = Field(
|
||||
min_length=1,
|
||||
max_length=100,
|
||||
)
|
||||
|
||||
member_ids: list[UUID] = Field(
|
||||
min_length=1,
|
||||
max_length=100,
|
||||
)
|
||||
|
||||
@field_validator("title")
|
||||
@classmethod
|
||||
def validate_title(cls, value: str) -> str:
|
||||
if not value.strip():
|
||||
raise ValueError("Chat title must not be blank")
|
||||
|
||||
return value
|
||||
|
||||
@field_validator("member_ids")
|
||||
@classmethod
|
||||
def validate_member_ids(cls, value: list[UUID]) -> list[UUID]:
|
||||
if len(value) != len(set(value)):
|
||||
raise ValueError("Member IDs must be unique")
|
||||
|
||||
return value
|
||||
|
||||
|
||||
class Chat(BaseModel):
|
||||
id: UUID
|
||||
title: str
|
||||
member_ids: list[UUID]
|
||||
created_at: datetime
|
||||
|
||||
|
||||
class ChatPage(BaseModel):
|
||||
items: list[Chat]
|
||||
next_cursor: str | None
|
||||
|
||||
|
||||
class CreateMessage(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
text: str = Field(max_length=4000)
|
||||
attachment_ids: list[UUID] = Field(default_factory=list, max_length=10)
|
||||
|
||||
@field_validator("attachment_ids")
|
||||
@classmethod
|
||||
def validate_attachment_ids(cls, value: list[UUID]) -> list[UUID]:
|
||||
if len(value) != len(set(value)):
|
||||
raise ValueError("Attachment IDs must be unique")
|
||||
|
||||
return value
|
||||
|
||||
@model_validator(mode="after")
|
||||
def validate_content(self):
|
||||
if not self.text.strip() and not self.attachment_ids:
|
||||
raise ValueError("Message must contain text or attachments")
|
||||
|
||||
return self
|
||||
|
||||
|
||||
class Message(BaseModel):
|
||||
id: UUID
|
||||
chat_id: UUID
|
||||
sender_id: UUID
|
||||
text: str
|
||||
created_at: datetime
|
||||
attachment_ids: list[UUID] = Field(default_factory=list)
|
||||
|
||||
|
||||
class MessagePage(BaseModel):
|
||||
items: list[Message]
|
||||
next_cursor: str | None
|
||||
|
||||
|
||||
class Variant(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
name: str = Field(pattern=r"^(thumbnail|crop|optimized)$")
|
||||
content_type: str = Field(pattern=r"^image/(jpeg|png|webp)$")
|
||||
size_bytes: int = Field(ge=1)
|
||||
width: int = Field(ge=1)
|
||||
height: int = Field(ge=1)
|
||||
sha256: str = Field(pattern=r"^[a-f0-9]{64}$")
|
||||
s3_key: str | None = Field(default=None, exclude=True)
|
||||
|
||||
|
||||
class Attachment(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
id: UUID
|
||||
chat_id: UUID
|
||||
owner_id: UUID
|
||||
filename: str = Field(min_length=1, max_length=255)
|
||||
content_type: str = Field(pattern=r"^image/(png|jpeg)$")
|
||||
size_bytes: int = Field(ge=1, le=10485760)
|
||||
sha256: str = Field(pattern=r"^[a-f0-9]{64}$")
|
||||
status: str = Field(pattern=r"^(queued|processing|ready|failed)$")
|
||||
variants: list[Variant]
|
||||
error_code: str | None
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
@@ -0,0 +1,78 @@
|
||||
import json
|
||||
import os
|
||||
from datetime import timezone
|
||||
from pathlib import Path
|
||||
|
||||
import pika
|
||||
|
||||
QUEUE = "attachment.process.v1"
|
||||
DLQ = "attachment.failed.v1"
|
||||
|
||||
|
||||
def connect():
|
||||
password = Path(
|
||||
os.environ["RABBITMQ_PASSWORD_FILE"]
|
||||
).read_text().strip()
|
||||
|
||||
credentials = pika.PlainCredentials(
|
||||
"messenger",
|
||||
password,
|
||||
)
|
||||
|
||||
connection = pika.BlockingConnection(
|
||||
pika.ConnectionParameters(
|
||||
host=os.getenv("RABBITMQ_HOST", "rabbitmq"),
|
||||
credentials=credentials,
|
||||
heartbeat=60,
|
||||
blocked_connection_timeout=30,
|
||||
connection_attempts=3,
|
||||
retry_delay=2,
|
||||
)
|
||||
)
|
||||
|
||||
channel = connection.channel()
|
||||
channel.queue_declare(queue=QUEUE, durable=True)
|
||||
channel.queue_declare(queue=DLQ, durable=True)
|
||||
|
||||
return connection, channel
|
||||
|
||||
|
||||
def publish(channel, row):
|
||||
payload = {
|
||||
"schema_version": 1,
|
||||
"job_id": str(row.id),
|
||||
"type": "attachment.process.v1",
|
||||
"attachment_id": str(row.attachment_id),
|
||||
"pipeline_version": 1,
|
||||
"attempt": row.attempt,
|
||||
"generation": row.generation,
|
||||
"created_at": row.created_at.astimezone(
|
||||
timezone.utc
|
||||
).isoformat().replace("+00:00", "Z"),
|
||||
}
|
||||
|
||||
channel.basic_publish(
|
||||
exchange="",
|
||||
routing_key=QUEUE,
|
||||
body=json.dumps(payload).encode(),
|
||||
properties=pika.BasicProperties(
|
||||
delivery_mode=2,
|
||||
content_type="application/json",
|
||||
message_id=str(row.id),
|
||||
),
|
||||
mandatory=True,
|
||||
)
|
||||
|
||||
|
||||
def publish_failed(channel, payload: dict):
|
||||
channel.basic_publish(
|
||||
exchange="",
|
||||
routing_key=DLQ,
|
||||
body=json.dumps(payload).encode(),
|
||||
properties=pika.BasicProperties(
|
||||
delivery_mode=2,
|
||||
content_type="application/json",
|
||||
message_id=payload["job_id"],
|
||||
),
|
||||
mandatory=True,
|
||||
)
|
||||
@@ -0,0 +1,875 @@
|
||||
from uuid import UUID, uuid4
|
||||
import hashlib
|
||||
import json
|
||||
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from src.models import Attachment, Chat, ChatPage, CreateChat, CreateUser, User, CreateMessage, Message, MessagePage
|
||||
from src.security import hash_password
|
||||
from src.errors import APIError
|
||||
from src.cursor import decode_cursor, encode_cursor, decode_chat_cursor, encode_chat_cursor
|
||||
|
||||
|
||||
async def create_user(
|
||||
session: AsyncSession,
|
||||
data: CreateUser,
|
||||
) -> User:
|
||||
user_id = uuid4()
|
||||
password_hash = hash_password(data.password)
|
||||
|
||||
result = await session.execute(
|
||||
text(
|
||||
"""
|
||||
INSERT INTO users (
|
||||
id,
|
||||
username,
|
||||
display_name,
|
||||
password_hash
|
||||
)
|
||||
VALUES (
|
||||
:id,
|
||||
:username,
|
||||
:display_name,
|
||||
:password_hash
|
||||
)
|
||||
RETURNING
|
||||
id,
|
||||
username,
|
||||
display_name,
|
||||
created_at
|
||||
"""
|
||||
),
|
||||
{
|
||||
"id": user_id,
|
||||
"username": data.username,
|
||||
"display_name": data.display_name,
|
||||
"password_hash": password_hash,
|
||||
},
|
||||
)
|
||||
|
||||
row = result.one()
|
||||
|
||||
return User(
|
||||
id=UUID(str(row.id)),
|
||||
username=row.username,
|
||||
display_name=row.display_name,
|
||||
created_at=row.created_at,
|
||||
)
|
||||
|
||||
async def get_user_credentials_by_username(
|
||||
session: AsyncSession,
|
||||
username: str,
|
||||
):
|
||||
result = await session.execute(
|
||||
text(
|
||||
"""
|
||||
SELECT
|
||||
id,
|
||||
username,
|
||||
display_name,
|
||||
password_hash,
|
||||
created_at
|
||||
FROM users
|
||||
WHERE username = :username
|
||||
"""
|
||||
),
|
||||
{
|
||||
"username": username,
|
||||
},
|
||||
)
|
||||
|
||||
return result.one_or_none()
|
||||
|
||||
|
||||
async def get_user_by_id(
|
||||
session: AsyncSession,
|
||||
user_id: UUID,
|
||||
) -> User | None:
|
||||
result = await session.execute(
|
||||
text(
|
||||
"""
|
||||
SELECT
|
||||
id,
|
||||
username,
|
||||
display_name,
|
||||
created_at
|
||||
FROM users
|
||||
WHERE id = :user_id
|
||||
"""
|
||||
),
|
||||
{
|
||||
"user_id": user_id,
|
||||
},
|
||||
)
|
||||
|
||||
row = result.one_or_none()
|
||||
|
||||
if row is None:
|
||||
return None
|
||||
|
||||
return User(
|
||||
id=UUID(str(row.id)),
|
||||
username=row.username,
|
||||
display_name=row.display_name,
|
||||
created_at=row.created_at,
|
||||
)
|
||||
|
||||
|
||||
async def create_chat(
|
||||
session: AsyncSession,
|
||||
data: CreateChat,
|
||||
current_user_id: UUID,
|
||||
) -> Chat:
|
||||
if current_user_id not in data.member_ids:
|
||||
raise APIError(
|
||||
status_code=400,
|
||||
code="invalid_request",
|
||||
message="Current user must be a chat member",
|
||||
)
|
||||
|
||||
users_result = await session.execute(
|
||||
text(
|
||||
"""
|
||||
SELECT id
|
||||
FROM users
|
||||
WHERE id = ANY(:member_ids)
|
||||
"""
|
||||
),
|
||||
{
|
||||
"member_ids": data.member_ids,
|
||||
},
|
||||
)
|
||||
|
||||
existing_user_ids = {
|
||||
UUID(str(row.id))
|
||||
for row in users_result
|
||||
}
|
||||
|
||||
if existing_user_ids != set(data.member_ids):
|
||||
raise APIError(
|
||||
status_code=400,
|
||||
code="invalid_request",
|
||||
message="Unknown chat member",
|
||||
)
|
||||
|
||||
chat_id = uuid4()
|
||||
|
||||
chat_result = await session.execute(
|
||||
text(
|
||||
"""
|
||||
INSERT INTO chats (
|
||||
id,
|
||||
title
|
||||
)
|
||||
VALUES (
|
||||
:id,
|
||||
:title
|
||||
)
|
||||
RETURNING
|
||||
id,
|
||||
title,
|
||||
created_at
|
||||
"""
|
||||
),
|
||||
{
|
||||
"id": chat_id,
|
||||
"title": data.title,
|
||||
},
|
||||
)
|
||||
|
||||
chat_row = chat_result.one()
|
||||
|
||||
for member_id in data.member_ids:
|
||||
await session.execute(
|
||||
text(
|
||||
"""
|
||||
INSERT INTO chat_members (
|
||||
chat_id,
|
||||
user_id
|
||||
)
|
||||
VALUES (
|
||||
:chat_id,
|
||||
:user_id
|
||||
)
|
||||
"""
|
||||
),
|
||||
{
|
||||
"chat_id": chat_id,
|
||||
"user_id": member_id,
|
||||
},
|
||||
)
|
||||
|
||||
return Chat(
|
||||
id=UUID(str(chat_row.id)),
|
||||
title=chat_row.title,
|
||||
member_ids=data.member_ids,
|
||||
created_at=chat_row.created_at,
|
||||
)
|
||||
|
||||
|
||||
async def get_chat_by_id(
|
||||
session: AsyncSession,
|
||||
chat_id: UUID,
|
||||
current_user_id: UUID,
|
||||
) -> Chat | None:
|
||||
chat_result = await session.execute(
|
||||
text(
|
||||
"""
|
||||
SELECT
|
||||
c.id,
|
||||
c.title,
|
||||
c.created_at
|
||||
FROM chats AS c
|
||||
JOIN chat_members AS cm
|
||||
ON cm.chat_id = c.id
|
||||
WHERE c.id = :chat_id
|
||||
AND cm.user_id = :current_user_id
|
||||
"""
|
||||
),
|
||||
{
|
||||
"chat_id": chat_id,
|
||||
"current_user_id": current_user_id,
|
||||
},
|
||||
)
|
||||
|
||||
chat_row = chat_result.one_or_none()
|
||||
|
||||
if chat_row is None:
|
||||
return None
|
||||
|
||||
members_result = await session.execute(
|
||||
text(
|
||||
"""
|
||||
SELECT user_id
|
||||
FROM chat_members
|
||||
WHERE chat_id = :chat_id
|
||||
ORDER BY user_id
|
||||
"""
|
||||
),
|
||||
{
|
||||
"chat_id": chat_id,
|
||||
},
|
||||
)
|
||||
|
||||
member_ids = [
|
||||
UUID(str(row.user_id))
|
||||
for row in members_result
|
||||
]
|
||||
|
||||
return Chat(
|
||||
id=UUID(str(chat_row.id)),
|
||||
title=chat_row.title,
|
||||
member_ids=member_ids,
|
||||
created_at=chat_row.created_at,
|
||||
)
|
||||
|
||||
|
||||
async def list_chats(
|
||||
session: AsyncSession,
|
||||
current_user_id: UUID,
|
||||
limit: int,
|
||||
cursor: str | None = None,
|
||||
) -> ChatPage:
|
||||
cursor_created_at = None
|
||||
cursor_chat_id = None
|
||||
if cursor is not None:
|
||||
try:
|
||||
cursor_user_id, cursor_created_at, cursor_chat_id = decode_chat_cursor(cursor)
|
||||
except ValueError as exc:
|
||||
raise APIError(status_code=400, code="invalid_request", message="Invalid cursor") from exc
|
||||
if cursor_user_id != current_user_id:
|
||||
raise APIError(status_code=400, code="invalid_request", message="Invalid cursor")
|
||||
|
||||
result = await session.execute(
|
||||
text(
|
||||
"""
|
||||
SELECT c.id, c.title, c.created_at
|
||||
FROM chats AS c
|
||||
JOIN chat_members AS cm ON cm.chat_id = c.id
|
||||
WHERE cm.user_id = :current_user_id
|
||||
AND (
|
||||
CAST(:cursor_created_at AS timestamptz) IS NULL
|
||||
OR (c.created_at, c.id) > (
|
||||
CAST(:cursor_created_at AS timestamptz),
|
||||
CAST(:cursor_chat_id AS uuid)
|
||||
)
|
||||
)
|
||||
ORDER BY c.created_at, c.id
|
||||
LIMIT :fetch_limit
|
||||
"""
|
||||
),
|
||||
{
|
||||
"current_user_id": current_user_id,
|
||||
"cursor_created_at": cursor_created_at,
|
||||
"cursor_chat_id": cursor_chat_id,
|
||||
"fetch_limit": limit + 1,
|
||||
},
|
||||
)
|
||||
rows = result.all()
|
||||
has_more = len(rows) > limit
|
||||
page_rows = rows[:limit]
|
||||
items = []
|
||||
for chat_row in page_rows:
|
||||
members_result = await session.execute(
|
||||
text(
|
||||
"""
|
||||
SELECT user_id
|
||||
FROM chat_members
|
||||
WHERE chat_id = :chat_id
|
||||
ORDER BY user_id
|
||||
"""
|
||||
),
|
||||
{"chat_id": chat_row.id},
|
||||
)
|
||||
items.append(
|
||||
Chat(
|
||||
id=UUID(str(chat_row.id)),
|
||||
title=chat_row.title,
|
||||
member_ids=[UUID(str(row.user_id)) for row in members_result],
|
||||
created_at=chat_row.created_at,
|
||||
)
|
||||
)
|
||||
next_cursor = None
|
||||
if has_more:
|
||||
last = page_rows[-1]
|
||||
next_cursor = encode_chat_cursor(
|
||||
current_user_id, last.created_at, UUID(str(last.id))
|
||||
)
|
||||
return ChatPage(items=items, next_cursor=next_cursor)
|
||||
|
||||
|
||||
async def create_message(
|
||||
session: AsyncSession,
|
||||
chat_id: UUID,
|
||||
current_user_id: UUID,
|
||||
data: CreateMessage,
|
||||
idempotency_key: str,
|
||||
) -> Message:
|
||||
membership_result = await session.execute(
|
||||
text("""
|
||||
SELECT 1
|
||||
FROM chat_members
|
||||
WHERE chat_id = :chat_id
|
||||
AND user_id = :user_id
|
||||
"""),
|
||||
{
|
||||
"chat_id": chat_id,
|
||||
"user_id": current_user_id,
|
||||
},
|
||||
)
|
||||
|
||||
if membership_result.scalar_one_or_none() is None:
|
||||
raise APIError(
|
||||
status_code=404,
|
||||
code="not_found",
|
||||
message="Chat not found",
|
||||
)
|
||||
|
||||
canonical_body = json.dumps(
|
||||
{
|
||||
"text": data.text,
|
||||
"attachment_ids": [str(attachment_id) for attachment_id in data.attachment_ids],
|
||||
},
|
||||
sort_keys=True,
|
||||
ensure_ascii=False,
|
||||
separators=(",", ":"),
|
||||
)
|
||||
|
||||
request_hash = hashlib.sha256(
|
||||
canonical_body.encode("utf-8")
|
||||
).hexdigest()
|
||||
|
||||
scope = (
|
||||
f"{current_user_id}:POST:"
|
||||
f"/api/v1/chats/{chat_id}/messages:"
|
||||
f"{idempotency_key}"
|
||||
)
|
||||
|
||||
await session.execute(
|
||||
text("""
|
||||
SELECT pg_advisory_xact_lock(
|
||||
hashtextextended(:scope, 0)
|
||||
)
|
||||
"""),
|
||||
{"scope": scope},
|
||||
)
|
||||
|
||||
await session.execute(
|
||||
text("""
|
||||
DELETE FROM message_idempotency
|
||||
WHERE user_id = :user_id
|
||||
AND chat_id = :chat_id
|
||||
AND key = :key
|
||||
AND created_at <=
|
||||
clock_timestamp() - INTERVAL '24 hours'
|
||||
"""),
|
||||
{
|
||||
"user_id": current_user_id,
|
||||
"chat_id": chat_id,
|
||||
"key": idempotency_key,
|
||||
},
|
||||
)
|
||||
|
||||
existing_result = await session.execute(
|
||||
text("""
|
||||
SELECT
|
||||
i.request_hash,
|
||||
m.id,
|
||||
m.chat_id,
|
||||
m.sender_id,
|
||||
m.text,
|
||||
m.created_at
|
||||
FROM message_idempotency AS i
|
||||
JOIN messages AS m
|
||||
ON m.id = i.message_id
|
||||
WHERE i.user_id = :user_id
|
||||
AND i.chat_id = :chat_id
|
||||
AND i.key = :key
|
||||
"""),
|
||||
{
|
||||
"user_id": current_user_id,
|
||||
"chat_id": chat_id,
|
||||
"key": idempotency_key,
|
||||
},
|
||||
)
|
||||
|
||||
existing = existing_result.one_or_none()
|
||||
|
||||
if existing is not None:
|
||||
if existing.request_hash != request_hash:
|
||||
raise APIError(
|
||||
status_code=409,
|
||||
code="conflict",
|
||||
message=(
|
||||
"Idempotency key already used "
|
||||
"with different request body"
|
||||
),
|
||||
)
|
||||
|
||||
attachment_result = await session.execute(
|
||||
text("""
|
||||
SELECT attachment_id
|
||||
FROM message_attachments
|
||||
WHERE message_id = :message_id
|
||||
ORDER BY attachment_id
|
||||
"""),
|
||||
{"message_id": existing.id},
|
||||
)
|
||||
|
||||
attachment_ids = [
|
||||
UUID(str(row.attachment_id))
|
||||
for row in attachment_result.all()
|
||||
]
|
||||
|
||||
return Message(
|
||||
id=UUID(str(existing.id)),
|
||||
chat_id=UUID(str(existing.chat_id)),
|
||||
sender_id=UUID(str(existing.sender_id)),
|
||||
text=existing.text,
|
||||
created_at=existing.created_at,
|
||||
attachment_ids=attachment_ids,
|
||||
)
|
||||
|
||||
if data.attachment_ids:
|
||||
attachments_result = await session.execute(
|
||||
text("""
|
||||
SELECT id
|
||||
FROM attachments
|
||||
WHERE id = ANY (:attachment_ids)
|
||||
AND chat_id = :chat_id
|
||||
AND status != 'failed'
|
||||
"""),
|
||||
{
|
||||
"attachment_ids": data.attachment_ids,
|
||||
"chat_id": chat_id,
|
||||
},
|
||||
)
|
||||
|
||||
valid_attachment_ids = {
|
||||
UUID(str(row.id))
|
||||
for row in attachments_result.all()
|
||||
}
|
||||
|
||||
if valid_attachment_ids != set(data.attachment_ids):
|
||||
raise APIError(
|
||||
status_code=400,
|
||||
code="invalid_request",
|
||||
message="Invalid attachments for this chat",
|
||||
)
|
||||
|
||||
message_id = uuid4()
|
||||
|
||||
message_result = await session.execute(
|
||||
text("""
|
||||
INSERT INTO messages (
|
||||
id,
|
||||
chat_id,
|
||||
sender_id,
|
||||
text
|
||||
)
|
||||
VALUES (
|
||||
:id,
|
||||
:chat_id,
|
||||
:sender_id,
|
||||
:text
|
||||
)
|
||||
RETURNING
|
||||
id,
|
||||
chat_id,
|
||||
sender_id,
|
||||
text,
|
||||
created_at
|
||||
"""),
|
||||
{
|
||||
"id": message_id,
|
||||
"chat_id": chat_id,
|
||||
"sender_id": current_user_id,
|
||||
"text": data.text,
|
||||
},
|
||||
)
|
||||
|
||||
row = message_result.one()
|
||||
|
||||
for attachment_id in data.attachment_ids:
|
||||
await session.execute(
|
||||
text("""
|
||||
INSERT INTO message_attachments (message_id,
|
||||
attachment_id)
|
||||
VALUES (:message_id,
|
||||
:attachment_id)
|
||||
"""),
|
||||
{
|
||||
"message_id": message_id,
|
||||
"attachment_id": attachment_id,
|
||||
},
|
||||
)
|
||||
|
||||
await session.execute(
|
||||
text("""
|
||||
INSERT INTO message_idempotency (
|
||||
user_id,
|
||||
chat_id,
|
||||
key,
|
||||
request_hash,
|
||||
message_id
|
||||
)
|
||||
VALUES (
|
||||
:user_id,
|
||||
:chat_id,
|
||||
:key,
|
||||
:request_hash,
|
||||
:message_id
|
||||
)
|
||||
"""),
|
||||
{
|
||||
"user_id": current_user_id,
|
||||
"chat_id": chat_id,
|
||||
"key": idempotency_key,
|
||||
"request_hash": request_hash,
|
||||
"message_id": message_id,
|
||||
},
|
||||
)
|
||||
|
||||
return Message(
|
||||
id=UUID(str(row.id)),
|
||||
chat_id=UUID(str(row.chat_id)),
|
||||
sender_id=UUID(str(row.sender_id)),
|
||||
text=row.text,
|
||||
created_at=row.created_at,
|
||||
attachment_ids=data.attachment_ids,
|
||||
)
|
||||
|
||||
|
||||
async def get_messages(
|
||||
session: AsyncSession,
|
||||
chat_id: UUID,
|
||||
current_user_id: UUID,
|
||||
limit: int,
|
||||
cursor: str | None,
|
||||
) -> MessagePage:
|
||||
membership_result = await session.execute(
|
||||
text(
|
||||
"""
|
||||
SELECT 1
|
||||
FROM chat_members
|
||||
WHERE chat_id = :chat_id
|
||||
AND user_id = :current_user_id
|
||||
"""
|
||||
),
|
||||
{
|
||||
"chat_id": chat_id,
|
||||
"current_user_id": current_user_id,
|
||||
},
|
||||
)
|
||||
|
||||
if membership_result.scalar_one_or_none() is None:
|
||||
raise APIError(
|
||||
status_code=404,
|
||||
code="not_found",
|
||||
message="Chat not found",
|
||||
)
|
||||
|
||||
cursor_created_at = None
|
||||
cursor_message_id = None
|
||||
|
||||
if cursor is not None:
|
||||
try:
|
||||
(
|
||||
cursor_chat_id,
|
||||
cursor_created_at,
|
||||
cursor_message_id,
|
||||
) = decode_cursor(cursor)
|
||||
except ValueError:
|
||||
raise APIError(
|
||||
status_code=400,
|
||||
code="invalid_request",
|
||||
message="Invalid cursor",
|
||||
)
|
||||
|
||||
if cursor_chat_id != chat_id:
|
||||
raise APIError(
|
||||
status_code=400,
|
||||
code="invalid_request",
|
||||
message="Cursor belongs to another chat",
|
||||
)
|
||||
|
||||
if cursor is None:
|
||||
messages_result = await session.execute(
|
||||
text(
|
||||
"""
|
||||
SELECT
|
||||
id,
|
||||
chat_id,
|
||||
sender_id,
|
||||
text,
|
||||
created_at
|
||||
FROM messages
|
||||
WHERE chat_id = :chat_id
|
||||
ORDER BY created_at, id
|
||||
LIMIT :fetch_limit
|
||||
"""
|
||||
),
|
||||
{
|
||||
"chat_id": chat_id,
|
||||
"fetch_limit": limit + 1,
|
||||
},
|
||||
)
|
||||
else:
|
||||
messages_result = await session.execute(
|
||||
text(
|
||||
"""
|
||||
SELECT
|
||||
id,
|
||||
chat_id,
|
||||
sender_id,
|
||||
text,
|
||||
created_at
|
||||
FROM messages
|
||||
WHERE chat_id = :chat_id
|
||||
AND (created_at, id) > (
|
||||
:cursor_created_at,
|
||||
:cursor_message_id
|
||||
)
|
||||
ORDER BY created_at, id
|
||||
LIMIT :fetch_limit
|
||||
"""
|
||||
),
|
||||
{
|
||||
"chat_id": chat_id,
|
||||
"cursor_created_at": cursor_created_at,
|
||||
"cursor_message_id": cursor_message_id,
|
||||
"fetch_limit": limit + 1,
|
||||
},
|
||||
)
|
||||
|
||||
rows = messages_result.all()
|
||||
|
||||
has_more = len(rows) > limit
|
||||
rows = rows[:limit]
|
||||
|
||||
attachments_by_message = {}
|
||||
|
||||
if rows:
|
||||
attachment_result = await session.execute(
|
||||
text("""
|
||||
SELECT message_id, attachment_id
|
||||
FROM message_attachments
|
||||
WHERE message_id = ANY(:message_ids)
|
||||
ORDER BY message_id, attachment_id
|
||||
"""),
|
||||
{
|
||||
"message_ids": [row.id for row in rows],
|
||||
},
|
||||
)
|
||||
|
||||
for attachment_row in attachment_result.all():
|
||||
message_id = UUID(str(attachment_row.message_id))
|
||||
attachment_id = UUID(str(attachment_row.attachment_id))
|
||||
|
||||
attachments_by_message.setdefault(
|
||||
message_id, []
|
||||
).append(attachment_id)
|
||||
|
||||
items = [
|
||||
Message(
|
||||
id=UUID(str(row.id)),
|
||||
chat_id=UUID(str(row.chat_id)),
|
||||
sender_id=UUID(str(row.sender_id)),
|
||||
text=row.text,
|
||||
created_at=row.created_at,
|
||||
attachment_ids=attachments_by_message.get(
|
||||
UUID(str(row.id)), []
|
||||
),
|
||||
)
|
||||
for row in rows
|
||||
]
|
||||
|
||||
next_cursor = None
|
||||
|
||||
if has_more:
|
||||
last_message = items[-1]
|
||||
|
||||
next_cursor = encode_cursor(
|
||||
chat_id=chat_id,
|
||||
created_at=last_message.created_at,
|
||||
message_id=last_message.id,
|
||||
)
|
||||
|
||||
return MessagePage(
|
||||
items=items,
|
||||
next_cursor=next_cursor,
|
||||
)
|
||||
|
||||
|
||||
def attachment_from_row(row) -> Attachment:
|
||||
return Attachment(
|
||||
id=UUID(str(row.id)),
|
||||
chat_id=UUID(str(row.chat_id)),
|
||||
owner_id=UUID(str(row.owner_id)),
|
||||
filename=row.filename,
|
||||
content_type=row.content_type,
|
||||
size_bytes=row.size_bytes,
|
||||
sha256=row.sha256,
|
||||
status=row.status,
|
||||
variants=row.variants,
|
||||
error_code=row.error_code,
|
||||
created_at=row.created_at,
|
||||
updated_at=row.updated_at,
|
||||
)
|
||||
|
||||
|
||||
async def create_attachment(
|
||||
session: AsyncSession,
|
||||
attachment_id: UUID,
|
||||
chat_id: UUID,
|
||||
owner_id: UUID,
|
||||
filename: str,
|
||||
content_type: str,
|
||||
size_bytes: int,
|
||||
sha256: str,
|
||||
s3_key: str,
|
||||
) -> Attachment:
|
||||
result = await session.execute(
|
||||
text(
|
||||
"""
|
||||
INSERT INTO attachments (
|
||||
id,
|
||||
chat_id,
|
||||
owner_id,
|
||||
filename,
|
||||
content_type,
|
||||
size_bytes,
|
||||
sha256,
|
||||
s3_key,
|
||||
status
|
||||
)
|
||||
VALUES (
|
||||
:id,
|
||||
:chat_id,
|
||||
:owner_id,
|
||||
:filename,
|
||||
:content_type,
|
||||
:size_bytes,
|
||||
:sha256,
|
||||
:s3_key,
|
||||
'queued'
|
||||
)
|
||||
RETURNING
|
||||
id,
|
||||
chat_id,
|
||||
owner_id,
|
||||
filename,
|
||||
content_type,
|
||||
size_bytes,
|
||||
sha256,
|
||||
status,
|
||||
variants,
|
||||
error_code,
|
||||
created_at,
|
||||
updated_at
|
||||
"""
|
||||
),
|
||||
{
|
||||
"id": attachment_id,
|
||||
"chat_id": chat_id,
|
||||
"owner_id": owner_id,
|
||||
"filename": filename,
|
||||
"content_type": content_type,
|
||||
"size_bytes": size_bytes,
|
||||
"sha256": sha256,
|
||||
"s3_key": s3_key,
|
||||
},
|
||||
)
|
||||
|
||||
return attachment_from_row(result.one())
|
||||
|
||||
|
||||
async def get_attachment(
|
||||
session: AsyncSession,
|
||||
attachment_id: UUID,
|
||||
current_user_id: UUID,
|
||||
):
|
||||
result = await session.execute(
|
||||
text(
|
||||
"""
|
||||
SELECT
|
||||
a.id,
|
||||
a.chat_id,
|
||||
a.owner_id,
|
||||
a.filename,
|
||||
a.content_type,
|
||||
a.size_bytes,
|
||||
a.sha256,
|
||||
a.s3_key,
|
||||
a.status,
|
||||
a.variants,
|
||||
a.error_code,
|
||||
a.created_at,
|
||||
a.updated_at
|
||||
FROM attachments AS a
|
||||
JOIN chat_members AS cm
|
||||
ON cm.chat_id = a.chat_id
|
||||
WHERE a.id = :attachment_id
|
||||
AND cm.user_id = :current_user_id
|
||||
"""
|
||||
),
|
||||
{
|
||||
"attachment_id": attachment_id,
|
||||
"current_user_id": current_user_id,
|
||||
},
|
||||
)
|
||||
|
||||
row = result.one_or_none()
|
||||
|
||||
if row is None:
|
||||
return None
|
||||
|
||||
return attachment_from_row(row), row.s3_key
|
||||
|
||||
async def attachment_key_exists(session: AsyncSession, s3_key: str) -> bool:
|
||||
result = await session.execute(
|
||||
text("SELECT 1 FROM attachments WHERE s3_key = :s3_key"),
|
||||
{"s3_key": s3_key},
|
||||
)
|
||||
return result.first() is not None
|
||||
@@ -0,0 +1,16 @@
|
||||
from argon2 import PasswordHasher
|
||||
from argon2.exceptions import VerifyMismatchError
|
||||
|
||||
|
||||
password_hasher = PasswordHasher()
|
||||
|
||||
|
||||
def hash_password(password: str) -> str:
|
||||
return password_hasher.hash(password)
|
||||
|
||||
|
||||
def verify_password(password_hash: str, password: str) -> bool:
|
||||
try:
|
||||
return password_hasher.verify(password_hash, password)
|
||||
except VerifyMismatchError:
|
||||
return False
|
||||
@@ -0,0 +1,87 @@
|
||||
import os
|
||||
import secrets
|
||||
from uuid import UUID
|
||||
|
||||
from redis.asyncio import Redis
|
||||
from redis.exceptions import RedisError
|
||||
|
||||
from src.errors import APIError
|
||||
|
||||
|
||||
CACHE_URL_FILE = "/run/secrets/cache_url"
|
||||
DEFAULT_SESSION_TTL_SECONDS = 3600
|
||||
|
||||
|
||||
def read_cache_url() -> str:
|
||||
with open(CACHE_URL_FILE, encoding="utf-8") as file:
|
||||
return file.read().strip()
|
||||
|
||||
|
||||
def get_session_ttl_seconds() -> int:
|
||||
return int(
|
||||
os.getenv(
|
||||
"SESSION_TTL_SECONDS",
|
||||
str(DEFAULT_SESSION_TTL_SECONDS),
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
redis_client = Redis.from_url(
|
||||
read_cache_url(),
|
||||
decode_responses=True,
|
||||
socket_connect_timeout=5,
|
||||
socket_timeout=5,
|
||||
)
|
||||
|
||||
|
||||
def session_store_unavailable() -> APIError:
|
||||
return APIError(
|
||||
status_code=503,
|
||||
code="unavailable",
|
||||
message="Session store unavailable",
|
||||
)
|
||||
|
||||
|
||||
async def create_session(user_id: UUID) -> tuple[str, int]:
|
||||
token = secrets.token_urlsafe(32)
|
||||
ttl = get_session_ttl_seconds()
|
||||
|
||||
try:
|
||||
await redis_client.set(
|
||||
f"session:{token}",
|
||||
str(user_id),
|
||||
ex=ttl,
|
||||
)
|
||||
except RedisError as exc:
|
||||
raise session_store_unavailable() from exc
|
||||
|
||||
return token, ttl
|
||||
|
||||
|
||||
async def get_session_user_id(token: str) -> UUID | None:
|
||||
try:
|
||||
value = await redis_client.get(f"session:{token}")
|
||||
except RedisError as exc:
|
||||
raise session_store_unavailable() from exc
|
||||
|
||||
if value is None:
|
||||
return None
|
||||
|
||||
try:
|
||||
return UUID(value)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
async def delete_session(token: str) -> None:
|
||||
try:
|
||||
await redis_client.delete(f"session:{token}")
|
||||
except RedisError as exc:
|
||||
raise session_store_unavailable() from exc
|
||||
|
||||
|
||||
async def ping_session_store() -> bool:
|
||||
try:
|
||||
return bool(await redis_client.ping())
|
||||
except RedisError:
|
||||
return False
|
||||
@@ -0,0 +1,87 @@
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import boto3
|
||||
from botocore.config import Config
|
||||
|
||||
|
||||
def read_secret(env_name: str) -> str:
|
||||
path = os.environ[env_name]
|
||||
return Path(path).read_text(encoding="utf-8").strip()
|
||||
|
||||
|
||||
s3_client = boto3.client(
|
||||
"s3",
|
||||
endpoint_url=os.environ["S3_ENDPOINT"],
|
||||
region_name=os.getenv("S3_REGION", "us-east-1"),
|
||||
aws_access_key_id=read_secret("S3_ACCESS_KEY_ID_FILE"),
|
||||
aws_secret_access_key=read_secret("S3_SECRET_ACCESS_KEY_FILE"),
|
||||
config=Config(
|
||||
signature_version="s3v4",
|
||||
connect_timeout=2,
|
||||
read_timeout=3,
|
||||
retries={"total_max_attempts": 1, "mode": "standard"},
|
||||
s3={"addressing_style": "path"},
|
||||
),
|
||||
)
|
||||
|
||||
S3_BUCKET = os.environ["S3_BUCKET"]
|
||||
|
||||
|
||||
async def put_object(key: str, body: bytes, content_type: str) -> None:
|
||||
await asyncio.to_thread(
|
||||
s3_client.put_object,
|
||||
Bucket=S3_BUCKET,
|
||||
Key=key,
|
||||
Body=body,
|
||||
ContentType=content_type,
|
||||
)
|
||||
|
||||
|
||||
async def get_object(key: str) -> bytes:
|
||||
def download() -> bytes:
|
||||
response = s3_client.get_object(Bucket=S3_BUCKET, Key=key)
|
||||
stream = response["Body"]
|
||||
try:
|
||||
return stream.read()
|
||||
finally:
|
||||
stream.close()
|
||||
|
||||
return await asyncio.to_thread(download)
|
||||
|
||||
|
||||
async def delete_object(key: str) -> None:
|
||||
await asyncio.to_thread(
|
||||
s3_client.delete_object,
|
||||
Bucket=S3_BUCKET,
|
||||
Key=key,
|
||||
)
|
||||
|
||||
|
||||
async def ping_storage() -> bool:
|
||||
try:
|
||||
await asyncio.to_thread(
|
||||
s3_client.head_bucket,
|
||||
Bucket=S3_BUCKET,
|
||||
)
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
async def object_exists(key: str) -> bool:
|
||||
from botocore.exceptions import ClientError
|
||||
|
||||
try:
|
||||
await asyncio.to_thread(
|
||||
s3_client.head_object,
|
||||
Bucket=S3_BUCKET,
|
||||
Key=key,
|
||||
)
|
||||
return True
|
||||
except ClientError as exc:
|
||||
if exc.response["Error"]["Code"] in {"404", "NoSuchKey", "NotFound"}:
|
||||
return False
|
||||
raise
|
||||
+450
@@ -0,0 +1,450 @@
|
||||
import asyncio
|
||||
import hashlib
|
||||
import io
|
||||
import os
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
import random
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from PIL import Image, ImageOps, UnidentifiedImageError
|
||||
from sqlalchemy import text
|
||||
|
||||
from src.database import SessionFactory
|
||||
from src.queue import QUEUE, connect
|
||||
from src.storage import get_object, put_object
|
||||
from src.images import MAX_IMAGE_PIXELS, MAX_UPLOAD_BYTES
|
||||
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
|
||||
MAX_PROCESSING_SECONDS = float(
|
||||
os.getenv("MAX_PROCESSING_SECONDS", "30")
|
||||
)
|
||||
|
||||
|
||||
class PermanentProcessingError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def retry_delay(attempt: int) -> float:
|
||||
base = min(60.0, 2.0 ** attempt)
|
||||
return base * random.uniform(0.75, 1.25)
|
||||
|
||||
|
||||
def validate_worker_image(body: bytes):
|
||||
if not body or len(body) > MAX_UPLOAD_BYTES:
|
||||
raise PermanentProcessingError("invalid_image_size")
|
||||
|
||||
try:
|
||||
with Image.open(io.BytesIO(body)) as image:
|
||||
if image.format not in {"PNG", "JPEG"}:
|
||||
raise PermanentProcessingError("unsupported_image_format")
|
||||
|
||||
if image.width * image.height > MAX_IMAGE_PIXELS:
|
||||
raise PermanentProcessingError("image_pixel_limit")
|
||||
|
||||
image.verify()
|
||||
|
||||
except (UnidentifiedImageError, OSError, ValueError, Image.DecompressionBombError) as exc:
|
||||
raise PermanentProcessingError("invalid_image") from exc
|
||||
|
||||
|
||||
def make_thumbnail(body: bytes):
|
||||
with Image.open(io.BytesIO(body)) as source:
|
||||
source.load()
|
||||
image = ImageOps.exif_transpose(source)
|
||||
image.thumbnail((256, 256), Image.Resampling.LANCZOS)
|
||||
|
||||
if image.mode != "RGB":
|
||||
rgba = image.convert("RGBA")
|
||||
background = Image.new("RGBA", rgba.size, "white")
|
||||
background.alpha_composite(rgba)
|
||||
image = background.convert("RGB")
|
||||
|
||||
output = io.BytesIO()
|
||||
image.save(output, format="JPEG", quality=85, optimize=True)
|
||||
|
||||
return output.getvalue(), image.width, image.height
|
||||
|
||||
|
||||
def make_crop(body: bytes):
|
||||
with Image.open(io.BytesIO(body)) as source:
|
||||
source.load()
|
||||
image = ImageOps.exif_transpose(source).convert("RGB")
|
||||
image = ImageOps.fit(
|
||||
image,
|
||||
(128, 128),
|
||||
method=Image.Resampling.LANCZOS,
|
||||
centering=(0.5, 0.5),
|
||||
)
|
||||
|
||||
output = io.BytesIO()
|
||||
image.save(output, format="JPEG", quality=85, optimize=True)
|
||||
|
||||
return output.getvalue(), image.width, image.height
|
||||
|
||||
|
||||
def make_optimized(body: bytes):
|
||||
with Image.open(io.BytesIO(body)) as source:
|
||||
source.load()
|
||||
image = ImageOps.exif_transpose(source).convert("RGB")
|
||||
|
||||
output = io.BytesIO()
|
||||
image.save(output, format="WEBP", quality=80, method=4)
|
||||
|
||||
return output.getvalue(), image.width, image.height
|
||||
|
||||
|
||||
def fault_injection(point: str):
|
||||
if os.getenv("FAULT_INJECTION_ENABLED") != "1":
|
||||
return
|
||||
|
||||
if os.getenv("FAULT_INJECTION_POINT") != point:
|
||||
return
|
||||
|
||||
marker = f"/tmp/lab5_fault_{point}"
|
||||
|
||||
try:
|
||||
fd = os.open(marker, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
|
||||
except FileExistsError:
|
||||
return
|
||||
|
||||
os.close(fd)
|
||||
logging.error("FAULT INJECTION: %s", point)
|
||||
os._exit(77)
|
||||
|
||||
|
||||
async def process(payload: dict):
|
||||
job_id = UUID(payload["job_id"])
|
||||
attachment_id = UUID(payload["attachment_id"])
|
||||
generation = payload.get("generation", 1)
|
||||
execution_id = uuid4()
|
||||
|
||||
|
||||
async with SessionFactory() as session:
|
||||
async with session.begin():
|
||||
result = await session.execute(
|
||||
text("""
|
||||
SELECT j.status, j.attempt, j.generation,
|
||||
a.s3_key, a.content_type, a.chat_id
|
||||
FROM attachment_jobs j
|
||||
JOIN attachments a ON a.id = j.attachment_id
|
||||
WHERE j.id = :job_id
|
||||
AND j.attachment_id = :attachment_id
|
||||
FOR UPDATE OF j
|
||||
"""),
|
||||
{"job_id": job_id, "attachment_id": attachment_id},
|
||||
)
|
||||
|
||||
row = result.one_or_none()
|
||||
|
||||
if row is None or row.status in ("ready", "failed"):
|
||||
return
|
||||
|
||||
if row.attempt != payload["attempt"]:
|
||||
return
|
||||
|
||||
if row.generation != generation:
|
||||
return
|
||||
|
||||
if row.status == "processing":
|
||||
return
|
||||
|
||||
await session.execute(
|
||||
text("""
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'processing',
|
||||
started_at = now(),
|
||||
execution_id = :execution_id
|
||||
WHERE id = :id
|
||||
"""),
|
||||
{"id": job_id, "execution_id": execution_id},
|
||||
)
|
||||
|
||||
await session.execute(
|
||||
text("""
|
||||
UPDATE attachments
|
||||
SET status = 'processing', updated_at = now()
|
||||
WHERE id = :id
|
||||
"""),
|
||||
{"id": attachment_id},
|
||||
)
|
||||
|
||||
try:
|
||||
body = await get_object(row.s3_key)
|
||||
validate_worker_image(body)
|
||||
thumbnail, width, height = await asyncio.to_thread(
|
||||
make_thumbnail, body
|
||||
)
|
||||
|
||||
crop, crop_width, crop_height = await asyncio.to_thread(
|
||||
make_crop, body
|
||||
)
|
||||
|
||||
optimized, optimized_width, optimized_height = await asyncio.to_thread(
|
||||
make_optimized, body
|
||||
)
|
||||
|
||||
original_key = (
|
||||
f"attachments/{row.chat_id}/{attachment_id}/g{generation}/original"
|
||||
)
|
||||
thumbnail_key = (
|
||||
f"variants/{row.chat_id}/{attachment_id}/g{generation}/thumbnail.jpg"
|
||||
)
|
||||
|
||||
fault_injection("before_side_effect")
|
||||
|
||||
await put_object(original_key, body, row.content_type)
|
||||
await put_object(thumbnail_key, thumbnail, "image/jpeg")
|
||||
crop_key = (
|
||||
f"variants/{row.chat_id}/{attachment_id}/g{generation}/crop.jpg"
|
||||
)
|
||||
optimized_key = (
|
||||
f"variants/{row.chat_id}/{attachment_id}/g{generation}/optimized.webp"
|
||||
)
|
||||
|
||||
await put_object(crop_key, crop, "image/jpeg")
|
||||
await put_object(optimized_key, optimized, "image/webp")
|
||||
|
||||
fault_injection("after_side_effect")
|
||||
|
||||
|
||||
if (
|
||||
os.getenv("FAULT_INJECTION_ENABLED") == "1"
|
||||
and os.getenv("FAULT_INJECTION_POINT") == "pause_before_commit"
|
||||
):
|
||||
marker = "/tmp/lab5_pause_before_commit"
|
||||
|
||||
with open(marker, "w", encoding="utf-8") as file:
|
||||
file.write(str(execution_id))
|
||||
|
||||
deadline = time.monotonic() + 180
|
||||
|
||||
while os.path.exists(marker):
|
||||
if time.monotonic() >= deadline:
|
||||
raise TimeoutError("pause_before_commit_timeout")
|
||||
await asyncio.sleep(0.1)
|
||||
|
||||
|
||||
variant = {
|
||||
"name": "thumbnail",
|
||||
"s3_key": thumbnail_key,
|
||||
"content_type": "image/jpeg",
|
||||
"size_bytes": len(thumbnail),
|
||||
"width": width,
|
||||
"height": height,
|
||||
"sha256": hashlib.sha256(thumbnail).hexdigest(),
|
||||
}
|
||||
|
||||
async with SessionFactory() as session:
|
||||
async with session.begin():
|
||||
result = await session.execute(
|
||||
text("""
|
||||
SELECT status, attempt, generation, execution_id
|
||||
FROM attachment_jobs
|
||||
WHERE id = :id
|
||||
FOR UPDATE
|
||||
"""),
|
||||
{"id": job_id},
|
||||
)
|
||||
|
||||
current = result.one_or_none()
|
||||
|
||||
if (
|
||||
current is None
|
||||
or current.status != "processing"
|
||||
or current.attempt != row.attempt
|
||||
or current.generation != generation
|
||||
or current.execution_id != execution_id
|
||||
):
|
||||
return
|
||||
await session.execute(
|
||||
text("""
|
||||
UPDATE attachments
|
||||
SET status = 'ready',
|
||||
s3_key = :key,
|
||||
variants = CAST(:variants AS json),
|
||||
error_code = NULL,
|
||||
updated_at = now()
|
||||
WHERE id = :id
|
||||
"""),
|
||||
{
|
||||
"id": attachment_id,
|
||||
"key": original_key,
|
||||
"variants": json.dumps([
|
||||
variant,
|
||||
{
|
||||
"name": "crop",
|
||||
"s3_key": crop_key,
|
||||
"content_type": "image/jpeg",
|
||||
"size_bytes": len(crop),
|
||||
"width": crop_width,
|
||||
"height": crop_height,
|
||||
"sha256": hashlib.sha256(crop).hexdigest(),
|
||||
},
|
||||
{
|
||||
"name": "optimized",
|
||||
"s3_key": optimized_key,
|
||||
"content_type": "image/webp",
|
||||
"size_bytes": len(optimized),
|
||||
"width": optimized_width,
|
||||
"height": optimized_height,
|
||||
"sha256": hashlib.sha256(optimized).hexdigest(),
|
||||
},
|
||||
]),
|
||||
},
|
||||
)
|
||||
|
||||
await session.execute(
|
||||
text("""
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'ready',
|
||||
next_attempt_at = NULL,
|
||||
last_error_code = NULL,
|
||||
finished_at = now()
|
||||
WHERE id = :id
|
||||
"""),
|
||||
{"id": job_id},
|
||||
)
|
||||
|
||||
|
||||
except Exception as exc:
|
||||
|
||||
logging.exception("Processing failed: %s", attachment_id)
|
||||
|
||||
permanent = isinstance(exc, PermanentProcessingError)
|
||||
|
||||
error_code = (
|
||||
|
||||
str(exc)[:64]
|
||||
|
||||
if permanent
|
||||
|
||||
else "temporary_processing_error"
|
||||
|
||||
)
|
||||
|
||||
async with SessionFactory() as session:
|
||||
async with session.begin():
|
||||
result = await session.execute(
|
||||
text("""
|
||||
SELECT status, attempt, generation, execution_id
|
||||
FROM attachment_jobs
|
||||
WHERE id = :id
|
||||
FOR UPDATE
|
||||
"""),
|
||||
{"id": job_id},
|
||||
)
|
||||
|
||||
current = result.one_or_none()
|
||||
|
||||
if (
|
||||
current is None
|
||||
or current.status != "processing"
|
||||
or current.attempt != row.attempt
|
||||
or current.generation != generation
|
||||
or current.execution_id != execution_id
|
||||
):
|
||||
return
|
||||
if permanent or row.attempt >= 5:
|
||||
await session.execute(
|
||||
text("""
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'failed',
|
||||
next_attempt_at = NULL,
|
||||
last_error_code = :error_code,
|
||||
finished_at = now()
|
||||
WHERE id = :id
|
||||
"""),
|
||||
{"id": job_id, "error_code": error_code},
|
||||
)
|
||||
|
||||
await session.execute(
|
||||
text("""
|
||||
UPDATE attachments
|
||||
SET status = 'failed',
|
||||
error_code = :error_code,
|
||||
updated_at = now()
|
||||
WHERE id = :id
|
||||
"""),
|
||||
{
|
||||
"id": attachment_id,
|
||||
"error_code": error_code,
|
||||
},
|
||||
)
|
||||
else:
|
||||
await session.execute(
|
||||
text("""
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'queued',
|
||||
attempt = attempt + 1,
|
||||
published_at = NULL,
|
||||
started_at = NULL,
|
||||
next_attempt_at = now() + (:delay * interval '1 second'),
|
||||
last_error_code = :error_code
|
||||
WHERE id = :id
|
||||
"""),
|
||||
{
|
||||
"id": job_id,
|
||||
"delay": retry_delay(row.attempt),
|
||||
"error_code": error_code,
|
||||
}
|
||||
)
|
||||
|
||||
await session.execute(
|
||||
text("""
|
||||
UPDATE attachments
|
||||
SET status = 'queued', updated_at = now()
|
||||
WHERE id = :id
|
||||
"""),
|
||||
{"id": attachment_id},
|
||||
)
|
||||
|
||||
|
||||
def main():
|
||||
while True:
|
||||
connection = None
|
||||
|
||||
try:
|
||||
connection, channel = connect()
|
||||
channel.basic_qos(prefetch_count=1)
|
||||
|
||||
loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(loop)
|
||||
|
||||
def callback(ch, method, properties, body):
|
||||
try:
|
||||
loop.run_until_complete(
|
||||
asyncio.wait_for(
|
||||
process(json.loads(body)),
|
||||
timeout=MAX_PROCESSING_SECONDS,
|
||||
)
|
||||
)
|
||||
fault_injection("before_ack")
|
||||
ch.basic_ack(delivery_tag=method.delivery_tag)
|
||||
except Exception:
|
||||
logging.exception("Worker processing error")
|
||||
ch.basic_nack(
|
||||
delivery_tag=method.delivery_tag,
|
||||
requeue=True,
|
||||
)
|
||||
time.sleep(2)
|
||||
|
||||
channel.basic_consume(
|
||||
queue=QUEUE,
|
||||
on_message_callback=callback,
|
||||
)
|
||||
channel.start_consuming()
|
||||
|
||||
except Exception:
|
||||
logging.exception("Worker reconnect")
|
||||
time.sleep(3)
|
||||
|
||||
finally:
|
||||
if connection and connection.is_open:
|
||||
connection.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,160 @@
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
from client import Client, check
|
||||
from fault_injection_grade5 import job_state
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
COMPOSE = ["docker", "compose", "-f", "compose.yaml", "-f", "compose.concurrent.yaml"]
|
||||
MARKER = "/tmp/lab5_pause_before_commit"
|
||||
|
||||
|
||||
def run(*args, check_result=True):
|
||||
return subprocess.run(
|
||||
args,
|
||||
cwd=ROOT,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=check_result,
|
||||
)
|
||||
|
||||
|
||||
def compose(*args):
|
||||
return run(*COMPOSE, *args)
|
||||
|
||||
|
||||
def sql(query):
|
||||
result = run(
|
||||
"docker", "compose", "exec", "-T", "postgres",
|
||||
"psql", "-U", "lab2", "-d", "messenger",
|
||||
"-t", "-A", "-c", query,
|
||||
)
|
||||
return result.stdout.strip()
|
||||
|
||||
|
||||
def wait_until(predicate, timeout, message):
|
||||
deadline = time.monotonic() + timeout
|
||||
|
||||
while time.monotonic() < deadline:
|
||||
if predicate():
|
||||
return
|
||||
time.sleep(0.25)
|
||||
|
||||
raise AssertionError(message)
|
||||
|
||||
|
||||
def marker_exists():
|
||||
result = run(
|
||||
*COMPOSE,
|
||||
"exec", "-T", "worker",
|
||||
"test", "-f", MARKER,
|
||||
check_result=False,
|
||||
)
|
||||
return result.returncode == 0
|
||||
|
||||
|
||||
def release_worker():
|
||||
compose("exec", "-T", "worker", "rm", "-f", MARKER)
|
||||
|
||||
|
||||
def main():
|
||||
try:
|
||||
compose("up", "-d", "--build", "--force-recreate", "worker")
|
||||
|
||||
client = Client()
|
||||
alice, bob, _, chat = client.bootstrap()
|
||||
attachment, original = client.attachment(alice, chat)
|
||||
attachment_id = attachment["id"]
|
||||
|
||||
wait_until(
|
||||
marker_exists,
|
||||
60,
|
||||
"first worker did not pause",
|
||||
)
|
||||
|
||||
first = job_state(attachment_id)
|
||||
|
||||
check(first["job_status"] == "processing", "first execution not processing")
|
||||
check(first["execution_id"] is not None, "missing execution ID")
|
||||
|
||||
sql(f"""
|
||||
UPDATE attachment_jobs
|
||||
SET started_at = now() - interval '120 seconds'
|
||||
WHERE id = '{first["job_id"]}'
|
||||
AND status = 'processing';
|
||||
""")
|
||||
|
||||
wait_until(
|
||||
lambda: (
|
||||
job_state(attachment_id)["execution_id"]
|
||||
!= first["execution_id"]
|
||||
),
|
||||
15,
|
||||
"dispatcher did not release the old execution",
|
||||
)
|
||||
|
||||
compose("up", "-d", "--build", "worker-second")
|
||||
|
||||
deadline = time.monotonic() + 120
|
||||
|
||||
while time.monotonic() < deadline:
|
||||
state = job_state(attachment_id)
|
||||
|
||||
if state["job_status"] == "ready":
|
||||
break
|
||||
|
||||
time.sleep(0.25)
|
||||
else:
|
||||
raise AssertionError("second worker did not complete the job")
|
||||
|
||||
check(state["execution_id"] != first["execution_id"], "execution ID unchanged")
|
||||
check(state["attachment_status"] == "ready", "attachment not ready")
|
||||
|
||||
release_worker()
|
||||
|
||||
time.sleep(3)
|
||||
|
||||
after = job_state(attachment_id)
|
||||
|
||||
check(after == state, "old execution overwrote newer result")
|
||||
|
||||
content = client.call(
|
||||
"GET",
|
||||
f"/api/v1/attachments/{attachment_id}/content",
|
||||
actor=bob,
|
||||
)
|
||||
|
||||
check(content["bytes"] == original, "original content changed")
|
||||
|
||||
print("PASS: stale execution rejected after newer execution completed")
|
||||
|
||||
finally:
|
||||
try:
|
||||
release_worker()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
try:
|
||||
compose("stop", "worker-second")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
run(
|
||||
"docker", "compose",
|
||||
"up", "-d", "--force-recreate", "worker",
|
||||
)
|
||||
|
||||
print("Normal worker configuration restored")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except Exception as exc:
|
||||
print(f"FAIL: {exc}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
@@ -0,0 +1,129 @@
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
from client import Client, check
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
OVERRIDE = ROOT / "compose.fault.yaml"
|
||||
POINTS = ("before_side_effect", "after_side_effect", "before_ack")
|
||||
CONTAINER = "mtusi2026lab5-worker-1"
|
||||
|
||||
|
||||
def command(*args):
|
||||
return subprocess.run(
|
||||
args,
|
||||
cwd=ROOT,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
|
||||
|
||||
def configure(point):
|
||||
OVERRIDE.write_text(
|
||||
"services:\n"
|
||||
" worker:\n"
|
||||
" environment:\n"
|
||||
' FAULT_INJECTION_ENABLED: "1"\n'
|
||||
f" FAULT_INJECTION_POINT: {point}\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
command(
|
||||
"docker", "compose",
|
||||
"-f", "compose.yaml",
|
||||
"-f", "compose.fault.yaml",
|
||||
"up", "-d", "--force-recreate", "worker",
|
||||
)
|
||||
|
||||
|
||||
def restore():
|
||||
command(
|
||||
"docker", "compose",
|
||||
"up", "-d", "--force-recreate", "worker",
|
||||
)
|
||||
|
||||
|
||||
def restart_count():
|
||||
result = command(
|
||||
"docker", "inspect",
|
||||
"-f", "{{.RestartCount}}",
|
||||
CONTAINER,
|
||||
)
|
||||
return int(result.stdout.strip())
|
||||
|
||||
|
||||
def crash_logged(point):
|
||||
result = command(
|
||||
"docker", "logs", CONTAINER,
|
||||
)
|
||||
return f"FAULT INJECTION: {point}" in result.stderr + result.stdout
|
||||
|
||||
|
||||
def run_case(point):
|
||||
print(f"RUN: {point}", flush=True)
|
||||
|
||||
configure(point)
|
||||
|
||||
client = Client()
|
||||
alice, bob, _, chat = client.bootstrap()
|
||||
attachment, original = client.attachment(alice, chat)
|
||||
|
||||
deadline = time.monotonic() + 30
|
||||
|
||||
while time.monotonic() < deadline:
|
||||
if restart_count() >= 1 and crash_logged(point):
|
||||
break
|
||||
time.sleep(0.5)
|
||||
else:
|
||||
raise AssertionError(f"{point}: worker did not crash")
|
||||
|
||||
ready = client.wait_ready(bob, attachment, wait_seconds=150)
|
||||
|
||||
check(ready["status"] == "ready", f"{point}: not ready")
|
||||
|
||||
content = client.call(
|
||||
"GET",
|
||||
f"/api/v1/attachments/{attachment['id']}/content",
|
||||
actor=bob,
|
||||
)
|
||||
|
||||
check(content["bytes"] == original, f"{point}: original changed")
|
||||
|
||||
for variant in ("thumbnail", "crop", "optimized"):
|
||||
response = client.call(
|
||||
"GET",
|
||||
f"/api/v1/attachments/{attachment['id']}/content?variant={variant}",
|
||||
actor=bob,
|
||||
)
|
||||
check(bool(response["bytes"]), f"{point}: {variant} is empty")
|
||||
|
||||
check(restart_count() == 1, f"{point}: unexpected restart count")
|
||||
|
||||
print(
|
||||
f"PASS: {point}, recovered, original and variants intact",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
|
||||
def main():
|
||||
try:
|
||||
for point in POINTS:
|
||||
run_case(point)
|
||||
finally:
|
||||
print("Restoring normal worker configuration...", flush=True)
|
||||
restore()
|
||||
print("FAULT INJECTION DISABLED", flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except (AssertionError, subprocess.CalledProcessError, OSError) as exc:
|
||||
print(f"FAIL: {exc}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
@@ -0,0 +1,463 @@
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import pika
|
||||
|
||||
from client import Client, check
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
BASE_URL = "http://localhost:8080"
|
||||
|
||||
|
||||
def sql(query):
|
||||
result = subprocess.run(
|
||||
[
|
||||
"docker", "compose", "exec", "-T", "postgres",
|
||||
"psql", "-U", "lab2", "-d", "messenger",
|
||||
"-t", "-A", "-c", query,
|
||||
],
|
||||
cwd=ROOT,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
return result.stdout.strip()
|
||||
|
||||
|
||||
def job_state(attachment_id):
|
||||
result = sql(f"""
|
||||
SELECT row_to_json(t)
|
||||
FROM (
|
||||
SELECT
|
||||
j.id AS job_id,
|
||||
j.attachment_id,
|
||||
j.status AS job_status,
|
||||
a.status AS attachment_status,
|
||||
j.attempt,
|
||||
j.generation,
|
||||
j.execution_id,
|
||||
j.finished_at,
|
||||
a.s3_key,
|
||||
a.variants
|
||||
FROM attachment_jobs j
|
||||
JOIN attachments a ON a.id = j.attachment_id
|
||||
WHERE a.id = '{attachment_id}'
|
||||
) t;
|
||||
""")
|
||||
check(bool(result), "job not found")
|
||||
return json.loads(result)
|
||||
|
||||
|
||||
def publish_duplicate(payload):
|
||||
result = subprocess.run(
|
||||
[
|
||||
"docker", "compose", "exec", "-T", "worker",
|
||||
"python", "-c",
|
||||
"""
|
||||
import json
|
||||
import sys
|
||||
import pika
|
||||
from src.queue import connect, QUEUE
|
||||
|
||||
payload = json.loads(sys.stdin.read())
|
||||
connection, channel = connect()
|
||||
channel.confirm_delivery()
|
||||
channel.basic_publish(
|
||||
exchange="",
|
||||
routing_key=QUEUE,
|
||||
body=json.dumps(payload).encode(),
|
||||
properties=pika.BasicProperties(
|
||||
delivery_mode=2,
|
||||
content_type="application/json",
|
||||
message_id=payload["job_id"],
|
||||
),
|
||||
mandatory=True,
|
||||
)
|
||||
connection.close()
|
||||
""",
|
||||
],
|
||||
cwd=ROOT,
|
||||
input=json.dumps(payload),
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def test_duplicate():
|
||||
client = Client(BASE_URL)
|
||||
alice, bob, _, chat = client.bootstrap()
|
||||
|
||||
attachment, original = client.attachment(alice, chat)
|
||||
attachment_id = attachment["id"]
|
||||
|
||||
client.wait_ready(bob, attachment, wait_seconds=120)
|
||||
|
||||
before = job_state(attachment_id)
|
||||
|
||||
check(before["job_status"] == "ready", "job is not ready")
|
||||
check(before["attachment_status"] == "ready", "attachment is not ready")
|
||||
|
||||
payload = {
|
||||
"schema_version": 1,
|
||||
"job_id": before["job_id"],
|
||||
"type": "attachment.process.v1",
|
||||
"attachment_id": attachment_id,
|
||||
"pipeline_version": 1,
|
||||
"attempt": before["attempt"],
|
||||
"generation": before["generation"],
|
||||
"created_at": "2026-01-01T00:00:00Z",
|
||||
}
|
||||
|
||||
publish_duplicate(payload)
|
||||
publish_duplicate(payload)
|
||||
|
||||
deadline = time.monotonic() + 30
|
||||
|
||||
while time.monotonic() < deadline:
|
||||
queue = subprocess.run(
|
||||
[
|
||||
"docker", "compose", "exec", "-T", "rabbitmq",
|
||||
"rabbitmqctl", "list_queues",
|
||||
"name", "messages_ready", "messages_unacknowledged",
|
||||
"--quiet",
|
||||
],
|
||||
cwd=ROOT,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
|
||||
lines = queue.stdout.splitlines()
|
||||
matching = [
|
||||
line.split()
|
||||
for line in lines
|
||||
if line.startswith("attachment.process.v1")
|
||||
]
|
||||
|
||||
if matching and matching[0][1:] == ["0", "0"]:
|
||||
break
|
||||
|
||||
time.sleep(1)
|
||||
else:
|
||||
raise AssertionError("duplicate messages were not consumed")
|
||||
|
||||
after = job_state(attachment_id)
|
||||
|
||||
check(before == after, "duplicate delivery changed committed result")
|
||||
|
||||
content = client.call(
|
||||
"GET",
|
||||
f"/api/v1/attachments/{attachment_id}/content",
|
||||
actor=bob,
|
||||
)
|
||||
|
||||
check(content["bytes"] == original, "original content changed")
|
||||
|
||||
print("PASS: duplicate delivery, state unchanged, original intact")
|
||||
|
||||
|
||||
|
||||
def test_reorder():
|
||||
client = Client(BASE_URL)
|
||||
alice, bob, _, chat = client.bootstrap()
|
||||
|
||||
attachment, original = client.attachment(alice, chat)
|
||||
attachment_id = attachment["id"]
|
||||
|
||||
client.wait_ready(bob, attachment, wait_seconds=120)
|
||||
|
||||
before = job_state(attachment_id)
|
||||
|
||||
check(before["job_status"] == "ready", "job is not ready")
|
||||
|
||||
payload = {
|
||||
"schema_version": 1,
|
||||
"job_id": before["job_id"],
|
||||
"type": "attachment.process.v1",
|
||||
"attachment_id": attachment_id,
|
||||
"pipeline_version": 1,
|
||||
"attempt": before["attempt"],
|
||||
"generation": before["generation"],
|
||||
"created_at": "2026-01-01T00:00:00Z",
|
||||
}
|
||||
|
||||
for attempt in (
|
||||
before["attempt"] + 1,
|
||||
before["attempt"] - 1,
|
||||
before["attempt"],
|
||||
):
|
||||
message = dict(payload)
|
||||
message["attempt"] = attempt
|
||||
publish_duplicate(message)
|
||||
|
||||
deadline = time.monotonic() + 30
|
||||
|
||||
while time.monotonic() < deadline:
|
||||
queue = subprocess.run(
|
||||
[
|
||||
"docker", "compose", "exec", "-T", "rabbitmq",
|
||||
"rabbitmqctl", "list_queues",
|
||||
"name", "messages_ready", "messages_unacknowledged",
|
||||
"--quiet",
|
||||
],
|
||||
cwd=ROOT,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
|
||||
matching = [
|
||||
line.split()
|
||||
for line in queue.stdout.splitlines()
|
||||
if line.startswith("attachment.process.v1")
|
||||
]
|
||||
|
||||
if matching and matching[0][1:] == ["0", "0"]:
|
||||
break
|
||||
|
||||
time.sleep(1)
|
||||
else:
|
||||
raise AssertionError("reordered messages were not consumed")
|
||||
|
||||
after = job_state(attachment_id)
|
||||
|
||||
check(before == after, "reordered delivery changed committed result")
|
||||
|
||||
content = client.call(
|
||||
"GET",
|
||||
f"/api/v1/attachments/{attachment_id}/content",
|
||||
actor=bob,
|
||||
)
|
||||
|
||||
check(content["bytes"] == original, "original content changed")
|
||||
|
||||
print("PASS: reordered attempts, state unchanged, original intact")
|
||||
|
||||
|
||||
|
||||
def test_stale_generation():
|
||||
client = Client(BASE_URL)
|
||||
alice, bob, _, chat = client.bootstrap()
|
||||
|
||||
attachment, original = client.attachment(alice, chat)
|
||||
attachment_id = attachment["id"]
|
||||
|
||||
client.wait_ready(bob, attachment, wait_seconds=120)
|
||||
|
||||
before = job_state(attachment_id)
|
||||
|
||||
check(before["job_status"] == "ready", "job is not ready")
|
||||
check(before["generation"] == 1, "unexpected initial generation")
|
||||
|
||||
payload = {
|
||||
"schema_version": 1,
|
||||
"job_id": before["job_id"],
|
||||
"type": "attachment.process.v1",
|
||||
"attachment_id": attachment_id,
|
||||
"pipeline_version": 1,
|
||||
"attempt": before["attempt"],
|
||||
"generation": before["generation"],
|
||||
"created_at": "2026-01-01T00:00:00Z",
|
||||
}
|
||||
|
||||
for generation in (0, 2, 1):
|
||||
message = dict(payload)
|
||||
message["generation"] = generation
|
||||
publish_duplicate(message)
|
||||
|
||||
deadline = time.monotonic() + 30
|
||||
|
||||
while time.monotonic() < deadline:
|
||||
queue = subprocess.run(
|
||||
[
|
||||
"docker", "compose", "exec", "-T", "rabbitmq",
|
||||
"rabbitmqctl", "list_queues",
|
||||
"name", "messages_ready", "messages_unacknowledged",
|
||||
"--quiet",
|
||||
],
|
||||
cwd=ROOT,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
|
||||
matching = [
|
||||
line.split()
|
||||
for line in queue.stdout.splitlines()
|
||||
if line.startswith("attachment.process.v1")
|
||||
]
|
||||
|
||||
if matching and matching[0][1:] == ["0", "0"]:
|
||||
break
|
||||
|
||||
time.sleep(1)
|
||||
else:
|
||||
raise AssertionError("generation messages were not consumed")
|
||||
|
||||
after = job_state(attachment_id)
|
||||
|
||||
check(before == after, "generation mismatch changed committed result")
|
||||
|
||||
content = client.call(
|
||||
"GET",
|
||||
f"/api/v1/attachments/{attachment_id}/content",
|
||||
actor=bob,
|
||||
)
|
||||
|
||||
check(content["bytes"] == original, "original content changed")
|
||||
|
||||
print("PASS: stale and future generations rejected, result unchanged")
|
||||
|
||||
|
||||
|
||||
def test_replay_generation():
|
||||
client = Client(BASE_URL)
|
||||
alice, bob, _, chat = client.bootstrap()
|
||||
|
||||
attachment, original = client.attachment(alice, chat)
|
||||
attachment_id = attachment["id"]
|
||||
|
||||
client.wait_ready(bob, attachment, wait_seconds=120)
|
||||
|
||||
before = job_state(attachment_id)
|
||||
|
||||
check(before["job_status"] == "ready", "initial job is not ready")
|
||||
check(before["generation"] == 1, "unexpected initial generation")
|
||||
|
||||
job_id = before["job_id"]
|
||||
|
||||
sql(f"""
|
||||
BEGIN;
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'failed',
|
||||
last_error_code = 'test_replay',
|
||||
finished_at = now(),
|
||||
failed_published_at = now()
|
||||
WHERE id = '{job_id}'
|
||||
AND status = 'ready';
|
||||
|
||||
UPDATE attachments
|
||||
SET status = 'failed',
|
||||
error_code = 'test_replay',
|
||||
updated_at = now()
|
||||
WHERE id = '{attachment_id}'
|
||||
AND status = 'ready';
|
||||
COMMIT;
|
||||
""")
|
||||
|
||||
result = subprocess.run(
|
||||
[
|
||||
"docker", "compose", "exec", "-T", "worker",
|
||||
"python", "-m", "scripts.replay_job", job_id,
|
||||
],
|
||||
cwd=ROOT,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
|
||||
check("REPLAY ACCEPTED" in result.stdout, "replay was rejected")
|
||||
|
||||
replayed = job_state(attachment_id)
|
||||
|
||||
check(replayed["generation"] == 2, "generation did not increase")
|
||||
|
||||
stale_payload = {
|
||||
"schema_version": 1,
|
||||
"job_id": job_id,
|
||||
"type": "attachment.process.v1",
|
||||
"attachment_id": attachment_id,
|
||||
"pipeline_version": 1,
|
||||
"attempt": 1,
|
||||
"generation": 1,
|
||||
"created_at": "2026-01-01T00:00:00Z",
|
||||
}
|
||||
|
||||
publish_duplicate(stale_payload)
|
||||
|
||||
deadline = time.monotonic() + 120
|
||||
|
||||
while time.monotonic() < deadline:
|
||||
current = job_state(attachment_id)
|
||||
|
||||
if current["job_status"] == "ready":
|
||||
break
|
||||
|
||||
check(current["job_status"] != "failed", "replay failed")
|
||||
time.sleep(0.5)
|
||||
else:
|
||||
raise AssertionError("replayed job did not become ready")
|
||||
|
||||
check(current["generation"] == 2, "generation did not increase")
|
||||
check(current["attachment_status"] == "ready", "attachment not ready")
|
||||
check(current["execution_id"] != before["execution_id"], "execution ID unchanged")
|
||||
|
||||
payload = {
|
||||
"schema_version": 1,
|
||||
"job_id": job_id,
|
||||
"type": "attachment.process.v1",
|
||||
"attachment_id": attachment_id,
|
||||
"pipeline_version": 1,
|
||||
"attempt": 1,
|
||||
"generation": 1,
|
||||
"created_at": "2026-01-01T00:00:00Z",
|
||||
}
|
||||
|
||||
publish_duplicate(payload)
|
||||
|
||||
deadline = time.monotonic() + 30
|
||||
|
||||
while time.monotonic() < deadline:
|
||||
queue = subprocess.run(
|
||||
[
|
||||
"docker", "compose", "exec", "-T", "rabbitmq",
|
||||
"rabbitmqctl", "list_queues",
|
||||
"name", "messages_ready", "messages_unacknowledged",
|
||||
"--quiet",
|
||||
],
|
||||
cwd=ROOT,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
|
||||
matching = [
|
||||
line.split()
|
||||
for line in queue.stdout.splitlines()
|
||||
if line.startswith("attachment.process.v1")
|
||||
]
|
||||
|
||||
if matching and matching[0][1:] == ["0", "0"]:
|
||||
break
|
||||
|
||||
time.sleep(1)
|
||||
else:
|
||||
raise AssertionError("stale message was not consumed")
|
||||
|
||||
after = job_state(attachment_id)
|
||||
|
||||
check(current == after, "stale generation overwrote newer result")
|
||||
|
||||
content = client.call(
|
||||
"GET",
|
||||
f"/api/v1/attachments/{attachment_id}/content",
|
||||
actor=bob,
|
||||
)
|
||||
|
||||
check(content["bytes"] == original, "original changed after replay")
|
||||
|
||||
print("PASS: replay generation 1 -> 2, stale message rejected")
|
||||
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
test_duplicate()
|
||||
test_reorder()
|
||||
test_stale_generation()
|
||||
test_replay_generation()
|
||||
@@ -0,0 +1,108 @@
|
||||
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
from client import Client, check
|
||||
from image_probe import jpeg_size
|
||||
|
||||
BASE_URL = "http://localhost:8080"
|
||||
STATE_FILE = Path("/tmp/lab5_grade3_attachment.txt")
|
||||
|
||||
|
||||
def prepare():
|
||||
c = Client(BASE_URL)
|
||||
alice, bob, _, chat = c.bootstrap()
|
||||
|
||||
attachment, original = c.attachment(alice, chat)
|
||||
attachment_id = attachment["id"]
|
||||
|
||||
current = c.call(
|
||||
"GET",
|
||||
f"/api/v1/attachments/{attachment_id}",
|
||||
actor=bob,
|
||||
schema="Attachment",
|
||||
)["json"]
|
||||
|
||||
check(current["status"] == "queued", "expected queued while worker is stopped")
|
||||
|
||||
message = c.message(
|
||||
alice,
|
||||
chat,
|
||||
"Проверка очереди",
|
||||
attachments=[attachment_id],
|
||||
)["json"]
|
||||
|
||||
check(
|
||||
message["attachment_ids"] == [attachment_id],
|
||||
"message did not retain attachment",
|
||||
)
|
||||
|
||||
STATE_FILE.write_text(
|
||||
"\n".join([attachment_id, bob["token"], original.hex()]),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
print("PASS: HTTP 202, queued, message linked")
|
||||
print("Attachment:", attachment_id)
|
||||
print("Now restart API and RabbitMQ, then start worker")
|
||||
|
||||
|
||||
def verify():
|
||||
attachment_id, token, original_hex = STATE_FILE.read_text(
|
||||
encoding="utf-8"
|
||||
).splitlines()
|
||||
|
||||
c = Client(BASE_URL)
|
||||
actor = {"token": token}
|
||||
deadline = time.monotonic() + 90
|
||||
|
||||
while time.monotonic() < deadline:
|
||||
attachment = c.call(
|
||||
"GET",
|
||||
f"/api/v1/attachments/{attachment_id}",
|
||||
actor=actor,
|
||||
schema="Attachment",
|
||||
)["json"]
|
||||
|
||||
check(attachment["status"] != "failed", "processing failed")
|
||||
|
||||
if attachment["status"] == "ready":
|
||||
break
|
||||
|
||||
time.sleep(0.5)
|
||||
else:
|
||||
raise AssertionError("attachment did not become ready")
|
||||
|
||||
original = c.call(
|
||||
"GET",
|
||||
f"/api/v1/attachments/{attachment_id}/content",
|
||||
actor=actor,
|
||||
)
|
||||
|
||||
check(original["bytes"] == bytes.fromhex(original_hex), "original changed")
|
||||
|
||||
thumbnail = c.call(
|
||||
"GET",
|
||||
f"/api/v1/attachments/{attachment_id}/content?variant=thumbnail",
|
||||
actor=actor,
|
||||
)
|
||||
|
||||
check(
|
||||
thumbnail["headers"].get_content_type() == "image/jpeg",
|
||||
"thumbnail is not JPEG",
|
||||
)
|
||||
|
||||
check(
|
||||
jpeg_size(thumbnail["bytes"]) == (256, 160),
|
||||
"incorrect thumbnail dimensions",
|
||||
)
|
||||
|
||||
print("PASS: restart recovery, ready, original, JPEG thumbnail 256x160")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 2 or sys.argv[1] not in {"prepare", "verify"}:
|
||||
raise SystemExit("Usage: python tests/queue_grade3.py prepare|verify")
|
||||
|
||||
{"prepare": prepare, "verify": verify}[sys.argv[1]]()
|
||||
Reference in New Issue
Block a user