ЛР 5 - БВТ2304 - Травкин - на 5 #57

Open
casablancawastaken wants to merge 4 commits from casablancawastaken/mtusi2026lab5:solution/bvt2304-travkin into main
46 changed files with 5544 additions and 24 deletions
+2
View File
@@ -30,3 +30,5 @@ coverage/
.vscode/
# Keep small, sanitized evidence summaries in evidence/ under version control.
# Inspect git diff --cached: ignore rules are not a secret scanner.
infra/rabbitmq/rabbitmq.conf
+27
View File
@@ -0,0 +1,27 @@
FROM python:3.13.15-slim
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY alembic.ini .
COPY migrations ./migrations
COPY src ./src
COPY scripts ./scripts
COPY tests ./tests
RUN useradd --create-home --uid 10001 appuser
USER appuser
EXPOSE 8080
HEALTHCHECK --interval=10s --timeout=3s --start-period=5s --retries=3 \
CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=2)" || exit 1
CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8080"]
+441 -24
View File
@@ -1,37 +1,454 @@
# Отчёт — лабораторная 5
# Отчёт — лабораторная работа №5
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
- **Автор, группа:** Станислав Травкин, БВТ2304
- **Целевая оценка:** 5
- **Выбранный трек оценки 5:** Трек 2 — Доказанная идемпотентность pipeline
- **Ветка:** `solution/bvt2304-travkin`
- **Commit/PR:** заполняется после публикации
- **Технологии:** Python 3.13, FastAPI, PostgreSQL, Redis, RabbitMQ, MinIO (S3), Docker Compose, Nginx, Pillow, SQLAlchemy, Alembic
- **Источники и помощь ИИ:** методические материалы лабораторной работы, документация используемых технологий; ChatGPT применялся для обсуждения архитектуры, написания и проверки кода, сценариев отказов и подготовки отчёта. Фактические результаты тестов получены при выполнении команд в локальном окружении.
## Запуск из чистого clone
## 1. Запуск из чистого clone
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
### 1.1. Требования
## Архитектура и решения
Для запуска необходимы Docker Engine, Docker Compose, Bash, Python 3.13 и доступ к реестрам контейнерных образов. Проверки выполнялись в Ubuntu 22.04 под WSL. PostgreSQL, Redis, RabbitMQ и MinIO локально устанавливать не нужно: они запускаются в контейнерах.
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab4, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
```bash
cd mtusi2026lab5
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
bash scripts/bootstrap.sh
docker compose config --quiet
docker compose up -d --build
docker compose ps
```
## Доказательства критериев
`bootstrap.sh` создаёт `.env`, каталог `secrets` и необходимые файлы секретов, включая `rabbitmq_password`, если они ещё не существуют. Повторный запуск не должен менять уже созданные секреты. Контейнер `migrate` применяет миграции Alembic, а инициализация S3 подготавливает хранилище. Данные PostgreSQL, Redis, RabbitMQ и MinIO размещаются в именованных томах Docker Compose.
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
Для проверки:
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
```bash
make check
make validate
make test
```
## Эксперименты и отказы
`make validate` требует установленных зависимостей проверки OpenAPI. `make check` и `make validate` проверяют структуру и контракт, но не заменяют интеграционный тест работающего приложения.
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
Результаты последней проверки:
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
```text
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.
```
## Ограничения и следующие шаги
Синтаксис `scripts/bootstrap.sh` проверен через `bash -n`, конфигурация — через `docker compose config --quiet`. На существующем стенде оба действия завершились без ошибок. Полный запуск на отдельном чистом хосте с пустыми томами не проводился; поэтому воспроизводимость чистого развёртывания подтверждена проверкой конфигурации, но не независимым испытанием.
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
## 2. Архитектура и решения
### 2.1. Компоненты
| Компонент | Назначение |
|---|---|
| Nginx | Входная HTTP-точка, распределение запросов между API-репликами |
| `api-1`, `api-2` (FastAPI) | Приём запросов, работа с пользователями, чатами, сообщениями и вложениями |
| PostgreSQL | Основные данные, состояния вложений, долговременные задания обработки |
| Redis | Кеш и вспомогательное хранилище |
| RabbitMQ | Доставка асинхронных заданий с возможными повторами |
| Dispatcher | Восстановимая публикация заданий из PostgreSQL в RabbitMQ |
| Worker | Проверка оригинала, создание изображений, запись результатов |
| MinIO | S3-совместимое хранилище оригиналов и вариантов |
API и worker выполняются отдельно. HTTP-запрос не ожидает создания производных изображений.
### 2.2. Путь задания
1. Клиент загружает PNG или JPEG через API.
2. API ограничивает размер и проверяет изображение; исходный объект записывается в `staging/` MinIO.
3. В одной транзакции PostgreSQL создаются запись вложения и задание обработки.
4. После успешного сохранения оригинала и COMMIT клиент получает HTTP 202.
5. Dispatcher находит неопубликованное задание, отправляет сообщение в RabbitMQ с publisher confirms и отмечает публикацию.
6. Worker принимает сообщение, проверяет актуальность задания и переводит его в `processing`.
7. Worker читает оригинал, создаёт thumbnail, crop и optimized, сохраняет объекты в MinIO.
8. Перед COMMIT результата worker повторно проверяет `generation`, `attempt` и `execution_id`.
9. Актуальное выполнение фиксирует `ready`; сообщение подтверждается RabbitMQ после обработки.
10. При временном сбое выполняется повторная попытка, при окончательном отказе — `failed` и публикация в DLQ.
Основные состояния: `queued`, `processing`, `ready`, `failed`. HTTP 202 означает принятие задания, а не завершение обработки. До готовности производные объекты не должны выдаваться как доступные.
### 2.3. Версионирование объектов
Для текущего pipeline используются ключи:
```text
attachments/{chat_id}/{attachment_id}/g{generation}/original
variants/{chat_id}/{attachment_id}/g{generation}/thumbnail.jpg
variants/{chat_id}/{attachment_id}/g{generation}/crop.jpg
variants/{chat_id}/{attachment_id}/g{generation}/optimized.webp
```
Разные поколения не используют один и тот же ключ результата. Устаревшие сообщения не могут легитимно заменить запись актуального поколения в PostgreSQL. В базе могут сохраняться старые вложения с ключами до введения `g{generation}`; это не означает, что они обработаны новым pipeline.
## 3. Критерии на оценку 3
### 3.1. Асинхронный приём и сохранность
Реализация: `src/main.py`, `src/dispatcher.py`, `src/worker.py`.
При загрузке API записывает исходный файл в MinIO и создаёт задание в PostgreSQL. Только после успешной фиксации возвращается HTTP 202. Даже при остановленном worker задание остаётся в долговременном хранилище. Сообщение с вложением можно создать до готовности производных изображений; это не блокирует остальной API.
### 3.2. Отдельная очередь и подтверждение обработки
RabbitMQ использует очередь `attachment.process.v1`. Сообщения доставляются по модели **at-least-once**: после сбоя возможна повторная доставка. Worker подтверждает сообщение после выполнения обработки, а не сразу после получения. Поэтому обработчик обязан проверять состояние и избегать повторного изменения готового результата.
### 3.3. Thumbnail, статусы и ошибки
Создаётся JPEG-thumbnail с сохранением пропорций и ограничением 256 × 256 пикселей. Оригинал остаётся доступным. Статусы обработки представлены как `queued`, `processing`, `ready`, `failed`. При временных ошибках предусмотрены ограниченные повторы; при исчерпании попыток сохраняется безопасный код ошибки.
### 3.4. Демонстрация остановленного worker
Сценарий реализован в `tests/queue_grade3.py`. При остановленном worker команда `prepare` загружает изображение и привязывает его к сообщению. После восстановления worker команда `verify` проверяет готовность, оригинал и thumbnail.
Полученные ранее результаты:
```text
PASS: HTTP 202, queued, message linked
PASS: restart recovery, ready, original, JPEG thumbnail 256x160
```
Перезапуск RabbitMQ следует проводить с сохранением тома; `docker compose down -v` в сценарии сохранности данных недопустим.
## 4. Критерии на оценку 4
### 4.1. Три варианта изображений
В `src/worker.py` реализованы:
| Вариант | Формат | Назначение |
|---|---|---|
| `thumbnail` | JPEG | Уменьшенное изображение с сохранением пропорций |
| `crop` | JPEG | Центральное кадрирование до 128 × 128 пикселей |
| `optimized` | WebP | Сжатый вариант с параметром качества 80 |
Варианты создаются заново, без намеренного переноса EXIF-метаданных исходного файла. Метаданные вариантов включают тип, размеры, размер в байтах, S3-ключ и контрольную сумму.
### 4.2. Ограничения обработки
Входные изображения ограничены 10 МиБ и 40 млн пикселей. Допускаются PNG и JPEG, формат проверяется по содержимому. Worker повторно валидирует полученный из MinIO оригинал. Число неподтверждённых сообщений ограничивается `prefetch_count=1` на consumer.
В worker применяется `asyncio.wait_for(..., timeout=MAX_PROCESSING_SECONDS)`; значение по умолчанию — 30 секунд. Это ограничивает ожидание асинхронной задачи, но **не гарантирует остановку** уже запущенного потока Pillow в `asyncio.to_thread()`. Для строгого принудительного ограничения CPU-времени потребовалась бы изоляция в отдельном процессе.
### 4.3. Разрыв между PostgreSQL и RabbitMQ
Вместо непосредственной публикации только из HTTP-запроса используется восстановимая схема с долговременным заданием в PostgreSQL. Dispatcher периодически выбирает неопубликованные задания, использует блокировку строк `FOR UPDATE SKIP LOCKED` и publisher confirms RabbitMQ. При сбое между публикацией и фиксацией отметки о публикации сообщение может появиться повторно, поэтому идемпотентность worker является обязательной частью решения.
Для зависших заданий `processing` dispatcher выполняет восстановление после истечения интервала ожидания (90 секунд), сбрасывая сведения об устаревшем выполнении. Это позволяет повторно выдать работу, но не делает внешние S3-операции частью транзакции PostgreSQL.
### 4.4. Retry, DLQ и replay
Временные ошибки обрабатываются повторными попытками с экспоненциальной задержкой и jitter; число попыток ограничено пятью. Постоянные ошибки не должны бесконечно повторяться. После окончательного отказа состояние становится `failed`, а сведения о задании публикуются в `attachment.failed.v1`.
Скрипт `scripts/replay_job.py` допускает повторный запуск, только если и задание, и вложение находятся в `failed`. При replay `generation` увеличивается, попытки и сведения о предыдущем выполнении сбрасываются, новое поколение возвращается в очередь.
Пример запуска для существующего неуспешного задания:
```bash
python -m scripts.replay_job <job_id>
```
### 4.5. Staging и его очистка
Временный оригинал записывается в `staging/{chat_id}/{attachment_id}.{extension}`. Для удаления временных копий успешно обработанных вложений реализован `scripts/cleanup_staging.py`. По умолчанию это dry-run; `--apply` выполняет удаление. Учитывается срок хранения `STAGING_RETENTION_SECONDS=86400` (24 часа) и фактическое наличие объекта в MinIO.
```bash
docker compose run --rm --no-deps worker python -m scripts.cleanup_staging
docker compose run --rm --no-deps worker python -m scripts.cleanup_staging --apply
```
В выполненном эксперименте обнаружены и удалены три временных объекта. Повторная проверка вернула `Candidates: 0`. Сохранность постоянного оригинала для проверенного старого вложения подтверждена отдельной проверкой `ORIGINAL EXISTS: True`.
Очистка запускается вручную, а не автоматически по расписанию. Скрипт не является универсальным сборщиком всех возможных S3-«сирот», в частности оставшихся после прерванных внешних операций.
### 4.6. Проверка метаданных и реальных S3-объектов
Последний интеграционный тест завершился:
```text
PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.
```
SQL-проверка пяти последних вложений показала `ready` для вложения и задания, `generation=1`, ключи `/g1/original` и `variants_count=3`. Для одного из этих вложений выполнена независимая проверка наличия объектов в MinIO:
```text
original: True
thumbnail: True
crop: True
optimized: True
PASS: original and all 3 variants exist in MinIO
```
Это проверяет как метаданные PostgreSQL, так и существование всех четырёх объектов.
## 5. Критерии на оценку 5 — Трек 2: Доказанная идемпотентность pipeline
### 5.1. Исследовательский вопрос и гипотеза
**Исследовательский вопрос:** можно ли сохранить согласованность результата при дублировании, нарушении порядка доставки, падении worker до/после побочного эффекта и конкурентном выполнении старой и новой попыток?
**Гипотеза:** сочетание `generation`, `attempt`, уникального `execution_id`, транзакционных проверок состояния и версионированных S3-ключей предотвращает фиксацию устаревшего результата в исследуемых сценариях, несмотря на повторную доставку сообщений.
### 5.2. Правила выполнения
- `job_id` и `attachment_id` идентифицируют работу и вложение.
- `generation` различает поколения после replay.
- `attempt` различает попытки внутри поколения.
- `execution_id` различает конкретные выполнения, в том числе конкурирующие.
- Worker проверяет актуальность сообщения до работы и непосредственно перед фиксацией результата.
- Старое сообщение не может обновить завершённое или уже переизданное задание.
- S3-ключи результатов включают поколение, поэтому разные поколения не пишут в один и тот же ключ.
Миграция `migrations/versions/c9b12026a005_execution_id.py` добавляет `execution_id` для fencing устаревших исполнителей.
### 5.3. Проверяемые инварианты
1. Повторная доставка уже завершённого задания не меняет его результат.
2. Устаревшая попытка не изменяет состояние более новой попытки.
3. Сообщение с неверным поколением не изменяет актуальное поколение.
4. Старое выполнение не может зафиксировать результат после завершения нового выполнения.
5. В исследуемых аварийных сценариях принятое задание восстанавливается и не теряется.
6. После успешного завершения оригинал и все три варианта существуют в MinIO.
### 5.4. Автоматическое внедрение отказов
Используются три самостоятельных набора тестов:
```bash
python tests/fault_injection_grade5.py
python tests/crash_grade5.py
python tests/concurrent_grade5.py
```
`fault_injection_grade5.py` проверяет дублирование, нарушение порядка, неверные поколения и replay. `crash_grade5.py` намеренно завершает worker в точках `before_side_effect`, `after_side_effect`, `before_ack`, затем проверяет восстановление. `concurrent_grade5.py` запускает конкурирующие выполнения, задерживает старое, передаёт работу новому и проверяет запрет устаревшего COMMIT.
### 5.5. Результаты
| Сценарий | Проверяемое свойство | Результат |
|---|---|---|
| Повторная доставка | Неизменность готового результата | PASS |
| Нарушение порядка попыток | Отклонение старой попытки | PASS |
| Старое и будущее поколение | Проверка `generation` | PASS |
| Replay `g1 → g2` | Старое сообщение не меняет новое поколение | PASS |
| Crash before side effect | Восстановление до записи в S3 | PASS |
| Crash after side effect | Восстановление после записи в S3 | PASS |
| Crash before ACK | Повторная доставка после завершения работы | PASS |
| Два конкурирующих worker | Отклонение устаревшего `execution_id` | PASS |
Фактический вывод финального запуска:
```text
PASS: duplicate delivery, state unchanged, original intact
PASS: reordered attempts, state unchanged, original intact
PASS: stale and future generations rejected, result unchanged
PASS: replay generation 1 -> 2, stale message rejected
RUN: before_side_effect
PASS: before_side_effect, recovered, original and variants intact
RUN: after_side_effect
PASS: after_side_effect, recovered, original and variants intact
RUN: before_ack
PASS: before_ack, recovered, original and variants intact
Restoring normal worker configuration...
FAULT INJECTION DISABLED
PASS: stale execution rejected after newer execution completed
Normal worker configuration restored
```
После испытаний дополнительно запущен `tests/smoke.py`, получен `PASS`. Команда `docker compose ps` показала девять работающих сервисов: две API-реплики, dispatcher, Nginx, PostgreSQL, RabbitMQ, Redis, MinIO и worker. Инфраструктурные сервисы с healthcheck имели статус `healthy`.
### 5.6. Почему это не exactly-once delivery
RabbitMQ может доставить сообщение повторно, если worker завершился после побочного эффекта, но до ACK. Между PostgreSQL, RabbitMQ и MinIO нет единой распределённой транзакции. Поэтому утверждать **exactly-once delivery** нельзя.
Реализована схема **at-least-once delivery + идемпотентные проверки + fencing старых исполнений**. Она предотвращает некорректное изменение итогового состояния в проверенных сценариях, но не исключает повторные физические вычисления и временные объекты в S3.
### 5.7. Интерпретация результатов
Все восемь автоматических сценариев завершились успешно. Результаты согласуются с гипотезой: в проверенных условиях повторная доставка и устаревшие выполнения не изменили актуальный результат, а задания восстановились после внедрённых отказов.
Это экспериментальное подтверждение перечисленных инвариантов для реализованных сценариев, а не математическое доказательство корректности при любой последовательности отказов.
## 6. Доказательства критериев README
| Критерий | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|---|---|---|---|
| HTTP 202 и задание при остановленном worker | `queue_grade3.py prepare` | Принято, `queued`, вложение привязано к сообщению | `tests/queue_grade3.py` |
| Восстановление и thumbnail | `queue_grade3.py verify` | `ready`, оригинал сохранён, JPEG 256 × 160 | `tests/queue_grade3.py` |
| Публичный API | `make test` | PASS | `tests/smoke.py` |
| Контракт | `make check`, `make validate` | PASS, PASS | `scripts/check_contract.py`, `scripts/validate_openapi.py` |
| Три варианта | SQL и `object_exists` в MinIO | 3 метаданных варианта, 4 объекта существуют | `src/worker.py`, `src/storage.py` |
| Ограничения входа | Валидация загрузки | 10 МиБ, 40 млн пикселей, PNG/JPEG | `src/images.py`, `src/main.py` |
| Восстановимая публикация | Dispatcher и publisher confirms | Долговременное задание и повторная публикация | `src/dispatcher.py` |
| Retry, DLQ | Ограниченные попытки и очередь отказов | `failed`, `attachment.failed.v1` | `src/worker.py`, `src/dispatcher.py` |
| Replay | Повторный запуск неуспешного задания | `generation` увеличивается | `scripts/replay_job.py` |
| Очистка staging | Dry-run и `--apply` | Удалены 3 объекта, затем 0 кандидатов | `scripts/cleanup_staging.py` |
| Дубликаты и reorder | `fault_injection_grade5.py` | 4 PASS | `tests/fault_injection_grade5.py` |
| Падение worker | `crash_grade5.py` | 3 PASS и восстановление | `tests/crash_grade5.py` |
| Устаревший worker | `concurrent_grade5.py` | PASS, старый COMMIT отклонён | `tests/concurrent_grade5.py` |
Для критериев, не имеющих отдельного сохранённого протокола в этом отчёте (например, независимый чистый запуск и количественный нагрузочный эксперимент), наличие реализации не следует путать с доказательством завершённого эксперимента.
## 7. Экспериментальные условия, метрики и ограничения
### 7.1. Окружение
| Компонент | Версия / окружение |
|---|---|
| ОС | Ubuntu 22.04 (WSL) |
| Python | 3.13.15 |
| PostgreSQL | 17.6 |
| Redis | 8.2.2 |
| RabbitMQ | 4.1.4 |
| Nginx | 1.29 |
| MinIO | RELEASE.2025-09-07T16-13-09Z |
Испытания выполнялись на локальном Docker Compose-стенде с реальными PostgreSQL, RabbitMQ и MinIO. Для fault injection использовались управляемые точки отказа worker. В конкурентном тесте использовались два worker и искусственно устаревшее выполнение.
### 7.2. Нагрузка и измерения
В проекте есть `make load`, `make test-persistence ACTION_SCRIPT=...` и `make test-failover ACTION_SCRIPT=...`, но в этой серии для оценки 5 проверялась прежде всего корректность состояний при отказах.
Не зафиксированы отдельные воспроизводимые измерения RPS, доли HTTP-ошибок, p50/p95/p99, CPU и RAM. Не зафиксированы также единая интенсивность поступления запросов, длительность прогрева, число повторений нагрузочного теста и набор сырых метрик. Поэтому численные результаты производительности в отчёте **не заявляются**.
### 7.3. Границы применимости
- Тесты не перебирают все возможные межпоточные расписания и комбинации отказов.
- Не исследованы повреждение диска, потеря постоянного тома, длительное сетевое разделение и одновременный отказ всех хранилищ.
- `asyncio.wait_for()` не прерывает уже выполняющуюся CPU-операцию Pillow в потоке.
- Между записью в S3 и COMMIT PostgreSQL могут оставаться неиспользуемые объекты; нужна отдельная периодическая сборка мусора.
- Очистка staging сейчас ручная и ограничена выбранными успешно завершёнными заданиями.
- Наличие версионированных ключей и fencing защищает логическое состояние, но не гарантирует отсутствие повторных физических операций.
- Независимый запуск на полностью чистой машине с пустыми томами в рамках итоговой проверки не выполнялся.
## 8. Демонстрация
Команды выполнять из корня проекта с активированным `.venv`. При демонстрации отказов дождаться завершения каждого сценария перед следующим.
### 8.1. Запуск и состояние инфраструктуры
```bash
docker compose up -d --build
docker compose ps
make check
make validate
make test
```
Показать две API-реплики, отдельные dispatcher и worker, PostgreSQL, Redis, RabbitMQ и MinIO. Объяснить, что `make check` и `make validate` не заменяют `make test`.
### 8.2. Очередь и обработка при остановленном worker
```bash
docker compose stop worker
python tests/queue_grade3.py prepare
docker compose exec -T rabbitmq rabbitmqctl list_queues name messages_ready messages_unacknowledged
docker compose start worker
python tests/queue_grade3.py verify
```
Показать HTTP 202, `queued`, связанное сообщение и последующее `ready`. Объяснить, почему обработка не блокирует API.
### 8.3. Сохранность после перезапуска брокера
```bash
docker compose stop worker
python tests/queue_grade3.py prepare
docker compose restart rabbitmq
docker compose start worker
python tests/queue_grade3.py verify
```
Проверить восстановление RabbitMQ перед запуском worker. Не удалять тома Docker.
### 8.4. Метаданные трёх вариантов
```bash
docker compose exec -T postgres psql -U lab2 -d messenger -c "
SELECT a.id, a.status, j.status AS job_status, j.generation,
a.s3_key, json_array_length(a.variants::json) AS variants_count
FROM attachments a
JOIN attachment_jobs j ON j.attachment_id = a.id
ORDER BY a.created_at DESC
LIMIT 5;
"
```
Показать `ready`, `/g1/original` и `variants_count=3` для свежих вложений. При необходимости дополнительно проверить существование ключей в MinIO через `src.storage.object_exists`.
### 8.5. Очередь ошибок и replay
```bash
docker compose exec -T rabbitmq rabbitmqctl list_queues name messages_ready messages_unacknowledged
docker compose logs --tail=100 worker
```
Объяснить ограничение числа попыток, backoff, jitter, DLQ `attachment.failed.v1`. Для существующего окончательно неуспешного задания:
```bash
python -m scripts.replay_job <job_id>
```
Replay нельзя применять к произвольному заданию в `ready` или `processing`.
### 8.6. Очистка временных файлов
```bash
docker compose run --rm --no-deps worker python -m scripts.cleanup_staging
```
При наличии подходящих кандидатов и после объяснения dry-run:
```bash
docker compose run --rm --no-deps worker python -m scripts.cleanup_staging --apply
```
### 8.7. Исследовательский трек 2: дубликаты и версии
```bash
python tests/fault_injection_grade5.py
```
Показать четыре PASS: duplicate delivery, reordered attempts, stale/future generations, replay `g1 → g2`.
### 8.8. Падения до и после побочного эффекта
```bash
python tests/crash_grade5.py
```
Показать три PASS и строку `FAULT INJECTION DISABLED`. Объяснить различие между записью в MinIO, COMMIT PostgreSQL и ACK RabbitMQ.
### 8.9. Конкурентные worker
```bash
python tests/concurrent_grade5.py
```
Показать `PASS: stale execution rejected after newer execution completed`. Объяснить `execution_id` и проверку перед COMMIT.
### 8.10. Финальная проверка после экспериментов
```bash
python tests/smoke.py --base-url http://localhost:8080
docker compose ps
git status --short
```
Проверить работоспособность API и восстановление штатного worker. `git status --short` после коммита должен быть пустым, если нет дополнительных локальных изменений.
## 9. Ограничения и следующие шаги
Для эксплуатации за пределами лабораторного стенда полезно добавить периодическую очистку staging и неиспользуемых версионированных объектов, изоляцию CPU-обработки в отдельные процессы, метрики времени обработки и очередей, алерты по DLQ и зависшим заданиям, полноценные нагрузочные серии с сохранением исходных результатов, а также тесты длительных сетевых разделений и одновременных отказов.
Текущая реализация подтверждена автоматическими испытаниями в заявленных сценариях; выводы не следует распространять на непроверенные режимы без дополнительных экспериментов.
+149
View File
@@ -0,0 +1,149 @@
# A generic, single database configuration.
[alembic]
# path to migration scripts.
# this is typically a path given in POSIX (e.g. forward slashes)
# format, relative to the token %(here)s which refers to the location of this
# ini file
script_location = %(here)s/migrations
# template used to generate migration file names; The default value is %%(rev)s_%%(slug)s
# Uncomment the line below if you want the files to be prepended with date and time
# see https://alembic.sqlalchemy.org/en/latest/tutorial.html#editing-the-ini-file
# for all available tokens
# file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s
# Or organize into date-based subdirectories (requires recursive_version_locations = true)
# file_template = %%(year)d/%%(month).2d/%%(day).2d_%%(hour).2d%%(minute).2d_%%(second).2d_%%(rev)s_%%(slug)s
# sys.path path, will be prepended to sys.path if present.
# defaults to the current working directory. for multiple paths, the path separator
# is defined by "path_separator" below.
prepend_sys_path = .
# timezone to use when rendering the date within the migration file
# as well as the filename.
# If specified, requires the tzdata library which can be installed by adding
# `alembic[tz]` to the pip requirements.
# string value is passed to ZoneInfo()
# leave blank for localtime
# timezone =
# max length of characters to apply to the "slug" field
# truncate_slug_length = 40
# set to 'true' to run the environment during
# the 'revision' command, regardless of autogenerate
# revision_environment = false
# set to 'true' to allow .pyc and .pyo files without
# a source .py file to be detected as revisions in the
# versions/ directory
# sourceless = false
# version location specification; This defaults
# to <script_location>/versions. When using multiple version
# directories, initial revisions must be specified with --version-path.
# The path separator used here should be the separator specified by "path_separator"
# below.
# version_locations = %(here)s/bar:%(here)s/bat:%(here)s/alembic/versions
# path_separator; This indicates what character is used to split lists of file
# paths, including version_locations and prepend_sys_path within configparser
# files such as alembic.ini.
# The default rendered in new alembic.ini files is "os", which uses os.pathsep
# to provide os-dependent path splitting.
#
# Note that in order to support legacy alembic.ini files, this default does NOT
# take place if path_separator is not present in alembic.ini. If this
# option is omitted entirely, fallback logic is as follows:
#
# 1. Parsing of the version_locations option falls back to using the legacy
# "version_path_separator" key, which if absent then falls back to the legacy
# behavior of splitting on spaces and/or commas.
# 2. Parsing of the prepend_sys_path option falls back to the legacy
# behavior of splitting on spaces, commas, or colons.
#
# Valid values for path_separator are:
#
# path_separator = :
# path_separator = ;
# path_separator = space
# path_separator = newline
#
# Use os.pathsep. Default configuration used for new projects.
path_separator = os
# set to 'true' to search source files recursively
# in each "version_locations" directory
# new in Alembic version 1.10
# recursive_version_locations = false
# the output encoding used when revision files
# are written from script.py.mako
# output_encoding = utf-8
# database URL. This is consumed by the user-maintained env.py script only.
# other means of configuring database URLs may be customized within the env.py
# file.
sqlalchemy.url =
[post_write_hooks]
# post_write_hooks defines scripts or Python functions that are run
# on newly generated revision scripts. See the documentation for further
# detail and examples
# format using "black" - use the console_scripts runner, against the "black" entrypoint
# hooks = black
# black.type = console_scripts
# black.entrypoint = black
# black.options = -l 79 REVISION_SCRIPT_FILENAME
# lint with attempts to fix using "ruff" - use the module runner, against the "ruff" module
# hooks = ruff
# ruff.type = module
# ruff.module = ruff
# ruff.options = check --fix REVISION_SCRIPT_FILENAME
# Alternatively, use the exec runner to execute a binary found on your PATH
# hooks = ruff
# ruff.type = exec
# ruff.executable = ruff
# ruff.options = check --fix REVISION_SCRIPT_FILENAME
# Logging configuration. This is also consumed by the user-maintained
# env.py script only.
[loggers]
keys = root,sqlalchemy,alembic
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = WARNING
handlers = console
qualname =
[logger_sqlalchemy]
level = WARNING
handlers =
qualname = sqlalchemy.engine
[logger_alembic]
level = INFO
handlers =
qualname = alembic
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S
+14
View File
@@ -0,0 +1,14 @@
services:
worker:
environment:
FAULT_INJECTION_ENABLED: "1"
FAULT_INJECTION_POINT: pause_before_commit
worker-second:
extends:
service: worker
file: compose.yaml
environment:
FAULT_INJECTION_ENABLED: "0"
FAULT_INJECTION_POINT: ""
restart: "no"
+5
View File
@@ -0,0 +1,5 @@
services:
worker:
environment:
FAULT_INJECTION_ENABLED: "1"
FAULT_INJECTION_POINT: before_ack
+228
View File
@@ -0,0 +1,228 @@
x-api: &api
build:
context: .
dockerfile: Dockerfile
environment: &api-environment
APP_ENV: lab
SESSION_TTL_SECONDS: 3600
DATABASE_URL_FILE: /run/secrets/database_url
CACHE_URL_FILE: /run/secrets/cache_url
S3_ENDPOINT: http://s3:9000
S3_REGION: us-east-1
S3_BUCKET: messenger-attachments
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
MAX_UPLOAD_BYTES: 10485760
MAX_IMAGE_PIXELS: 40000000
RABBITMQ_HOST: rabbitmq
RABBITMQ_PASSWORD_FILE: /run/secrets/rabbitmq_password
secrets:
- database_url
- cache_url
- s3_access_key_id
- s3_secret_access_key
- rabbitmq_password
depends_on:
migrate:
condition: service_completed_successfully
redis:
condition: service_healthy
s3-init:
condition: service_completed_successfully
expose:
- "8080"
restart: unless-stopped
stop_grace_period: 15s
mem_limit: 512m
cpus: 1.0
services:
postgres:
image: postgres:17.6
environment:
POSTGRES_DB: messenger
POSTGRES_USER: lab2
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
secrets:
- postgres_password
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U lab2 -d messenger"]
interval: 5s
timeout: 3s
retries: 10
restart: unless-stopped
cpus: 1.0
mem_limit: 768m
redis:
image: redis:8.2.2
command:
- redis-server
- --appendonly
- "yes"
- --appendfsync
- everysec
volumes:
- redis_data:/data
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 3s
retries: 10
restart: unless-stopped
cpus: 0.5
mem_limit: 256m
s3:
image: dappros/minio:RELEASE.2025-09-07T16-13-09Z
command: server /data --console-address ":9001"
environment:
MINIO_ROOT_USER_FILE: /run/secrets/minio_root_user
MINIO_ROOT_PASSWORD_FILE: /run/secrets/minio_root_password
secrets:
- minio_root_user
- minio_root_password
- s3_access_key_id
- s3_secret_access_key
volumes:
- s3_data:/data
ports:
- "127.0.0.1:9001:9001"
healthcheck:
test: [ "CMD", "mc", "ready", "local" ]
interval: 10s
timeout: 5s
retries: 10
start_period: 15s
restart: unless-stopped
cpus: 1.0
mem_limit: 512m
s3-init:
image: dappros/minio:RELEASE.2025-09-07T16-13-09Z
entrypoint: [ "/bin/sh", "/config/init.sh" ]
volumes:
- ./infra/minio/init.sh:/config/init.sh:ro
- ./infra/minio/attachments-policy.json:/config/attachments-policy.json:ro
secrets:
- minio_root_user
- minio_root_password
- s3_access_key_id
- s3_secret_access_key
depends_on:
s3:
condition: service_healthy
restart: "no"
cpus: 0.5
mem_limit: 256m
migrate:
build:
context: .
dockerfile: Dockerfile
command: ["alembic", "upgrade", "head"]
secrets:
- database_url
depends_on:
postgres:
condition: service_healthy
restart: "no"
cpus: 1.0
mem_limit: 512m
api-1:
<<: *api
environment:
<<: *api-environment
INSTANCE_ID: api-1
api-2:
<<: *api
environment:
<<: *api-environment
INSTANCE_ID: api-2
nginx:
image: nginx:1.29-alpine
ports:
- "8080:8080"
volumes:
- ./infra/nginx.conf:/etc/nginx/nginx.conf:ro
depends_on:
- api-1
- api-2
restart: unless-stopped
cpus: 0.5
mem_limit: 256m
rabbitmq:
image: rabbitmq:4.1.4-management-alpine
hostname: rabbitmq
environment:
RABBITMQ_CONFIG_FILE: /etc/rabbitmq/rabbitmq
secrets:
- rabbitmq_password
volumes:
- rabbitmq_data:/var/lib/rabbitmq
- ./infra/rabbitmq/rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf:ro
healthcheck:
test: ["CMD", "rabbitmq-diagnostics", "-q", "check_running"]
interval: 10s
timeout: 15s
retries: 10
start_period: 40s
restart: unless-stopped
mem_limit: 512m
cpus: 0.5
dispatcher:
<<: *api
command: [ "python", "-m", "src.dispatcher" ]
depends_on:
migrate:
condition: service_completed_successfully
rabbitmq:
condition: service_healthy
s3-init:
condition: service_completed_successfully
healthcheck:
disable: true
worker:
<<: *api
command: [ "python", "-m", "src.worker" ]
depends_on:
migrate:
condition: service_completed_successfully
rabbitmq:
condition: service_healthy
s3-init:
condition: service_completed_successfully
healthcheck:
disable: true
secrets:
postgres_password:
file: ./secrets/postgres_password
database_url:
file: ./secrets/database_url
cache_url:
file: ./secrets/cache_url
minio_root_user:
file: ./secrets/minio_root_user
minio_root_password:
file: ./secrets/minio_root_password
s3_access_key_id:
file: ./secrets/s3_access_key_id
s3_secret_access_key:
file: ./secrets/s3_secret_access_key
rabbitmq_password:
file: ./secrets/rabbitmq_password
volumes:
postgres_data:
redis_data:
s3_data:
rabbitmq_data:
+26
View File
@@ -0,0 +1,26 @@
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::messenger-attachments"
]
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::messenger-attachments/*"
]
}
]
}
+19
View File
@@ -0,0 +1,19 @@
#!/bin/sh
set -eu
ROOT_USER="$(cat /run/secrets/minio_root_user)"
ROOT_PASSWORD="$(cat /run/secrets/minio_root_password)"
APP_USER="$(cat /run/secrets/s3_access_key_id)"
APP_PASSWORD="$(cat /run/secrets/s3_secret_access_key)"
mc alias set local http://s3:9000 "$ROOT_USER" "$ROOT_PASSWORD"
mc mb --ignore-existing local/messenger-attachments
mc anonymous set private local/messenger-attachments
mc admin policy create local messenger-attachments-rw /config/attachments-policy.json
mc admin user add local "$APP_USER" "$APP_PASSWORD"
mc admin policy attach local messenger-attachments-rw --user "$APP_USER"
mc admin user info local "$APP_USER"
mc ls local/messenger-attachments
+37
View File
@@ -0,0 +1,37 @@
events {
worker_connections 1024;
}
http {
resolver 127.0.0.11 valid=5s ipv6=off;
resolver_timeout 2s;
upstream api_backend {
zone api_backend 64k;
server api-1:8080 resolve max_fails=1 fail_timeout=5s;
server api-2:8080 resolve max_fails=1 fail_timeout=5s;
}
server {
listen 8080;
client_max_body_size 12m;
location / {
proxy_pass http://api_backend;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 2s;
proxy_read_timeout 10s;
proxy_send_timeout 10s;
proxy_next_upstream error timeout http_502 http_503 http_504;
proxy_next_upstream_tries 2;
}
}
}
+1
View File
@@ -0,0 +1 @@
Generic single-database configuration with an async dbapi.
+50
View File
@@ -0,0 +1,50 @@
import asyncio
from alembic import context
from sqlalchemy import pool
from sqlalchemy.ext.asyncio import async_engine_from_config
config = context.config
DATABASE_URL_FILE = "/run/secrets/database_url"
def read_database_url() -> str:
with open(DATABASE_URL_FILE, encoding="utf-8") as file:
return file.read().strip()
config.set_main_option("sqlalchemy.url", read_database_url())
target_metadata = None
def do_run_migrations(connection) -> None:
context.configure(
connection=connection,
target_metadata=target_metadata,
)
with context.begin_transaction():
context.run_migrations()
async def run_async_migrations() -> None:
connectable = async_engine_from_config(
config.get_section(config.config_ini_section, {}),
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
async with connectable.connect() as connection:
await connection.run_sync(do_run_migrations)
await connectable.dispose()
def run_migrations_online() -> None:
asyncio.run(run_async_migrations())
run_migrations_online()
+28
View File
@@ -0,0 +1,28 @@
"""${message}
Revision ID: ${up_revision}
Revises: ${down_revision | comma,n}
Create Date: ${create_date}
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
${imports if imports else ""}
# revision identifiers, used by Alembic.
revision: str = ${repr(up_revision)}
down_revision: Union[str, Sequence[str], None] = ${repr(down_revision)}
branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)}
depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)}
def upgrade() -> None:
"""Upgrade schema."""
${upgrades if upgrades else "pass"}
def downgrade() -> None:
"""Downgrade schema."""
${downgrades if downgrades else "pass"}
@@ -0,0 +1,42 @@
from alembic import op
import sqlalchemy as sa
revision = "a6b12026a002"
down_revision = "e5a12026a001"
branch_labels = None
depends_on = None
def upgrade():
op.add_column(
"attachment_jobs",
sa.Column("next_attempt_at", sa.DateTime(timezone=True)),
)
op.add_column(
"attachment_jobs",
sa.Column("last_error_code", sa.String(64)),
)
op.add_column(
"attachment_jobs",
sa.Column("finished_at", sa.DateTime(timezone=True)),
)
op.create_index(
"ix_attachment_jobs_retry_schedule",
"attachment_jobs",
["status", "next_attempt_at"],
)
def downgrade():
op.drop_index(
"ix_attachment_jobs_retry_schedule",
table_name="attachment_jobs",
)
op.drop_column("attachment_jobs", "finished_at")
op.drop_column("attachment_jobs", "last_error_code")
op.drop_column("attachment_jobs", "next_attempt_at")
@@ -0,0 +1,33 @@
from alembic import op
import sqlalchemy as sa
revision = "a7b12026a003"
down_revision = "a6b12026a002"
branch_labels = None
depends_on = None
def upgrade():
op.add_column(
"attachment_jobs",
sa.Column("failed_published_at", sa.DateTime(timezone=True)),
)
op.create_index(
"ix_attachment_jobs_failed_unpublished",
"attachment_jobs",
["created_at"],
postgresql_where=sa.text(
"status = 'failed' AND failed_published_at IS NULL"
),
)
def downgrade():
op.drop_index(
"ix_attachment_jobs_failed_unpublished",
table_name="attachment_jobs",
)
op.drop_column("attachment_jobs", "failed_published_at")
@@ -0,0 +1,61 @@
"""Message idempotency for lab 3.
Revision ID: b3a91c0e7d42
Revises: f7c51e1e945b
"""
from alembic import op
import sqlalchemy as sa
revision = "b3a91c0e7d42"
down_revision = "f7c51e1e945b"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"message_idempotency",
sa.Column("user_id", sa.Uuid(), nullable=False),
sa.Column("chat_id", sa.Uuid(), nullable=False),
sa.Column("key", sa.String(length=128), nullable=False),
sa.Column("request_hash", sa.String(length=64), nullable=False),
sa.Column("message_id", sa.Uuid(), nullable=False),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.text("CURRENT_TIMESTAMP"),
nullable=False,
),
sa.ForeignKeyConstraint(
["user_id"],
["users.id"],
ondelete="CASCADE",
),
sa.ForeignKeyConstraint(
["chat_id"],
["chats.id"],
ondelete="CASCADE",
),
sa.ForeignKeyConstraint(
["message_id"],
["messages.id"],
ondelete="CASCADE",
),
sa.PrimaryKeyConstraint("user_id", "chat_id", "key"),
)
op.create_index(
"ix_message_idempotency_created_at",
"message_idempotency",
["created_at"],
)
def downgrade() -> None:
op.drop_index(
"ix_message_idempotency_created_at",
table_name="message_idempotency",
)
op.drop_table("message_idempotency")
@@ -0,0 +1,24 @@
from alembic import op
import sqlalchemy as sa
revision = "b8b12026a004"
down_revision = "a7b12026a003"
branch_labels = None
depends_on = None
def upgrade():
op.add_column(
"attachment_jobs",
sa.Column(
"generation",
sa.Integer(),
nullable=False,
server_default="1",
),
)
def downgrade():
op.drop_column("attachment_jobs", "generation")
@@ -0,0 +1,78 @@
from alembic import op
import sqlalchemy as sa
revision = "c8f42a19e6b0"
down_revision = "b3a91c0e7d42"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"attachments",
sa.Column("id", sa.Uuid(), nullable=False),
sa.Column("chat_id", sa.Uuid(), nullable=False),
sa.Column("uploader_id", sa.Uuid(), nullable=False),
sa.Column("filename", sa.String(length=255), nullable=False),
sa.Column("content_type", sa.String(length=100), nullable=False),
sa.Column("size_bytes", sa.BigInteger(), nullable=False),
sa.Column("s3_key", sa.String(length=512), nullable=False),
sa.Column("status", sa.String(length=20), nullable=False),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.text("CURRENT_TIMESTAMP"),
nullable=False,
),
sa.ForeignKeyConstraint(
["chat_id"],
["chats.id"],
ondelete="CASCADE",
),
sa.ForeignKeyConstraint(
["uploader_id"],
["users.id"],
ondelete="RESTRICT",
),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("s3_key", name="uq_attachments_s3_key"),
sa.CheckConstraint(
"size_bytes > 0 AND size_bytes <= 10485760",
name="ck_attachments_size",
),
sa.CheckConstraint(
"status IN ('pending', 'ready')",
name="ck_attachments_status",
),
)
op.create_index(
"ix_attachments_chat_id",
"attachments",
["chat_id"],
)
op.create_table(
"message_attachments",
sa.Column("message_id", sa.Uuid(), nullable=False),
sa.Column("attachment_id", sa.Uuid(), nullable=False),
sa.ForeignKeyConstraint(
["message_id"],
["messages.id"],
ondelete="CASCADE",
),
sa.ForeignKeyConstraint(
["attachment_id"],
["attachments.id"],
ondelete="RESTRICT",
),
sa.PrimaryKeyConstraint("message_id", "attachment_id"),
)
def downgrade() -> None:
op.drop_table("message_attachments")
op.drop_index("ix_attachments_chat_id", table_name="attachments")
op.drop_table("attachments")
@@ -0,0 +1,19 @@
from alembic import op
import sqlalchemy as sa
revision = "c9b12026a005"
down_revision = "b8b12026a004"
branch_labels = None
depends_on = None
def upgrade():
op.add_column(
"attachment_jobs",
sa.Column("execution_id", sa.UUID(), nullable=True),
)
def downgrade():
op.drop_column("attachment_jobs", "execution_id")
@@ -0,0 +1,80 @@
from alembic import op
import sqlalchemy as sa
revision = "d9e53b20f7c1"
down_revision = "c8f42a19e6b0"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.alter_column(
"attachments",
"uploader_id",
new_column_name="owner_id",
)
op.add_column(
"attachments",
sa.Column(
"sha256",
sa.String(length=64),
nullable=False,
),
)
op.add_column(
"attachments",
sa.Column(
"variants",
sa.JSON(),
nullable=False,
server_default=sa.text("'[]'::json"),
),
)
op.add_column(
"attachments",
sa.Column(
"error_code",
sa.String(length=100),
nullable=True,
),
)
op.add_column(
"attachments",
sa.Column(
"updated_at",
sa.DateTime(timezone=True),
server_default=sa.text("CURRENT_TIMESTAMP"),
nullable=False,
),
)
op.create_check_constraint(
"ck_attachments_sha256",
"attachments",
"sha256 ~ '^[a-f0-9]{64}$'",
)
def downgrade() -> None:
op.drop_constraint(
"ck_attachments_sha256",
"attachments",
type_="check",
)
op.drop_column("attachments", "updated_at")
op.drop_column("attachments", "error_code")
op.drop_column("attachments", "variants")
op.drop_column("attachments", "sha256")
op.alter_column(
"attachments",
"owner_id",
new_column_name="uploader_id",
)
@@ -0,0 +1,77 @@
from alembic import op
import sqlalchemy as sa
revision = "e5a12026a001"
down_revision = "d9e53b20f7c1"
branch_labels = None
depends_on = None
def upgrade():
op.drop_constraint(
"ck_attachments_status",
"attachments",
type_="check",
)
op.create_check_constraint(
"ck_attachments_status",
"attachments",
"status IN ('queued', 'processing', 'ready', 'failed')",
)
op.create_table(
"attachment_jobs",
sa.Column("id", sa.Uuid(), primary_key=True),
sa.Column(
"attachment_id",
sa.Uuid(),
sa.ForeignKey("attachments.id", ondelete="CASCADE"),
nullable=False,
unique=True,
),
sa.Column(
"attempt",
sa.Integer(),
nullable=False,
server_default="1",
),
sa.Column(
"status",
sa.String(20),
nullable=False,
server_default="queued",
),
sa.Column("published_at", sa.DateTime(timezone=True)),
sa.Column("started_at", sa.DateTime(timezone=True)),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("CURRENT_TIMESTAMP"),
),
)
op.create_index(
"ix_attachment_jobs_unpublished",
"attachment_jobs",
["published_at"],
)
def downgrade():
op.drop_index(
"ix_attachment_jobs_unpublished",
table_name="attachment_jobs",
)
op.drop_table("attachment_jobs")
op.drop_constraint(
"ck_attachments_status",
"attachments",
type_="check",
)
op.create_check_constraint(
"ck_attachments_status",
"attachments",
"status IN ('pending', 'ready')",
)
@@ -0,0 +1,110 @@
"""initial schema
Revision ID: f7c51e1e945b
Revises:
Create Date: 2026-10-07 18:33:37.771593
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = "f7c51e1e945b"
down_revision: Union[str, Sequence[str], None] = None
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"users",
sa.Column("id", sa.Uuid(), nullable=False),
sa.Column("username", sa.String(length=32), nullable=False),
sa.Column("display_name", sa.String(length=100), nullable=False),
sa.Column("password_hash", sa.Text(), nullable=False),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.text("CURRENT_TIMESTAMP"),
nullable=False,
),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("username", name="uq_users_username"),
)
op.create_table(
"chats",
sa.Column("id", sa.Uuid(), nullable=False),
sa.Column("title", sa.String(length=100), nullable=False),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.text("CURRENT_TIMESTAMP"),
nullable=False,
),
sa.PrimaryKeyConstraint("id"),
)
op.create_table(
"chat_members",
sa.Column("chat_id", sa.Uuid(), nullable=False),
sa.Column("user_id", sa.Uuid(), nullable=False),
sa.ForeignKeyConstraint(
["chat_id"],
["chats.id"],
ondelete="CASCADE",
),
sa.ForeignKeyConstraint(
["user_id"],
["users.id"],
ondelete="CASCADE",
),
sa.PrimaryKeyConstraint("chat_id", "user_id"),
)
op.create_table(
"messages",
sa.Column("id", sa.Uuid(), nullable=False),
sa.Column("chat_id", sa.Uuid(), nullable=False),
sa.Column("sender_id", sa.Uuid(), nullable=False),
sa.Column("text", sa.String(length=4000), nullable=False),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.text("CURRENT_TIMESTAMP"),
nullable=False,
),
sa.ForeignKeyConstraint(
["chat_id"],
["chats.id"],
ondelete="CASCADE",
),
sa.ForeignKeyConstraint(
["sender_id"],
["users.id"],
ondelete="RESTRICT",
),
sa.PrimaryKeyConstraint("id"),
)
op.create_index(
"ix_messages_chat_created_id",
"messages",
["chat_id", "created_at", "id"],
unique=False,
)
def downgrade() -> None:
op.drop_index(
"ix_messages_chat_created_id",
table_name="messages",
)
op.drop_table("messages")
op.drop_table("chat_members")
op.drop_table("chats")
op.drop_table("users")
+23
View File
@@ -0,0 +1,23 @@
annotated-doc==0.0.5
annotated-types==0.8.0
anyio==4.15.1
click==8.5.0
fastapi==0.141.1
h11==0.16.0
idna==3.20
pydantic==2.13.5
pydantic_core==2.46.5
starlette==1.7.0
typing-inspection==0.4.4
typing_extensions==4.16.0
uvicorn==0.53.0
SQLAlchemy==2.1.4
asyncpg==0.32.0
alembic==1.20.0
greenlet==3.5.6
argon2-cffi==25.1.0
redis==7.0.1
boto3==1.43.120
Pillow==13.0.0
python-multipart==0.0.32
pika==1.3.2
+41
View File
@@ -0,0 +1,41 @@
#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")/.."
mkdir -p secrets
chmod 700 secrets
umask 077
create_random_secret() {
local name="$1"
local value="$2"
if [[ ! -s "secrets/$name" ]]; then
printf '%s\n' "$value" > "secrets/$name"
fi
chmod 644 "secrets/$name"
}
create_random_secret postgres_password "$(python3 -c 'import secrets; print(secrets.token_hex(24))')"
create_random_secret rabbitmq_password "$(python3 -c 'import secrets; print(secrets.token_hex(24))')"
if [[ ! -s secrets/database_url ]]; then
PASSWORD="$(cat secrets/postgres_password)"
printf 'postgresql+asyncpg://lab2:%s@postgres:5432/messenger\n' "$PASSWORD" > secrets/database_url
fi
create_random_secret cache_url 'redis://redis:6379/0'
create_random_secret minio_root_user "minio-$(python3 -c 'import secrets; print(secrets.token_hex(12))')"
create_random_secret minio_root_password "$(python3 -c 'import secrets; print(secrets.token_hex(32))')"
create_random_secret s3_access_key_id "$(python3 -c 'import secrets; print(secrets.token_hex(20))')"
create_random_secret s3_secret_access_key "$(python3 -c 'import secrets; print(secrets.token_hex(32))')"
chmod 644 secrets/*
if [[ ! -f .env ]]; then
cp .env.example .env
fi
docker compose config --quiet
+68
View File
@@ -0,0 +1,68 @@
import argparse
import asyncio
import os
from datetime import datetime, timedelta, timezone
from sqlalchemy import text
from src.database import SessionFactory, engine
from src.storage import delete_object, object_exists
RETENTION_SECONDS = int(os.getenv("STAGING_RETENTION_SECONDS", "86400"))
async def cleanup(apply: bool):
cutoff = datetime.now(timezone.utc) - timedelta(seconds=RETENTION_SECONDS)
async with SessionFactory() as session:
result = await session.execute(
text("""
SELECT a.id, a.chat_id, a.content_type, a.s3_key,
a.created_at
FROM attachments a
JOIN attachment_jobs j ON j.attachment_id = a.id
WHERE a.status = 'ready'
AND j.status = 'ready'
AND a.s3_key LIKE 'attachments/%'
AND a.created_at < :cutoff
"""),
{"cutoff": cutoff},
)
rows = result.all()
candidates = 0
for row in rows:
extension = "png" if row.content_type == "image/png" else "jpg"
staging_key = f"staging/{row.chat_id}/{row.id}.{extension}"
if not await object_exists(staging_key):
continue
candidates += 1
if not apply:
print(f"DRY RUN: {staging_key}")
continue
await delete_object(staging_key)
print(f"DELETED: {staging_key}")
print(f"Candidates: {candidates}")
async def main():
parser = argparse.ArgumentParser()
parser.add_argument("--apply", action="store_true")
args = parser.parse_args()
try:
await cleanup(args.apply)
finally:
await engine.dispose()
if __name__ == "__main__":
asyncio.run(main())
+83
View File
@@ -0,0 +1,83 @@
import argparse
import asyncio
from uuid import UUID
from sqlalchemy import text
from src.database import SessionFactory, engine
async def replay(job_id: UUID):
async with SessionFactory() as session:
async with session.begin():
result = await session.execute(
text("""
SELECT j.id, j.attachment_id, j.status,
j.generation, a.status AS attachment_status
FROM attachment_jobs j
JOIN attachments a ON a.id = j.attachment_id
WHERE j.id = :job_id
FOR UPDATE OF j, a
"""),
{"job_id": job_id},
)
row = result.one_or_none()
if row is None:
print("REJECTED: job not found")
return
if row.status != "failed" or row.attachment_status != "failed":
print("REJECTED: job and attachment must be failed")
return
await session.execute(
text("""
UPDATE attachment_jobs
SET status = 'queued',
generation = generation + 1,
attempt = 1,
published_at = NULL,
started_at = NULL,
execution_id = NULL,
next_attempt_at = NULL,
last_error_code = NULL,
finished_at = NULL,
failed_published_at = NULL
WHERE id = :job_id
"""),
{"job_id": job_id},
)
await session.execute(
text("""
UPDATE attachments
SET status = 'queued',
error_code = NULL,
updated_at = now()
WHERE id = :attachment_id
"""),
{"attachment_id": row.attachment_id},
)
print(
f"REPLAY ACCEPTED: job={job_id}, "
f"generation={row.generation + 1}"
)
async def main():
parser = argparse.ArgumentParser()
parser.add_argument("job_id", type=UUID)
args = parser.parse_args()
try:
await replay(args.job_id)
finally:
await engine.dispose()
if __name__ == "__main__":
asyncio.run(main())
+4
View File
@@ -0,0 +1,4 @@
#!/usr/bin/env bash
set -euo pipefail
docker compose restart postgres redis api-1 api-2
+14
View File
@@ -0,0 +1,14 @@
#!/usr/bin/env bash
set -euo pipefail
: "${TARGET_INSTANCE_ID:?}"
case "$TARGET_INSTANCE_ID" in
api-1|api-2)
docker compose stop "$TARGET_INSTANCE_ID"
;;
*)
echo "Unknown instance: $TARGET_INSTANCE_ID" >&2
exit 1
;;
esac
View File
+86
View File
@@ -0,0 +1,86 @@
import base64
import binascii
import json
from datetime import datetime
from uuid import UUID
def encode_cursor(
chat_id: UUID,
created_at: datetime,
message_id: UUID,
) -> str:
data = {
"chat_id": str(chat_id),
"created_at": created_at.isoformat(),
"message_id": str(message_id),
}
raw = json.dumps(
data,
separators=(",", ":"),
).encode("utf-8")
return base64.urlsafe_b64encode(raw).decode("ascii")
def decode_cursor(cursor: str) -> tuple[UUID, datetime, UUID]:
try:
raw = base64.urlsafe_b64decode(cursor.encode("ascii"))
data = json.loads(raw.decode("utf-8"))
if not isinstance(data, dict):
raise ValueError("Invalid cursor")
if set(data.keys()) != {
"chat_id",
"created_at",
"message_id",
}:
raise ValueError("Invalid cursor")
chat_id = UUID(data["chat_id"])
created_at = datetime.fromisoformat(data["created_at"])
message_id = UUID(data["message_id"])
if created_at.tzinfo is None or created_at.utcoffset() is None:
raise ValueError("Invalid cursor")
return chat_id, created_at, message_id
except (
ValueError,
KeyError,
TypeError,
UnicodeError,
binascii.Error,
json.JSONDecodeError,
) as exc:
raise ValueError("Invalid cursor") from exc
def encode_chat_cursor(user_id: UUID, created_at: datetime, chat_id: UUID) -> str:
data = {
"user_id": str(user_id),
"created_at": created_at.isoformat(),
"chat_id": str(chat_id),
}
raw = json.dumps(data, separators=(",", ":")).encode("utf-8")
return base64.urlsafe_b64encode(raw).decode("ascii")
def decode_chat_cursor(cursor: str) -> tuple[UUID, datetime, UUID]:
try:
raw = base64.b64decode(cursor.encode("ascii"), altchars=b"-_", validate=True)
data = json.loads(raw.decode("utf-8"))
if not isinstance(data, dict) or set(data) != {"user_id", "created_at", "chat_id"}:
raise ValueError("Invalid cursor")
user_id = UUID(data["user_id"])
created_at = datetime.fromisoformat(data["created_at"])
chat_id = UUID(data["chat_id"])
if created_at.tzinfo is None or created_at.utcoffset() is None:
raise ValueError("Invalid cursor")
return user_id, created_at, chat_id
except (ValueError, TypeError, KeyError, UnicodeError, binascii.Error, json.JSONDecodeError) as exc:
raise ValueError("Invalid cursor") from exc
+57
View File
@@ -0,0 +1,57 @@
from collections.abc import AsyncIterator
import socket
from sqlalchemy import text
from sqlalchemy.exc import SQLAlchemyError
from sqlalchemy.ext.asyncio import (
AsyncSession,
async_sessionmaker,
create_async_engine,
)
DATABASE_URL_FILE = "/run/secrets/database_url"
POOL_SIZE = 5
MAX_OVERFLOW = 5
POOL_TIMEOUT_SECONDS = 5
DB_CONNECT_TIMEOUT_SECONDS = 5
def read_database_url() -> str:
with open(DATABASE_URL_FILE, encoding="utf-8") as file:
return file.read().strip()
database_url = read_database_url()
engine = create_async_engine(
database_url,
pool_size=POOL_SIZE,
max_overflow=MAX_OVERFLOW,
pool_timeout=POOL_TIMEOUT_SECONDS,
connect_args={
"timeout": DB_CONNECT_TIMEOUT_SECONDS,
},
pool_pre_ping=True,
)
SessionFactory = async_sessionmaker(
bind=engine,
expire_on_commit=False,
)
async def get_db_session() -> AsyncIterator[AsyncSession]:
async with SessionFactory() as session:
yield session
async def ping_database() -> bool:
try:
async with engine.connect() as connection:
await connection.execute(text("SELECT 1"))
return True
except (SQLAlchemyError, OSError):
return False
+149
View File
@@ -0,0 +1,149 @@
import asyncio
import logging
from sqlalchemy import text
from src.database import SessionFactory, engine
from src.queue import connect, publish, publish_failed
logging.basicConfig(level=logging.INFO)
async def dispatch(channel):
async with SessionFactory() as session:
async with session.begin():
await session.execute(
text("""
UPDATE attachment_jobs
SET status = 'queued',
published_at = NULL,
started_at = NULL,
execution_id = NULL
WHERE status = 'processing'
AND started_at < now() - interval '90 seconds'
""")
)
await session.execute(
text("""
UPDATE attachments
SET status = 'queued',
updated_at = now()
WHERE status = 'processing'
AND id IN (
SELECT attachment_id
FROM attachment_jobs
WHERE status = 'queued'
)
""")
)
result = await session.execute(
text("""
SELECT id, attachment_id, attempt, generation, created_at
FROM attachment_jobs
WHERE published_at IS NULL
AND status = 'queued'
AND (
next_attempt_at IS NULL
OR next_attempt_at <= now()
)
ORDER BY created_at LIMIT 25
FOR
UPDATE SKIP LOCKED
""")
)
rows = result.all()
for row in rows:
publish(channel, row)
await session.execute(
text("""
UPDATE attachment_jobs
SET published_at = now()
WHERE id = :id
"""),
{"id": row.id},
)
return len(rows)
async def dispatch_failed(channel):
async with SessionFactory() as session:
async with session.begin():
result = await session.execute(
text("""
SELECT id, attachment_id, attempt, generation,
last_error_code, created_at, finished_at
FROM attachment_jobs
WHERE status = 'failed'
AND failed_published_at IS NULL
ORDER BY created_at
LIMIT 25
FOR UPDATE SKIP LOCKED
""")
)
rows = result.all()
for row in rows:
payload = {
"schema_version": 1,
"type": "attachment.failed.v1",
"job_id": str(row.id),
"attachment_id": str(row.attachment_id),
"attempt": row.attempt,
"generation": row.generation,
"error_code": row.last_error_code,
"created_at": row.created_at.isoformat(),
"finished_at": (
row.finished_at.isoformat()
if row.finished_at
else None
),
}
publish_failed(channel, payload)
await session.execute(
text("""
UPDATE attachment_jobs
SET failed_published_at = now()
WHERE id = :id
"""),
{"id": row.id},
)
return len(rows)
async def main():
while True:
connection = None
try:
connection, channel = connect()
channel.confirm_delivery()
while True:
count = await dispatch(channel)
failed_count = await dispatch_failed(channel)
await asyncio.sleep(0.2 if count or failed_count else 1)
except Exception:
logging.exception("Dispatcher reconnect")
await asyncio.sleep(3)
finally:
if connection and connection.is_open:
connection.close()
if __name__ == "__main__":
try:
asyncio.run(main())
finally:
asyncio.run(engine.dispose())
+12
View File
@@ -0,0 +1,12 @@
class APIError(Exception):
def __init__(
self,
status_code: int,
code: str,
message: str,
headers: dict[str, str] | None = None,
):
self.status_code = status_code
self.code = code
self.message = message
self.headers = headers
+71
View File
@@ -0,0 +1,71 @@
import hashlib
import io
import os
from PIL import Image, UnidentifiedImageError
from src.errors import APIError
MAX_UPLOAD_BYTES = int(os.getenv("MAX_UPLOAD_BYTES", "10485760"))
MAX_IMAGE_PIXELS = int(os.getenv("MAX_IMAGE_PIXELS", "40000000"))
ALLOWED_FORMATS = {
"PNG": "image/png",
"JPEG": "image/jpeg",
}
def validate_image(body: bytes, content_type: str) -> tuple[int, str]:
if content_type not in ALLOWED_FORMATS.values():
raise APIError(
status_code=415,
code="unsupported_media_type",
message="Only PNG and JPEG images are supported",
)
size_bytes = len(body)
if size_bytes > MAX_UPLOAD_BYTES:
raise APIError(
status_code=413,
code="payload_too_large",
message="Image exceeds maximum upload size",
)
if size_bytes == 0:
raise APIError(
status_code=400,
code="invalid_request",
message="Image must not be empty",
)
try:
with Image.open(io.BytesIO(body)) as image:
if image.format not in ALLOWED_FORMATS:
raise ValueError("Unsupported image format")
if ALLOWED_FORMATS[image.format] != content_type:
raise ValueError("Content-Type does not match image format")
width, height = image.size
if width * height > MAX_IMAGE_PIXELS:
raise APIError(
status_code=413,
code="payload_too_large",
message="Image exceeds maximum pixel count",
)
image.load()
except APIError:
raise
except (UnidentifiedImageError, OSError, ValueError, Image.DecompressionBombError) as exc:
raise APIError(
status_code=400,
code="invalid_request",
message="Invalid image data",
) from exc
return size_bytes, hashlib.sha256(body).hexdigest()
+779
View File
@@ -0,0 +1,779 @@
from datetime import datetime, timedelta, timezone
from uuid import UUID, uuid4
import os
import re
import base64
import binascii
import asyncio
from fastapi import Depends, FastAPI, Request, Header, Query
from fastapi.exceptions import RequestValidationError
from fastapi.responses import JSONResponse, Response
from fastapi.security import (
HTTPAuthorizationCredentials,
HTTPBearer,
)
from botocore.exceptions import BotoCoreError, ClientError
from sqlalchemy import text
from sqlalchemy.exc import IntegrityError, SQLAlchemyError
from sqlalchemy.ext.asyncio import AsyncSession
from starlette.exceptions import HTTPException as StarletteHTTPException
from contextlib import asynccontextmanager
from src.database import engine, get_db_session, ping_database, SessionFactory
from src.errors import APIError
from src.models import CreateUser, SessionResponse, User, CreateChat, Chat, ChatPage, CreateMessage, Message, MessagePage, Attachment
from src.repositories import (
create_chat,
list_chats,
create_user as create_user_in_db,
get_user_by_id,
get_chat_by_id,
get_user_credentials_by_username,
create_message,
get_messages,
create_attachment,
get_attachment,
attachment_key_exists,
)
from src.security import verify_password
from src.sessions import (
create_session,
delete_session,
get_session_user_id,
ping_session_store,
redis_client,
)
from src.images import MAX_UPLOAD_BYTES, validate_image
from src.storage import delete_object, put_object, get_object
@asynccontextmanager
async def lifespan(app: FastAPI):
yield
await redis_client.aclose()
await engine.dispose()
app = FastAPI(lifespan=lifespan)
INSTANCE_ID = os.getenv("INSTANCE_ID") or str(uuid4())
bearer_security = HTTPBearer(auto_error=False)
async def get_current_user(
credentials: HTTPAuthorizationCredentials | None = Depends(bearer_security),
session: AsyncSession = Depends(get_db_session),
) -> User:
if credentials is None or credentials.scheme.lower() != "bearer":
raise APIError(
status_code=401,
code="unauthorized",
message="Unauthorized",
headers={
"WWW-Authenticate": "Bearer",
},
)
user_id = await get_session_user_id(credentials.credentials)
if user_id is None:
raise APIError(
status_code=401,
code="unauthorized",
message="Unauthorized",
headers={
"WWW-Authenticate": "Bearer",
},
)
user = await get_user_by_id(session, user_id)
if user is None:
raise APIError(
status_code=401,
code="unauthorized",
message="Unauthorized",
headers={
"WWW-Authenticate": "Bearer",
},
)
return user
@app.middleware("http")
async def request_middleware(request: Request, call_next):
request_id = str(uuid4())
request.state.request_id = request_id
if request.method in {"POST", "PUT", "PATCH"}:
content_type = request.headers.get("content-type")
if content_type is not None:
media_type = content_type.split(";", 1)[0].strip().lower()
is_attachment_upload = (
request.method == "POST"
and re.fullmatch(
r"/api/v1/chats/[^/]+/attachments",
request.url.path,
) is not None
)
allowed_media_types = (
{"image/png", "image/jpeg"}
if is_attachment_upload
else {"application/json"}
)
if media_type not in allowed_media_types:
return JSONResponse(
status_code=415,
content={
"error": {
"code": "unsupported_media_type",
"message": "Unsupported media type",
"request_id": request_id,
}
},
headers={
"X-Request-Id": request_id,
"X-Instance-Id": INSTANCE_ID,
},
)
response = await call_next(request)
response.headers["X-Request-Id"] = request_id
response.headers["X-Instance-Id"] = INSTANCE_ID
return response
@app.exception_handler(RequestValidationError)
async def validation_exception_handler(
request: Request,
exc: RequestValidationError,
):
request_id = request.state.request_id
return JSONResponse(
status_code=400,
content={
"error": {
"code": "invalid_request",
"message": "Invalid request",
"request_id": request_id,
}
},
)
@app.exception_handler(APIError)
async def api_error_handler(request: Request, exc: APIError):
request_id = request.state.request_id
return JSONResponse(
status_code=exc.status_code,
content={
"error": {
"code": exc.code,
"message": exc.message,
"request_id": request_id,
}
},
headers=exc.headers,
)
@app.exception_handler(SQLAlchemyError)
async def database_exception_handler(
request: Request,
exc: SQLAlchemyError,
):
request_id = request.state.request_id
return JSONResponse(
status_code=503,
content={
"error": {
"code": "unavailable",
"message": "Database unavailable",
"request_id": request_id,
}
},
)
@app.exception_handler(OSError)
async def database_connection_exception_handler(
request: Request,
exc: OSError,
):
request_id = request.state.request_id
return JSONResponse(
status_code=503,
content={
"error": {
"code": "unavailable",
"message": "Database unavailable",
"request_id": request_id,
}
},
)
@app.exception_handler(StarletteHTTPException)
async def http_exception_handler(
request: Request,
exc: StarletteHTTPException,
):
request_id = request.state.request_id
if exc.status_code == 404:
code = "not_found"
message = "Not found"
else:
code = "invalid_request"
message = "Invalid request"
return JSONResponse(
status_code=exc.status_code,
content={
"error": {
"code": code,
"message": message,
"request_id": request_id,
}
},
headers=exc.headers,
)
@app.exception_handler(Exception)
async def unexpected_exception_handler(
request: Request,
exc: Exception,
):
request_id = request.state.request_id
return JSONResponse(
status_code=500,
content={
"error": {
"code": "internal_error",
"message": "Internal server error",
"request_id": request_id,
}
},
)
@app.get("/health/live")
async def health_live():
return {"status": "ok"}
@app.get("/health/ready")
async def health_ready():
database_ready = await ping_database()
session_store_ready = await ping_session_store()
if not database_ready or not session_store_ready:
raise APIError(
status_code=503,
code="unavailable",
message="Service dependencies unavailable",
)
return {"status": "ok"}
@app.post("/api/v1/users", response_model=User, status_code=201)
async def create_user(
data: CreateUser,
session: AsyncSession = Depends(get_db_session),
):
try:
async with session.begin():
return await create_user_in_db(session, data)
except IntegrityError as exc:
raise APIError(
status_code=409,
code="conflict",
message="Username already exists",
) from exc
@app.get("/api/v1/users/me", response_model=User)
async def get_me(
current_user: User = Depends(get_current_user),
):
return current_user
@app.post(
"/api/v1/auth/sessions",
response_model=SessionResponse,
status_code=201,
)
async def create_auth_session(
authorization: str | None = Header(default=None),
session: AsyncSession = Depends(get_db_session),
):
credentials = None
if authorization is not None:
scheme, separator, encoded = authorization.partition(" ")
if separator and scheme.lower() == "basic":
try:
decoded = base64.b64decode(encoded.strip(), validate=True).decode("utf-8")
if ":" in decoded:
credentials = decoded.split(":", 1)
except (ValueError, UnicodeError, binascii.Error):
pass
if credentials is None:
raise APIError(
status_code=401,
code="unauthorized",
message="Unauthorized",
headers={
"WWW-Authenticate": 'Basic realm="messenger", charset="UTF-8"',
},
)
user_credentials = await get_user_credentials_by_username(
session,
credentials[0],
)
if user_credentials is None:
raise APIError(
status_code=401,
code="unauthorized",
message="Unauthorized",
headers={
"WWW-Authenticate": 'Basic realm="messenger", charset="UTF-8"',
},
)
if not verify_password(
user_credentials.password_hash,
credentials[1],
):
raise APIError(
status_code=401,
code="unauthorized",
message="Unauthorized",
headers={
"WWW-Authenticate": 'Basic realm="messenger", charset="UTF-8"',
},
)
user = User(
id=user_credentials.id,
username=user_credentials.username,
display_name=user_credentials.display_name,
created_at=user_credentials.created_at,
)
token, ttl = await create_session(user.id)
expires_at = datetime.now(timezone.utc) + timedelta(seconds=ttl)
return SessionResponse(
token=token,
token_type="Bearer",
expires_at=expires_at,
user=user,
)
@app.delete(
"/api/v1/auth/sessions/current",
status_code=204,
)
async def delete_current_session(
credentials: HTTPAuthorizationCredentials | None = Depends(bearer_security),
current_user: User = Depends(get_current_user),
):
await delete_session(credentials.credentials)
@app.post(
"/api/v1/chats",
response_model=Chat,
status_code=201,
)
async def create_chat_endpoint(
data: CreateChat,
current_user: User = Depends(get_current_user),
session: AsyncSession = Depends(get_db_session),
) -> Chat:
try:
chat = await create_chat(
session=session,
data=data,
current_user_id=current_user.id,
)
await session.commit()
except Exception:
await session.rollback()
raise
return chat
@app.get(
"/api/v1/chats",
response_model=ChatPage,
)
async def get_chats(
limit: int = 50,
cursor: str | None = None,
current_user: User = Depends(get_current_user),
session: AsyncSession = Depends(get_db_session),
) -> ChatPage:
if limit < 1 or limit > 100:
raise APIError(
status_code=400,
code="invalid_request",
message="Limit must be between 1 and 100",
)
return await list_chats(
session=session,
current_user_id=current_user.id,
limit=limit,
cursor=cursor,
)
@app.get(
"/api/v1/chats/{chat_id}",
response_model=Chat,
)
async def get_chat_endpoint(
chat_id: UUID,
current_user: User = Depends(get_current_user),
session: AsyncSession = Depends(get_db_session),
) -> Chat:
chat = await get_chat_by_id(
session=session,
chat_id=chat_id,
current_user_id=current_user.id,
)
if chat is None:
raise APIError(
status_code=404,
code="not_found",
message="Chat not found",
)
return chat
@app.post(
"/api/v1/chats/{chat_id}/messages",
response_model=Message,
status_code=201,
)
async def create_message_endpoint(
chat_id: UUID,
data: CreateMessage,
idempotency_key: str | None = Header(
default=None,
alias="Idempotency-Key",
),
current_user: User = Depends(get_current_user),
session: AsyncSession = Depends(get_db_session),
) -> Message:
if (
idempotency_key is None
or re.fullmatch(
r"[A-Za-z0-9._:-]{1,128}",
idempotency_key,
) is None
):
raise APIError(
status_code=400,
code="invalid_request",
message="Invalid or missing Idempotency-Key",
)
try:
message = await create_message(
session=session,
chat_id=chat_id,
current_user_id=current_user.id,
data=data,
idempotency_key=idempotency_key,
)
await session.commit()
except Exception:
await session.rollback()
raise
return message
@app.get(
"/api/v1/chats/{chat_id}/messages",
response_model=MessagePage,
)
async def get_messages_endpoint(
chat_id: UUID,
limit: int = 50,
cursor: str | None = None,
current_user: User = Depends(get_current_user),
session: AsyncSession = Depends(get_db_session),
) -> MessagePage:
if limit < 1 or limit > 100:
raise APIError(
status_code=400,
code="invalid_request",
message="Limit must be between 1 and 100",
)
return await get_messages(
session=session,
chat_id=chat_id,
current_user_id=current_user.id,
limit=limit,
cursor=cursor,
)
@app.post(
"/api/v1/chats/{chat_id}/attachments",
response_model=Attachment,
status_code=202,
)
async def upload_attachment_endpoint(
chat_id: UUID,
request: Request,
filename: str = Query(min_length=1, max_length=255),
current_user: User = Depends(get_current_user),
session: AsyncSession = Depends(get_db_session),
) -> Attachment:
chat = await get_chat_by_id(
session=session,
chat_id=chat_id,
current_user_id=current_user.id,
)
if chat is None:
raise APIError(
status_code=404,
code="not_found",
message="Chat not found",
)
content_type = request.headers.get("content-type", "").split(";", 1)[0].strip().lower()
if content_type not in {"image/png", "image/jpeg"}:
raise APIError(
status_code=415,
code="unsupported_media_type",
message="Only PNG and JPEG images are supported",
)
body = bytearray()
async for chunk in request.stream():
if len(body) + len(chunk) > MAX_UPLOAD_BYTES:
raise APIError(
status_code=413,
code="payload_too_large",
message="Image exceeds maximum upload size",
)
body.extend(chunk)
image_bytes = bytes(body)
size_bytes, sha256 = validate_image(image_bytes, content_type)
attachment_id = uuid4()
extension = "png" if content_type == "image/png" else "jpg"
s3_key = f"staging/{chat_id}/{attachment_id}.{extension}"
try:
await put_object(
key=s3_key,
body=image_bytes,
content_type=content_type,
)
except (BotoCoreError, ClientError, OSError, TimeoutError) as exc:
raise APIError(
status_code=503,
code="unavailable",
message="Object storage unavailable",
) from exc
try:
attachment = await create_attachment(
session=session,
attachment_id=attachment_id,
chat_id=chat_id,
owner_id=current_user.id,
filename=filename,
content_type=content_type,
size_bytes=size_bytes,
sha256=sha256,
s3_key=s3_key,
)
await session.execute(
text("""
INSERT INTO attachment_jobs (id, attachment_id)
VALUES (:job_id, :attachment_id)
"""),
{
"job_id": uuid4(),
"attachment_id": attachment_id,
},
)
await session.commit()
except BaseException:
try:
await session.rollback()
except Exception:
pass
try:
async with SessionFactory() as verification_session:
exists = await attachment_key_exists(
verification_session,
s3_key,
)
except Exception:
exists = True
if not exists:
try:
await asyncio.shield(delete_object(s3_key))
except Exception:
pass
raise
return attachment
@app.get(
"/api/v1/attachments/{attachment_id}",
response_model=Attachment,
)
async def get_attachment_endpoint(
attachment_id: UUID,
current_user: User = Depends(get_current_user),
session: AsyncSession = Depends(get_db_session),
) -> Attachment:
result = await get_attachment(
session=session,
attachment_id=attachment_id,
current_user_id=current_user.id,
)
if result is None:
raise APIError(
status_code=404,
code="not_found",
message="Attachment not found",
)
attachment, s3_key = result
return attachment
@app.get("/api/v1/attachments/{attachment_id}/content")
async def get_attachment_content_endpoint(
attachment_id: UUID,
current_user: User = Depends(get_current_user),
session: AsyncSession = Depends(get_db_session),
variant: str = Query(default="original"),
):
result = await get_attachment(
session=session,
attachment_id=attachment_id,
current_user_id=current_user.id,
)
if result is None:
raise APIError(
status_code=404,
code="not_found",
message="Attachment not found",
)
attachment, s3_key = result
if attachment.status != "ready":
raise APIError(
status_code=409,
code="conflict",
message="Attachment is not ready",
)
if variant == "original":
media_type = attachment.content_type
elif variant in {"thumbnail", "crop", "optimized"}:
selected = next(
(item for item in attachment.variants if item.name == variant),
None,
)
if selected is None:
raise APIError(
status_code=404,
code="not_found",
message="Variant not found",
)
extension = {
"thumbnail": "jpg",
"crop": "jpg",
"optimized": "webp",
}[variant]
s3_key = selected.s3_key or (
f"variants/{attachment.chat_id}/{attachment_id}/"
f"{variant}.{extension}"
)
media_type = selected.content_type
else:
raise APIError(
status_code=404,
code="not_found",
message="Variant not found",
)
try:
body = await get_object(s3_key)
except (BotoCoreError, ClientError, OSError, TimeoutError) as exc:
raise APIError(
status_code=503,
code="unavailable",
message="Object storage unavailable",
) from exc
return Response(
content=body,
media_type=media_type,
headers={
"Cache-Control": "private, no-store",
"X-Content-Type-Options": "nosniff",
},
)
+153
View File
@@ -0,0 +1,153 @@
from datetime import datetime
from uuid import UUID
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
class CreateUser(BaseModel):
model_config = ConfigDict(extra="forbid")
username: str = Field(
min_length=3,
max_length=32,
pattern=r"^[a-z0-9_]+$",
)
display_name: str = Field(
min_length=1,
max_length=100,
)
password: str = Field(
min_length=12,
max_length=128,
)
@field_validator("display_name")
@classmethod
def validate_display_name(cls, value: str) -> str:
if not value.strip():
raise ValueError("Display name must not be blank")
return value
class User(BaseModel):
id: UUID
username: str
display_name: str
created_at: datetime
class SessionResponse(BaseModel):
token: str
token_type: str
expires_at: datetime
user: User
class CreateChat(BaseModel):
model_config = ConfigDict(extra="forbid")
title: str = Field(
min_length=1,
max_length=100,
)
member_ids: list[UUID] = Field(
min_length=1,
max_length=100,
)
@field_validator("title")
@classmethod
def validate_title(cls, value: str) -> str:
if not value.strip():
raise ValueError("Chat title must not be blank")
return value
@field_validator("member_ids")
@classmethod
def validate_member_ids(cls, value: list[UUID]) -> list[UUID]:
if len(value) != len(set(value)):
raise ValueError("Member IDs must be unique")
return value
class Chat(BaseModel):
id: UUID
title: str
member_ids: list[UUID]
created_at: datetime
class ChatPage(BaseModel):
items: list[Chat]
next_cursor: str | None
class CreateMessage(BaseModel):
model_config = ConfigDict(extra="forbid")
text: str = Field(max_length=4000)
attachment_ids: list[UUID] = Field(default_factory=list, max_length=10)
@field_validator("attachment_ids")
@classmethod
def validate_attachment_ids(cls, value: list[UUID]) -> list[UUID]:
if len(value) != len(set(value)):
raise ValueError("Attachment IDs must be unique")
return value
@model_validator(mode="after")
def validate_content(self):
if not self.text.strip() and not self.attachment_ids:
raise ValueError("Message must contain text or attachments")
return self
class Message(BaseModel):
id: UUID
chat_id: UUID
sender_id: UUID
text: str
created_at: datetime
attachment_ids: list[UUID] = Field(default_factory=list)
class MessagePage(BaseModel):
items: list[Message]
next_cursor: str | None
class Variant(BaseModel):
model_config = ConfigDict(extra="forbid")
name: str = Field(pattern=r"^(thumbnail|crop|optimized)$")
content_type: str = Field(pattern=r"^image/(jpeg|png|webp)$")
size_bytes: int = Field(ge=1)
width: int = Field(ge=1)
height: int = Field(ge=1)
sha256: str = Field(pattern=r"^[a-f0-9]{64}$")
s3_key: str | None = Field(default=None, exclude=True)
class Attachment(BaseModel):
model_config = ConfigDict(extra="forbid")
id: UUID
chat_id: UUID
owner_id: UUID
filename: str = Field(min_length=1, max_length=255)
content_type: str = Field(pattern=r"^image/(png|jpeg)$")
size_bytes: int = Field(ge=1, le=10485760)
sha256: str = Field(pattern=r"^[a-f0-9]{64}$")
status: str = Field(pattern=r"^(queued|processing|ready|failed)$")
variants: list[Variant]
error_code: str | None
created_at: datetime
updated_at: datetime
+78
View File
@@ -0,0 +1,78 @@
import json
import os
from datetime import timezone
from pathlib import Path
import pika
QUEUE = "attachment.process.v1"
DLQ = "attachment.failed.v1"
def connect():
password = Path(
os.environ["RABBITMQ_PASSWORD_FILE"]
).read_text().strip()
credentials = pika.PlainCredentials(
"messenger",
password,
)
connection = pika.BlockingConnection(
pika.ConnectionParameters(
host=os.getenv("RABBITMQ_HOST", "rabbitmq"),
credentials=credentials,
heartbeat=60,
blocked_connection_timeout=30,
connection_attempts=3,
retry_delay=2,
)
)
channel = connection.channel()
channel.queue_declare(queue=QUEUE, durable=True)
channel.queue_declare(queue=DLQ, durable=True)
return connection, channel
def publish(channel, row):
payload = {
"schema_version": 1,
"job_id": str(row.id),
"type": "attachment.process.v1",
"attachment_id": str(row.attachment_id),
"pipeline_version": 1,
"attempt": row.attempt,
"generation": row.generation,
"created_at": row.created_at.astimezone(
timezone.utc
).isoformat().replace("+00:00", "Z"),
}
channel.basic_publish(
exchange="",
routing_key=QUEUE,
body=json.dumps(payload).encode(),
properties=pika.BasicProperties(
delivery_mode=2,
content_type="application/json",
message_id=str(row.id),
),
mandatory=True,
)
def publish_failed(channel, payload: dict):
channel.basic_publish(
exchange="",
routing_key=DLQ,
body=json.dumps(payload).encode(),
properties=pika.BasicProperties(
delivery_mode=2,
content_type="application/json",
message_id=payload["job_id"],
),
mandatory=True,
)
+875
View File
@@ -0,0 +1,875 @@
from uuid import UUID, uuid4
import hashlib
import json
from sqlalchemy import text
from sqlalchemy.ext.asyncio import AsyncSession
from src.models import Attachment, Chat, ChatPage, CreateChat, CreateUser, User, CreateMessage, Message, MessagePage
from src.security import hash_password
from src.errors import APIError
from src.cursor import decode_cursor, encode_cursor, decode_chat_cursor, encode_chat_cursor
async def create_user(
session: AsyncSession,
data: CreateUser,
) -> User:
user_id = uuid4()
password_hash = hash_password(data.password)
result = await session.execute(
text(
"""
INSERT INTO users (
id,
username,
display_name,
password_hash
)
VALUES (
:id,
:username,
:display_name,
:password_hash
)
RETURNING
id,
username,
display_name,
created_at
"""
),
{
"id": user_id,
"username": data.username,
"display_name": data.display_name,
"password_hash": password_hash,
},
)
row = result.one()
return User(
id=UUID(str(row.id)),
username=row.username,
display_name=row.display_name,
created_at=row.created_at,
)
async def get_user_credentials_by_username(
session: AsyncSession,
username: str,
):
result = await session.execute(
text(
"""
SELECT
id,
username,
display_name,
password_hash,
created_at
FROM users
WHERE username = :username
"""
),
{
"username": username,
},
)
return result.one_or_none()
async def get_user_by_id(
session: AsyncSession,
user_id: UUID,
) -> User | None:
result = await session.execute(
text(
"""
SELECT
id,
username,
display_name,
created_at
FROM users
WHERE id = :user_id
"""
),
{
"user_id": user_id,
},
)
row = result.one_or_none()
if row is None:
return None
return User(
id=UUID(str(row.id)),
username=row.username,
display_name=row.display_name,
created_at=row.created_at,
)
async def create_chat(
session: AsyncSession,
data: CreateChat,
current_user_id: UUID,
) -> Chat:
if current_user_id not in data.member_ids:
raise APIError(
status_code=400,
code="invalid_request",
message="Current user must be a chat member",
)
users_result = await session.execute(
text(
"""
SELECT id
FROM users
WHERE id = ANY(:member_ids)
"""
),
{
"member_ids": data.member_ids,
},
)
existing_user_ids = {
UUID(str(row.id))
for row in users_result
}
if existing_user_ids != set(data.member_ids):
raise APIError(
status_code=400,
code="invalid_request",
message="Unknown chat member",
)
chat_id = uuid4()
chat_result = await session.execute(
text(
"""
INSERT INTO chats (
id,
title
)
VALUES (
:id,
:title
)
RETURNING
id,
title,
created_at
"""
),
{
"id": chat_id,
"title": data.title,
},
)
chat_row = chat_result.one()
for member_id in data.member_ids:
await session.execute(
text(
"""
INSERT INTO chat_members (
chat_id,
user_id
)
VALUES (
:chat_id,
:user_id
)
"""
),
{
"chat_id": chat_id,
"user_id": member_id,
},
)
return Chat(
id=UUID(str(chat_row.id)),
title=chat_row.title,
member_ids=data.member_ids,
created_at=chat_row.created_at,
)
async def get_chat_by_id(
session: AsyncSession,
chat_id: UUID,
current_user_id: UUID,
) -> Chat | None:
chat_result = await session.execute(
text(
"""
SELECT
c.id,
c.title,
c.created_at
FROM chats AS c
JOIN chat_members AS cm
ON cm.chat_id = c.id
WHERE c.id = :chat_id
AND cm.user_id = :current_user_id
"""
),
{
"chat_id": chat_id,
"current_user_id": current_user_id,
},
)
chat_row = chat_result.one_or_none()
if chat_row is None:
return None
members_result = await session.execute(
text(
"""
SELECT user_id
FROM chat_members
WHERE chat_id = :chat_id
ORDER BY user_id
"""
),
{
"chat_id": chat_id,
},
)
member_ids = [
UUID(str(row.user_id))
for row in members_result
]
return Chat(
id=UUID(str(chat_row.id)),
title=chat_row.title,
member_ids=member_ids,
created_at=chat_row.created_at,
)
async def list_chats(
session: AsyncSession,
current_user_id: UUID,
limit: int,
cursor: str | None = None,
) -> ChatPage:
cursor_created_at = None
cursor_chat_id = None
if cursor is not None:
try:
cursor_user_id, cursor_created_at, cursor_chat_id = decode_chat_cursor(cursor)
except ValueError as exc:
raise APIError(status_code=400, code="invalid_request", message="Invalid cursor") from exc
if cursor_user_id != current_user_id:
raise APIError(status_code=400, code="invalid_request", message="Invalid cursor")
result = await session.execute(
text(
"""
SELECT c.id, c.title, c.created_at
FROM chats AS c
JOIN chat_members AS cm ON cm.chat_id = c.id
WHERE cm.user_id = :current_user_id
AND (
CAST(:cursor_created_at AS timestamptz) IS NULL
OR (c.created_at, c.id) > (
CAST(:cursor_created_at AS timestamptz),
CAST(:cursor_chat_id AS uuid)
)
)
ORDER BY c.created_at, c.id
LIMIT :fetch_limit
"""
),
{
"current_user_id": current_user_id,
"cursor_created_at": cursor_created_at,
"cursor_chat_id": cursor_chat_id,
"fetch_limit": limit + 1,
},
)
rows = result.all()
has_more = len(rows) > limit
page_rows = rows[:limit]
items = []
for chat_row in page_rows:
members_result = await session.execute(
text(
"""
SELECT user_id
FROM chat_members
WHERE chat_id = :chat_id
ORDER BY user_id
"""
),
{"chat_id": chat_row.id},
)
items.append(
Chat(
id=UUID(str(chat_row.id)),
title=chat_row.title,
member_ids=[UUID(str(row.user_id)) for row in members_result],
created_at=chat_row.created_at,
)
)
next_cursor = None
if has_more:
last = page_rows[-1]
next_cursor = encode_chat_cursor(
current_user_id, last.created_at, UUID(str(last.id))
)
return ChatPage(items=items, next_cursor=next_cursor)
async def create_message(
session: AsyncSession,
chat_id: UUID,
current_user_id: UUID,
data: CreateMessage,
idempotency_key: str,
) -> Message:
membership_result = await session.execute(
text("""
SELECT 1
FROM chat_members
WHERE chat_id = :chat_id
AND user_id = :user_id
"""),
{
"chat_id": chat_id,
"user_id": current_user_id,
},
)
if membership_result.scalar_one_or_none() is None:
raise APIError(
status_code=404,
code="not_found",
message="Chat not found",
)
canonical_body = json.dumps(
{
"text": data.text,
"attachment_ids": [str(attachment_id) for attachment_id in data.attachment_ids],
},
sort_keys=True,
ensure_ascii=False,
separators=(",", ":"),
)
request_hash = hashlib.sha256(
canonical_body.encode("utf-8")
).hexdigest()
scope = (
f"{current_user_id}:POST:"
f"/api/v1/chats/{chat_id}/messages:"
f"{idempotency_key}"
)
await session.execute(
text("""
SELECT pg_advisory_xact_lock(
hashtextextended(:scope, 0)
)
"""),
{"scope": scope},
)
await session.execute(
text("""
DELETE FROM message_idempotency
WHERE user_id = :user_id
AND chat_id = :chat_id
AND key = :key
AND created_at <=
clock_timestamp() - INTERVAL '24 hours'
"""),
{
"user_id": current_user_id,
"chat_id": chat_id,
"key": idempotency_key,
},
)
existing_result = await session.execute(
text("""
SELECT
i.request_hash,
m.id,
m.chat_id,
m.sender_id,
m.text,
m.created_at
FROM message_idempotency AS i
JOIN messages AS m
ON m.id = i.message_id
WHERE i.user_id = :user_id
AND i.chat_id = :chat_id
AND i.key = :key
"""),
{
"user_id": current_user_id,
"chat_id": chat_id,
"key": idempotency_key,
},
)
existing = existing_result.one_or_none()
if existing is not None:
if existing.request_hash != request_hash:
raise APIError(
status_code=409,
code="conflict",
message=(
"Idempotency key already used "
"with different request body"
),
)
attachment_result = await session.execute(
text("""
SELECT attachment_id
FROM message_attachments
WHERE message_id = :message_id
ORDER BY attachment_id
"""),
{"message_id": existing.id},
)
attachment_ids = [
UUID(str(row.attachment_id))
for row in attachment_result.all()
]
return Message(
id=UUID(str(existing.id)),
chat_id=UUID(str(existing.chat_id)),
sender_id=UUID(str(existing.sender_id)),
text=existing.text,
created_at=existing.created_at,
attachment_ids=attachment_ids,
)
if data.attachment_ids:
attachments_result = await session.execute(
text("""
SELECT id
FROM attachments
WHERE id = ANY (:attachment_ids)
AND chat_id = :chat_id
AND status != 'failed'
"""),
{
"attachment_ids": data.attachment_ids,
"chat_id": chat_id,
},
)
valid_attachment_ids = {
UUID(str(row.id))
for row in attachments_result.all()
}
if valid_attachment_ids != set(data.attachment_ids):
raise APIError(
status_code=400,
code="invalid_request",
message="Invalid attachments for this chat",
)
message_id = uuid4()
message_result = await session.execute(
text("""
INSERT INTO messages (
id,
chat_id,
sender_id,
text
)
VALUES (
:id,
:chat_id,
:sender_id,
:text
)
RETURNING
id,
chat_id,
sender_id,
text,
created_at
"""),
{
"id": message_id,
"chat_id": chat_id,
"sender_id": current_user_id,
"text": data.text,
},
)
row = message_result.one()
for attachment_id in data.attachment_ids:
await session.execute(
text("""
INSERT INTO message_attachments (message_id,
attachment_id)
VALUES (:message_id,
:attachment_id)
"""),
{
"message_id": message_id,
"attachment_id": attachment_id,
},
)
await session.execute(
text("""
INSERT INTO message_idempotency (
user_id,
chat_id,
key,
request_hash,
message_id
)
VALUES (
:user_id,
:chat_id,
:key,
:request_hash,
:message_id
)
"""),
{
"user_id": current_user_id,
"chat_id": chat_id,
"key": idempotency_key,
"request_hash": request_hash,
"message_id": message_id,
},
)
return Message(
id=UUID(str(row.id)),
chat_id=UUID(str(row.chat_id)),
sender_id=UUID(str(row.sender_id)),
text=row.text,
created_at=row.created_at,
attachment_ids=data.attachment_ids,
)
async def get_messages(
session: AsyncSession,
chat_id: UUID,
current_user_id: UUID,
limit: int,
cursor: str | None,
) -> MessagePage:
membership_result = await session.execute(
text(
"""
SELECT 1
FROM chat_members
WHERE chat_id = :chat_id
AND user_id = :current_user_id
"""
),
{
"chat_id": chat_id,
"current_user_id": current_user_id,
},
)
if membership_result.scalar_one_or_none() is None:
raise APIError(
status_code=404,
code="not_found",
message="Chat not found",
)
cursor_created_at = None
cursor_message_id = None
if cursor is not None:
try:
(
cursor_chat_id,
cursor_created_at,
cursor_message_id,
) = decode_cursor(cursor)
except ValueError:
raise APIError(
status_code=400,
code="invalid_request",
message="Invalid cursor",
)
if cursor_chat_id != chat_id:
raise APIError(
status_code=400,
code="invalid_request",
message="Cursor belongs to another chat",
)
if cursor is None:
messages_result = await session.execute(
text(
"""
SELECT
id,
chat_id,
sender_id,
text,
created_at
FROM messages
WHERE chat_id = :chat_id
ORDER BY created_at, id
LIMIT :fetch_limit
"""
),
{
"chat_id": chat_id,
"fetch_limit": limit + 1,
},
)
else:
messages_result = await session.execute(
text(
"""
SELECT
id,
chat_id,
sender_id,
text,
created_at
FROM messages
WHERE chat_id = :chat_id
AND (created_at, id) > (
:cursor_created_at,
:cursor_message_id
)
ORDER BY created_at, id
LIMIT :fetch_limit
"""
),
{
"chat_id": chat_id,
"cursor_created_at": cursor_created_at,
"cursor_message_id": cursor_message_id,
"fetch_limit": limit + 1,
},
)
rows = messages_result.all()
has_more = len(rows) > limit
rows = rows[:limit]
attachments_by_message = {}
if rows:
attachment_result = await session.execute(
text("""
SELECT message_id, attachment_id
FROM message_attachments
WHERE message_id = ANY(:message_ids)
ORDER BY message_id, attachment_id
"""),
{
"message_ids": [row.id for row in rows],
},
)
for attachment_row in attachment_result.all():
message_id = UUID(str(attachment_row.message_id))
attachment_id = UUID(str(attachment_row.attachment_id))
attachments_by_message.setdefault(
message_id, []
).append(attachment_id)
items = [
Message(
id=UUID(str(row.id)),
chat_id=UUID(str(row.chat_id)),
sender_id=UUID(str(row.sender_id)),
text=row.text,
created_at=row.created_at,
attachment_ids=attachments_by_message.get(
UUID(str(row.id)), []
),
)
for row in rows
]
next_cursor = None
if has_more:
last_message = items[-1]
next_cursor = encode_cursor(
chat_id=chat_id,
created_at=last_message.created_at,
message_id=last_message.id,
)
return MessagePage(
items=items,
next_cursor=next_cursor,
)
def attachment_from_row(row) -> Attachment:
return Attachment(
id=UUID(str(row.id)),
chat_id=UUID(str(row.chat_id)),
owner_id=UUID(str(row.owner_id)),
filename=row.filename,
content_type=row.content_type,
size_bytes=row.size_bytes,
sha256=row.sha256,
status=row.status,
variants=row.variants,
error_code=row.error_code,
created_at=row.created_at,
updated_at=row.updated_at,
)
async def create_attachment(
session: AsyncSession,
attachment_id: UUID,
chat_id: UUID,
owner_id: UUID,
filename: str,
content_type: str,
size_bytes: int,
sha256: str,
s3_key: str,
) -> Attachment:
result = await session.execute(
text(
"""
INSERT INTO attachments (
id,
chat_id,
owner_id,
filename,
content_type,
size_bytes,
sha256,
s3_key,
status
)
VALUES (
:id,
:chat_id,
:owner_id,
:filename,
:content_type,
:size_bytes,
:sha256,
:s3_key,
'queued'
)
RETURNING
id,
chat_id,
owner_id,
filename,
content_type,
size_bytes,
sha256,
status,
variants,
error_code,
created_at,
updated_at
"""
),
{
"id": attachment_id,
"chat_id": chat_id,
"owner_id": owner_id,
"filename": filename,
"content_type": content_type,
"size_bytes": size_bytes,
"sha256": sha256,
"s3_key": s3_key,
},
)
return attachment_from_row(result.one())
async def get_attachment(
session: AsyncSession,
attachment_id: UUID,
current_user_id: UUID,
):
result = await session.execute(
text(
"""
SELECT
a.id,
a.chat_id,
a.owner_id,
a.filename,
a.content_type,
a.size_bytes,
a.sha256,
a.s3_key,
a.status,
a.variants,
a.error_code,
a.created_at,
a.updated_at
FROM attachments AS a
JOIN chat_members AS cm
ON cm.chat_id = a.chat_id
WHERE a.id = :attachment_id
AND cm.user_id = :current_user_id
"""
),
{
"attachment_id": attachment_id,
"current_user_id": current_user_id,
},
)
row = result.one_or_none()
if row is None:
return None
return attachment_from_row(row), row.s3_key
async def attachment_key_exists(session: AsyncSession, s3_key: str) -> bool:
result = await session.execute(
text("SELECT 1 FROM attachments WHERE s3_key = :s3_key"),
{"s3_key": s3_key},
)
return result.first() is not None
+16
View File
@@ -0,0 +1,16 @@
from argon2 import PasswordHasher
from argon2.exceptions import VerifyMismatchError
password_hasher = PasswordHasher()
def hash_password(password: str) -> str:
return password_hasher.hash(password)
def verify_password(password_hash: str, password: str) -> bool:
try:
return password_hasher.verify(password_hash, password)
except VerifyMismatchError:
return False
+87
View File
@@ -0,0 +1,87 @@
import os
import secrets
from uuid import UUID
from redis.asyncio import Redis
from redis.exceptions import RedisError
from src.errors import APIError
CACHE_URL_FILE = "/run/secrets/cache_url"
DEFAULT_SESSION_TTL_SECONDS = 3600
def read_cache_url() -> str:
with open(CACHE_URL_FILE, encoding="utf-8") as file:
return file.read().strip()
def get_session_ttl_seconds() -> int:
return int(
os.getenv(
"SESSION_TTL_SECONDS",
str(DEFAULT_SESSION_TTL_SECONDS),
)
)
redis_client = Redis.from_url(
read_cache_url(),
decode_responses=True,
socket_connect_timeout=5,
socket_timeout=5,
)
def session_store_unavailable() -> APIError:
return APIError(
status_code=503,
code="unavailable",
message="Session store unavailable",
)
async def create_session(user_id: UUID) -> tuple[str, int]:
token = secrets.token_urlsafe(32)
ttl = get_session_ttl_seconds()
try:
await redis_client.set(
f"session:{token}",
str(user_id),
ex=ttl,
)
except RedisError as exc:
raise session_store_unavailable() from exc
return token, ttl
async def get_session_user_id(token: str) -> UUID | None:
try:
value = await redis_client.get(f"session:{token}")
except RedisError as exc:
raise session_store_unavailable() from exc
if value is None:
return None
try:
return UUID(value)
except ValueError:
return None
async def delete_session(token: str) -> None:
try:
await redis_client.delete(f"session:{token}")
except RedisError as exc:
raise session_store_unavailable() from exc
async def ping_session_store() -> bool:
try:
return bool(await redis_client.ping())
except RedisError:
return False
+87
View File
@@ -0,0 +1,87 @@
import asyncio
import os
from pathlib import Path
import boto3
from botocore.config import Config
def read_secret(env_name: str) -> str:
path = os.environ[env_name]
return Path(path).read_text(encoding="utf-8").strip()
s3_client = boto3.client(
"s3",
endpoint_url=os.environ["S3_ENDPOINT"],
region_name=os.getenv("S3_REGION", "us-east-1"),
aws_access_key_id=read_secret("S3_ACCESS_KEY_ID_FILE"),
aws_secret_access_key=read_secret("S3_SECRET_ACCESS_KEY_FILE"),
config=Config(
signature_version="s3v4",
connect_timeout=2,
read_timeout=3,
retries={"total_max_attempts": 1, "mode": "standard"},
s3={"addressing_style": "path"},
),
)
S3_BUCKET = os.environ["S3_BUCKET"]
async def put_object(key: str, body: bytes, content_type: str) -> None:
await asyncio.to_thread(
s3_client.put_object,
Bucket=S3_BUCKET,
Key=key,
Body=body,
ContentType=content_type,
)
async def get_object(key: str) -> bytes:
def download() -> bytes:
response = s3_client.get_object(Bucket=S3_BUCKET, Key=key)
stream = response["Body"]
try:
return stream.read()
finally:
stream.close()
return await asyncio.to_thread(download)
async def delete_object(key: str) -> None:
await asyncio.to_thread(
s3_client.delete_object,
Bucket=S3_BUCKET,
Key=key,
)
async def ping_storage() -> bool:
try:
await asyncio.to_thread(
s3_client.head_bucket,
Bucket=S3_BUCKET,
)
return True
except Exception:
return False
async def object_exists(key: str) -> bool:
from botocore.exceptions import ClientError
try:
await asyncio.to_thread(
s3_client.head_object,
Bucket=S3_BUCKET,
Key=key,
)
return True
except ClientError as exc:
if exc.response["Error"]["Code"] in {"404", "NoSuchKey", "NotFound"}:
return False
raise
+450
View File
@@ -0,0 +1,450 @@
import asyncio
import hashlib
import io
import os
import json
import logging
import time
import random
from uuid import UUID, uuid4
from PIL import Image, ImageOps, UnidentifiedImageError
from sqlalchemy import text
from src.database import SessionFactory
from src.queue import QUEUE, connect
from src.storage import get_object, put_object
from src.images import MAX_IMAGE_PIXELS, MAX_UPLOAD_BYTES
logging.basicConfig(level=logging.INFO)
MAX_PROCESSING_SECONDS = float(
os.getenv("MAX_PROCESSING_SECONDS", "30")
)
class PermanentProcessingError(Exception):
pass
def retry_delay(attempt: int) -> float:
base = min(60.0, 2.0 ** attempt)
return base * random.uniform(0.75, 1.25)
def validate_worker_image(body: bytes):
if not body or len(body) > MAX_UPLOAD_BYTES:
raise PermanentProcessingError("invalid_image_size")
try:
with Image.open(io.BytesIO(body)) as image:
if image.format not in {"PNG", "JPEG"}:
raise PermanentProcessingError("unsupported_image_format")
if image.width * image.height > MAX_IMAGE_PIXELS:
raise PermanentProcessingError("image_pixel_limit")
image.verify()
except (UnidentifiedImageError, OSError, ValueError, Image.DecompressionBombError) as exc:
raise PermanentProcessingError("invalid_image") from exc
def make_thumbnail(body: bytes):
with Image.open(io.BytesIO(body)) as source:
source.load()
image = ImageOps.exif_transpose(source)
image.thumbnail((256, 256), Image.Resampling.LANCZOS)
if image.mode != "RGB":
rgba = image.convert("RGBA")
background = Image.new("RGBA", rgba.size, "white")
background.alpha_composite(rgba)
image = background.convert("RGB")
output = io.BytesIO()
image.save(output, format="JPEG", quality=85, optimize=True)
return output.getvalue(), image.width, image.height
def make_crop(body: bytes):
with Image.open(io.BytesIO(body)) as source:
source.load()
image = ImageOps.exif_transpose(source).convert("RGB")
image = ImageOps.fit(
image,
(128, 128),
method=Image.Resampling.LANCZOS,
centering=(0.5, 0.5),
)
output = io.BytesIO()
image.save(output, format="JPEG", quality=85, optimize=True)
return output.getvalue(), image.width, image.height
def make_optimized(body: bytes):
with Image.open(io.BytesIO(body)) as source:
source.load()
image = ImageOps.exif_transpose(source).convert("RGB")
output = io.BytesIO()
image.save(output, format="WEBP", quality=80, method=4)
return output.getvalue(), image.width, image.height
def fault_injection(point: str):
if os.getenv("FAULT_INJECTION_ENABLED") != "1":
return
if os.getenv("FAULT_INJECTION_POINT") != point:
return
marker = f"/tmp/lab5_fault_{point}"
try:
fd = os.open(marker, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
except FileExistsError:
return
os.close(fd)
logging.error("FAULT INJECTION: %s", point)
os._exit(77)
async def process(payload: dict):
job_id = UUID(payload["job_id"])
attachment_id = UUID(payload["attachment_id"])
generation = payload.get("generation", 1)
execution_id = uuid4()
async with SessionFactory() as session:
async with session.begin():
result = await session.execute(
text("""
SELECT j.status, j.attempt, j.generation,
a.s3_key, a.content_type, a.chat_id
FROM attachment_jobs j
JOIN attachments a ON a.id = j.attachment_id
WHERE j.id = :job_id
AND j.attachment_id = :attachment_id
FOR UPDATE OF j
"""),
{"job_id": job_id, "attachment_id": attachment_id},
)
row = result.one_or_none()
if row is None or row.status in ("ready", "failed"):
return
if row.attempt != payload["attempt"]:
return
if row.generation != generation:
return
if row.status == "processing":
return
await session.execute(
text("""
UPDATE attachment_jobs
SET status = 'processing',
started_at = now(),
execution_id = :execution_id
WHERE id = :id
"""),
{"id": job_id, "execution_id": execution_id},
)
await session.execute(
text("""
UPDATE attachments
SET status = 'processing', updated_at = now()
WHERE id = :id
"""),
{"id": attachment_id},
)
try:
body = await get_object(row.s3_key)
validate_worker_image(body)
thumbnail, width, height = await asyncio.to_thread(
make_thumbnail, body
)
crop, crop_width, crop_height = await asyncio.to_thread(
make_crop, body
)
optimized, optimized_width, optimized_height = await asyncio.to_thread(
make_optimized, body
)
original_key = (
f"attachments/{row.chat_id}/{attachment_id}/g{generation}/original"
)
thumbnail_key = (
f"variants/{row.chat_id}/{attachment_id}/g{generation}/thumbnail.jpg"
)
fault_injection("before_side_effect")
await put_object(original_key, body, row.content_type)
await put_object(thumbnail_key, thumbnail, "image/jpeg")
crop_key = (
f"variants/{row.chat_id}/{attachment_id}/g{generation}/crop.jpg"
)
optimized_key = (
f"variants/{row.chat_id}/{attachment_id}/g{generation}/optimized.webp"
)
await put_object(crop_key, crop, "image/jpeg")
await put_object(optimized_key, optimized, "image/webp")
fault_injection("after_side_effect")
if (
os.getenv("FAULT_INJECTION_ENABLED") == "1"
and os.getenv("FAULT_INJECTION_POINT") == "pause_before_commit"
):
marker = "/tmp/lab5_pause_before_commit"
with open(marker, "w", encoding="utf-8") as file:
file.write(str(execution_id))
deadline = time.monotonic() + 180
while os.path.exists(marker):
if time.monotonic() >= deadline:
raise TimeoutError("pause_before_commit_timeout")
await asyncio.sleep(0.1)
variant = {
"name": "thumbnail",
"s3_key": thumbnail_key,
"content_type": "image/jpeg",
"size_bytes": len(thumbnail),
"width": width,
"height": height,
"sha256": hashlib.sha256(thumbnail).hexdigest(),
}
async with SessionFactory() as session:
async with session.begin():
result = await session.execute(
text("""
SELECT status, attempt, generation, execution_id
FROM attachment_jobs
WHERE id = :id
FOR UPDATE
"""),
{"id": job_id},
)
current = result.one_or_none()
if (
current is None
or current.status != "processing"
or current.attempt != row.attempt
or current.generation != generation
or current.execution_id != execution_id
):
return
await session.execute(
text("""
UPDATE attachments
SET status = 'ready',
s3_key = :key,
variants = CAST(:variants AS json),
error_code = NULL,
updated_at = now()
WHERE id = :id
"""),
{
"id": attachment_id,
"key": original_key,
"variants": json.dumps([
variant,
{
"name": "crop",
"s3_key": crop_key,
"content_type": "image/jpeg",
"size_bytes": len(crop),
"width": crop_width,
"height": crop_height,
"sha256": hashlib.sha256(crop).hexdigest(),
},
{
"name": "optimized",
"s3_key": optimized_key,
"content_type": "image/webp",
"size_bytes": len(optimized),
"width": optimized_width,
"height": optimized_height,
"sha256": hashlib.sha256(optimized).hexdigest(),
},
]),
},
)
await session.execute(
text("""
UPDATE attachment_jobs
SET status = 'ready',
next_attempt_at = NULL,
last_error_code = NULL,
finished_at = now()
WHERE id = :id
"""),
{"id": job_id},
)
except Exception as exc:
logging.exception("Processing failed: %s", attachment_id)
permanent = isinstance(exc, PermanentProcessingError)
error_code = (
str(exc)[:64]
if permanent
else "temporary_processing_error"
)
async with SessionFactory() as session:
async with session.begin():
result = await session.execute(
text("""
SELECT status, attempt, generation, execution_id
FROM attachment_jobs
WHERE id = :id
FOR UPDATE
"""),
{"id": job_id},
)
current = result.one_or_none()
if (
current is None
or current.status != "processing"
or current.attempt != row.attempt
or current.generation != generation
or current.execution_id != execution_id
):
return
if permanent or row.attempt >= 5:
await session.execute(
text("""
UPDATE attachment_jobs
SET status = 'failed',
next_attempt_at = NULL,
last_error_code = :error_code,
finished_at = now()
WHERE id = :id
"""),
{"id": job_id, "error_code": error_code},
)
await session.execute(
text("""
UPDATE attachments
SET status = 'failed',
error_code = :error_code,
updated_at = now()
WHERE id = :id
"""),
{
"id": attachment_id,
"error_code": error_code,
},
)
else:
await session.execute(
text("""
UPDATE attachment_jobs
SET status = 'queued',
attempt = attempt + 1,
published_at = NULL,
started_at = NULL,
next_attempt_at = now() + (:delay * interval '1 second'),
last_error_code = :error_code
WHERE id = :id
"""),
{
"id": job_id,
"delay": retry_delay(row.attempt),
"error_code": error_code,
}
)
await session.execute(
text("""
UPDATE attachments
SET status = 'queued', updated_at = now()
WHERE id = :id
"""),
{"id": attachment_id},
)
def main():
while True:
connection = None
try:
connection, channel = connect()
channel.basic_qos(prefetch_count=1)
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
def callback(ch, method, properties, body):
try:
loop.run_until_complete(
asyncio.wait_for(
process(json.loads(body)),
timeout=MAX_PROCESSING_SECONDS,
)
)
fault_injection("before_ack")
ch.basic_ack(delivery_tag=method.delivery_tag)
except Exception:
logging.exception("Worker processing error")
ch.basic_nack(
delivery_tag=method.delivery_tag,
requeue=True,
)
time.sleep(2)
channel.basic_consume(
queue=QUEUE,
on_message_callback=callback,
)
channel.start_consuming()
except Exception:
logging.exception("Worker reconnect")
time.sleep(3)
finally:
if connection and connection.is_open:
connection.close()
if __name__ == "__main__":
main()
+160
View File
@@ -0,0 +1,160 @@
import json
import subprocess
import sys
import time
from pathlib import Path
from client import Client, check
from fault_injection_grade5 import job_state
ROOT = Path(__file__).resolve().parents[1]
COMPOSE = ["docker", "compose", "-f", "compose.yaml", "-f", "compose.concurrent.yaml"]
MARKER = "/tmp/lab5_pause_before_commit"
def run(*args, check_result=True):
return subprocess.run(
args,
cwd=ROOT,
capture_output=True,
text=True,
check=check_result,
)
def compose(*args):
return run(*COMPOSE, *args)
def sql(query):
result = run(
"docker", "compose", "exec", "-T", "postgres",
"psql", "-U", "lab2", "-d", "messenger",
"-t", "-A", "-c", query,
)
return result.stdout.strip()
def wait_until(predicate, timeout, message):
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if predicate():
return
time.sleep(0.25)
raise AssertionError(message)
def marker_exists():
result = run(
*COMPOSE,
"exec", "-T", "worker",
"test", "-f", MARKER,
check_result=False,
)
return result.returncode == 0
def release_worker():
compose("exec", "-T", "worker", "rm", "-f", MARKER)
def main():
try:
compose("up", "-d", "--build", "--force-recreate", "worker")
client = Client()
alice, bob, _, chat = client.bootstrap()
attachment, original = client.attachment(alice, chat)
attachment_id = attachment["id"]
wait_until(
marker_exists,
60,
"first worker did not pause",
)
first = job_state(attachment_id)
check(first["job_status"] == "processing", "first execution not processing")
check(first["execution_id"] is not None, "missing execution ID")
sql(f"""
UPDATE attachment_jobs
SET started_at = now() - interval '120 seconds'
WHERE id = '{first["job_id"]}'
AND status = 'processing';
""")
wait_until(
lambda: (
job_state(attachment_id)["execution_id"]
!= first["execution_id"]
),
15,
"dispatcher did not release the old execution",
)
compose("up", "-d", "--build", "worker-second")
deadline = time.monotonic() + 120
while time.monotonic() < deadline:
state = job_state(attachment_id)
if state["job_status"] == "ready":
break
time.sleep(0.25)
else:
raise AssertionError("second worker did not complete the job")
check(state["execution_id"] != first["execution_id"], "execution ID unchanged")
check(state["attachment_status"] == "ready", "attachment not ready")
release_worker()
time.sleep(3)
after = job_state(attachment_id)
check(after == state, "old execution overwrote newer result")
content = client.call(
"GET",
f"/api/v1/attachments/{attachment_id}/content",
actor=bob,
)
check(content["bytes"] == original, "original content changed")
print("PASS: stale execution rejected after newer execution completed")
finally:
try:
release_worker()
except Exception:
pass
try:
compose("stop", "worker-second")
except Exception:
pass
run(
"docker", "compose",
"up", "-d", "--force-recreate", "worker",
)
print("Normal worker configuration restored")
if __name__ == "__main__":
try:
main()
except Exception as exc:
print(f"FAIL: {exc}", file=sys.stderr)
raise SystemExit(1)
+129
View File
@@ -0,0 +1,129 @@
import json
import subprocess
import sys
import time
from pathlib import Path
from client import Client, check
ROOT = Path(__file__).resolve().parents[1]
OVERRIDE = ROOT / "compose.fault.yaml"
POINTS = ("before_side_effect", "after_side_effect", "before_ack")
CONTAINER = "mtusi2026lab5-worker-1"
def command(*args):
return subprocess.run(
args,
cwd=ROOT,
capture_output=True,
text=True,
check=True,
)
def configure(point):
OVERRIDE.write_text(
"services:\n"
" worker:\n"
" environment:\n"
' FAULT_INJECTION_ENABLED: "1"\n'
f" FAULT_INJECTION_POINT: {point}\n",
encoding="utf-8",
)
command(
"docker", "compose",
"-f", "compose.yaml",
"-f", "compose.fault.yaml",
"up", "-d", "--force-recreate", "worker",
)
def restore():
command(
"docker", "compose",
"up", "-d", "--force-recreate", "worker",
)
def restart_count():
result = command(
"docker", "inspect",
"-f", "{{.RestartCount}}",
CONTAINER,
)
return int(result.stdout.strip())
def crash_logged(point):
result = command(
"docker", "logs", CONTAINER,
)
return f"FAULT INJECTION: {point}" in result.stderr + result.stdout
def run_case(point):
print(f"RUN: {point}", flush=True)
configure(point)
client = Client()
alice, bob, _, chat = client.bootstrap()
attachment, original = client.attachment(alice, chat)
deadline = time.monotonic() + 30
while time.monotonic() < deadline:
if restart_count() >= 1 and crash_logged(point):
break
time.sleep(0.5)
else:
raise AssertionError(f"{point}: worker did not crash")
ready = client.wait_ready(bob, attachment, wait_seconds=150)
check(ready["status"] == "ready", f"{point}: not ready")
content = client.call(
"GET",
f"/api/v1/attachments/{attachment['id']}/content",
actor=bob,
)
check(content["bytes"] == original, f"{point}: original changed")
for variant in ("thumbnail", "crop", "optimized"):
response = client.call(
"GET",
f"/api/v1/attachments/{attachment['id']}/content?variant={variant}",
actor=bob,
)
check(bool(response["bytes"]), f"{point}: {variant} is empty")
check(restart_count() == 1, f"{point}: unexpected restart count")
print(
f"PASS: {point}, recovered, original and variants intact",
flush=True,
)
def main():
try:
for point in POINTS:
run_case(point)
finally:
print("Restoring normal worker configuration...", flush=True)
restore()
print("FAULT INJECTION DISABLED", flush=True)
if __name__ == "__main__":
try:
main()
except (AssertionError, subprocess.CalledProcessError, OSError) as exc:
print(f"FAIL: {exc}", file=sys.stderr)
raise SystemExit(1)
+463
View File
@@ -0,0 +1,463 @@
import json
import subprocess
import time
from pathlib import Path
import pika
from client import Client, check
ROOT = Path(__file__).resolve().parents[1]
BASE_URL = "http://localhost:8080"
def sql(query):
result = subprocess.run(
[
"docker", "compose", "exec", "-T", "postgres",
"psql", "-U", "lab2", "-d", "messenger",
"-t", "-A", "-c", query,
],
cwd=ROOT,
capture_output=True,
text=True,
check=True,
)
return result.stdout.strip()
def job_state(attachment_id):
result = sql(f"""
SELECT row_to_json(t)
FROM (
SELECT
j.id AS job_id,
j.attachment_id,
j.status AS job_status,
a.status AS attachment_status,
j.attempt,
j.generation,
j.execution_id,
j.finished_at,
a.s3_key,
a.variants
FROM attachment_jobs j
JOIN attachments a ON a.id = j.attachment_id
WHERE a.id = '{attachment_id}'
) t;
""")
check(bool(result), "job not found")
return json.loads(result)
def publish_duplicate(payload):
result = subprocess.run(
[
"docker", "compose", "exec", "-T", "worker",
"python", "-c",
"""
import json
import sys
import pika
from src.queue import connect, QUEUE
payload = json.loads(sys.stdin.read())
connection, channel = connect()
channel.confirm_delivery()
channel.basic_publish(
exchange="",
routing_key=QUEUE,
body=json.dumps(payload).encode(),
properties=pika.BasicProperties(
delivery_mode=2,
content_type="application/json",
message_id=payload["job_id"],
),
mandatory=True,
)
connection.close()
""",
],
cwd=ROOT,
input=json.dumps(payload),
capture_output=True,
text=True,
check=True,
)
return result
def test_duplicate():
client = Client(BASE_URL)
alice, bob, _, chat = client.bootstrap()
attachment, original = client.attachment(alice, chat)
attachment_id = attachment["id"]
client.wait_ready(bob, attachment, wait_seconds=120)
before = job_state(attachment_id)
check(before["job_status"] == "ready", "job is not ready")
check(before["attachment_status"] == "ready", "attachment is not ready")
payload = {
"schema_version": 1,
"job_id": before["job_id"],
"type": "attachment.process.v1",
"attachment_id": attachment_id,
"pipeline_version": 1,
"attempt": before["attempt"],
"generation": before["generation"],
"created_at": "2026-01-01T00:00:00Z",
}
publish_duplicate(payload)
publish_duplicate(payload)
deadline = time.monotonic() + 30
while time.monotonic() < deadline:
queue = subprocess.run(
[
"docker", "compose", "exec", "-T", "rabbitmq",
"rabbitmqctl", "list_queues",
"name", "messages_ready", "messages_unacknowledged",
"--quiet",
],
cwd=ROOT,
capture_output=True,
text=True,
check=True,
)
lines = queue.stdout.splitlines()
matching = [
line.split()
for line in lines
if line.startswith("attachment.process.v1")
]
if matching and matching[0][1:] == ["0", "0"]:
break
time.sleep(1)
else:
raise AssertionError("duplicate messages were not consumed")
after = job_state(attachment_id)
check(before == after, "duplicate delivery changed committed result")
content = client.call(
"GET",
f"/api/v1/attachments/{attachment_id}/content",
actor=bob,
)
check(content["bytes"] == original, "original content changed")
print("PASS: duplicate delivery, state unchanged, original intact")
def test_reorder():
client = Client(BASE_URL)
alice, bob, _, chat = client.bootstrap()
attachment, original = client.attachment(alice, chat)
attachment_id = attachment["id"]
client.wait_ready(bob, attachment, wait_seconds=120)
before = job_state(attachment_id)
check(before["job_status"] == "ready", "job is not ready")
payload = {
"schema_version": 1,
"job_id": before["job_id"],
"type": "attachment.process.v1",
"attachment_id": attachment_id,
"pipeline_version": 1,
"attempt": before["attempt"],
"generation": before["generation"],
"created_at": "2026-01-01T00:00:00Z",
}
for attempt in (
before["attempt"] + 1,
before["attempt"] - 1,
before["attempt"],
):
message = dict(payload)
message["attempt"] = attempt
publish_duplicate(message)
deadline = time.monotonic() + 30
while time.monotonic() < deadline:
queue = subprocess.run(
[
"docker", "compose", "exec", "-T", "rabbitmq",
"rabbitmqctl", "list_queues",
"name", "messages_ready", "messages_unacknowledged",
"--quiet",
],
cwd=ROOT,
capture_output=True,
text=True,
check=True,
)
matching = [
line.split()
for line in queue.stdout.splitlines()
if line.startswith("attachment.process.v1")
]
if matching and matching[0][1:] == ["0", "0"]:
break
time.sleep(1)
else:
raise AssertionError("reordered messages were not consumed")
after = job_state(attachment_id)
check(before == after, "reordered delivery changed committed result")
content = client.call(
"GET",
f"/api/v1/attachments/{attachment_id}/content",
actor=bob,
)
check(content["bytes"] == original, "original content changed")
print("PASS: reordered attempts, state unchanged, original intact")
def test_stale_generation():
client = Client(BASE_URL)
alice, bob, _, chat = client.bootstrap()
attachment, original = client.attachment(alice, chat)
attachment_id = attachment["id"]
client.wait_ready(bob, attachment, wait_seconds=120)
before = job_state(attachment_id)
check(before["job_status"] == "ready", "job is not ready")
check(before["generation"] == 1, "unexpected initial generation")
payload = {
"schema_version": 1,
"job_id": before["job_id"],
"type": "attachment.process.v1",
"attachment_id": attachment_id,
"pipeline_version": 1,
"attempt": before["attempt"],
"generation": before["generation"],
"created_at": "2026-01-01T00:00:00Z",
}
for generation in (0, 2, 1):
message = dict(payload)
message["generation"] = generation
publish_duplicate(message)
deadline = time.monotonic() + 30
while time.monotonic() < deadline:
queue = subprocess.run(
[
"docker", "compose", "exec", "-T", "rabbitmq",
"rabbitmqctl", "list_queues",
"name", "messages_ready", "messages_unacknowledged",
"--quiet",
],
cwd=ROOT,
capture_output=True,
text=True,
check=True,
)
matching = [
line.split()
for line in queue.stdout.splitlines()
if line.startswith("attachment.process.v1")
]
if matching and matching[0][1:] == ["0", "0"]:
break
time.sleep(1)
else:
raise AssertionError("generation messages were not consumed")
after = job_state(attachment_id)
check(before == after, "generation mismatch changed committed result")
content = client.call(
"GET",
f"/api/v1/attachments/{attachment_id}/content",
actor=bob,
)
check(content["bytes"] == original, "original content changed")
print("PASS: stale and future generations rejected, result unchanged")
def test_replay_generation():
client = Client(BASE_URL)
alice, bob, _, chat = client.bootstrap()
attachment, original = client.attachment(alice, chat)
attachment_id = attachment["id"]
client.wait_ready(bob, attachment, wait_seconds=120)
before = job_state(attachment_id)
check(before["job_status"] == "ready", "initial job is not ready")
check(before["generation"] == 1, "unexpected initial generation")
job_id = before["job_id"]
sql(f"""
BEGIN;
UPDATE attachment_jobs
SET status = 'failed',
last_error_code = 'test_replay',
finished_at = now(),
failed_published_at = now()
WHERE id = '{job_id}'
AND status = 'ready';
UPDATE attachments
SET status = 'failed',
error_code = 'test_replay',
updated_at = now()
WHERE id = '{attachment_id}'
AND status = 'ready';
COMMIT;
""")
result = subprocess.run(
[
"docker", "compose", "exec", "-T", "worker",
"python", "-m", "scripts.replay_job", job_id,
],
cwd=ROOT,
capture_output=True,
text=True,
check=True,
)
check("REPLAY ACCEPTED" in result.stdout, "replay was rejected")
replayed = job_state(attachment_id)
check(replayed["generation"] == 2, "generation did not increase")
stale_payload = {
"schema_version": 1,
"job_id": job_id,
"type": "attachment.process.v1",
"attachment_id": attachment_id,
"pipeline_version": 1,
"attempt": 1,
"generation": 1,
"created_at": "2026-01-01T00:00:00Z",
}
publish_duplicate(stale_payload)
deadline = time.monotonic() + 120
while time.monotonic() < deadline:
current = job_state(attachment_id)
if current["job_status"] == "ready":
break
check(current["job_status"] != "failed", "replay failed")
time.sleep(0.5)
else:
raise AssertionError("replayed job did not become ready")
check(current["generation"] == 2, "generation did not increase")
check(current["attachment_status"] == "ready", "attachment not ready")
check(current["execution_id"] != before["execution_id"], "execution ID unchanged")
payload = {
"schema_version": 1,
"job_id": job_id,
"type": "attachment.process.v1",
"attachment_id": attachment_id,
"pipeline_version": 1,
"attempt": 1,
"generation": 1,
"created_at": "2026-01-01T00:00:00Z",
}
publish_duplicate(payload)
deadline = time.monotonic() + 30
while time.monotonic() < deadline:
queue = subprocess.run(
[
"docker", "compose", "exec", "-T", "rabbitmq",
"rabbitmqctl", "list_queues",
"name", "messages_ready", "messages_unacknowledged",
"--quiet",
],
cwd=ROOT,
capture_output=True,
text=True,
check=True,
)
matching = [
line.split()
for line in queue.stdout.splitlines()
if line.startswith("attachment.process.v1")
]
if matching and matching[0][1:] == ["0", "0"]:
break
time.sleep(1)
else:
raise AssertionError("stale message was not consumed")
after = job_state(attachment_id)
check(current == after, "stale generation overwrote newer result")
content = client.call(
"GET",
f"/api/v1/attachments/{attachment_id}/content",
actor=bob,
)
check(content["bytes"] == original, "original changed after replay")
print("PASS: replay generation 1 -> 2, stale message rejected")
if __name__ == "__main__":
test_duplicate()
test_reorder()
test_stale_generation()
test_replay_generation()
+108
View File
@@ -0,0 +1,108 @@
import sys
import time
from pathlib import Path
from client import Client, check
from image_probe import jpeg_size
BASE_URL = "http://localhost:8080"
STATE_FILE = Path("/tmp/lab5_grade3_attachment.txt")
def prepare():
c = Client(BASE_URL)
alice, bob, _, chat = c.bootstrap()
attachment, original = c.attachment(alice, chat)
attachment_id = attachment["id"]
current = c.call(
"GET",
f"/api/v1/attachments/{attachment_id}",
actor=bob,
schema="Attachment",
)["json"]
check(current["status"] == "queued", "expected queued while worker is stopped")
message = c.message(
alice,
chat,
"Проверка очереди",
attachments=[attachment_id],
)["json"]
check(
message["attachment_ids"] == [attachment_id],
"message did not retain attachment",
)
STATE_FILE.write_text(
"\n".join([attachment_id, bob["token"], original.hex()]),
encoding="utf-8",
)
print("PASS: HTTP 202, queued, message linked")
print("Attachment:", attachment_id)
print("Now restart API and RabbitMQ, then start worker")
def verify():
attachment_id, token, original_hex = STATE_FILE.read_text(
encoding="utf-8"
).splitlines()
c = Client(BASE_URL)
actor = {"token": token}
deadline = time.monotonic() + 90
while time.monotonic() < deadline:
attachment = c.call(
"GET",
f"/api/v1/attachments/{attachment_id}",
actor=actor,
schema="Attachment",
)["json"]
check(attachment["status"] != "failed", "processing failed")
if attachment["status"] == "ready":
break
time.sleep(0.5)
else:
raise AssertionError("attachment did not become ready")
original = c.call(
"GET",
f"/api/v1/attachments/{attachment_id}/content",
actor=actor,
)
check(original["bytes"] == bytes.fromhex(original_hex), "original changed")
thumbnail = c.call(
"GET",
f"/api/v1/attachments/{attachment_id}/content?variant=thumbnail",
actor=actor,
)
check(
thumbnail["headers"].get_content_type() == "image/jpeg",
"thumbnail is not JPEG",
)
check(
jpeg_size(thumbnail["bytes"]) == (256, 160),
"incorrect thumbnail dimensions",
)
print("PASS: restart recovery, ready, original, JPEG thumbnail 256x160")
if __name__ == "__main__":
if len(sys.argv) != 2 or sys.argv[1] not in {"prepare", "verify"}:
raise SystemExit("Usage: python tests/queue_grade3.py prepare|verify")
{"prepare": prepare, "verify": verify}[sys.argv[1]]()