ЛР 4 — БПИ2302 — Перевалова — на 5 #98

Closed
perevalova wants to merge 2 commits from perevalova/mtusi2026lab4:solution/lab4-implementation into main
81 changed files with 5899 additions and 46 deletions
+24
View File
@@ -0,0 +1,24 @@
# Local Docker Compose configuration for Lab 4.
# Copy this file to .env and replace every CHANGE_ME value locally.
# Never commit .env or share its contents.
POSTGRES_DB=messenger
POSTGRES_USER=messenger
POSTGRES_PASSWORD=CHANGE_ME_POSTGRES_PASSWORD
MINIO_ROOT_USER=CHANGE_ME_S3_ADMIN_USER
MINIO_ROOT_PASSWORD=CHANGE_ME_S3_ADMIN_PASSWORD
S3_ACCESS_KEY_ID=CHANGE_ME_APP_ACCESS_KEY
S3_SECRET_ACCESS_KEY=CHANGE_ME_APP_SECRET_KEY
S3_REGION=us-east-1
S3_BUCKET=messenger-attachments
S3_PUBLIC_ENDPOINT_URL=http://localhost:9000
SESSION_TTL_SECONDS=3600
MESSAGE_CACHE_TTL_SECONDS=60
IDEMPOTENCY_TTL_SECONDS=86400
DB_POOL_SIZE=3
DB_MAX_OVERFLOW=2
DB_POOL_TIMEOUT=10
MAX_UPLOAD_BYTES=10485760
MAX_IMAGE_PIXELS=40000000
ATTACHMENT_CLEANUP_INTERVAL_SECONDS=60
ATTACHMENT_CLEANUP_GRACE_SECONDS=600
ATTACHMENT_CLEANUP_BATCH_SIZE=100
+45 -15
View File
@@ -1,18 +1,48 @@
## Лабораторная 4
## Лабораторная 4 — S3-хранилище и вложения
Автор / группа: TODO
Целевая оценка (3/4/5): TODO
Трек 5, если заявлен: TODO
**Автор / группа:** Перевалова Виктория, БПИ2302
Что реализовано и что перенесено: TODO
Команды запуска из чистого clone: TODO
Результаты проверок и ссылка на REPORT.md: TODO
Ограничения / известные ошибки: TODO
Источники и использование ИИ: TODO
**Целевая оценка:** 5
- [ ] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
- [ ] Контракт и публичные тесты не ослаблены.
- [ ] make check и make test выполнены; приложен реальный вывод.
- [ ] Сценарии отказов/безопасности своего уровня воспроизведены.
- [ ] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
- [ ] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
**Трек:** Track 1 — Presigned Upload
### Что реализовано
- Перенесена основа предыдущей лабораторной; сохранены публичные API-сценарии и контракт.
- Реализованы Proxy Upload и Presigned Upload с проверкой членства в чате, уникальными ключами, TTL, проверкой размера, SHA-256 и фактического формата изображения при complete.
- Метаданные хранятся в PostgreSQL, бинарные данные — в приватном S3-compatible bucket.
- Добавлены cleanup незавершённых presigned-загрузок, best-effort компенсация при ошибке записи метаданных, ограниченные S3 credentials и bootstrap bucket.
- Стенд включает PostgreSQL, Redis, S3-compatible storage, миграции, Traefik и две API-реплики.
### Запуск
См. [`REPORT.md`](../REPORT.md), раздел «Запуск из чистого clone». Для локального Compose используется `.env.compose.example`; после копирования в `.env` необходимо заменить демонстрационные значения.
### Проверки и доказательства
- `scripts/check_contract.py` — PASS.
- Python compileall — PASS.
- Публичный smoke — PASS: повторно выполнен после исправления маршрутизации Traefik; результат сохранён в `evidence/smoke.txt`.
- Presigned upload, completion, idempotency и download — PASS.
- Все негативные сценарии Presigned Upload — PASS (409/415/413/404 согласно проверяемому сценарию).
- Негативные сценарии, cleanup, компенсация S3/PostgreSQL, S3 unavailable/security, persistence, attachment failover и нагрузочные эксперименты описаны в `REPORT.md`; сводки находятся в `evidence/`.
- Эксперимент Track 1 подтвердил сокращение тела запроса к API до 159 байт для файла 5,5 MiB; снижение RAM не подтверждено и не заявляется.
### Ограничения и отклонения
- Между PostgreSQL и S3 нет общей распределённой транзакции; компенсация Proxy Upload best-effort. Незарегистрированные orphan-объекты требуют отдельной сверки/lifecycle policy.
- Пиковая RAM измерялась через периодический `docker stats`; короткие пики могли быть пропущены.
- В Compose используется Silo с S3 API, хотя задание буквально перечисляет MinIO, SeaweedFS или RustFS. Это отклонение явно раскрыто в `REPORT.md`; требуется согласование с преподавателем.
### Источники и использование ИИ
Использованы исходный skeleton, контракт, документация зависимостей и материалы лабораторной. ChatGPT применялся для анализа требований, диагностики, подготовки отдельных тестов и редактирования документации. Результаты проверялись локальным запуском и тестами; ИИ не заменял выполнение проверок.
### Checklist
- [x] Контракт и публичные tests не ослаблялись ради прохождения.
- [x] Сценарии отказов и безопасности уровня 4/5 воспроизведены; результаты приложены в `evidence/`.
- [x] Локальные секреты исключены из Git через `.gitignore`; `.env` не должен добавляться в PR.
- [x] Исходный README задания сохранён, инструкция конкретной реализации находится в `REPORT.md`.
- [ ] Чистый запуск из clone и повторный `make test` после последних изменений подтверждены.
- [ ] `make validate` выполнен с `requirements-dev.txt`; приложен вывод.
+1
View File
@@ -2,6 +2,7 @@
.env
.env.*
!.env.example
!.env.compose.example
secrets/
certs/private/
*.key
+23
View File
@@ -0,0 +1,23 @@
FROM python:3.11.9-slim-bookworm
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PIP_NO_CACHE_DIR=1
WORKDIR /app
RUN groupadd --system app && useradd --system --gid app --home-dir /app app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app ./app
COPY migrations ./migrations
COPY scripts ./scripts
COPY tests ./tests
COPY contracts ./contracts
COPY lab.json .
COPY alembic.ini .
RUN chown -R app:app /app
USER app
EXPOSE 8080
HEALTHCHECK --interval=5s --timeout=3s --start-period=20s --retries=3 \
CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=2)"
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8080", "--timeout-keep-alive", "5", "--timeout-graceful-shutdown", "20"]
+475 -20
View File
@@ -1,37 +1,492 @@
# Отчёт — лабораторная 4
# Отчёт — лабораторная работа №4
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
- **Автор, группа:** Перевалова Виктория, БПИ2302.
- **Целевая оценка:** 5.
- **Выбранный трек:** Track 1 — Presigned Upload (прямая загрузка в S3).
- **Основная гипотеза:** прямая загрузка файлов в S3-compatible хранилище по временной подписанной ссылке позволяет уменьшить объём данных, проходящих через API, и сократить время обработки больших файлов на стороне API.
- **Commit/PR:** будет указан после итогового commit и публикации изменений в Gitea.
- **Использованные источники, библиотеки и помощь ИИ:** исходный репозиторий лабораторной работы и материалы преподавателя; Python 3, FastAPI, SQLAlchemy, Alembic, PostgreSQL, Redis, boto3, Pillow, Docker Compose, S3-compatible object storage. ChatGPT использовался для анализа требований, проектирования отдельных компонентов, подготовки тестов, диагностики ошибок и оформления отчёта. Реализация и результаты проверялись запуском приложения и тестовых сценариев.
---
## Запуск из чистого clone
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
### 1. Требования к окружению
Для запуска проекта необходимы:
- Git;
- Docker Desktop с поддержкой Docker Compose;
- Python 3.13 для локального запуска вспомогательных тестовых скриптов;
- PowerShell (для Windows) или совместимая командная оболочка;
- доступ к Gitea и возможность клонировать репозиторий.
Основные сервисы запускаются в Docker Compose:
| Компонент | Используемая версия |
|---|---|
| PostgreSQL | 16.6-bookworm |
| Redis | 7.4.2-bookworm |
| S3-compatible storage (Silo) | RELEASE.2026-10-26T00-00-00Z |
| Traefik | 3.6.2 |
| API | Docker image проекта |
### 2. Клонирование проекта
```powershell
git clone https://gitea.privcloud.tech/perevalova/mtusi2026lab4.git
cd mtusi2026lab4
```
Для перехода на ветку с реализацией:
```powershell
git checkout solution/lab4-implementation
```
### 3. Подготовка конфигурации
Создать локальный файл `.env` на основе предоставленного примера:
```powershell
Copy-Item .env.compose.example .env
```
Перед запуском необходимо проверить соответствие переменных конфигурации требованиям `compose.yaml`.
Для работы инфраструктуры необходимо задать следующие переменные:
- `POSTGRES_PASSWORD` — пароль PostgreSQL;
- `MINIO_ROOT_USER` — административный пользователь S3-хранилища;
- `MINIO_ROOT_PASSWORD` — административный пароль S3-хранилища;
- `S3_ACCESS_KEY_ID` — отдельный пользователь приложения для доступа к S3;
- `S3_SECRET_ACCESS_KEY` — секретный ключ пользователя приложения;
- `S3_REGION` — регион S3;
- `S3_BUCKET` — имя bucket.
Дополнительные параметры ограничений загрузки и фоновой очистки имеют значения по умолчанию в Compose:
- `MAX_UPLOAD_BYTES` — 10 MiB;
- `MAX_IMAGE_PIXELS` — 40 000 000;
- `ATTACHMENT_CLEANUP_INTERVAL_SECONDS` — 60 секунд;
- `ATTACHMENT_CLEANUP_GRACE_SECONDS` — 600 секунд;
- `ATTACHMENT_CLEANUP_BATCH_SIZE` — 100 записей.
Секреты должны храниться только в локальном `.env`. Файл не должен попадать в Git.
Файл `.env.compose.example` содержит имена переменных, используемые непосредственно в `compose.yaml`. После копирования необходимо заменить все демонстрационные значения `CHANGE_ME` локальными секретами. Сам `.env` не добавляется в Git.
### 4. Проверка конфигурации и сборка
```powershell
docker compose config --quiet
docker compose build
```
### 5. Запуск инфраструктуры и приложения
```powershell
docker compose up -d --build --scale api=2
```
В проекте предусмотрены следующие компоненты:
- PostgreSQL;
- Redis;
- S3-compatible storage;
- сервис первоначальной настройки S3;
- Alembic migrations;
- две реплики API;
- Traefik.
Миграции выполняются отдельным сервисом `migrate` перед запуском API.
### 6. Ожидание готовности
Проверить состояние контейнеров:
```powershell
docker compose ps
```
Проверить readiness API:
```powershell
Invoke-RestMethod http://localhost:8080/health/ready
```
Ожидаемый результат:
```json
{
"status": "ok"
}
```
### 7. Проверка приложения
Проверка структуры и контракта:
```powershell
python -X utf8 scripts/check_contract.py
python -X utf8 scripts/validate_openapi.py
```
Для полной валидации OpenAPI установите dev-зависимости: `python -m pip install -r requirements-dev.txt`. Затем выполните `python -X utf8 scripts/validate_openapi.py`.
Проверка синтаксиса:
```powershell
python -X utf8 -m compileall -q app migrations
```
Публичный smoke-тест:
```powershell
python -X utf8 tests/smoke.py --base-url http://localhost:8080
```
Дополнительные проверки:
```powershell
python -X utf8 scripts/test_presigned_upload.py
python -X utf8 scripts/test_presigned_negative.py
```
В Dockerfile вспомогательные каталоги `scripts/`, `tests/`, `contracts/` и файл `lab.json` должны копироваться в образ API. После пересборки тест очистки незавершённых загрузок выполняется внутри API-контейнера:
```powershell
docker compose exec -T api python /app/scripts/test_attachment_cleanup.py
```
Тест компенсационного удаления при отказе PostgreSQL выполняется внутри API-контейнера:
```powershell
docker compose exec -T api python /app/scripts/test_s3_db_compensation.py
```
### 8. Остановка проекта
Безопасная остановка контейнеров с сохранением данных:
```powershell
docker compose down
```
Повторный запуск:
```powershell
docker compose up -d --build --scale api=2
```
**Внимание:** намеренное удаление учебных данных выполняется отдельно:
```powershell
docker compose down -v
```
Эта команда удаляет Docker volumes, включая данные PostgreSQL, Redis и S3-хранилища. Использовать её следует только при осознанном сбросе учебного окружения.
---
## Архитектура и решения
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab3, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
### 1. Общая архитектура
Система состоит из нескольких взаимодействующих компонентов.
```text
Клиент
|
v
Traefik
|
----------------
| |
v v
API-1 API-2
| |
---------------------------
| | |
v v v
PostgreSQL Redis S3 Storage
(метаданные) (кэш) (файлы)
```
Дополнительно используются:
- Alembic для управления миграциями;
- отдельный bootstrap-сервис для первоначальной настройки S3;
- фоновая задача очистки незавершённых загрузок, работающая в API-репликах.
**Примечание о выборе S3-сервера.** В Compose закреплён образ `pgsty/silo:RELEASE.2026-10-26T00-00-00Z`, совместимый с S3 API. В формулировке задания прямо перечислены MinIO, SeaweedFS и RustFS; Silo в этом перечне не назван. Замена сервера не выполнялась, поскольку текущий стенд и доказательства собраны на Silo. Это потенциальное формальное отклонение, которое следует согласовать с преподавателем.
### 2. Назначение компонентов
**API (FastAPI)**
Обрабатывает запросы пользователей, проверяет авторизацию и членство в чатах, выполняет загрузку и скачивание вложений, проверяет метаданные и взаимодействует с хранилищами.
**PostgreSQL**
Является источником истины для метаданных вложений и незавершённых загрузок.
В базе хранятся:
- идентификатор вложения;
- идентификатор чата;
- идентификатор владельца;
- имя файла;
- MIME-тип;
- размер;
- SHA-256;
- ключ объекта в S3;
- статус обработки;
- даты создания и изменения.
Для presigned-загрузок дополнительно сохраняются ожидаемые размер и контрольная сумма, срок действия и состояние загрузочной сессии.
**Redis**
Используется для кэширования истории сообщений и связанных с ней операций. Не является источником истины для вложений.
**S3-compatible storage**
Хранит бинарное содержимое файлов. Bucket `messenger-attachments` настроен как приватный.
API обращается к S3 по внутреннему адресу Docker-сети. Клиент использует публичный адрес S3 для прямой загрузки по подписанной ссылке.
**Traefik**
Распределяет запросы между двумя экземплярами API.
### 3. Источник истины и транзакционные границы
PostgreSQL хранит сведения о вложениях, а S3 — бинарные данные.
Между PostgreSQL и S3 нет общей атомарной транзакции. Поэтому согласованность обеспечивается последовательностью операций и компенсационными действиями.
Для Proxy Upload:
1. API принимает и проверяет файл.
2. Загружает содержимое в S3.
3. Создаёт запись метаданных в PostgreSQL.
4. Если сохранение метаданных завершается ошибкой, API выполняет rollback и пытается удалить объект из S3.
Для Presigned Upload:
1. API проверяет пользователя и членство в чате.
2. Создаёт уникальный ключ объекта и запись `pending` в PostgreSQL.
3. Возвращает временную подписанную ссылку.
4. Клиент загружает файл непосредственно в S3.
5. Клиент вызывает endpoint завершения загрузки.
6. API получает объект из S3, проверяет размер, SHA-256 и содержимое изображения.
7. После успешной проверки создаёт запись вложения и переводит загрузку в состояние `completed`.
При ошибках проверки загрузка переводится в `failed`, а объект удаляется из S3 при успешном выполнении компенсационного действия.
Для незавершённых загрузок используется фоновая очистка с периодической проверкой просроченных записей. Для конкурентной работы нескольких API-реплик применяется блокировка PostgreSQL `FOR UPDATE SKIP LOCKED`.
### 4. Ограничения загрузки и безопасность
Поддерживаются PNG и JPEG.
Ограничения:
- максимальный размер файла — 10 MiB;
- максимальное количество пикселей — 40 000 000;
- проверяется фактический формат содержимого;
- проверяется соответствие MIME-типа;
- проверяется SHA-256;
- используется безопасное имя файла;
- тело запроса читается потоково;
- для больших файлов применяется временный файл вместо неограниченного хранения в RAM;
- скачивание доступно только авторизованным участникам соответствующего чата.
S3 bucket закрыт для анонимного доступа. Для приложения создан отдельный S3-пользователь с отдельной политикой доступа.
Административная консоль S3 не опубликована на хостовой машине.
---
## Доказательства критериев
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
В таблице приведены выполненные сценарии и фактические результаты.
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
| Критерий | Команда / сценарий | Наблюдаемый результат | Файл доказательства |
|---|---|---|---|
| Проверка структуры проекта | `python -X utf8 scripts/check_contract.py` | PASS: структура, локальные ссылки и синтаксис | `evidence/check-contract.txt` (если сохранён; иначе вывод команды) |
| Валидация OpenAPI | `python -X utf8 scripts/validate_openapi.py` | PASS: OpenAPI 3.1 и JSON Schema | Вывод команды |
| Публичный smoke | `python -X utf8 tests/smoke.py --base-url http://localhost:8080` | PASS: публичный API-сценарий; повторно подтверждён после исправления маршрутизации Traefik | `evidence/smoke.txt` |
| Готовность API | `Invoke-RestMethod http://localhost:8080/health/ready` | `status: ok` | Вывод команды |
| Две API-реплики | `docker compose ps` | Обе реплики имеют статус healthy | `evidence/persistence-before.txt`, `evidence/failover-before.txt` |
| Хранение бинарных данных в S3 | Proxy Upload и проверка скачивания | Файл сохраняется и возвращается с совпадающим SHA-256 | `evidence/track1-results.csv` |
| Метаданные в PostgreSQL | Загрузка и получение вложения | Сохраняются владелец, чат, размер, MIME, SHA-256 и статус | `evidence/track1-results.csv` |
| Ограничение размера | Загрузка файла больше 10 MiB | HTTP 413 | `evidence/negative-tests-results.txt` |
| Повреждённое изображение | Загрузка некорректных байтов | HTTP 415 | `evidence/negative-tests-results.txt` |
| Несовпадение MIME | PNG с `image/jpeg` | HTTP 415 | `evidence/negative-tests-results.txt` |
| Проверка SHA-256 | Неверная ожидаемая контрольная сумма | HTTP 409 | `evidence/negative-tests-results.txt` |
| Проверка размера | Неверный ожидаемый размер | HTTP 409 | `evidence/negative-tests-results.txt` |
| Защита от чужой загрузки | Чужой пользователь вызывает complete | HTTP 404 | `evidence/negative-tests-results.txt` |
| Защита метаданных | Неучастник чата запрашивает вложение | HTTP 404 | `evidence/negative-tests-results.txt` |
| Защита скачивания | Неучастник чата скачивает вложение | HTTP 404 | `evidence/negative-tests-results.txt` |
| Запрет межчатового прикрепления | Вложение из одного чата отправляется в другой | HTTP 404 | `evidence/negative-tests-results.txt` |
| Идемпотентность завершения | Повторное завершение загрузки | Возвращается существующее вложение | Вывод `scripts/test_presigned_upload.py` |
| Прямой Presigned Upload | Создание ссылки, PUT в S3, complete | Последний повторный запуск: PASS; загрузка завершена, SHA-256 совпадает | Вывод `scripts/test_presigned_upload.py` |
| Очистка незавершённых загрузок | Ожидание фонового cleanup | Статус `expired`, объект удалён из S3 | `evidence/attachment-cleanup-results.txt` |
| Компенсация Proxy Upload | Искусственная ошибка PostgreSQL после S3 PUT | HTTP 503, объект удалён из S3 | `evidence/s3-db-compensation-results.txt` |
| Приватность bucket | `curl.exe -i http://localhost:9000/messenger-attachments/` без авторизации | HTTP 403 Forbidden (`AccessDenied`) | `evidence/s3-security-results.txt` |
| Недоступность S3 Console | `docker compose port minio 9001` | Порт 9001 не опубликован на хост | `evidence/s3-security-results.txt` |
| Сохранность данных | Перезапуск PostgreSQL, Redis, S3 и API без удаления volumes | Подтверждённые данные и существующая сессия сохранены | `evidence/persistence-results.txt` |
| Отказоустойчивость API | Остановка одной API-реплики | 108 успешных записей, 1 неуспешная попытка, первое восстановление за 2,607 с (общий сценарий сообщений) | `evidence/failover-results.txt` |
| Отказоустойчивость вложений | Остановка именно API-реплики, обслужившей Proxy Upload; скачивание через оставшуюся реплику | 2 реплики до отказа; 87 успешных записей, 4 неуспешные попытки; первое восстановление за 4,597 с; вложение скачано побайтно идентичным | `evidence/attachment-failover-results.txt` |
| Недоступность S3 | Остановка S3-сервиса | Readiness сообщил `Required storage unavailable`; после запуска S3 обе API-реплики восстановили readiness | `evidence/s3-unavailable-results.txt` |
| Эксперимент Track 1 | Сравнение Proxy и Presigned | Измерены полное время, время API и размер тела запросов | `evidence/track1-results.csv` |
| Параллельная загрузка | 10 одновременных Proxy и 10 Presigned загрузок | 10/10 успешных в обоих режимах | `evidence/concurrent-upload-results.txt`, `evidence/concurrent-upload-results.csv` |
### Подтверждение инфраструктурных hook
Для проверки сохранности данных использовался сценарий перезапуска:
```sh
docker compose restart postgres redis minio api
```
Таким образом, перезапускались:
- PostgreSQL;
- Redis;
- S3-compatible storage;
- API.
Docker volumes при этом не удалялись.
Для общего сценария отказоустойчивости API останавливался один из двух экземпляров API. Наблюдаемый результат: 108 успешных записей, 1 неуспешная попытка и первое восстановление за 2,607 секунды (`evidence/failover-results.txt`).
Отдельно проверялся отказ именно той API-реплики, которая обслужила загрузку вложения. Тест получил идентификатор обслужившей реплики из ответа API, остановил именно её, продолжил работу через оставшуюся реплику и скачал вложение. Результат: 87 успешных записей, 4 неуспешные попытки, первое восстановление за 4,597 секунды; содержимое вложения совпало побайтно (`evidence/attachment-failover-results.txt`).
---
## Эксперименты и отказы
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
### Исследовательский трек 1 — Presigned Upload
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
#### Гипотеза
## Ограничения и следующие шаги
Прямая загрузка файлов из клиента в S3-compatible хранилище по временной подписанной ссылке позволяет существенно сократить объём данных, передаваемых через API. Предполагается, что это уменьшит объём передаваемых через API данных и время обработки запроса на стороне API при работе с крупными вложениями.
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
#### Методика эксперимента
Сравнивались два способа загрузки:
1. **Proxy Upload** — клиент передаёт бинарное содержимое файла API, после чего API сохраняет его в S3.
2. **Presigned Upload** — API выдаёт временную подписанную ссылку, клиент загружает файл непосредственно в S3, затем вызывает endpoint завершения загрузки.
Использовались два PNG-файла:
- `sample.png` — 111 817 байт (0,11 MiB);
- `sample_large.png` — 5 769 330 байт (5,50 MiB).
Для каждого сочетания способа загрузки и размера файла выполнено по три повтора. Всего проведено 12 измерений.
Среда эксперимента: Docker Compose, два экземпляра API, PostgreSQL 16.6, Redis 7.4.2, Silo S3-compatible storage. API-контейнеры имели ограничение памяти 512 MiB.
#### Результаты
| Способ загрузки | Размер файла | Среднее полное время | Среднее время запросов к API | Тело запросов к API |
|---|---:|---:|---:|---:|
| Proxy | 111 817 Б | 0,0786 с | 0,0785 с | 111 817 Б |
| Presigned | 111 817 Б | 0,1296 с | 0,0826 с | 158 Б |
| Proxy | 5 769 330 Б | 0,2666 с | 0,2665 с | 5 769 330 Б |
| Presigned | 5 769 330 Б | 0,2564 с | 0,1275 с | 159 Б |
При загрузке большого файла объём тела запросов клиента к API сократился примерно на 99,997%. При загрузке небольшого файла сокращение составило около 99,86%.
Полное время загрузки большого файла при Presigned Upload оказалось примерно на 3,8% меньше, чем при Proxy Upload. Суммарное время запросов к API сократилось примерно на 52,2%.
#### Параллельная загрузка
Дополнительно проведён эксперимент с десятью одновременными загрузками файла размером 5 769 330 байт.
Для каждого способа загрузки выполнялось по десять параллельных операций.
| Показатель | Proxy | Presigned |
|---|---:|---:|
| Количество загрузок | 10 | 10 |
| Успешные загрузки | 10 | 10 |
| Ошибки | 0 | 0 |
| Общее время | 3,996 с | 4,941 с |
| Пиковая суммарная RAM API | 220,2 MiB | 304,4 MiB |
| Прирост RAM относительно базового уровня | 16,1 MiB | 84,2 MiB |
Параллельный эксперимент подтвердил, что оба способа справляются с десятью одновременными загрузками без ошибок.
При этом в данном запуске Presigned Upload показал более высокий пик памяти. Поэтому утверждать, что прямая загрузка снижает потребление RAM, нельзя.
#### Потребление памяти в последовательном эксперименте
В ходе последовательного эксперимента фиксировалась память API-контейнеров с помощью `docker stats`. Измерения выполнялись с интервалом около 100 мс.
Зафиксированные максимальные значения среди API-реплик:
- Proxy, маленький файл — 103,4 MiB;
- Presigned, маленький файл — 112,7 MiB;
- Proxy, большой файл — 109,2 MiB;
- Presigned, большой файл — 128,9 MiB.
Результаты не подтверждают снижение RAM при Presigned Upload в текущем эксперименте. Измерения ограничены интервалом опроса Docker Stats и не позволяют исключить кратковременные пики между замерами.
#### Вывод
Гипотеза о сокращении объёма данных, передаваемых через API, подтверждена.
Presigned Upload позволяет передавать содержимое файла непосредственно в S3, оставляя API операции выдачи временной ссылки, проверки загрузки и сохранения метаданных.
Уменьшение времени обработки запросов API для большого файла также наблюдалось. Однако снижение общего времени загрузки небольших файлов не подтверждено, а снижение потребления RAM требует дополнительного исследования.
**Файлы с исходными измерениями:**
- `evidence/track1-results.csv`;
- `evidence/concurrent-upload-results.csv`;
- `evidence/concurrent-upload-results.txt`.
### Проверка отказов и восстановления
| Воздействие | Что ожидалось | Что наблюдалось | Время восстановления / потеря данных |
|---|---|---|---|
| Перезапуск PostgreSQL, Redis, S3 и API | Сохранение подтверждённых данных при сохранении volumes | Подтверждённые сообщения, вложения и существующая сессия сохранились | Данные не потеряны |
| Остановка одной API-реплики (общий сценарий) | Продолжение работы через оставшуюся реплику | 108 успешных записей, 1 неуспешная попытка | Первое восстановление — 2,607 с |
| Остановка реплики, обслужившей загрузку вложения | Доступность вложения через другую реплику | 87 успешных записей, 4 неуспешные попытки; вложение скачано побайтно идентичным | Первое восстановление — 4,597 с |
| Незавершённая Presigned Upload | Удаление просроченного объекта и обновление статуса | Запись получила статус `expired`, объект удалён | Ожидание фоновой очистки |
| Недоступность S3 | Остановка S3-сервиса | Readiness сообщил `Required storage unavailable`; после восстановления S3 readiness снова `ok` | Сервис восстановлен без пересоздания API |
| Ошибка PostgreSQL после загрузки в S3 через Proxy | Компенсационное удаление объекта | HTTP 503, объект отсутствует в S3 | Объект удалён |
| Ошибка проверки SHA-256 | Отклонение загрузки и удаление некорректного объекта | HTTP 409 | Ошибка обработана |
| Несовпадение размера файла | Отклонение загрузки | HTTP 409 | Ошибка обработана |
| Повреждённое изображение | Отклонение содержимого | HTTP 415 | Ошибка обработана |
| Превышение лимита размера | Отклонение загрузки до неограниченного чтения | HTTP 413 | Ошибка обработана |
| Несовпадение MIME-типа | Отклонение файла | HTTP 415 | Ошибка обработана |
| Чужой пользователь завершает загрузку | Запрет доступа | HTTP 404 | Доступ запрещён |
| Попытка прикрепить вложение из другого чата | Запрет межчатового доступа | HTTP 404 | Доступ запрещён |
### Известные ограничения
1. PostgreSQL и S3 не используют общую распределённую транзакцию. Согласованность обеспечивается компенсационными операциями и фоновой очисткой.
2. Удаление объекта из S3 является best-effort операцией. При одновременном отказе хранилища и компенсационного удаления может потребоваться повторная проверка состояния.
3. Фоновая очистка работает с зарегистрированными незавершёнными загрузками. Объекты, для которых вообще не была создана запись загрузки, требуют отдельного механизма сверки или политики жизненного цикла S3.
4. Эксперимент по памяти проводился с периодическим опросом Docker Stats. В параллельном запуске мониторинг успел получить лишь 4–5 измерений, поэтому кратковременные пики могли быть пропущены.
5. Эксперименты проведены в локальной Docker-среде, а не в распределённой производственной инфраструктуре. Результаты нельзя автоматически переносить на промышленную нагрузку.
6. Для прямой загрузки необходима доступность S3 endpoint со стороны клиента. В локальном окружении используется опубликованный адрес S3 API.
7. Фактическое снижение потребления RAM при Presigned Upload экспериментально не подтверждено.
### Итоговый вывод
В лабораторной работе реализовано S3-хранилище вложений для приложения обмена сообщениями. Бинарные данные хранятся отдельно от метаданных, которые сохраняются в PostgreSQL.
Реализованы два способа загрузки файлов: Proxy Upload и Presigned Upload. Для прямой загрузки разработан механизм выдачи временных подписанных ссылок и последующей проверки загруженного объекта.
Проведены проверки безопасности, негативные тесты, проверка очистки незавершённых загрузок, компенсации при отказе PostgreSQL, сохранности данных после перезапуска и отказоустойчивости двух API-реплик.
В исследовательском треке 1 подтверждено существенное сокращение объёма данных, передаваемых через API. Для файла размером 5,5 MiB тело запроса к API уменьшилось с 5 769 330 до 159 байт. При этом эксперимент не подтвердил снижение потребления оперативной памяти.
Таким образом, реализация демонстрирует преимущества прямой загрузки в S3, сохраняет совместимость с обязательным Proxy Upload и включает механизмы проверки и обработки ошибок при работе с двумя независимыми хранилищами.
+117
View File
@@ -0,0 +1,117 @@
# A generic, single database configuration.
[alembic]
# path to migration scripts
# Use forward slashes (/) also on windows to provide an os agnostic path
script_location = migrations
# template used to generate migration file names; The default value is %%(rev)s_%%(slug)s
# Uncomment the line below if you want the files to be prepended with date and time
# see https://alembic.sqlalchemy.org/en/latest/tutorial.html#editing-the-ini-file
# for all available tokens
# file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s
# sys.path path, will be prepended to sys.path if present.
# defaults to the current working directory.
prepend_sys_path = .
# timezone to use when rendering the date within the migration file
# as well as the filename.
# If specified, requires the python>=3.9 or backports.zoneinfo library.
# Any required deps can installed by adding `alembic[tz]` to the pip requirements
# string value is passed to ZoneInfo()
# leave blank for localtime
# timezone =
# max length of characters to apply to the "slug" field
# truncate_slug_length = 40
# set to 'true' to run the environment during
# the 'revision' command, regardless of autogenerate
# revision_environment = false
# set to 'true' to allow .pyc and .pyo files without
# a source .py file to be detected as revisions in the
# versions/ directory
# sourceless = false
# version location specification; This defaults
# to migrations/versions. When using multiple version
# directories, initial revisions must be specified with --version-path.
# The path separator used here should be the separator specified by "version_path_separator" below.
# version_locations = %(here)s/bar:%(here)s/bat:migrations/versions
# version path separator; As mentioned above, this is the character used to split
# version_locations. The default within new alembic.ini files is "os", which uses os.pathsep.
# If this key is omitted entirely, it falls back to the legacy behavior of splitting on spaces and/or commas.
# Valid values for version_path_separator are:
#
# version_path_separator = :
# version_path_separator = ;
# version_path_separator = space
# version_path_separator = newline
version_path_separator = os # Use os.pathsep. Default configuration used for new projects.
# set to 'true' to search source files recursively
# in each "version_locations" directory
# new in Alembic version 1.10
# recursive_version_locations = false
# the output encoding used when revision files
# are written from script.py.mako
# output_encoding = utf-8
sqlalchemy.url = driver://user:pass@localhost/dbname
[post_write_hooks]
# post_write_hooks defines scripts or Python functions that are run
# on newly generated revision scripts. See the documentation for further
# detail and examples
# format using "black" - use the console_scripts runner, against the "black" entrypoint
# hooks = black
# black.type = console_scripts
# black.entrypoint = black
# black.options = -l 79 REVISION_SCRIPT_FILENAME
# lint with attempts to fix using "ruff" - use the exec runner, execute a binary
# hooks = ruff
# ruff.type = exec
# ruff.executable = %(here)s/.venv/bin/ruff
# ruff.options = --fix REVISION_SCRIPT_FILENAME
# Logging configuration
[loggers]
keys = root,sqlalchemy,alembic
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = WARNING
handlers = console
qualname =
[logger_sqlalchemy]
level = WARNING
handlers =
qualname = sqlalchemy.engine
[logger_alembic]
level = INFO
handlers =
qualname = alembic
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S
View File
View File
+422
View File
@@ -0,0 +1,422 @@
import hashlib
import tempfile
import uuid
from datetime import datetime, timezone, timedelta
from urllib.parse import quote
from fastapi import APIRouter, Depends, Header, HTTPException, Query, Request, Response
from fastapi.responses import StreamingResponse
from sqlalchemy import select
from sqlalchemy.orm import Session
from botocore.exceptions import BotoCoreError, ClientError
from app.api.deps import get_db, get_current_user
from app.core.config import settings
from app.models import CreateAttachmentUpload
from app.db.models import Attachment, AttachmentUpload, User
from app.services.object_storage import client, create_presigned_upload_url
from app.services.image_validation import MAX_BYTES, validate_image_file
from app.repositories.chat_repository import ChatRepository
router = APIRouter(prefix="/api/v1", tags=["Attachments"])
def iso_utc(value):
return value.astimezone(timezone.utc).isoformat(timespec="microseconds").replace("+00:00", "Z")
def attachment_json(item):
return {
"id": str(item.id), "chat_id": str(item.chat_id), "owner_id": str(item.owner_id),
"filename": item.filename, "content_type": item.content_type,
"size_bytes": item.size_bytes, "sha256": item.sha256, "status": item.status,
"variants": [], "error_code": item.error_code,
"created_at": iso_utc(item.created_at), "updated_at": iso_utc(item.updated_at),
}
def member_attachment(db, attachment_id, user_id):
item = db.get(Attachment, attachment_id)
if item is None or not ChatRepository(db).is_member(item.chat_id, user_id):
raise HTTPException(404, "Attachment not found")
return item
async def read_limited(request: Request):
declared = request.headers.get("content-length")
if declared and declared.isdigit() and int(declared) > MAX_BYTES:
raise HTTPException(413, "Image exceeds 10 MiB")
size = 0
# Keep at most 1 MiB in RAM; larger bodies spill to a temporary file.
spool = tempfile.SpooledTemporaryFile(max_size=1024 * 1024, mode="w+b")
try:
async for chunk in request.stream():
size += len(chunk)
if size > MAX_BYTES:
raise HTTPException(413, "Image exceeds 10 MiB")
spool.write(chunk)
spool.seek(0)
return spool, size
except Exception:
spool.close()
raise
def sha256_file(fileobj):
digest = hashlib.sha256()
fileobj.seek(0)
while True:
chunk = fileobj.read(1024 * 1024)
if not chunk: break
digest.update(chunk)
fileobj.seek(0)
return digest.hexdigest()
@router.post("/chats/{chat_id}/attachments/presigned", status_code=201)
def create_attachment_upload(
chat_id: str,
payload: CreateAttachmentUpload,
db: Session = Depends(get_db),
user: User = Depends(get_current_user),
):
try:
chat_uuid = uuid.UUID(chat_id)
except ValueError:
raise HTTPException(400, "Invalid chat UUID") from None
if not ChatRepository(db).is_member(chat_uuid, user.id):
raise HTTPException(404, "Chat not found")
if payload.size_bytes > settings.max_upload_bytes:
raise HTTPException(413, "Image exceeds configured upload limit")
upload_id = uuid.uuid4()
object_key = f"uploads/{chat_uuid}/{upload_id}"
expires_at = datetime.now(timezone.utc) + timedelta(
seconds=settings.presigned_upload_ttl_seconds
)
try:
upload_url = create_presigned_upload_url(
object_key=object_key,
content_type=payload.content_type,
)
except (BotoCoreError, ClientError):
raise HTTPException(503, "Object storage unavailable") from None
upload = AttachmentUpload(
id=upload_id,
chat_id=chat_uuid,
owner_id=user.id,
filename=payload.filename,
content_type=payload.content_type,
expected_size_bytes=payload.size_bytes,
expected_sha256=payload.sha256.lower(),
object_key=object_key,
status="pending",
expires_at=expires_at,
)
try:
db.add(upload)
db.commit()
except Exception:
db.rollback()
raise HTTPException(503, "Could not save upload metadata") from None
return {
"upload_id": str(upload_id),
"upload_url": upload_url,
"method": "PUT",
"headers": {
"Content-Type": payload.content_type,
},
"expires_in": settings.presigned_upload_ttl_seconds,
"expires_at": iso_utc(expires_at),
}
@router.post("/attachments/uploads/{upload_id}/complete", status_code=200)
def complete_attachment_upload(
upload_id: str,
db: Session = Depends(get_db),
user: User = Depends(get_current_user),
):
try:
parsed_id = uuid.UUID(upload_id)
except ValueError:
raise HTTPException(400, "Invalid upload UUID") from None
# Lock the upload row to prevent concurrent completion.
upload = db.scalar(
select(AttachmentUpload)
.where(AttachmentUpload.id == parsed_id)
.with_for_update()
)
if upload is None:
raise HTTPException(404, "Upload not found")
if (
upload.owner_id != user.id
or not ChatRepository(db).is_member(upload.chat_id, user.id)
):
raise HTTPException(404, "Upload not found")
# Idempotent completion: return the existing attachment.
if upload.status == "completed":
item = db.scalar(
select(Attachment).where(
Attachment.object_key == upload.object_key
)
)
if item is None:
raise HTTPException(409, "Completed attachment not found")
return attachment_json(item)
if upload.status != "pending":
raise HTTPException(409, "Upload is not pending")
now = datetime.now(timezone.utc)
if upload.expires_at <= now:
upload.status = "expired"
db.commit()
try:
client().delete_object(
Bucket=settings.s3_bucket,
Key=upload.object_key,
)
except Exception:
pass
raise HTTPException(409, "Upload has expired")
s3 = client()
# Read the object from S3.
try:
result = s3.get_object(
Bucket=settings.s3_bucket,
Key=upload.object_key,
)
except ClientError as exc:
code = exc.response.get("Error", {}).get("Code", "")
if code in {"NoSuchKey", "404", "NotFound"}:
raise HTTPException(409, "Uploaded object not found") from None
raise HTTPException(503, "Object storage unavailable") from None
except BotoCoreError:
raise HTTPException(503, "Object storage unavailable") from None
body = result["Body"]
spool = tempfile.SpooledTemporaryFile(
max_size=1024 * 1024,
mode="w+b",
)
def reject_upload(status_code: int, detail: str):
upload.status = "failed"
db.commit()
try:
s3.delete_object(
Bucket=settings.s3_bucket,
Key=upload.object_key,
)
except Exception:
pass
raise HTTPException(status_code=status_code, detail=detail)
try:
actual_size = result.get("ContentLength")
if actual_size != upload.expected_size_bytes:
reject_upload(409, "Uploaded file size mismatch")
if actual_size > settings.max_upload_bytes:
reject_upload(413, "Image exceeds configured upload limit")
total = 0
digest = hashlib.sha256()
try:
while True:
chunk = body.read(64 * 1024)
if not chunk:
break
total += len(chunk)
if total > settings.max_upload_bytes:
reject_upload(413, "Image exceeds configured upload limit")
if total > upload.expected_size_bytes:
reject_upload(409, "Uploaded file size mismatch")
digest.update(chunk)
spool.write(chunk)
finally:
body.close()
if total != upload.expected_size_bytes:
reject_upload(409, "Uploaded file size mismatch")
if digest.hexdigest() != upload.expected_sha256:
reject_upload(409, "Uploaded file checksum mismatch")
# Validate the actual image content, not only S3 metadata.
try:
actual_type, _, _ = validate_image_file(
spool,
upload.content_type,
)
except HTTPException as exc:
reject_upload(exc.status_code, str(exc.detail))
attachment = Attachment(
id=uuid.uuid4(),
chat_id=upload.chat_id,
owner_id=upload.owner_id,
filename=upload.filename,
content_type=actual_type,
size_bytes=total,
sha256=digest.hexdigest(),
object_key=upload.object_key,
status="ready",
error_code=None,
)
upload.status = "completed"
upload.completed_at = now
db.add(attachment)
db.commit()
db.refresh(attachment)
return attachment_json(attachment)
except HTTPException:
raise
except (BotoCoreError, ClientError):
db.rollback()
raise HTTPException(503, "Object storage unavailable") from None
except Exception:
db.rollback()
raise
finally:
spool.close()
@router.post("/chats/{chat_id}/attachments", status_code=201)
async def upload_attachment(
chat_id: str, request: Request, filename: str = Query(..., min_length=1, max_length=255),
db: Session = Depends(get_db), user: User = Depends(get_current_user),
):
try:
chat_uuid = uuid.UUID(chat_id)
except ValueError:
raise HTTPException(400, "Invalid chat UUID") from None
if not ChatRepository(db).is_member(chat_uuid, user.id):
raise HTTPException(404, "Chat not found")
if not filename.strip() or any(c in filename for c in ("/", "\\", "\x00")) or any(ord(c) < 32 or ord(c) == 127 for c in filename):
raise HTTPException(400, "Unsafe filename")
content_type = request.headers.get("content-type", "").split(";", 1)[0].strip().lower()
if content_type not in {"image/png", "image/jpeg"}:
raise HTTPException(415, "Only PNG and JPEG are supported")
spool, size = await read_limited(request)
try:
actual_type, _, _ = validate_image_file(spool, content_type)
digest = sha256_file(spool)
attachment_id = uuid.uuid4()
object_key = f"original/{chat_uuid}/{attachment_id}"
s3 = client()
try:
s3.put_object(Bucket=settings.s3_bucket, Key=object_key, Body=spool, ContentType=actual_type)
except (BotoCoreError, ClientError):
raise HTTPException(503, "Object storage unavailable") from None
item = Attachment(
id=attachment_id, chat_id=chat_uuid, owner_id=user.id, filename=filename,
content_type=actual_type, size_bytes=size, sha256=digest,
object_key=object_key, status="ready", error_code=None,
)
try:
db.add(item)
db.commit()
db.refresh(item)
except Exception:
db.rollback()
try: s3.delete_object(Bucket=settings.s3_bucket, Key=object_key)
except Exception: pass
raise HTTPException(503, "Could not save attachment metadata") from None
return attachment_json(item)
finally:
spool.close()
@router.get("/attachments/{attachment_id}")
def get_attachment(attachment_id: str, db: Session = Depends(get_db), user: User = Depends(get_current_user)):
try: parsed = uuid.UUID(attachment_id)
except ValueError: raise HTTPException(400, "Invalid attachment UUID") from None
item = member_attachment(db, parsed, user.id)
if item.status != "ready":
raise HTTPException(409, "Attachment upload is not completed")
return attachment_json(item)
@router.get("/attachments/{attachment_id}/content")
def download_attachment(attachment_id: str, variant: str = "original", db: Session = Depends(get_db), user: User = Depends(get_current_user)):
if variant != "original":
raise HTTPException(404, "Variant not found")
try: parsed = uuid.UUID(attachment_id)
except ValueError: raise HTTPException(400, "Invalid attachment UUID") from None
item = member_attachment(db, parsed, user.id)
if item.status != "ready":
raise HTTPException(409, "Attachment is not ready")
spool = tempfile.SpooledTemporaryFile(max_size=1024 * 1024, mode="w+b")
try:
result = client().get_object(Bucket=settings.s3_bucket, Key=item.object_key)
body = result["Body"]
total = 0
digest = hashlib.sha256()
try:
while True:
chunk = body.read(64 * 1024)
if not chunk:
break
total += len(chunk)
if total > MAX_BYTES:
raise HTTPException(503, "Stored object exceeds configured limit")
digest.update(chunk)
spool.write(chunk)
finally:
body.close()
if total != item.size_bytes or digest.hexdigest() != item.sha256:
raise HTTPException(503, "Stored object checksum mismatch")
spool.seek(0)
except (BotoCoreError, ClientError):
spool.close()
raise HTTPException(503, "Object storage unavailable") from None
except Exception:
spool.close()
raise
def stream_verified_file():
try:
while True:
chunk = spool.read(64 * 1024)
if not chunk:
break
yield chunk
finally:
spool.close()
return StreamingResponse(
stream_verified_file(),
media_type=item.content_type,
headers={"Content-Disposition": f"inline; filename*=UTF-8''{quote(item.filename)}"},
)
+96
View File
@@ -0,0 +1,96 @@
from datetime import timezone
from fastapi import APIRouter, Depends, Header, HTTPException, status
from fastapi.security import HTTPBasic, HTTPBasicCredentials
from sqlalchemy.orm import Session
from app.api.deps import get_db
from app.api.users import user_to_response
from app.services.auth_service import (
AuthService,
InvalidCredentialsError,
InvalidSessionError,
)
router = APIRouter(prefix="/api/v1/auth", tags=["Authentication"])
basic_security = HTTPBasic(auto_error=False)
@router.post("/sessions", status_code=status.HTTP_201_CREATED)
def create_session(
credentials: HTTPBasicCredentials = Depends(basic_security),
db: Session = Depends(get_db),
):
if credentials is None:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Basic authentication required",
headers={
"WWW-Authenticate": 'Basic realm="messenger", charset="UTF-8"'
},
)
try:
result = AuthService(db).create_session(
username=credentials.username,
password=credentials.password,
)
expires_at = result["expires_at"]
if expires_at.tzinfo is None:
expires_at = expires_at.replace(tzinfo=timezone.utc)
else:
expires_at = expires_at.astimezone(timezone.utc)
return {
"token": result["token"],
"token_type": result["token_type"],
"expires_at": expires_at.isoformat(
timespec="microseconds"
).replace("+00:00", "Z"),
"user": user_to_response(result["user"]),
}
except InvalidCredentialsError:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid username or password",
headers={
"WWW-Authenticate": 'Basic realm="messenger", charset="UTF-8"'
},
) from None
@router.delete("/sessions/current", status_code=status.HTTP_204_NO_CONTENT)
def delete_current_session(
authorization: str | None = Header(default=None),
db: Session = Depends(get_db),
):
if not authorization or not authorization.startswith("Bearer "):
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Bearer token required",
headers={"WWW-Authenticate": 'Bearer realm="messenger"'},
)
token = authorization[7:].strip()
if not token:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Bearer token required",
headers={"WWW-Authenticate": 'Bearer realm="messenger"'},
)
try:
AuthService(db).delete_session(token)
except InvalidSessionError:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid or expired session",
headers={"WWW-Authenticate": 'Bearer realm="messenger"'},
) from None
+90
View File
@@ -0,0 +1,90 @@
import uuid
from fastapi import APIRouter, Depends, HTTPException, Query, status
from sqlalchemy.orm import Session
from app.api.deps import get_db, get_current_user
from app.db.models import User as UserModel
from app.models import Chat as ChatResponse, CreateChat
from app.services.chat_service import (
ChatService,
ChatNotFoundError,
InvalidChatDataError,
)
router = APIRouter(prefix="/api/v1/chats", tags=["Chats"])
def chat_to_response(chat) -> ChatResponse:
return ChatResponse(
id=str(chat.id),
title=chat.title,
member_ids=[str(member.user_id) for member in chat.members],
created_at=chat.created_at,
)
@router.post("", response_model=ChatResponse, status_code=status.HTTP_201_CREATED)
def create_chat(
body: CreateChat,
db: Session = Depends(get_db),
user: UserModel = Depends(get_current_user),
):
try:
chat = ChatService(db).create_chat(
title=body.title,
member_ids=body.member_ids,
creator_id=user.id,
)
return chat_to_response(chat)
except InvalidChatDataError as exc:
raise HTTPException(
status_code=400,
detail=str(exc),
) from None
@router.get("")
def list_chats(
limit: int = Query(default=50, ge=1, le=100),
cursor: str | None = None,
db: Session = Depends(get_db),
user: UserModel = Depends(get_current_user),
):
if cursor is not None:
raise HTTPException(
status_code=400,
detail="Chat cursor pagination is not implemented yet",
)
chats = ChatService(db).list_chats(user.id)
chats = chats[:limit]
return {
"items": [chat_to_response(chat) for chat in chats],
"next_cursor": None,
}
@router.get("/{chat_id}", response_model=ChatResponse)
def get_chat(
chat_id: str,
db: Session = Depends(get_db),
user: UserModel = Depends(get_current_user),
):
try:
parsed_id = uuid.UUID(chat_id)
except ValueError:
raise HTTPException(status_code=400, detail="Invalid chat UUID") from None
try:
chat = ChatService(db).get_chat(parsed_id, user.id)
return chat_to_response(chat)
except ChatNotFoundError:
raise HTTPException(
status_code=404,
detail="Chat not found",
) from None
+40
View File
@@ -0,0 +1,40 @@
from fastapi import Depends, HTTPException, status
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
from sqlalchemy.orm import Session
from app.db.session import SessionLocal
from app.services.auth_service import AuthService, InvalidSessionError
bearer_scheme = HTTPBearer(auto_error=False)
def get_db():
db = SessionLocal()
try:
yield db
finally:
db.close()
def get_current_user(
credentials: HTTPAuthorizationCredentials | None = Depends(bearer_scheme),
db: Session = Depends(get_db),
):
if credentials is None or credentials.scheme.lower() != "bearer":
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Bearer token required",
headers={"WWW-Authenticate": "Bearer"},
)
try:
user = AuthService(db).authenticate(credentials.credentials)
return user
except InvalidSessionError:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid or expired token",
headers={"WWW-Authenticate": "Bearer"},
) from None
+73
View File
@@ -0,0 +1,73 @@
import re
import uuid
from fastapi import APIRouter, Depends, Header, HTTPException, Query, status
from sqlalchemy.orm import Session
from app.api.deps import get_db, get_current_user
from app.db.models import User as UserModel
from app.models import CreateMessage, Message as MessageResponse
from app.services.message_service import (
MessageService, MessageNotFoundError, InvalidMessageDataError,
IdempotencyConflictError,
)
router = APIRouter(prefix="/api/v1/chats/{chat_id}/messages", tags=["Messages"])
IDEMPOTENCY_KEY_RE = re.compile(r"^[A-Za-z0-9._:-]{1,128}$")
def message_to_response(message) -> MessageResponse:
return MessageResponse(
id=str(message.id), chat_id=str(message.chat_id),
sender_id=str(message.sender_id), text=message.text,
created_at=message.created_at, attachment_ids=message.attachment_ids or [],
)
def parse_chat_id(chat_id: str) -> uuid.UUID:
try:
return uuid.UUID(chat_id)
except ValueError:
raise HTTPException(status_code=400, detail="Invalid chat UUID") from None
@router.post("", response_model=MessageResponse, status_code=status.HTTP_201_CREATED)
def create_message(
chat_id: str, body: CreateMessage,
idempotency_key: str = Header(..., alias="Idempotency-Key", min_length=1, max_length=128),
db: Session = Depends(get_db),
user: UserModel = Depends(get_current_user),
):
if not IDEMPOTENCY_KEY_RE.fullmatch(idempotency_key):
raise HTTPException(status_code=400, detail="Invalid Idempotency-Key")
parsed_id = parse_chat_id(chat_id)
try:
message = MessageService(db).create_message(
chat_id=parsed_id, sender_id=user.id, text=body.text,
idempotency_key=idempotency_key, attachment_ids=body.attachment_ids,
)
return message_to_response(message)
except MessageNotFoundError:
raise HTTPException(status_code=404, detail="Chat not found") from None
except InvalidMessageDataError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from None
except IdempotencyConflictError:
raise HTTPException(status_code=409, detail="Idempotency-Key was already used with a different request") from None
@router.get("")
def list_messages(
chat_id: str, limit: int = Query(default=50, ge=1, le=100),
cursor: str | None = None, db: Session = Depends(get_db),
user: UserModel = Depends(get_current_user),
):
parsed_id = parse_chat_id(chat_id)
try:
messages, next_cursor = MessageService(db).get_history(
chat_id=parsed_id, user_id=user.id, limit=limit, cursor=cursor
)
return {"items": [message_to_response(item) for item in messages], "next_cursor": next_cursor}
except MessageNotFoundError:
raise HTTPException(status_code=404, detail="Chat not found") from None
except InvalidMessageDataError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from None
+55
View File
@@ -0,0 +1,55 @@
from fastapi import APIRouter, Depends, HTTPException, status
from sqlalchemy.orm import Session
from app.api.deps import get_db, get_current_user
from app.db.models import User as UserModel
from app.models import CreateUser, User
from app.services.user_service import (
UserService,
UsernameAlreadyExistsError,
)
router = APIRouter(prefix="/api/v1/users", tags=["Users"])
def user_to_response(user: UserModel) -> User:
return User(
id=str(user.id),
username=user.username,
display_name=user.display_name,
created_at=user.created_at,
)
@router.post("", response_model=User, status_code=status.HTTP_201_CREATED)
def register_user(
body: CreateUser,
db: Session = Depends(get_db),
):
try:
user = UserService(db).register(
username=body.username,
display_name=body.display_name,
password=body.password,
)
return user_to_response(user)
except UsernameAlreadyExistsError:
raise HTTPException(
status_code=status.HTTP_409_CONFLICT,
detail="Username already exists",
) from None
except ValueError as exc:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=str(exc),
) from None
@router.get("/me", response_model=User)
def get_me(
current_user: UserModel = Depends(get_current_user),
):
return user_to_response(current_user)
View File
+49
View File
@@ -0,0 +1,49 @@
import os
from pydantic import AliasChoices, Field
from pydantic_settings import BaseSettings, SettingsConfigDict
class Settings(BaseSettings):
app_name: str = "Messenger API"
app_env: str = "lab"
database_url: str = "postgresql+psycopg://messenger:messenger@postgres:5432/messenger"
db_pool_size: int = 5
db_max_overflow: int = 5
db_pool_timeout: int = 10
session_ttl_seconds: int = 3600
cache_url: str = "redis://redis:6379/0"
message_cache_ttl_seconds: int = 60
idempotency_ttl_seconds: int = 86400
instance_id: str = os.getenv("INSTANCE_ID") or os.getenv("HOSTNAME") or "api-local"
shutdown_grace_seconds: int = 20
s3_endpoint_url: str = Field(
default="http://minio:9000",
validation_alias=AliasChoices("S3_ENDPOINT_URL", "S3_ENDPOINT"),
)
s3_public_endpoint_url: str = Field(
default="http://localhost:9000",
validation_alias=AliasChoices("S3_PUBLIC_ENDPOINT_URL"),
)
presigned_upload_ttl_seconds: int = 300
attachment_cleanup_interval_seconds: int = Field(default=60, ge=5, le=3600)
attachment_cleanup_grace_seconds: int = Field(default=600, ge=0, le=86400)
attachment_cleanup_batch_size: int = Field(default=100, ge=1, le=1000)
s3_region: str = Field(default="us-east-1", validation_alias=AliasChoices("S3_REGION"))
s3_bucket: str = Field(default="messenger-attachments", validation_alias=AliasChoices("S3_BUCKET"))
s3_access_key: str = Field(
default="messenger-api",
validation_alias=AliasChoices("S3_ACCESS_KEY", "S3_ACCESS_KEY_ID"),
)
s3_secret_key: str = Field(
default="change-me-api-secret",
validation_alias=AliasChoices("S3_SECRET_KEY", "S3_SECRET_ACCESS_KEY"),
)
max_upload_bytes: int = 10 * 1024 * 1024
max_image_pixels: int = 40_000_000
model_config = SettingsConfigDict(
env_file=".env", env_file_encoding="utf-8", extra="ignore"
)
settings = Settings()
View File
+5
View File
@@ -0,0 +1,5 @@
from sqlalchemy.orm import DeclarativeBase
class Base(DeclarativeBase):
pass
+298
View File
@@ -0,0 +1,298 @@
import uuid
from datetime import datetime, timezone
from sqlalchemy import (
DateTime,
JSON,
ForeignKey,
Index,
String,
Text,
UniqueConstraint,
)
from sqlalchemy.dialects.postgresql import UUID
from sqlalchemy.orm import Mapped, mapped_column, relationship
from app.db.base import Base
def utc_now() -> datetime:
return datetime.now(timezone.utc)
class User(Base):
__tablename__ = "users"
id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
primary_key=True,
default=uuid.uuid4,
)
username: Mapped[str] = mapped_column(
String(32),
unique=True,
nullable=False,
index=True,
)
display_name: Mapped[str] = mapped_column(
String(100),
nullable=False,
)
password_hash: Mapped[str] = mapped_column(
String(255),
nullable=False,
)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
nullable=False,
)
memberships: Mapped[list["ChatMember"]] = relationship(
back_populates="user",
cascade="all, delete-orphan",
)
messages: Mapped[list["Message"]] = relationship(
back_populates="sender",
)
class Chat(Base):
__tablename__ = "chats"
id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
primary_key=True,
default=uuid.uuid4,
)
title: Mapped[str] = mapped_column(
String(100),
nullable=False,
)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
nullable=False,
)
members: Mapped[list["ChatMember"]] = relationship(
back_populates="chat",
cascade="all, delete-orphan",
)
messages: Mapped[list["Message"]] = relationship(
back_populates="chat",
cascade="all, delete-orphan",
)
class ChatMember(Base):
__tablename__ = "chat_members"
chat_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("chats.id", ondelete="CASCADE"),
primary_key=True,
)
user_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("users.id", ondelete="CASCADE"),
primary_key=True,
)
chat: Mapped["Chat"] = relationship(
back_populates="members",
)
user: Mapped["User"] = relationship(
back_populates="memberships",
)
class Message(Base):
__tablename__ = "messages"
id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
primary_key=True,
default=uuid.uuid4,
)
chat_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("chats.id", ondelete="CASCADE"),
nullable=False,
)
sender_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("users.id"),
nullable=False,
)
text: Mapped[str] = mapped_column(
Text,
nullable=False,
)
attachment_ids: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
nullable=False,
)
chat: Mapped["Chat"] = relationship(
back_populates="messages",
)
sender: Mapped["User"] = relationship(
back_populates="messages",
)
__table_args__ = (
Index(
"ix_messages_chat_created_id",
"chat_id",
"created_at",
"id",
),
)
class Session(Base):
__tablename__ = "sessions"
id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
primary_key=True,
default=uuid.uuid4,
)
token_hash: Mapped[str] = mapped_column(
String(64),
unique=True,
nullable=False,
index=True,
)
user_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
)
expires_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
nullable=False,
)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
nullable=False,
)
user: Mapped["User"] = relationship()
class IdempotencyRecord(Base):
__tablename__ = "idempotency_records"
id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True), primary_key=True, default=uuid.uuid4
)
user_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True), ForeignKey("users.id", ondelete="CASCADE"), nullable=False
)
method: Mapped[str] = mapped_column(String(10), nullable=False)
path: Mapped[str] = mapped_column(String(255), nullable=False)
idempotency_key: Mapped[str] = mapped_column(String(128), nullable=False)
request_hash: Mapped[str] = mapped_column(String(64), nullable=False)
# NULL is visible only inside the creating transaction; the message and
# idempotency row are committed atomically.
message_id: Mapped[uuid.UUID | None] = mapped_column(
UUID(as_uuid=True), ForeignKey("messages.id", ondelete="CASCADE"), nullable=True
)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), default=utc_now, nullable=False
)
expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
__table_args__ = (
UniqueConstraint(
"user_id", "method", "path", "idempotency_key",
name="uq_idempotency_scope_key",
),
Index("ix_idempotency_records_expires_at", "expires_at"),
)
class Attachment(Base):
__tablename__ = "attachments"
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
chat_id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), ForeignKey("chats.id", ondelete="CASCADE"), nullable=False, index=True)
owner_id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), ForeignKey("users.id", ondelete="CASCADE"), nullable=False)
filename: Mapped[str] = mapped_column(String(255), nullable=False)
content_type: Mapped[str] = mapped_column(String(32), nullable=False)
size_bytes: Mapped[int] = mapped_column(nullable=False)
sha256: Mapped[str] = mapped_column(String(64), nullable=False)
object_key: Mapped[str] = mapped_column(String(512), unique=True, nullable=False)
status: Mapped[str] = mapped_column(String(16), nullable=False, default="ready")
error_code: Mapped[str | None] = mapped_column(String(64), nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utc_now, nullable=False)
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utc_now, onupdate=utc_now, nullable=False)
class AttachmentUpload(Base):
__tablename__ = "attachment_uploads"
id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
primary_key=True,
default=uuid.uuid4,
)
chat_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("chats.id", ondelete="CASCADE"),
nullable=False,
)
owner_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
)
filename: Mapped[str] = mapped_column(
String(255),
nullable=False,
)
content_type: Mapped[str] = mapped_column(
String(32),
nullable=False,
)
expected_size_bytes: Mapped[int] = mapped_column(
nullable=False,
)
expected_sha256: Mapped[str] = mapped_column(
String(64),
nullable=False,
)
object_key: Mapped[str] = mapped_column(
String(512),
unique=True,
nullable=False,
)
status: Mapped[str] = mapped_column(
String(16),
nullable=False,
default="pending",
)
expires_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
nullable=False,
)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=utc_now,
nullable=False,
)
completed_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True),
nullable=True,
)
__table_args__ = (
Index("ix_attachment_uploads_chat_id", "chat_id"),
Index("ix_attachment_uploads_owner_id", "owner_id"),
Index("ix_attachment_uploads_expires_at", "expires_at"),
)
+15
View File
@@ -0,0 +1,15 @@
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from app.core.config import settings
engine = create_engine(
settings.database_url,
pool_pre_ping=True,
pool_size=settings.db_pool_size,
max_overflow=settings.db_max_overflow,
pool_timeout=settings.db_pool_timeout,
pool_recycle=1800,
)
SessionLocal = sessionmaker(bind=engine, autoflush=False, autocommit=False)
+106
View File
@@ -0,0 +1,106 @@
import asyncio
import logging
import uuid
from contextlib import asynccontextmanager, suppress
from fastapi import FastAPI, HTTPException, Request
from fastapi.exceptions import RequestValidationError
from fastapi.responses import JSONResponse
from pathlib import Path
from sqlalchemy import text
from sqlalchemy.exc import SQLAlchemyError
from app.api.auth import router as auth_router
from app.api.users import router as users_router
from app.api.chats import router as chats_router
from app.api.messages import router as messages_router
from app.api.attachments import router as attachments_router
from app.core.config import settings
from app.db.session import engine
from app.services.object_storage import client as s3_client
from app.services.attachment_cleanup import cleanup_expired_uploads
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s %(message)s")
logger = logging.getLogger(__name__)
async def attachment_cleanup_loop():
while True:
try:
# Synchronous boto3/SQLAlchemy I/O must not block the event loop.
await asyncio.to_thread(cleanup_expired_uploads)
except Exception:
logger.exception("Attachment cleanup worker iteration failed")
await asyncio.sleep(settings.attachment_cleanup_interval_seconds)
@asynccontextmanager
async def lifespan(app: FastAPI):
logger.info("Application startup instance_id=%s", settings.instance_id)
cleanup_task = asyncio.create_task(attachment_cleanup_loop())
try:
yield
finally:
cleanup_task.cancel()
with suppress(asyncio.CancelledError):
await cleanup_task
engine.dispose()
logger.info("Application shutdown instance_id=%s", settings.instance_id)
app = FastAPI(title=settings.app_name, version="1.4.0", lifespan=lifespan)
app.include_router(users_router)
app.include_router(auth_router)
app.include_router(chats_router)
app.include_router(messages_router)
app.include_router(attachments_router)
@app.middleware("http")
async def response_headers(request: Request, call_next):
request_id = str(uuid.uuid4())
request.state.request_id = request_id
try:
response = await call_next(request)
except Exception:
logger.exception("Unhandled request error; request_id=%s", request_id)
response = JSONResponse(status_code=500, content={"error": {"code": "internal_error", "message": "Unexpected error", "request_id": request_id}})
response.headers["X-Request-Id"] = request_id
response.headers["X-Instance-Id"] = settings.instance_id
response.headers["X-Content-Type-Options"] = "nosniff"
return response
@app.exception_handler(HTTPException)
async def http_exception_handler(request: Request, exc: HTTPException):
request_id = getattr(request.state, "request_id", str(uuid.uuid4()))
code = {400: "invalid_request", 401: "unauthorized", 404: "not_found", 409: "conflict", 413: "payload_too_large", 415: "unsupported_media_type", 429: "rate_limited", 500: "internal_error", 503: "unavailable"}.get(exc.status_code, "internal_error")
headers = dict(exc.headers or {})
headers["X-Request-Id"] = request_id
headers["X-Instance-Id"] = settings.instance_id
return JSONResponse(status_code=exc.status_code, content={"error": {"code": code, "message": exc.detail if isinstance(exc.detail, str) else "Request failed", "request_id": request_id}}, headers=headers)
@app.exception_handler(RequestValidationError)
async def validation_exception_handler(request: Request, exc: RequestValidationError):
request_id = getattr(request.state, "request_id", str(uuid.uuid4()))
return JSONResponse(status_code=400, content={"error": {"code": "invalid_request", "message": "Validation error", "request_id": request_id}}, headers={"X-Request-Id": request_id, "X-Instance-Id": settings.instance_id})
@app.get("/health/live")
def liveness():
return {"status": "ok"}
DRAIN_FILE = Path("/tmp/api-draining")
@app.get("/health/ready")
def readiness():
if DRAIN_FILE.exists():
raise HTTPException(
status_code=503,
detail="Instance is draining",
)
try:
with engine.connect() as connection:
connection.execute(text("SELECT 1"))
s3_client().head_bucket(Bucket=settings.s3_bucket)
return {"status": "ok"}
except Exception:
logger.exception("PostgreSQL or S3 readiness check failed")
raise HTTPException(
status_code=503,
detail="Required storage unavailable",
) from None
+138
View File
@@ -0,0 +1,138 @@
from datetime import datetime, timezone
from typing import List, Optional, Literal
from pydantic import BaseModel, ConfigDict, Field, field_validator
class BaseConfig(BaseModel):
model_config = ConfigDict(
from_attributes=True,
json_encoders={
datetime: lambda value: (
value.astimezone(timezone.utc)
.isoformat(timespec="microseconds")
.replace("+00:00", "Z")
)
},
)
class CreateUser(BaseConfig):
username: str = Field(..., pattern=r"^[a-z0-9_]{3,32}$")
display_name: str = Field(..., min_length=1, max_length=100)
password: str = Field(..., min_length=12, max_length=128)
@field_validator("display_name")
@classmethod
def validate_display_name(cls, value: str) -> str:
value = value.strip()
if not value:
raise ValueError("display_name must not be blank")
return value
class User(BaseConfig):
id: str
username: str
display_name: str
created_at: datetime
class CreateChat(BaseConfig):
title: str = Field(..., min_length=1, max_length=100)
member_ids: List[str] = Field(..., min_length=1, max_length=100)
@field_validator("title")
@classmethod
def validate_title(cls, value: str) -> str:
value = value.strip()
if not value:
raise ValueError("title must not be blank")
return value
@field_validator("member_ids")
@classmethod
def validate_unique_members(cls, value: List[str]) -> List[str]:
if len(value) != len(set(value)):
raise ValueError("member_ids must be unique")
return value
class Chat(BaseConfig):
id: str
title: str
member_ids: List[str]
created_at: datetime
class CreateMessage(BaseConfig):
text: str = Field(..., max_length=4000)
attachment_ids: List[str] = Field(default_factory=list, max_length=10)
@field_validator("attachment_ids")
@classmethod
def validate_attachment_ids(cls, value):
if len(value) != len(set(value)):
raise ValueError("attachment_ids must be unique")
return value
@field_validator("text")
@classmethod
def validate_text(cls, value: str) -> str:
value = value.strip()
return value
class Message(BaseConfig):
id: str
chat_id: str
sender_id: str
text: str
created_at: datetime
attachment_ids: List[str] = Field(default_factory=list)
class Health(BaseConfig):
status: str = "ok"
class ErrorDetail(BaseConfig):
code: str
message: str
request_id: str
class ErrorResponse(BaseConfig):
error: ErrorDetail
class ChatPage(BaseConfig):
items: List[Chat]
next_cursor: Optional[str] = None
class MessagePage(BaseConfig):
items: List[Message]
next_cursor: Optional[str] = None
class CreateAttachmentUpload(BaseConfig):
filename: str = Field(..., min_length=1, max_length=255)
content_type: Literal["image/png", "image/jpeg"]
size_bytes: int = Field(..., gt=0, le=10 * 1024 * 1024)
sha256: str = Field(..., pattern=r"^[a-fA-F0-9]{64}$")
@field_validator("filename")
@classmethod
def validate_filename(cls, value: str) -> str:
value = value.strip()
if (
not value
or any(c in value for c in ("/", "\\", "\x00"))
or any(ord(c) < 32 or ord(c) == 127 for c in value)
):
raise ValueError("Unsafe filename")
return value
View File
+71
View File
@@ -0,0 +1,71 @@
import uuid
from sqlalchemy import select
from sqlalchemy.orm import Session, selectinload
from app.db.models import Chat, ChatMember
class ChatRepository:
def __init__(self, db: Session):
self.db = db
def get_by_id(self, chat_id: uuid.UUID) -> Chat | None:
statement = (
select(Chat)
.options(selectinload(Chat.members))
.where(Chat.id == chat_id)
)
return self.db.scalar(statement)
def get_by_user(
self,
user_id: uuid.UUID,
limit: int = 50,
) -> list[Chat]:
statement = (
select(Chat)
.join(ChatMember, ChatMember.chat_id == Chat.id)
.where(ChatMember.user_id == user_id)
.order_by(Chat.created_at.asc(), Chat.id.asc())
.limit(limit)
)
return list(self.db.scalars(statement).all())
def is_member(
self,
chat_id: uuid.UUID,
user_id: uuid.UUID,
) -> bool:
statement = select(ChatMember).where(
ChatMember.chat_id == chat_id,
ChatMember.user_id == user_id,
)
return self.db.scalar(statement) is not None
def create(
self,
title: str,
member_ids: list[uuid.UUID],
) -> Chat:
chat = Chat(title=title)
self.db.add(chat)
self.db.flush()
memberships = [
ChatMember(
chat_id=chat.id,
user_id=user_id,
)
for user_id in member_ids
]
self.db.add_all(memberships)
self.db.flush()
return chat
+73
View File
@@ -0,0 +1,73 @@
import uuid
from datetime import datetime
from sqlalchemy import and_, or_, select
from sqlalchemy.orm import Session
from app.db.models import Message
class MessageRepository:
def __init__(self, db: Session):
self.db = db
def get_by_id(
self,
message_id: uuid.UUID,
) -> Message | None:
statement = select(Message).where(
Message.id == message_id
)
return self.db.scalar(statement)
def create(
self,
chat_id: uuid.UUID,
sender_id: uuid.UUID,
text: str,
attachment_ids: list[str] | None = None,
) -> Message:
message = Message(
chat_id=chat_id,
sender_id=sender_id,
text=text,
attachment_ids=attachment_ids or [],
)
self.db.add(message)
self.db.flush()
return message
def get_history(
self,
chat_id: uuid.UUID,
limit: int = 50,
cursor_created_at: datetime | None = None,
cursor_id: uuid.UUID | None = None,
) -> list[Message]:
statement = select(Message).where(
Message.chat_id == chat_id
)
if cursor_created_at is not None and cursor_id is not None:
statement = statement.where(
or_(
Message.created_at > cursor_created_at,
and_(
Message.created_at == cursor_created_at,
Message.id > cursor_id,
),
)
)
statement = (
statement
.order_by(Message.created_at.asc(), Message.id.asc())
.limit(limit)
)
return list(self.db.scalars(statement).all())
+78
View File
@@ -0,0 +1,78 @@
import uuid
from datetime import datetime, timezone
from sqlalchemy import delete, select
from sqlalchemy.orm import Session
from app.db.models import Session as UserSession
class SessionRepository:
def __init__(self, db: Session):
self.db = db
def create(
self,
user_id: uuid.UUID,
token_hash: str,
expires_at: datetime,
) -> UserSession:
session = UserSession(
user_id=user_id,
token_hash=token_hash,
expires_at=expires_at,
)
self.db.add(session)
self.db.flush()
return session
def get_by_token_hash(
self,
token_hash: str,
) -> UserSession | None:
statement = select(UserSession).where(
UserSession.token_hash == token_hash
)
return self.db.scalar(statement)
def get_valid_session(
self,
token_hash: str,
) -> UserSession | None:
now = datetime.now(timezone.utc)
statement = select(UserSession).where(
UserSession.token_hash == token_hash,
UserSession.expires_at > now,
)
return self.db.scalar(statement)
def delete_by_token_hash(
self,
token_hash: str,
) -> bool:
statement = (
delete(UserSession)
.where(UserSession.token_hash == token_hash)
.returning(UserSession.id)
)
return self.db.scalar(statement) is not None
def delete_expired(self) -> int:
now = datetime.now(timezone.utc)
statement = (
delete(UserSession)
.where(UserSession.expires_at <= now)
)
result = self.db.execute(statement)
return result.rowcount
+41
View File
@@ -0,0 +1,41 @@
import uuid
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.db.models import User
class UserRepository:
def __init__(self, db: Session):
self.db = db
def get_by_id(self, user_id: uuid.UUID) -> User | None:
statement = select(User).where(User.id == user_id)
return self.db.scalar(statement)
def get_by_username(self, username: str) -> User | None:
statement = select(User).where(
User.username == username
)
return self.db.scalar(statement)
def create(
self,
username: str,
display_name: str,
password_hash: str,
) -> User:
user = User(
username=username,
display_name=display_name,
password_hash=password_hash,
)
self.db.add(user)
self.db.flush()
return user
View File
+66
View File
@@ -0,0 +1,66 @@
"""Best-effort cleanup for abandoned presigned uploads."""
import logging
from datetime import datetime, timedelta, timezone
from sqlalchemy import select
from app.core.config import settings
from app.db.models import AttachmentUpload
from app.db.session import SessionLocal
from app.services.object_storage import client
logger = logging.getLogger(__name__)
def cleanup_expired_uploads() -> int:
"""Delete old pending objects and mark their upload records expired.
A grace period protects clients completing an upload near URL expiry.
PostgreSQL SKIP LOCKED makes this safe across multiple API replicas.
"""
cutoff = datetime.now(timezone.utc) - timedelta(
seconds=settings.attachment_cleanup_grace_seconds
)
deleted_count = 0
with SessionLocal() as db:
uploads = db.scalars(
select(AttachmentUpload)
.where(
AttachmentUpload.status == "pending",
AttachmentUpload.expires_at <= cutoff,
)
.order_by(AttachmentUpload.expires_at.asc())
.limit(settings.attachment_cleanup_batch_size)
.with_for_update(skip_locked=True)
).all()
if not uploads:
return 0
s3 = client()
try:
for upload in uploads:
# DeleteObject is idempotent: a missing object is also safe.
s3.delete_object(
Bucket=settings.s3_bucket,
Key=upload.object_key,
)
upload.status = "expired"
deleted_count += 1
db.commit()
except Exception:
db.rollback()
logger.exception(
"Attachment cleanup batch failed; pending records will be retried"
)
return 0
if deleted_count:
logger.info(
"Expired abandoned attachment uploads cleaned: count=%d",
deleted_count,
)
return deleted_count
+159
View File
@@ -0,0 +1,159 @@
import hashlib
import secrets
from datetime import datetime, timedelta, timezone
from redis import Redis
from redis.backoff import NoBackoff
from redis.exceptions import RedisError
from redis.retry import Retry
from sqlalchemy.orm import Session
from app.core.config import settings
from app.db.models import User
from app.repositories.session_repository import SessionRepository
from app.repositories.user_repository import UserRepository
from app.services.user_service import password_hasher
class InvalidCredentialsError(Exception):
pass
class InvalidSessionError(Exception):
pass
class AuthService:
def __init__(self, db: Session):
self.db = db
self.sessions = SessionRepository(db)
self.users = UserRepository(db)
self._redis_available = True
self.redis = Redis.from_url(
settings.cache_url,
decode_responses=True,
socket_connect_timeout=0.5,
socket_timeout=0.5,
retry_on_timeout=False,
retry=Retry(NoBackoff(), 0),
)
@staticmethod
def _hash_token(token: str) -> str:
return hashlib.sha256(token.encode("utf-8")).hexdigest()
def create_session(self, username: str, password: str) -> dict:
user = self.users.get_by_username(username)
if user is None or not password_hasher.verify(
password, user.password_hash
):
raise InvalidCredentialsError("Invalid username or password")
token = secrets.token_urlsafe(32)
token_hash = self._hash_token(token)
expires_at = datetime.now(timezone.utc) + timedelta(
seconds=settings.session_ttl_seconds
)
try:
self.sessions.create(
user_id=user.id,
token_hash=token_hash,
expires_at=expires_at,
)
self.db.commit()
except Exception:
self.db.rollback()
raise
self._cache_session(token_hash, user.id, expires_at)
return {
"token": token,
"token_type": "Bearer",
"expires_at": expires_at,
"user": user,
}
def _cache_session(
self,
token_hash: str,
user_id,
expires_at,
) -> None:
if not self._redis_available:
return
ttl = max(
1,
int((expires_at - datetime.now(timezone.utc)).total_seconds()),
)
try:
self.redis.setex(
f"session:{token_hash}",
ttl,
str(user_id),
)
except RedisError:
self._redis_available = False
def authenticate(self, token: str) -> User:
token_hash = self._hash_token(token)
cached_user_id = None
if self._redis_available:
try:
cached_user_id = self.redis.get(
f"session:{token_hash}"
)
except RedisError:
self._redis_available = False
session = self.sessions.get_valid_session(token_hash)
if session is None:
if self._redis_available:
try:
self.redis.delete(f"session:{token_hash}")
except RedisError:
self._redis_available = False
raise InvalidSessionError(
"Session is invalid or expired"
)
if cached_user_id is None:
self._cache_session(
token_hash,
session.user_id,
session.expires_at,
)
return session.user
def delete_session(self, token: str) -> None:
token_hash = self._hash_token(token)
try:
deleted = self.sessions.delete_by_token_hash(token_hash)
if not deleted:
raise InvalidSessionError(
"Session is invalid or expired"
)
self.db.commit()
except Exception:
self.db.rollback()
raise
if self._redis_available:
try:
self.redis.delete(f"session:{token_hash}")
except RedisError:
self._redis_available = False
+94
View File
@@ -0,0 +1,94 @@
import uuid
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from app.db.models import Chat
from app.repositories.chat_repository import ChatRepository
from app.repositories.user_repository import UserRepository
class ChatNotFoundError(Exception):
pass
class ChatAccessError(Exception):
pass
class InvalidChatDataError(Exception):
pass
class ChatService:
def __init__(self, db: Session):
self.db = db
self.chats = ChatRepository(db)
self.users = UserRepository(db)
def create_chat(
self,
title: str,
member_ids: list[str],
creator_id: uuid.UUID,
) -> Chat:
title = title.strip()
if not title or len(title) > 100:
raise InvalidChatDataError("Invalid chat title")
if not member_ids or len(member_ids) > 100:
raise InvalidChatDataError("Invalid members")
try:
parsed_ids = [uuid.UUID(str(value)) for value in member_ids]
except (ValueError, TypeError, AttributeError):
raise InvalidChatDataError("Invalid member UUID") from None
if len(parsed_ids) != len(set(parsed_ids)):
raise InvalidChatDataError("Duplicate members")
if creator_id not in parsed_ids:
raise InvalidChatDataError("Creator must be a member")
for user_id in parsed_ids:
if self.users.get_by_id(user_id) is None:
raise InvalidChatDataError("Unknown member")
try:
chat = self.chats.create(
title=title,
member_ids=parsed_ids,
)
self.db.commit()
self.db.refresh(chat)
return chat
except IntegrityError as exc:
self.db.rollback()
raise InvalidChatDataError("Unable to create chat") from exc
except Exception:
self.db.rollback()
raise
def get_chat(self, chat_id: uuid.UUID, user_id: uuid.UUID) -> Chat:
chat = self.chats.get_by_id(chat_id)
if chat is None:
raise ChatNotFoundError()
if not self.chats.is_member(chat_id, user_id):
raise ChatNotFoundError()
return chat
def list_chats(self, user_id: uuid.UUID) -> list[Chat]:
return self.chats.get_by_user(user_id)
def check_membership(
self,
chat_id: uuid.UUID,
user_id: uuid.UUID,
) -> bool:
return self.chats.is_member(chat_id, user_id)
+41
View File
@@ -0,0 +1,41 @@
from io import BytesIO
from PIL import Image, UnidentifiedImageError
from fastapi import HTTPException
MAX_BYTES = 10 * 1024 * 1024
MAX_PIXELS = 40_000_000
ALLOWED = {"PNG": "image/png", "JPEG": "image/jpeg"}
def validate_image_file(fileobj, declared_type: str):
"""Validate a seekable, size-limited file without loading its full encoded bytes."""
try:
fileobj.seek(0)
with Image.open(fileobj) as image:
fmt = image.format
width, height = image.size
if fmt not in ALLOWED:
raise HTTPException(415, "Only PNG and JPEG are supported")
if width * height > MAX_PIXELS:
raise HTTPException(413, "Image exceeds 40 million pixels")
image.verify()
fileobj.seek(0)
with Image.open(fileobj) as image:
image.load()
except HTTPException:
raise
except (UnidentifiedImageError, OSError, ValueError, SyntaxError, Image.DecompressionBombError):
raise HTTPException(415, "Invalid or corrupted image") from None
actual_type = ALLOWED[fmt]
if declared_type != actual_type:
raise HTTPException(415, "Content-Type does not match image content")
fileobj.seek(0)
return actual_type, width, height
def validate_image(data: bytes, declared_type: str):
if not data:
raise HTTPException(415, "Empty image")
if len(data) > MAX_BYTES:
raise HTTPException(413, "Image exceeds 10 MiB")
return validate_image_file(BytesIO(data), declared_type)
+435
View File
@@ -0,0 +1,435 @@
import base64
import hashlib
import json
import time
import uuid
from datetime import datetime, timedelta, timezone
from types import SimpleNamespace
from redis import Redis
from redis.backoff import NoBackoff
from redis.exceptions import RedisError
from redis.retry import Retry
from sqlalchemy import delete, select, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.orm import Session
from app.core.config import settings
from app.db.models import Attachment, IdempotencyRecord, Message
from app.repositories.chat_repository import ChatRepository
from app.repositories.message_repository import MessageRepository
class MessageNotFoundError(Exception):
pass
class MessageAccessError(Exception):
pass
class InvalidMessageDataError(Exception):
pass
class IdempotencyConflictError(Exception):
pass
class MessageService:
def __init__(self, db: Session):
self.db = db
self.messages = MessageRepository(db)
self.chats = ChatRepository(db)
self._redis_available = True
self.redis = Redis.from_url(
settings.cache_url,
decode_responses=True,
socket_connect_timeout=0.5,
socket_timeout=0.5,
retry_on_timeout=False,
retry=Retry(NoBackoff(), 0),
)
@staticmethod
def encode_cursor(message: Message) -> str:
payload = {
"created_at": message.created_at.isoformat(),
"id": str(message.id),
}
raw = json.dumps(payload).encode("utf-8")
return (
base64.urlsafe_b64encode(raw)
.decode("ascii")
.rstrip("=")
)
@staticmethod
def decode_cursor(cursor: str | None):
if not cursor:
return None, None
try:
padded = cursor + "=" * (-len(cursor) % 4)
raw = base64.urlsafe_b64decode(padded.encode("ascii"))
payload = json.loads(raw.decode("utf-8"))
created_at = datetime.fromisoformat(payload["created_at"])
message_id = uuid.UUID(payload["id"])
if created_at.tzinfo is None:
raise ValueError("Cursor timestamp must include timezone")
return created_at, message_id
except (ValueError, TypeError, KeyError, UnicodeError):
raise InvalidMessageDataError("Invalid cursor") from None
def _version_key(self, chat_id: uuid.UUID) -> str:
return f"messages:version:{chat_id}"
def _cache_key(
self,
chat_id: uuid.UUID,
version: int,
limit: int,
cursor: str | None,
) -> str:
cursor_hash = hashlib.sha256(
(cursor or "first-page").encode("utf-8")
).hexdigest()[:16]
return (
f"messages:{chat_id}:v{version}:"
f"limit:{limit}:cursor:{cursor_hash}"
)
def _get_version(self, chat_id: uuid.UUID) -> int:
if not self._redis_available:
return 0
try:
value = self.redis.get(self._version_key(chat_id))
return int(value or 0)
except (RedisError, ValueError):
self._redis_available = False
return 0
def _bump_version(self, chat_id: uuid.UUID) -> None:
if not self._redis_available:
return
try:
self.redis.incr(self._version_key(chat_id))
except RedisError:
self._redis_available = False
@staticmethod
def _serialize(messages, next_cursor):
return json.dumps({
"items": [
{
"id": str(message.id),
"chat_id": str(message.chat_id),
"sender_id": str(message.sender_id),
"text": message.text,
"attachment_ids": message.attachment_ids or [],
"created_at": message.created_at.isoformat(),
}
for message in messages
],
"next_cursor": next_cursor,
})
@staticmethod
def _deserialize(payload: str):
data = json.loads(payload)
messages = [
SimpleNamespace(
id=uuid.UUID(item["id"]),
chat_id=uuid.UUID(item["chat_id"]),
sender_id=uuid.UUID(item["sender_id"]),
text=item["text"],
attachment_ids=item.get("attachment_ids", []),
created_at=datetime.fromisoformat(item["created_at"]),
)
for item in data["items"]
]
return messages, data["next_cursor"]
def _read_cache(self, key: str):
if not self._redis_available:
return None
try:
payload = self.redis.get(key)
if payload is None:
return None
return self._deserialize(payload)
except RedisError:
self._redis_available = False
return None
except (ValueError, TypeError, KeyError):
return None
def _load_from_database(
self,
chat_id,
limit,
cursor_created_at,
cursor_id,
):
rows = self.messages.get_history(
chat_id=chat_id,
limit=limit + 1,
cursor_created_at=cursor_created_at,
cursor_id=cursor_id,
)
has_more = len(rows) > limit
items = rows[:limit]
next_cursor = (
self.encode_cursor(items[-1])
if has_more and items
else None
)
return items, next_cursor
def create_message(
self,
chat_id: uuid.UUID,
sender_id: uuid.UUID,
text: str,
idempotency_key: str,
attachment_ids: list[str] | None = None,
) -> Message:
text = text.strip()
attachment_ids = attachment_ids or []
if (not text and not attachment_ids) or len(text) > 4000 or len(attachment_ids) > 10:
raise InvalidMessageDataError("Invalid message text")
# Access is checked for every retry, before consulting the key.
if not self.chats.is_member(chat_id, sender_id):
raise MessageNotFoundError()
if len(attachment_ids) != len(set(attachment_ids)):
raise InvalidMessageDataError("Duplicate attachment IDs")
if attachment_ids:
try:
parsed_attachment_ids = [uuid.UUID(value) for value in attachment_ids]
except (ValueError, TypeError):
raise InvalidMessageDataError("Invalid attachment UUID") from None
attachments = list(self.db.scalars(select(Attachment).where(Attachment.id.in_(parsed_attachment_ids))).all())
if len(attachments) != len(parsed_attachment_ids) or any(a.chat_id != chat_id for a in attachments):
raise MessageNotFoundError()
if any(a.status != "ready" for a in attachments):
raise InvalidMessageDataError("Attachment is not ready")
method = "POST"
path = f"/api/v1/chats/{chat_id}/messages"
request_hash = hashlib.sha256(
json.dumps({"text": text, "attachment_ids": attachment_ids}, sort_keys=True, separators=(",", ":")).encode("utf-8")
).hexdigest()
now = datetime.now(timezone.utc)
expires_at = now + timedelta(seconds=max(settings.idempotency_ttl_seconds, 86400))
scope = (
IdempotencyRecord.user_id == sender_id,
IdempotencyRecord.method == method,
IdempotencyRecord.path == path,
IdempotencyRecord.idempotency_key == idempotency_key,
)
try:
# An expired key may be reused. Deletion and claim are serialized by
# PostgreSQL's unique constraint for the scope.
self.db.execute(
delete(IdempotencyRecord).where(*scope, IdempotencyRecord.expires_at <= now)
)
claim = (
pg_insert(IdempotencyRecord)
.values(
id=uuid.uuid4(), user_id=sender_id, method=method, path=path,
idempotency_key=idempotency_key, request_hash=request_hash,
message_id=None, created_at=now, expires_at=expires_at,
)
.on_conflict_do_nothing(constraint="uq_idempotency_scope_key")
.returning(IdempotencyRecord.id)
)
claimed_id = self.db.execute(claim).scalar_one_or_none()
if claimed_id is None:
existing = self.db.execute(
select(IdempotencyRecord).where(*scope).with_for_update()
).scalar_one_or_none()
if existing is None:
# Rare expiration race: retry once in a fresh transaction.
self.db.rollback()
return self.create_message(chat_id, sender_id, text, idempotency_key, attachment_ids)
if existing.request_hash != request_hash:
self.db.rollback()
raise IdempotencyConflictError()
if existing.message_id is None:
raise RuntimeError("Committed idempotency record has no message")
message = self.db.get(Message, existing.message_id)
if message is None:
raise RuntimeError("Idempotency record references a missing message")
self.db.commit()
return message
message = self.messages.create(
chat_id=chat_id, sender_id=sender_id, text=text, attachment_ids=attachment_ids
)
self.db.execute(
update(IdempotencyRecord)
.where(IdempotencyRecord.id == claimed_id)
.values(message_id=message.id)
)
self.db.commit()
self.db.refresh(message)
self._bump_version(chat_id)
return message
except IdempotencyConflictError:
raise
except Exception:
self.db.rollback()
raise
def get_history(
self,
chat_id: uuid.UUID,
user_id: uuid.UUID,
limit: int = 50,
cursor: str | None = None,
):
if not 1 <= limit <= 100:
raise InvalidMessageDataError(
"Limit must be between 1 and 100"
)
if not self.chats.is_member(chat_id, user_id):
raise MessageNotFoundError()
cursor_created_at, cursor_id = self.decode_cursor(cursor)
version = self._get_version(chat_id)
cache_key = self._cache_key(
chat_id,
version,
limit,
cursor,
)
cached = self._read_cache(cache_key)
if cached is not None:
return cached
if not self._redis_available:
return self._load_from_database(
chat_id,
limit,
cursor_created_at,
cursor_id,
)
lock_key = f"lock:{cache_key}"
lock_token = str(uuid.uuid4())
try:
acquired = self.redis.set(
lock_key,
lock_token,
nx=True,
ex=10,
)
except RedisError:
self._redis_available = False
return self._load_from_database(
chat_id,
limit,
cursor_created_at,
cursor_id,
)
if not acquired:
for _ in range(10):
time.sleep(0.05)
cached = self._read_cache(cache_key)
if cached is not None:
return cached
if not self._redis_available:
break
return self._load_from_database(
chat_id,
limit,
cursor_created_at,
cursor_id,
)
try:
cached = self._read_cache(cache_key)
if cached is not None:
return cached
if not self._redis_available:
return self._load_from_database(
chat_id,
limit,
cursor_created_at,
cursor_id,
)
items, next_cursor = self._load_from_database(
chat_id,
limit,
cursor_created_at,
cursor_id,
)
if self._redis_available and self._get_version(chat_id) == version:
try:
self.redis.setex(
cache_key,
settings.message_cache_ttl_seconds,
self._serialize(items, next_cursor),
)
except RedisError:
self._redis_available = False
return items, next_cursor
finally:
if self._redis_available:
try:
self.redis.eval(
"""
if redis.call('get', KEYS[1]) == ARGV[1]
then
return redis.call('del', KEYS[1])
else
return 0
end
""",
1,
lock_key,
lock_token,
)
except RedisError:
self._redis_available = False
+39
View File
@@ -0,0 +1,39 @@
import boto3
from botocore.config import Config
from app.core.config import settings
def client(endpoint_url=None):
return boto3.client(
"s3", endpoint_url=endpoint_url or settings.s3_endpoint_url,
aws_access_key_id=settings.s3_access_key,
aws_secret_access_key=settings.s3_secret_key,
region_name=settings.s3_region,
config=Config(signature_version="s3v4", connect_timeout=3, read_timeout=15, retries={"max_attempts": 2}, s3={"addressing_style": "path"}),
)
def ensure_bucket() -> None:
s3 = client()
try:
s3.head_bucket(Bucket=settings.s3_bucket)
except Exception:
s3.create_bucket(Bucket=settings.s3_bucket)
def presigned_client():
return client(endpoint_url=settings.s3_public_endpoint_url)
def create_presigned_upload_url(object_key: str, content_type: str) -> str:
s3 = presigned_client()
return s3.generate_presigned_url(
ClientMethod="put_object",
Params={
"Bucket": settings.s3_bucket,
"Key": object_key,
"ContentType": content_type,
},
ExpiresIn=settings.presigned_upload_ttl_seconds,
)
+81
View File
@@ -0,0 +1,81 @@
import re
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from pwdlib import PasswordHash
from app.db.models import User
from app.repositories.user_repository import UserRepository
password_hasher = PasswordHash.recommended()
class UsernameAlreadyExistsError(Exception):
pass
class UserService:
def __init__(self, db: Session):
self.db = db
self.users = UserRepository(db)
def register(
self,
username: str,
display_name: str,
password: str,
) -> User:
if not re.fullmatch(r"[a-z0-9_]{3,32}", username):
raise ValueError("Invalid username")
if not display_name or not display_name.strip():
raise ValueError("Display name cannot be empty")
if len(display_name) > 100:
raise ValueError("Display name is too long")
if not 12 <= len(password) <= 128:
raise ValueError("Invalid password length")
if self.users.get_by_username(username) is not None:
raise UsernameAlreadyExistsError(
"Username already exists"
)
password_hash = password_hasher.hash(password)
try:
user = self.users.create(
username=username,
display_name=display_name,
password_hash=password_hash,
)
self.db.commit()
self.db.refresh(user)
return user
except IntegrityError as exc:
self.db.rollback()
raise UsernameAlreadyExistsError(
"Username already exists"
) from exc
except Exception:
self.db.rollback()
raise
def verify_password(
self,
user: User,
password: str,
) -> bool:
return password_hasher.verify(
password,
user.password_hash,
)
+193
View File
@@ -0,0 +1,193 @@
import base64
import bisect
import binascii
import json
import os
import threading
import uuid
from datetime import datetime, timezone
from typing import Dict, List, Optional, Tuple
from .models import Chat, Message, User
Key = Tuple[datetime, str]
def _encode_cursor(kind: str, scope: str, item) -> str:
payload = {
"v": 1,
"kind": kind,
"scope": scope,
"created_at": item.created_at.isoformat(),
"id": item.id,
}
raw = json.dumps(payload, separators=(",", ":")).encode("utf-8")
return base64.urlsafe_b64encode(raw).decode("ascii").rstrip("=")
def _decode_cursor(cursor: str, kind: str, scope: str) -> Key:
try:
if not cursor or len(cursor) > 4096:
raise ValueError
padded = cursor + "=" * (-len(cursor) % 4)
raw = base64.b64decode(padded, altchars=b"-_", validate=True)
payload = json.loads(raw.decode("utf-8"))
if not isinstance(payload, dict) or payload.get("v") != 1:
raise ValueError
if payload.get("kind") != kind or payload.get("scope") != scope:
raise ValueError
item_id = payload.get("id")
timestamp = payload.get("created_at")
if not isinstance(item_id, str) or not isinstance(timestamp, str):
raise ValueError
created_at = datetime.fromisoformat(timestamp)
if created_at.tzinfo is None:
raise ValueError
created_at = created_at.astimezone(timezone.utc)
return created_at, item_id
except (
ValueError,
TypeError,
KeyError,
UnicodeDecodeError,
json.JSONDecodeError,
binascii.Error,
):
raise ValueError("invalid_request") from None
class InMemoryStore:
def __init__(self):
self._lock = threading.RLock()
self.users: Dict[str, User] = {}
self.chats: Dict[str, Chat] = {}
self.messages: Dict[str, List[Message]] = {}
self.usernames: Dict[str, str] = {}
self.message_keys: Dict[str, List[Key]] = {}
self.message_index_enabled = os.getenv("MESSAGE_INDEX_ENABLED", "1") != "0"
def create_user(self, username: str, display_name: str) -> User:
with self._lock:
if username in self.usernames:
raise ValueError("conflict")
user_id = str(uuid.uuid4())
user = User(
id=user_id,
username=username,
display_name=display_name,
created_at=datetime.now(timezone.utc),
)
self.users[user_id] = user
self.usernames[username] = user_id
return user
def get_user(self, user_id: str) -> Optional[User]:
with self._lock:
return self.users.get(user_id)
def create_chat(self, title: str, member_ids: List[str], creator_id: str) -> Chat:
with self._lock:
if creator_id not in member_ids or len(member_ids) != len(set(member_ids)):
raise ValueError("invalid_request")
if any(member_id not in self.users for member_id in member_ids):
raise ValueError("invalid_request")
chat_id = str(uuid.uuid4())
chat = Chat(
id=chat_id,
title=title,
member_ids=list(member_ids),
created_at=datetime.now(timezone.utc),
)
self.chats[chat_id] = chat
self.messages[chat_id] = []
self.message_keys[chat_id] = []
return chat
def get_chat(self, chat_id: str, user_id: str) -> Optional[Chat]:
with self._lock:
chat = self.chats.get(chat_id)
if chat is None or user_id not in chat.member_ids:
return None
return chat
def list_chats(
self, user_id: str, limit: int, cursor: Optional[str]
) -> Tuple[List[Chat], Optional[str]]:
with self._lock:
user_chats = [chat for chat in self.chats.values() if user_id in chat.member_ids]
user_chats.sort(key=lambda chat: (chat.created_at, chat.id))
keys = [(chat.created_at, chat.id) for chat in user_chats]
start_index = 0
if cursor:
target_key = _decode_cursor(cursor, kind="chats", scope=user_id)
start_index = bisect.bisect_right(keys, target_key)
page_with_extra = user_chats[start_index : start_index + limit + 1]
has_next = len(page_with_extra) > limit
page = page_with_extra[:limit]
next_cursor = None
if has_next and page:
next_cursor = _encode_cursor("chats", user_id, page[-1])
return page, next_cursor
def create_message(self, chat_id: str, sender_id: str, text: str) -> Message:
with self._lock:
chat = self.chats.get(chat_id)
if chat is None or sender_id not in chat.member_ids:
raise ValueError("not_found")
message = Message(
id=str(uuid.uuid4()),
chat_id=chat_id,
sender_id=sender_id,
text=text,
created_at=datetime.now(timezone.utc),
)
messages = self.messages[chat_id]
keys = self.message_keys[chat_id]
key = (message.created_at, message.id)
if self.message_index_enabled:
index = bisect.bisect_right(keys, key)
keys.insert(index, key)
messages.insert(index, message)
else:
current_keys = [(item.created_at, item.id) for item in messages]
index = bisect.bisect_right(current_keys, key)
messages.insert(index, message)
keys.insert(index, key)
return message
def list_messages(
self,
chat_id: str,
user_id: str,
limit: int,
cursor: Optional[str],
) -> Tuple[List[Message], Optional[str]]:
with self._lock:
chat = self.chats.get(chat_id)
if chat is None or user_id not in chat.member_ids:
raise ValueError("not_found")
messages = self.messages[chat_id]
if self.message_index_enabled:
keys = self.message_keys[chat_id]
else:
keys = [(item.created_at, item.id) for item in messages]
start_index = 0
if cursor:
target_key = _decode_cursor(cursor, kind="messages", scope=chat_id)
start_index = bisect.bisect_right(keys, target_key)
page_with_extra = messages[start_index : start_index + limit + 1]
has_next = len(page_with_extra) > limit
page = page_with_extra[:limit]
next_cursor = None
if has_next and page:
next_cursor = _encode_cursor("messages", chat_id, page[-1])
return page, next_cursor
+183
View File
@@ -0,0 +1,183 @@
name: mtusi-lab4
services:
postgres:
image: postgres:16.6-bookworm
restart: unless-stopped
environment:
POSTGRES_DB: ${POSTGRES_DB:-messenger}
POSTGRES_USER: ${POSTGRES_USER:-messenger}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U \"$$POSTGRES_USER\" -d \"$$POSTGRES_DB\""]
interval: 5s
timeout: 3s
retries: 15
start_period: 10s
networks: [backend]
redis:
image: redis:7.4.2-bookworm
restart: unless-stopped
command: ["redis-server", "--appendonly", "yes", "--appendfsync", "everysec"]
volumes:
- redis_data:/data
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 3s
retries: 10
networks: [backend]
minio:
image: pgsty/silo:RELEASE.2026-10-26T00-00-00Z
command: ["server", "/data", "--address", ":9000", "--console-address", ":9001"]
restart: unless-stopped
ports:
- "127.0.0.1:9000:9000"
environment:
MINIO_ROOT_USER: ${MINIO_ROOT_USER:?Set MINIO_ROOT_USER in .env}
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:?Set MINIO_ROOT_PASSWORD in .env}
volumes:
- minio_data:/data
# MinIO is intentionally internal: neither S3 API nor admin console is published on the host.
networks: [backend]
minio-bootstrap:
image: pgsty/mc:RELEASE.2026-10-26T00-00-00Z
depends_on:
minio:
condition: service_started
entrypoint: ["/bin/sh", "-ec"]
command:
- |
mc --disable-pager alias set local http://minio:9000 "$${MINIO_ROOT_USER}" "$${MINIO_ROOT_PASSWORD}"
until mc --disable-pager ready local; do echo 'Waiting for MinIO...'; sleep 2; done
mc --disable-pager mb --ignore-existing local/messenger-attachments
mc --disable-pager anonymous set none local/messenger-attachments
if ! mc --disable-pager admin policy info local messenger-attachments-policy >/dev/null 2>&1; then
mc --disable-pager admin policy create local messenger-attachments-policy /policy.json
fi
if ! mc --disable-pager admin user info local "$${S3_ACCESS_KEY_ID}" >/dev/null 2>&1; then
mc --disable-pager admin user add local "$${S3_ACCESS_KEY_ID}" "$${S3_SECRET_ACCESS_KEY}"
fi
mc --disable-pager admin policy attach local messenger-attachments-policy --user "$${S3_ACCESS_KEY_ID}"
environment:
MINIO_ROOT_USER: ${MINIO_ROOT_USER:?Set MINIO_ROOT_USER in .env}
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:?Set MINIO_ROOT_PASSWORD in .env}
S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID:?Set S3_ACCESS_KEY_ID in .env}
S3_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY:?Set S3_SECRET_ACCESS_KEY in .env}
volumes:
- ./infra/minio-policy.json:/policy.json:ro
restart: "no"
networks: [backend]
migrate:
build:
context: .
dockerfile: Dockerfile
command: ["alembic", "upgrade", "head"]
environment:
DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-messenger}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB:-messenger}
depends_on:
postgres:
condition: service_healthy
restart: "no"
networks: [backend]
api:
build:
context: .
dockerfile: Dockerfile
restart: unless-stopped
environment:
APP_ENV: lab
DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-messenger}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB:-messenger}
CACHE_URL: redis://redis:6379/0
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
MESSAGE_CACHE_TTL_SECONDS: ${MESSAGE_CACHE_TTL_SECONDS:-60}
IDEMPOTENCY_TTL_SECONDS: ${IDEMPOTENCY_TTL_SECONDS:-86400}
DB_POOL_SIZE: ${DB_POOL_SIZE:-3}
DB_MAX_OVERFLOW: ${DB_MAX_OVERFLOW:-2}
DB_POOL_TIMEOUT: ${DB_POOL_TIMEOUT:-10}
S3_ENDPOINT: http://minio:9000
S3_PUBLIC_ENDPOINT_URL: ${S3_PUBLIC_ENDPOINT_URL:-http://localhost:9000}
S3_REGION: ${S3_REGION:-us-east-1}
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID:?Set S3_ACCESS_KEY_ID in .env}
S3_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY:?Set S3_SECRET_ACCESS_KEY in .env}
MAX_UPLOAD_BYTES: ${MAX_UPLOAD_BYTES:-10485760}
MAX_IMAGE_PIXELS: ${MAX_IMAGE_PIXELS:-40000000}
ATTACHMENT_CLEANUP_INTERVAL_SECONDS: ${ATTACHMENT_CLEANUP_INTERVAL_SECONDS:-60}
ATTACHMENT_CLEANUP_GRACE_SECONDS: ${ATTACHMENT_CLEANUP_GRACE_SECONDS:-600}
ATTACHMENT_CLEANUP_BATCH_SIZE: ${ATTACHMENT_CLEANUP_BATCH_SIZE:-100}
depends_on:
migrate:
condition: service_completed_successfully
postgres:
condition: service_healthy
redis:
condition: service_healthy
minio-bootstrap:
condition: service_completed_successfully
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=2)"]
interval: 5s
timeout: 3s
retries: 3
start_period: 20s
deploy:
replicas: 2
resources:
limits:
cpus: "1.0"
memory: 512M
labels:
- traefik.enable=true
- traefik.docker.network=mtusi-lab4-edge
- traefik.http.routers.messenger.rule=PathPrefix(`/`)
- traefik.http.routers.messenger.entrypoints=web
- traefik.http.services.messenger.loadbalancer.server.port=8080
- traefik.http.services.messenger.loadbalancer.healthcheck.path=/health/ready
- traefik.http.services.messenger.loadbalancer.healthcheck.interval=3s
- traefik.http.services.messenger.loadbalancer.healthcheck.timeout=2s
- traefik.http.services.messenger.loadbalancer.passhostheader=true
networks: [backend, edge]
traefik:
image: traefik:v3.6.2
restart: unless-stopped
command:
- --providers.docker=true
- --providers.docker.exposedbydefault=false
- --providers.docker.network=mtusi-lab4-edge
- --entrypoints.web.address=:8080
- --ping=true
- --ping.entrypoint=ping
- --entrypoints.ping.address=:8082
- --accesslog=true
- --log.level=INFO
ports:
- "8080:8080"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
healthcheck:
test: ["CMD", "wget", "-q", "--spider", "http://127.0.0.1:8082/ping"]
interval: 10s
timeout: 3s
retries: 5
networks: [edge]
networks:
backend:
driver: bridge
edge:
name: mtusi-lab4-edge
driver: bridge
volumes:
postgres_data:
redis_data:
minio_data:
Binary file not shown.
+14
View File
@@ -0,0 +1,14 @@
ATTACHMENT FAILOVER TEST
Mode: failover
Observed API instances before failure: 2
Successful writes: 87
Failed attempts: 4
First recovery: 4.597 seconds
The API replica that handled the attachment upload was stopped.
The attachment remained accessible through the other API replica.
Downloaded content matched the original bytes.
Existing session and confirmed messages were preserved.
RESULT: PASS
+3
View File
@@ -0,0 +1,3 @@
method,concurrency,file_size_bytes,successful_uploads,failed_uploads,wall_time_s,throughput_uploads_per_s,peak_api_total_mib,baseline_api_total_mib,peak_delta_api_total_mib,api1_peak_mib,api2_peak_mib,memory_samples,memory_sampling_errors,avg_upload_time_s,max_upload_time_s,api_body_bytes_per_upload,s3_body_bytes_per_upload
proxy,10,5769330,10,0,3.9963,2.502,220.2,204.1,16.1,110.2,110.0,4,0,2.3315,2.6514,5769330,0
presigned,10,5769330,10,0,4.941,2.024,304.4,220.2,84.2,156.8,147.6,5,0,3.6495,3.9018,160,5769330
1 method concurrency file_size_bytes successful_uploads failed_uploads wall_time_s throughput_uploads_per_s peak_api_total_mib baseline_api_total_mib peak_delta_api_total_mib api1_peak_mib api2_peak_mib memory_samples memory_sampling_errors avg_upload_time_s max_upload_time_s api_body_bytes_per_upload s3_body_bytes_per_upload
2 proxy 10 5769330 10 0 3.9963 2.502 220.2 204.1 16.1 110.2 110.0 4 0 2.3315 2.6514 5769330 0
3 presigned 10 5769330 10 0 4.941 2.024 304.4 220.2 84.2 156.8 147.6 5 0 3.6495 3.9018 160 5769330
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
+15
View File
@@ -0,0 +1,15 @@
S3 SECURITY TEST
1. Admin console port:
Command: docker compose port minio 9001
Result: no port 9001/tcp published.
2. Anonymous bucket access:
Command: curl.exe -i http://localhost:9000/messenger-attachments/
Result: HTTP 403 Forbidden
S3 error: AccessDenied
3. Bucket:
messenger-attachments
RESULT: PASS
+25
View File
@@ -0,0 +1,25 @@
S3 UNAVAILABLE TEST
1. Stopped S3:
docker compose stop minio
2. Readiness response while S3 was unavailable:
HTTP 503
{
"error": {
"code": "unavailable",
"message": "Required storage unavailable"
}
}
3. Restarted S3:
docker compose start minio
4. Container status:
Both API replicas healthy.
S3 container running.
5. Readiness response after recovery:
status: ok
RESULT: PASS
Binary file not shown.
+13
View File
@@ -0,0 +1,13 @@
method,file,file_size_bytes,repeat,elapsed_total_s,elapsed_api_s,client_to_api_body_bytes,client_to_s3_upload_bytes,api_peak_rss_mib,api_peak_delta_mib
proxy,sample.png,111817,1,0.0838,0.0837,111817,0,103.4,0.0
proxy,sample.png,111817,2,0.0773,0.0772,111817,0,103.4,0.0
proxy,sample.png,111817,3,0.0746,0.0746,111817,0,103.4,0.0
presigned,sample.png,111817,1,0.1103,0.084,158,111817,103.8,1.5
presigned,sample.png,111817,2,0.1003,0.0556,158,111817,103.8,0.0
presigned,sample.png,111817,3,0.1782,0.1082,158,111817,112.7,8.9
proxy,sample_large.png,5769330,1,0.2416,0.2415,5769330,0,103.8,1.5
proxy,sample_large.png,5769330,2,0.2777,0.2776,5769330,0,105.3,1.5
proxy,sample_large.png,5769330,3,0.2806,0.2805,5769330,0,109.2,5.2
presigned,sample_large.png,5769330,1,0.3062,0.1403,159,5769330,128.9,19.7
presigned,sample_large.png,5769330,2,0.2171,0.1092,159,5769330,116.3,0.0
presigned,sample_large.png,5769330,3,0.2459,0.1329,159,5769330,116.3,0.0
1 method file file_size_bytes repeat elapsed_total_s elapsed_api_s client_to_api_body_bytes client_to_s3_upload_bytes api_peak_rss_mib api_peak_delta_mib
2 proxy sample.png 111817 1 0.0838 0.0837 111817 0 103.4 0.0
3 proxy sample.png 111817 2 0.0773 0.0772 111817 0 103.4 0.0
4 proxy sample.png 111817 3 0.0746 0.0746 111817 0 103.4 0.0
5 presigned sample.png 111817 1 0.1103 0.084 158 111817 103.8 1.5
6 presigned sample.png 111817 2 0.1003 0.0556 158 111817 103.8 0.0
7 presigned sample.png 111817 3 0.1782 0.1082 158 111817 112.7 8.9
8 proxy sample_large.png 5769330 1 0.2416 0.2415 5769330 0 103.8 1.5
9 proxy sample_large.png 5769330 2 0.2777 0.2776 5769330 0 105.3 1.5
10 proxy sample_large.png 5769330 3 0.2806 0.2805 5769330 0 109.2 5.2
11 presigned sample_large.png 5769330 1 0.3062 0.1403 159 5769330 128.9 19.7
12 presigned sample_large.png 5769330 2 0.2171 0.1092 159 5769330 116.3 0.0
13 presigned sample_large.png 5769330 3 0.2459 0.1329 159 5769330 116.3 0.0
+19
View File
@@ -0,0 +1,19 @@
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation",
"s3:ListBucket",
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::messenger-attachments",
"arn:aws:s3:::messenger-attachments/*"
]
}
]
}
+1
View File
@@ -0,0 +1 @@
Generic single-database configuration.
+60
View File
@@ -0,0 +1,60 @@
from logging.config import fileConfig
from alembic import context
from sqlalchemy import engine_from_config, pool
from app.core.config import settings
from app.db.base import Base
from app.db import models
config = context.config
if config.config_file_name is not None:
fileConfig(config.config_file_name)
config.set_main_option(
"sqlalchemy.url",
settings.database_url.replace("%", "%%"),
)
target_metadata = Base.metadata
def run_migrations_offline() -> None:
url = settings.database_url
context.configure(
url=url,
target_metadata=target_metadata,
literal_binds=True,
dialect_opts={"paramstyle": "named"},
compare_type=True,
)
with context.begin_transaction():
context.run_migrations()
def run_migrations_online() -> None:
connectable = engine_from_config(
config.get_section(config.config_ini_section, {}),
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
with connectable.connect() as connection:
context.configure(
connection=connection,
target_metadata=target_metadata,
compare_type=True,
)
with context.begin_transaction():
context.run_migrations()
if context.is_offline_mode():
run_migrations_offline()
else:
run_migrations_online()
+26
View File
@@ -0,0 +1,26 @@
"""${message}
Revision ID: ${up_revision}
Revises: ${down_revision | comma,n}
Create Date: ${create_date}
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
${imports if imports else ""}
# revision identifiers, used by Alembic.
revision: str = ${repr(up_revision)}
down_revision: Union[str, None] = ${repr(down_revision)}
branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)}
depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)}
def upgrade() -> None:
${upgrades if upgrades else "pass"}
def downgrade() -> None:
${downgrades if downgrades else "pass"}
@@ -0,0 +1,79 @@
"""create initial database schema
Revision ID: 21ad89656327
Revises:
Create Date: 2026-11-04 17:31:39.336368
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = '21ad89656327'
down_revision: Union[str, None] = None
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('chats',
sa.Column('id', sa.UUID(), nullable=False),
sa.Column('title', sa.String(length=100), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.PrimaryKeyConstraint('id')
)
op.create_table('users',
sa.Column('id', sa.UUID(), nullable=False),
sa.Column('username', sa.String(length=32), nullable=False),
sa.Column('display_name', sa.String(length=100), nullable=False),
sa.Column('password_hash', sa.String(length=255), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.PrimaryKeyConstraint('id')
)
op.create_index(op.f('ix_users_username'), 'users', ['username'], unique=True)
op.create_table('chat_members',
sa.Column('chat_id', sa.UUID(), nullable=False),
sa.Column('user_id', sa.UUID(), nullable=False),
sa.ForeignKeyConstraint(['chat_id'], ['chats.id'], ondelete='CASCADE'),
sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('chat_id', 'user_id')
)
op.create_table('messages',
sa.Column('id', sa.UUID(), nullable=False),
sa.Column('chat_id', sa.UUID(), nullable=False),
sa.Column('sender_id', sa.UUID(), nullable=False),
sa.Column('text', sa.Text(), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(['chat_id'], ['chats.id'], ondelete='CASCADE'),
sa.ForeignKeyConstraint(['sender_id'], ['users.id'], ),
sa.PrimaryKeyConstraint('id')
)
op.create_index('ix_messages_chat_created_id', 'messages', ['chat_id', 'created_at', 'id'], unique=False)
op.create_table('sessions',
sa.Column('id', sa.UUID(), nullable=False),
sa.Column('token_hash', sa.String(length=64), nullable=False),
sa.Column('user_id', sa.UUID(), nullable=False),
sa.Column('expires_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id')
)
op.create_index(op.f('ix_sessions_token_hash'), 'sessions', ['token_hash'], unique=True)
# ### end Alembic commands ###
def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.drop_index(op.f('ix_sessions_token_hash'), table_name='sessions')
op.drop_table('sessions')
op.drop_index('ix_messages_chat_created_id', table_name='messages')
op.drop_table('messages')
op.drop_table('chat_members')
op.drop_index(op.f('ix_users_username'), table_name='users')
op.drop_table('users')
op.drop_table('chats')
# ### end Alembic commands ###
@@ -0,0 +1,37 @@
"""add attachment metadata"""
from alembic import op
import sqlalchemy as sa
from typing import Sequence, Union
revision: str = "4d9a1f2c7b10"
down_revision: Union[str, None] = "8c3d9f4a21b7"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.add_column("messages", sa.Column("attachment_ids", sa.JSON(), nullable=False, server_default="[]"))
op.create_table(
"attachments",
sa.Column("id", sa.UUID(), nullable=False),
sa.Column("chat_id", sa.UUID(), nullable=False),
sa.Column("owner_id", sa.UUID(), nullable=False),
sa.Column("filename", sa.String(255), nullable=False),
sa.Column("content_type", sa.String(32), nullable=False),
sa.Column("size_bytes", sa.Integer(), nullable=False),
sa.Column("sha256", sa.String(64), nullable=False),
sa.Column("object_key", sa.String(512), nullable=False),
sa.Column("status", sa.String(16), nullable=False),
sa.Column("error_code", sa.String(64), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(["chat_id"], ["chats.id"], ondelete="CASCADE"),
sa.ForeignKeyConstraint(["owner_id"], ["users.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("object_key"),
)
op.create_index("ix_attachments_chat_id", "attachments", ["chat_id"])
def downgrade() -> None:
op.drop_index("ix_attachments_chat_id", table_name="attachments")
op.drop_table("attachments")
op.drop_column("messages", "attachment_ids")
@@ -0,0 +1,39 @@
"""add shared PostgreSQL idempotency records
Revision ID: 8c3d9f4a21b7
Revises: 21ad89656327
Create Date: 2026-11-05
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
revision: str = "8c3d9f4a21b7"
down_revision: Union[str, None] = "21ad89656327"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"idempotency_records",
sa.Column("id", sa.UUID(), nullable=False),
sa.Column("user_id", sa.UUID(), nullable=False),
sa.Column("method", sa.String(length=10), nullable=False),
sa.Column("path", sa.String(length=255), nullable=False),
sa.Column("idempotency_key", sa.String(length=128), nullable=False),
sa.Column("request_hash", sa.String(length=64), nullable=False),
sa.Column("message_id", sa.UUID(), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(["message_id"], ["messages.id"], ondelete="CASCADE"),
sa.ForeignKeyConstraint(["user_id"], ["users.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("user_id", "method", "path", "idempotency_key", name="uq_idempotency_scope_key"),
)
op.create_index("ix_idempotency_records_expires_at", "idempotency_records", ["expires_at"], unique=False)
def downgrade() -> None:
op.drop_index("ix_idempotency_records_expires_at", table_name="idempotency_records")
op.drop_table("idempotency_records")
@@ -0,0 +1,78 @@
"""add attachment uploads
Revision ID: a7c2e91d5f40
Revises: 4d9a1f2c7b10
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
revision: str = "a7c2e91d5f40"
down_revision: Union[str, None] = "4d9a1f2c7b10"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"attachment_uploads",
sa.Column("id", sa.UUID(), nullable=False),
sa.Column("chat_id", sa.UUID(), nullable=False),
sa.Column("owner_id", sa.UUID(), nullable=False),
sa.Column("filename", sa.String(length=255), nullable=False),
sa.Column("content_type", sa.String(length=32), nullable=False),
sa.Column("expected_size_bytes", sa.Integer(), nullable=False),
sa.Column("expected_sha256", sa.String(length=64), nullable=False),
sa.Column("object_key", sa.String(length=512), nullable=False),
sa.Column("status", sa.String(length=16), nullable=False),
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("completed_at", sa.DateTime(timezone=True), nullable=True),
sa.ForeignKeyConstraint(
["chat_id"],
["chats.id"],
ondelete="CASCADE",
),
sa.ForeignKeyConstraint(
["owner_id"],
["users.id"],
ondelete="CASCADE",
),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("object_key"),
)
op.create_index(
"ix_attachment_uploads_chat_id",
"attachment_uploads",
["chat_id"],
)
op.create_index(
"ix_attachment_uploads_owner_id",
"attachment_uploads",
["owner_id"],
)
op.create_index(
"ix_attachment_uploads_expires_at",
"attachment_uploads",
["expires_at"],
)
def downgrade() -> None:
op.drop_index(
"ix_attachment_uploads_expires_at",
table_name="attachment_uploads",
)
op.drop_index(
"ix_attachment_uploads_owner_id",
table_name="attachment_uploads",
)
op.drop_index(
"ix_attachment_uploads_chat_id",
table_name="attachment_uploads",
)
op.drop_table("attachment_uploads")
+11
View File
@@ -0,0 +1,11 @@
fastapi==0.115.6
uvicorn[standard]==0.34.0
pydantic==2.10.4
pydantic-settings==2.7.1
SQLAlchemy==2.0.36
psycopg[binary]==3.2.3
alembic==1.14.0
redis==5.2.1
pwdlib[argon2]==0.2.1
boto3==1.35.99
Pillow==11.0.0
+30
View File
@@ -0,0 +1,30 @@
import os
import runpy
import subprocess
import sys
bash = r"C:\Program Files\Git\bin\bash.exe"
original_popen = subprocess.Popen
def compatible_popen(args, **kwargs):
if isinstance(args, (list, tuple)) and len(args) == 1:
args = [bash, args[0]]
return original_popen(args, **kwargs)
subprocess.Popen = compatible_popen
sys.path.insert(0, os.path.abspath("tests"))
mode = sys.argv[1] if len(sys.argv) > 1 else "failover"
if mode not in ("persistence", "failover"):
raise SystemExit("Mode must be persistence or failover")
sys.argv = [
"tests/resilience.py",
"--mode", mode,
"--action-script",
"scripts/restart.sh" if mode == "persistence" else "scripts/stop-one.sh",
"--base-url", "http://localhost:8080",
]
runpy.run_path("tests/resilience.py", run_name="__main__")
+52
View File
@@ -0,0 +1,52 @@
#!/usr/bin/env python3
"""Create charts from real Track 1 summary.csv data; does not invent measurements."""
import csv
from pathlib import Path
import matplotlib.pyplot as plt
root = Path(__file__).resolve().parents[1]
source = root / 'evidence' / 'scaling' / 'summary.csv'
if not source.exists():
raise SystemExit('Run scripts/run-scaling-experiment.py first')
with source.open(encoding='utf-8', newline='') as f:
rows = list(csv.DictReader(f))
if not rows:
raise SystemExit('summary.csv is empty')
out = source.parent
for key, title, ylabel, filename in [
('successful_rps', 'Track 1: successful throughput',
'Successful requests/sec', 'throughput.png'),
('p95_ms', 'Track 1: p95 latency',
'Milliseconds', 'p95-latency.png'),
('error_ratio', 'Track 1: error ratio',
'Error ratio', 'errors.png'),
]:
labels = [r['scenario'] for r in rows]
values = [float(r[key]) if r[key] else 0 for r in rows]
plt.figure(figsize=(11, 5))
plt.bar(labels, values)
if key == 'error_ratio':
plt.ylim(0, 0.01)
for i, value in enumerate(values):
plt.text(
i, 0.0002, f'{value:.1%}',
ha='center', va='bottom'
)
plt.title(title)
plt.ylabel(ylabel)
plt.xticks(rotation=25, ha='right')
plt.tight_layout()
plt.savefig(out / filename, dpi=160)
plt.close()
print('Charts written to', out)
+304
View File
@@ -0,0 +1,304 @@
import csv
import hashlib
import json
import subprocess
import sys
import threading
import time
import urllib.request
from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / "tests"))
from client import Client, check
BASE_URL = "http://localhost:8080"
FIXTURE = ROOT / "tests" / "fixtures" / "sample_large.png"
API_CONTAINERS = ("mtusi-lab4-api-1", "mtusi-lab4-api-2")
CONCURRENCY = 10
OUTPUT = ROOT / "evidence" / "concurrent-upload-results.csv"
def parse_mib(value):
units = {
"B": 1 / (1024 * 1024),
"KiB": 1 / 1024,
"MiB": 1,
"GiB": 1024,
"Ki": 1 / 1024,
"Mi": 1,
"Gi": 1024,
"kB": 1000 / (1024 * 1024),
"MB": 1000 * 1000 / (1024 * 1024),
"GB": 1000 * 1000 * 1000 / (1024 * 1024),
}
for unit in sorted(units, key=len, reverse=True):
if value.endswith(unit):
return float(value[:-len(unit)].strip()) * units[unit]
raise ValueError(f"Unknown Docker memory unit: {value}")
def docker_memory():
result = subprocess.run(
[
"docker", "stats", "--no-stream",
"--format", "{{.Name}}|{{.MemUsage}}",
],
capture_output=True,
text=True,
check=True,
)
values = {}
for line in result.stdout.splitlines():
if "|" not in line:
continue
name, usage = line.split("|", 1)
if name in API_CONTAINERS:
values[name] = parse_mib(usage.split("/", 1)[0].strip())
missing = set(API_CONTAINERS) - set(values)
if missing:
raise RuntimeError("Missing API containers: " + ", ".join(missing))
return values
class MemoryMonitor:
def __init__(self):
self.running = False
self.samples = []
self.thread = None
self.errors = []
def sample(self):
while self.running:
try:
values = docker_memory()
self.samples.append(values)
except Exception as exc:
self.errors.append(type(exc).__name__)
time.sleep(0.1)
def __enter__(self):
self.baseline = docker_memory()
self.samples = [dict(self.baseline)]
self.running = True
self.thread = threading.Thread(target=self.sample, daemon=True)
self.thread.start()
return self
def __exit__(self, exc_type, exc_value, traceback):
self.running = False
self.thread.join(timeout=5)
try:
self.samples.append(docker_memory())
except Exception as exc:
self.errors.append(type(exc).__name__)
def summary(self):
peak_by_container = {
name: max(sample[name] for sample in self.samples)
for name in API_CONTAINERS
}
total_samples = [
sum(sample[name] for name in API_CONTAINERS)
for sample in self.samples
]
baseline_total = sum(self.baseline.values())
peak_total = max(total_samples)
return {
"baseline_total_mib": round(baseline_total, 2),
"peak_total_mib": round(peak_total, 2),
"peak_delta_total_mib": round(peak_total - baseline_total, 2),
"api1_peak_mib": round(peak_by_container[API_CONTAINERS[0]], 2),
"api2_peak_mib": round(peak_by_container[API_CONTAINERS[1]], 2),
"samples": len(self.samples),
"sampling_errors": len(self.errors),
}
def response_json(response):
return json.loads(response["bytes"].decode("utf-8"))
def upload_one(index, method, actor, chat, content, digest, barrier):
client = Client(BASE_URL)
barrier.wait(timeout=30)
started = time.monotonic()
if method == "proxy":
response = client.call(
"POST",
f"/api/v1/chats/{chat['id']}/attachments"
f"?filename=parallel-{index}.png",
201,
actor=actor,
raw=content,
headers={"Content-Type": "image/png"},
)
attachment = response_json(response)
api_elapsed = response["elapsed"]
api_body_bytes = len(content)
s3_body_bytes = 0
else:
body = {
"filename": f"parallel-{index}.png",
"content_type": "image/png",
"size_bytes": len(content),
"sha256": digest,
}
create_response = client.call(
"POST",
f"/api/v1/chats/{chat['id']}/attachments/presigned",
201,
actor=actor,
body=body,
)
created = response_json(create_response)
request = urllib.request.Request(
created["upload_url"],
data=content,
headers=created["headers"],
method="PUT",
)
with urllib.request.urlopen(request, timeout=90) as response:
check(response.status in (200, 201, 204), "S3 PUT failed")
complete_response = client.call(
"POST",
f"/api/v1/attachments/uploads/{created['upload_id']}/complete",
200,
actor=actor,
)
attachment = response_json(complete_response)
api_elapsed = create_response["elapsed"] + complete_response["elapsed"]
api_body_bytes = len(json.dumps(body).encode("utf-8"))
s3_body_bytes = len(content)
elapsed = time.monotonic() - started
check(attachment["status"] == "ready", "attachment is not ready")
check(attachment["size_bytes"] == len(content), "wrong attachment size")
check(attachment["sha256"] == digest, "wrong attachment checksum")
return {
"worker": index,
"elapsed_total_s": round(elapsed, 4),
"elapsed_api_s": round(api_elapsed, 4),
"client_to_api_body_bytes": api_body_bytes,
"client_to_s3_upload_bytes": s3_body_bytes,
"attachment_id": attachment["id"],
}
def run_method(method, actor, chat, content, digest):
print(f"\nStarting {CONCURRENCY} simultaneous {method} uploads...")
barrier = threading.Barrier(CONCURRENCY)
results = []
errors = []
with MemoryMonitor() as memory:
started = time.monotonic()
with ThreadPoolExecutor(max_workers=CONCURRENCY) as executor:
futures = [
executor.submit(
upload_one, index, method, actor, chat,
content, digest, barrier,
)
for index in range(1, CONCURRENCY + 1)
]
for future in as_completed(futures):
try:
results.append(future.result())
except Exception as exc:
errors.append(f"{type(exc).__name__}: {exc}")
wall_time = time.monotonic() - started
mem = memory.summary()
successful = len(results)
print(
f"{method}: success={successful}/{CONCURRENCY}, "
f"errors={len(errors)}, wall={wall_time:.3f}s, "
f"peak API RAM={mem['peak_total_mib']} MiB, "
f"delta={mem['peak_delta_total_mib']} MiB, "
f"samples={mem['samples']}"
)
for error in errors:
print("ERROR:", error)
return {
"method": method,
"concurrency": CONCURRENCY,
"file_size_bytes": len(content),
"successful_uploads": successful,
"failed_uploads": len(errors),
"wall_time_s": round(wall_time, 4),
"throughput_uploads_per_s": round(successful / wall_time, 3),
"peak_api_total_mib": mem["peak_total_mib"],
"baseline_api_total_mib": mem["baseline_total_mib"],
"peak_delta_api_total_mib": mem["peak_delta_total_mib"],
"api1_peak_mib": mem["api1_peak_mib"],
"api2_peak_mib": mem["api2_peak_mib"],
"memory_samples": mem["samples"],
"memory_sampling_errors": mem["sampling_errors"],
"avg_upload_time_s": round(
sum(item["elapsed_total_s"] for item in results) / successful, 4
) if successful else None,
"max_upload_time_s": max(
(item["elapsed_total_s"] for item in results), default=None
),
"api_body_bytes_per_upload": (
results[0]["client_to_api_body_bytes"] if results else None
),
"s3_body_bytes_per_upload": (
results[0]["client_to_s3_upload_bytes"] if results else None
),
}
def main():
if not FIXTURE.exists():
raise FileNotFoundError(FIXTURE)
content = FIXTURE.read_bytes()
if len(content) > 10 * 1024 * 1024:
raise RuntimeError("Fixture exceeds 10 MiB")
digest = hashlib.sha256(content).hexdigest()
with urllib.request.urlopen(BASE_URL + "/health/ready", timeout=10) as response:
check(response.status == 200, "API is not ready")
# Prepare identities and chat before measuring concurrent uploads.
setup = Client(BASE_URL)
alice, _, _, chat = setup.bootstrap()
rows = []
for method in ("proxy", "presigned"):
rows.append(run_method(method, alice, chat, content, digest))
OUTPUT.parent.mkdir(parents=True, exist_ok=True)
with OUTPUT.open("w", newline="", encoding="utf-8-sig") as file:
writer = csv.DictWriter(file, fieldnames=list(rows[0].keys()))
writer.writeheader()
writer.writerows(rows)
print("\nResults:", OUTPUT)
print("Fixture:", FIXTURE.name)
print("File size:", len(content), "bytes")
print("SHA-256:", digest)
if any(row["successful_uploads"] != CONCURRENCY for row in rows):
raise SystemExit("FAIL: at least one concurrent upload failed")
print("PASS: 10 concurrent uploads completed for both methods")
if __name__ == "__main__":
main()
+375
View File
@@ -0,0 +1,375 @@
import csv
import json
import subprocess
import sys
import threading
import time
import urllib.request
import hashlib
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / "tests"))
from client import Client, check
BASE_URL = "http://localhost:8080"
FIXTURES = [
ROOT / "tests" / "fixtures" / "sample.png",
ROOT / "tests" / "fixtures" / "sample_large.png",
]
API_CONTAINERS = [
"mtusi-lab4-api-1",
"mtusi-lab4-api-2",
]
REPEATS = 3
OUTPUT = ROOT / "evidence" / "track1-results.csv"
def docker_memory():
"""Return current memory usage in MiB for both API containers."""
result = subprocess.run(
[
"docker",
"stats",
"--no-stream",
"--format",
"{{.Name}}|{{.MemUsage}}",
],
capture_output=True,
text=True,
check=True,
)
values = {}
for line in result.stdout.splitlines():
if "|" not in line:
continue
name, usage = line.split("|", 1)
if name not in API_CONTAINERS:
continue
memory_value = usage.split("/", 1)[0].strip()
values[name] = parse_memory_mib(memory_value)
missing = set(API_CONTAINERS) - set(values)
if missing:
raise RuntimeError(
"Could not read memory for containers: "
+ ", ".join(sorted(missing))
)
return values
def parse_memory_mib(value):
"""Convert Docker memory values to MiB."""
value = value.strip()
units = {
"B": 1 / (1024 * 1024),
"KiB": 1 / 1024,
"MiB": 1,
"GiB": 1024,
"kB": 1000 / (1024 * 1024),
"MB": 1000 * 1000 / (1024 * 1024),
"GB": 1000 * 1000 * 1000 / (1024 * 1024),
"Ki": 1 / 1024,
"Mi": 1,
"Gi": 1024,
}
for unit in sorted(units, key=len, reverse=True):
if value.endswith(unit):
number = value[:-len(unit)].strip()
return float(number) * units[unit]
raise ValueError(f"Unknown Docker memory unit: {value}")
class MemoryMonitor:
"""Sample API container memory while one upload is running."""
def __init__(self):
self.baseline = {}
self.peak = {}
self.running = False
self.thread = None
def _sample(self):
while self.running:
try:
current = docker_memory()
for name, value in current.items():
self.peak[name] = max(
self.peak.get(name, value),
value,
)
except Exception as exc:
print(
"WARNING: memory sampling failed:",
type(exc).__name__,
)
time.sleep(0.1)
def __enter__(self):
self.baseline = docker_memory()
self.peak = dict(self.baseline)
self.running = True
self.thread = threading.Thread(
target=self._sample,
daemon=True,
)
self.thread.start()
return self
def __exit__(self, exc_type, exc_value, traceback):
self.running = False
if self.thread:
self.thread.join(timeout=3)
try:
current = docker_memory()
for name, value in current.items():
self.peak[name] = max(
self.peak.get(name, value),
value,
)
except Exception:
pass
@property
def peak_rss_mib(self):
return max(self.peak.values())
@property
def peak_delta_mib(self):
return max(
self.peak[name] - self.baseline[name]
for name in API_CONTAINERS
)
def response_json(response):
return json.loads(response["bytes"].decode("utf-8"))
def proxy_upload(client, actor, chat, content):
"""Upload a file through the API proxy endpoint."""
started = time.monotonic()
response = client.call(
"POST",
f"/api/v1/chats/{chat['id']}/attachments?filename=benchmark.png",
201,
actor=actor,
raw=content,
headers={"Content-Type": "image/png"},
)
elapsed = time.monotonic() - started
attachment = response_json(response)
check(attachment["status"] == "ready", "proxy attachment is not ready")
check(
attachment["size_bytes"] == len(content),
"proxy attachment size mismatch",
)
check(
attachment["sha256"] == hashlib.sha256(content).hexdigest(),
"proxy attachment checksum mismatch",
)
return {
"elapsed_total_s": elapsed,
"elapsed_api_s": response["elapsed"],
"api_request_body_bytes": len(content),
"s3_upload_bytes": 0,
}
def presigned_upload(client, actor, chat, content):
"""Upload a file directly to S3 using a presigned URL."""
digest = hashlib.sha256(content).hexdigest()
create_body = {
"filename": "benchmark.png",
"content_type": "image/png",
"size_bytes": len(content),
"sha256": digest,
}
create_body_bytes = len(
json.dumps(create_body, ensure_ascii=False).encode("utf-8")
)
started = time.monotonic()
create_response = client.call(
"POST",
f"/api/v1/chats/{chat['id']}/attachments/presigned",
201,
actor=actor,
body=create_body,
)
created = response_json(create_response)
check(created["method"] == "PUT", "unexpected upload method")
s3_request = urllib.request.Request(
created["upload_url"],
data=content,
headers=created["headers"],
method="PUT",
)
with urllib.request.urlopen(s3_request, timeout=60) as response:
check(
response.status in (200, 201, 204),
"direct S3 upload failed",
)
complete_response = client.call(
"POST",
f"/api/v1/attachments/uploads/{created['upload_id']}/complete",
200,
actor=actor,
)
attachment = response_json(complete_response)
elapsed = time.monotonic() - started
check(attachment["status"] == "ready", "presigned attachment is not ready")
check(
attachment["size_bytes"] == len(content),
"presigned attachment size mismatch",
)
check(
attachment["sha256"] == digest,
"presigned attachment checksum mismatch",
)
return {
"elapsed_total_s": elapsed,
"elapsed_api_s": (
create_response["elapsed"] + complete_response["elapsed"]
),
"api_request_body_bytes": create_body_bytes,
"s3_upload_bytes": len(content),
}
def main():
OUTPUT.parent.mkdir(parents=True, exist_ok=True)
for path in FIXTURES:
if not path.exists():
raise FileNotFoundError(path)
if path.stat().st_size > 10 * 1024 * 1024:
raise RuntimeError(f"Fixture exceeds 10 MiB: {path.name}")
print("Checking API readiness...")
with urllib.request.urlopen(
BASE_URL + "/health/ready",
timeout=10,
) as response:
check(response.status == 200, "API is not ready")
client = Client(BASE_URL)
alice, bob, eve, chat = client.bootstrap()
rows = []
for fixture in FIXTURES:
content = fixture.read_bytes()
size_bytes = len(content)
for method in ("proxy", "presigned"):
for repeat in range(1, REPEATS + 1):
print(
f"Running {method}: {fixture.name}, "
f"repeat {repeat}/{REPEATS}"
)
with MemoryMonitor() as memory:
if method == "proxy":
metrics = proxy_upload(
client,
alice,
chat,
content,
)
else:
metrics = presigned_upload(
client,
alice,
chat,
content,
)
row = {
"method": method,
"file": fixture.name,
"file_size_bytes": size_bytes,
"repeat": repeat,
"elapsed_total_s": round(
metrics["elapsed_total_s"], 4
),
"elapsed_api_s": round(
metrics["elapsed_api_s"], 4
),
"client_to_api_body_bytes": (
metrics["api_request_body_bytes"]
),
"client_to_s3_upload_bytes": (
metrics["s3_upload_bytes"]
),
"api_peak_rss_mib": round(
memory.peak_rss_mib, 2
),
"api_peak_delta_mib": round(
memory.peak_delta_mib, 2
),
}
rows.append(row)
print(
f" total={row['elapsed_total_s']}s, "
f"API={row['elapsed_api_s']}s, "
f"API body={row['client_to_api_body_bytes']} bytes, "
f"peak RAM={row['api_peak_rss_mib']} MiB, "
f"RAM delta={row['api_peak_delta_mib']} MiB"
)
fieldnames = list(rows[0].keys())
with OUTPUT.open("w", newline="", encoding="utf-8-sig") as file:
writer = csv.DictWriter(file, fieldnames=fieldnames)
writer.writeheader()
writer.writerows(rows)
print()
print("PASS: Track 1 benchmark completed")
print(f"Results: {OUTPUT}")
print(f"Measurements: {len(rows)}")
if __name__ == "__main__":
main()
+40
View File
@@ -0,0 +1,40 @@
#!/usr/bin/env python3
"""Concurrent idempotency verification through the public load-balancer URL."""
from concurrent.futures import ThreadPoolExecutor
import uuid
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tests"))
from client import Client, Failure, arguments, check, run
def main():
def configure(p):
p.add_argument('--requests', type=int, default=20)
args = arguments('Concurrent cross-replica idempotency test', configure)
check(2 <= args.requests <= 100, 'requests must be 2..100')
c = Client(args.base_url, args.ca_file, timeout=10)
alice, bob, eve, chat = c.bootstrap()
key = 'race-' + uuid.uuid4().hex
body = 'same concurrent request ' + uuid.uuid4().hex
def send(_):
return Client(args.base_url, args.ca_file, timeout=10).message(alice, chat, body, key=key)
with ThreadPoolExecutor(max_workers=args.requests) as pool:
responses = list(pool.map(send, range(args.requests)))
messages = [r['json'] for r in responses]
check(all(m == messages[0] for m in messages), 'concurrent retries returned different messages')
instances = {r['headers'].get('X-Instance-Id') for r in responses}
check(None not in instances, 'missing X-Instance-Id')
check(len(instances) >= 2, 'load balancer did not expose at least two API replicas')
check(len({m['id'] for m in messages}) == 1, 'concurrent retries created duplicates')
conflict = c.call('POST', '/api/v1/chats/' + chat['id'] + '/messages', 409,
actor=alice, body={'text': body + ' changed'},
headers={'Idempotency-Key': key})
print({'requests': args.requests, 'unique_message_ids': 1,
'observed_instances': len(instances), 'different_body_status': conflict['status']})
print('PASS: concurrent same-key requests produced exactly one message')
if __name__ == '__main__':
run(main)
+116
View File
@@ -0,0 +1,116 @@
#!/usr/bin/env python3
"""Diagnostic closed-loop load test. Does not modify public tests."""
from concurrent.futures import ThreadPoolExecutor
from collections import Counter
from pathlib import Path
import json
import math
import re
import sys
import threading
import time
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / "tests"))
from client import Client, Failure, LAB, arguments, check, run
def main():
def configure(p):
p.add_argument('--duration', type=float, default=90)
p.add_argument('--concurrency', type=int, default=8)
p.add_argument('--output', default='evidence/rolling-update-diagnostic.json')
args = arguments('Diagnostic synthetic POST message load', configure)
check(1 <= args.duration <= 3600, 'duration must be 1..3600 seconds')
check(1 <= args.concurrency <= 128, 'concurrency must be 1..128')
c = Client(args.base_url, args.ca_file)
alice, bob, eve, chat = c.bootstrap()
barrier = threading.Barrier(args.concurrency)
def worker(worker_id):
local = Client(args.base_url, args.ca_file, timeout=5)
barrier.wait()
deadline = time.monotonic() + args.duration
latencies = []
errors = Counter()
examples = []
index = 0
while time.monotonic() < deadline:
start = time.monotonic()
try:
local.message(alice, chat, 'diagnostic-%s-%s' % (worker_id, index))
latencies.append(time.monotonic() - start)
except Failure as exc:
message = str(exc)
match = re.search(r'received (\d{3})', message)
if match:
category = 'http_' + match.group(1)
elif 'connection/TLS failure' in message:
kind = re.search(r'\(([^)]+)\)', message)
category = 'connection_' + (kind.group(1) if kind else 'unknown')
else:
category = 'other_failure'
errors[category] += 1
if len(examples) < 10:
examples.append(message[:300])
index += 1
return latencies, errors, examples
started = time.monotonic()
with ThreadPoolExecutor(max_workers=args.concurrency) as pool:
results = list(pool.map(worker, range(args.concurrency)))
elapsed = time.monotonic() - started
latencies = sorted(x for values, _, _ in results for x in values)
errors = Counter()
examples = []
for _, counts, samples in results:
errors.update(counts)
examples.extend(samples)
total_errors = sum(errors.values())
total = len(latencies) + total_errors
def percentile(p):
if not latencies:
return None
return round(1000 * latencies[max(0, math.ceil(p * len(latencies)) - 1)], 3)
report = {
'lab': LAB,
'workload': 'closed-loop POST messages; no automatic retries',
'duration_seconds': round(elapsed, 3),
'concurrency': args.concurrency,
'successful_requests': len(latencies),
'failed_requests': total_errors,
'error_ratio': round(total_errors / total, 6) if total else None,
'errors_by_type': dict(errors),
'successful_rps': round(len(latencies) / elapsed, 3),
'successful_latency_ms': {
'p50': percentile(.50),
'p95': percentile(.95),
'p99': percentile(.99)
},
'error_examples': examples[:20],
'note': 'Latency percentiles cover successful requests only.'
}
text = json.dumps(report, ensure_ascii=False, indent=2)
dest = Path(args.output)
dest.parent.mkdir(parents=True, exist_ok=True)
dest.write_text(text + '\n', encoding='utf-8')
print(text)
check(bool(latencies), 'no successful writes')
if __name__ == '__main__':
run(main)
+45
View File
@@ -0,0 +1,45 @@
import os
from pathlib import Path
from PIL import Image
ROOT = Path(__file__).resolve().parents[1]
FIXTURES = ROOT / "tests" / "fixtures"
WIDTH = 1600
HEIGHT = 1200
def main():
FIXTURES.mkdir(parents=True, exist_ok=True)
source = FIXTURES / "sample.png"
large = FIXTURES / "sample_large.png"
if not source.exists():
raise FileNotFoundError(
f"Small fixture not found: {source}"
)
pixels = os.urandom(WIDTH * HEIGHT * 3)
image = Image.frombytes(
"RGB",
(WIDTH, HEIGHT),
pixels,
)
image.save(large, format="PNG", optimize=False)
for path in (source, large):
size = path.stat().st_size
print(f"{path.name}: {size} bytes ({size / 1024 / 1024:.2f} MiB)")
if large.stat().st_size > 10 * 1024 * 1024:
raise RuntimeError("Large fixture exceeds the 10 MiB upload limit")
print("PASS: Track 1 fixtures prepared")
if __name__ == "__main__":
main()
+4
View File
@@ -0,0 +1,4 @@
#!/bin/sh
set -eu
docker compose restart postgres redis minio api
+46
View File
@@ -0,0 +1,46 @@
#!/usr/bin/env bash
set -eu
# Compose does not provide a native rolling-update primitive. This script
# performs a surge replacement: add one new healthy replica, then remove one old.
desired="${API_REPLICAS:-2}"
load_duration="${ROLLING_LOAD_DURATION:-90}"
load_script="${LOAD_SCRIPT:-tests/load.py}"
load_output="${ROLLING_LOAD_OUTPUT:-evidence/rolling-update-load.json}"
load_log="${ROLLING_LOAD_LOG:-evidence/rolling-update-load.log}"
if [ "$desired" -lt 2 ]; then echo 'API_REPLICAS must be >= 2' >&2; exit 2; fi
docker compose build api
python "$load_script" --base-url http://localhost:8080 --duration "$load_duration" --concurrency 8 --output "$load_output" >"$load_log" 2>&1 &
load_pid=$!
cleanup() { kill "$load_pid" 2>/dev/null || true; wait "$load_pid" 2>/dev/null || true; }
trap cleanup EXIT INT TERM
# Capture the old replica ids. Each iteration adds a new container before removing an old one.
old_ids=$(docker compose ps -q api)
for old_id in $old_ids; do
before=$(docker compose ps -q api | sort)
docker compose up -d --no-deps --no-recreate --scale "api=$((desired + 1))" api
new_id=''
attempt=0
while [ "$attempt" -lt 60 ]; do
after=$(docker compose ps -q api | sort)
new_id=$(comm -13 <(printf '%s\n' "$before") <(printf '%s\n' "$after") | head -n 1 || true)
if [ -n "$new_id" ]; then
health=$(docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' "$new_id")
if [ "$health" = healthy ]; then break; fi
fi
attempt=$((attempt + 1)); sleep 2
done
if [ -z "$new_id" ] || [ "${health:-}" != healthy ]; then
echo 'New replica did not become healthy; leaving old replicas running' >&2
exit 1
fi
docker exec "$old_id" sh -c 'touch /tmp/api-draining'
sleep 10
docker stop --timeout 20 "$old_id" >/dev/null
docker rm "$old_id" >/dev/null
docker compose up -d --no-deps --no-recreate --scale "api=$desired" api
python scripts/wait-ready.py http://localhost:8080/health/ready 30
done
wait "$load_pid" || { echo 'Load test failed; inspect evidence/rolling-update-load.log' >&2; exit 1; }
trap - EXIT INT TERM
+131
View File
@@ -0,0 +1,131 @@
#!/usr/bin/env python3
"""Run Track 1 scenarios on the user's Docker host; records only measured results."""
import argparse
import csv
import json
from pathlib import Path
import subprocess
import sys
import time
ROOT = Path(__file__).resolve().parents[1]
OUT = ROOT / 'evidence' / 'scaling'
BASE_ENV = ROOT / '.env'
def run(cmd, env=None, check=True, capture=False):
return subprocess.run(cmd, cwd=ROOT, env=env, check=check,
text=True, stdout=subprocess.PIPE if capture else None,
stderr=subprocess.STDOUT if capture else None)
def stats():
p = run(['docker', 'stats', '--no-stream', '--format', '{{.Name}};{{.CPUPerc}};{{.MemUsage}}'], capture=True)
return p.stdout.strip().splitlines()
def env_value(source, key, default):
prefix = key + '='
for line in source.splitlines():
if line.startswith(prefix):
return line[len(prefix):].strip().strip('\"').strip("'")
return default
def db_activity(env_path, db_user, db_name):
query = "SELECT count(*) FILTER (WHERE state='active'), count(*), current_setting('max_connections') FROM pg_stat_activity"
result = run(['docker', 'compose', '--env-file', str(env_path), 'exec', '-T', 'postgres',
'psql', '-U', db_user, '-d', db_name, '-At', '-F', ',', '-c', query],
capture=True, check=False)
return result.stdout.strip() if result.returncode == 0 else 'unavailable'
def scenario(name, replicas, total_cpu, total_mem, duration, concurrency):
source = BASE_ENV.read_text(encoding='utf-8')
db_user = env_value(source, 'POSTGRES_USER', 'messenger')
db_name = env_value(source, 'POSTGRES_DB', 'messenger')
api_cpu = total_cpu / replicas
api_mem_mb = total_mem // replicas
env_text = source.rstrip() + f'\nAPI_CPU_LIMIT={api_cpu:.3f}\nAPI_MEMORY_LIMIT={api_mem_mb}M\n'
env_path = OUT / ('.env-' + name)
env_path.write_text(env_text, encoding='utf-8')
compose = ['docker', 'compose', '--env-file', str(env_path), 'up', '-d', '--build', '--scale', f'api={replicas}']
run(compose)
deadline = time.time() + 120
while time.time() < deadline:
result = run(['docker', 'compose', '--env-file', str(env_path), 'ps', '-q', 'api'], capture=True, check=False)
ids = [line.strip() for line in result.stdout.splitlines() if line.strip()]
healthy = 0
for container_id in ids:
inspected = run(['docker', 'inspect', '--format', '{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}', container_id], capture=True, check=False)
if inspected.returncode == 0 and inspected.stdout.strip() == 'healthy':
healthy += 1
if len(ids) >= replicas and healthy >= replicas:
break
time.sleep(2)
else:
raise RuntimeError(f'{name}: API replicas did not become healthy')
output = OUT / (name + '.json')
cmd = [sys.executable, 'tests/load.py', '--base-url', 'http://localhost:8080',
'--duration', str(duration), '--concurrency', str(concurrency), '--output', str(output)]
proc = subprocess.Popen(cmd, cwd=ROOT, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
samples = []
while proc.poll() is None:
timestamp = time.time()
try:
samples.append({'timestamp_epoch': timestamp, 'containers': stats(),
'postgres_active_total_max_connections': db_activity(env_path, db_user, db_name)})
except Exception as exc:
samples.append({'timestamp_epoch': timestamp, 'stats_error': type(exc).__name__})
time.sleep(1)
stdout = proc.stdout.read() if proc.stdout else ''
if proc.returncode != 0:
raise RuntimeError(f'load failed for {name}: {stdout}')
metrics = json.loads(output.read_text(encoding='utf-8'))
metrics.update({'scenario': name, 'api_replicas': replicas,
'total_api_cpu_budget': total_cpu,
'total_api_memory_budget_mb': total_mem,
'per_replica_cpu_limit': api_cpu,
'per_replica_memory_limit_mb': api_mem_mb,
'container_stats_samples': samples})
output.write_text(json.dumps(metrics, ensure_ascii=False, indent=2) + '\n', encoding='utf-8')
run(['docker', 'compose', '--env-file', str(env_path), 'down'])
return metrics
def main():
p = argparse.ArgumentParser(description=__doc__)
p.add_argument('--duration', type=int, default=30)
p.add_argument('--concurrency', type=int, default=8)
p.add_argument('--fixed-cpu', type=float, default=2.0)
p.add_argument('--fixed-memory-mb', type=int, default=2048)
p.add_argument('--expanded-cpu', type=float, nargs=3, default=[2.0, 4.0, 8.0])
p.add_argument('--expanded-memory-mb', type=int, nargs=3, default=[2048, 4096, 8192])
args = p.parse_args()
if not BASE_ENV.exists():
p.error('Create .env from .env.example before running')
OUT.mkdir(parents=True, exist_ok=True)
results = []
for n in (1, 2, 4):
results.append(scenario(f'fixed_{n}replicas', n, args.fixed_cpu,
args.fixed_memory_mb, args.duration, args.concurrency))
for cpu, mem in zip(args.expanded_cpu, args.expanded_memory_mb):
results.append(scenario(f'expanded_4replicas_{cpu:g}cpu_{mem}mb', 4, cpu,
mem, args.duration, args.concurrency))
with (OUT / 'summary.csv').open('w', newline='', encoding='utf-8') as f:
fields = ['scenario', 'api_replicas', 'total_api_cpu_budget', 'total_api_memory_budget_mb',
'successful_rps', 'failed_requests', 'error_ratio', 'p50_ms', 'p95_ms', 'p99_ms']
writer = csv.DictWriter(f, fieldnames=fields)
writer.writeheader()
for x in results:
lat = x['successful_latency_ms']
writer.writerow({'scenario': x['scenario'], 'api_replicas': x['api_replicas'],
'total_api_cpu_budget': x['total_api_cpu_budget'],
'total_api_memory_budget_mb': x['total_api_memory_budget_mb'],
'successful_rps': x['successful_rps'], 'failed_requests': x['failed_requests'],
'error_ratio': x['error_ratio'], 'p50_ms': lat['p50'], 'p95_ms': lat['p95'], 'p99_ms': lat['p99']})
print(f'Wrote measured results to {OUT}')
if __name__ == '__main__':
main()
+23
View File
@@ -0,0 +1,23 @@
#!/bin/sh
set -eu
: "${TARGET_INSTANCE_ID:?TARGET_INSTANCE_ID is required}"
project="${COMPOSE_PROJECT_NAME:-mtusi-lab4}"
found=""
for id in $(docker ps -q --filter "label=com.docker.compose.project=${project}" --filter "label=com.docker.compose.service=api"); do
hostname=$(docker inspect --format '{{.Config.Hostname}}' "$id")
if [ "$hostname" = "$TARGET_INSTANCE_ID" ]; then
found="$id"
break
fi
done
if [ -z "$found" ]; then
echo "No running API container matches TARGET_INSTANCE_ID" >&2
exit 1
fi
docker stop --time 3 "$found" >/dev/null
+101
View File
@@ -0,0 +1,101 @@
import hashlib
import sys
import time
import uuid
from datetime import datetime, timedelta, timezone
from pathlib import Path
from botocore.exceptions import ClientError
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT))
sys.path.insert(0, str(ROOT / "tests"))
from client import Client
from app.core.config import settings
from app.db.models import AttachmentUpload
from app.db.session import SessionLocal
from app.services.object_storage import client as s3_client
BASE_URL = "http://localhost:8080"
FIXTURE = ROOT / "tests" / "fixtures" / "sample.png"
def main():
content = FIXTURE.read_bytes()
digest = hashlib.sha256(content).hexdigest()
api = Client(BASE_URL)
alice, _, _, chat = api.bootstrap()
upload_id = uuid.uuid4()
object_key = f"cleanup-test/{chat['id']}/{upload_id}"
# Host-side test uses the published S3 endpoint.
s3 = s3_client()
s3.put_object(
Bucket=settings.s3_bucket,
Key=object_key,
Body=content,
ContentType="image/png",
)
with SessionLocal() as db:
upload = AttachmentUpload(
id=upload_id,
chat_id=uuid.UUID(chat["id"]),
owner_id=uuid.UUID(alice["id"]),
filename="cleanup-test.png",
content_type="image/png",
expected_size_bytes=len(content),
expected_sha256=digest,
object_key=object_key,
status="pending",
# Older than expiry + 10-minute grace period.
expires_at=datetime.now(timezone.utc) - timedelta(minutes=11),
)
db.add(upload)
db.commit()
print(f"Created synthetic pending upload: {upload_id}")
print("Waiting for background cleanup (up to 90 seconds)...")
deadline = time.monotonic() + 90
final_status = None
while time.monotonic() < deadline:
with SessionLocal() as db:
row = db.get(AttachmentUpload, upload_id)
final_status = row.status if row else None
if final_status == "expired":
break
time.sleep(2)
if final_status != "expired":
raise SystemExit(
f"FAIL: expected status expired, got {final_status!r}"
)
try:
s3.head_object(Bucket=settings.s3_bucket, Key=object_key)
except ClientError as exc:
code = exc.response.get("Error", {}).get("Code", "")
status = exc.response.get("ResponseMetadata", {}).get("HTTPStatusCode")
if code not in ("404", "NoSuchKey", "NotFound") and status != 404:
raise
else:
raise SystemExit("FAIL: S3 object still exists")
print("PASS: abandoned upload marked expired")
print("PASS: corresponding S3 object deleted")
print(f"Upload ID: {upload_id}")
print(f"Object size: {len(content)} bytes")
print(f"SHA-256: {digest}")
if __name__ == "__main__":
main()
+285
View File
@@ -0,0 +1,285 @@
import uuid
import hashlib
import json
import sys
import urllib.request
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / "tests"))
from client import Client, check
BASE_URL = "http://localhost:8080"
IMAGE = ROOT / "tests" / "fixtures" / "sample.png"
def response_json(response):
return json.loads(response["bytes"].decode("utf-8"))
def create_upload(client, actor, chat, content, filename="sample.png"):
"""Create a presigned upload and return its metadata."""
response = client.call(
"POST",
f"/api/v1/chats/{chat['id']}/attachments/presigned",
201,
actor=actor,
body={
"filename": filename,
"content_type": "image/png",
"size_bytes": len(content),
"sha256": hashlib.sha256(content).hexdigest(),
},
)
return response_json(response)
def create_upload_with_metadata(
client,
actor,
chat,
content,
expected_size=None,
expected_sha256=None,
):
"""Create an upload with deliberately supplied expected metadata."""
response = client.call(
"POST",
f"/api/v1/chats/{chat['id']}/attachments/presigned",
201,
actor=actor,
body={
"filename": "sample.png",
"content_type": "image/png",
"size_bytes": (
len(content) if expected_size is None else expected_size
),
"sha256": (
hashlib.sha256(content).hexdigest()
if expected_sha256 is None
else expected_sha256
),
},
)
return response_json(response)
def put_to_s3(upload, content):
"""Upload bytes directly to S3 using the presigned URL."""
request = urllib.request.Request(
upload["upload_url"],
data=content,
headers=upload["headers"],
method="PUT",
)
with urllib.request.urlopen(request, timeout=30) as response:
check(
response.status in (200, 201, 204),
"S3 upload failed",
)
def complete_upload(client, actor, upload_id, expected_status=200):
return client.call(
"POST",
f"/api/v1/attachments/uploads/{upload_id}/complete",
expected_status,
actor=actor,
)
def main():
content = IMAGE.read_bytes()
digest = hashlib.sha256(content).hexdigest()
client = Client(BASE_URL)
alice, bob, eve, chat = client.bootstrap()
# 1. Incorrect SHA-256 must be rejected.
wrong_digest = "0" * 64
check(wrong_digest != digest, "test checksum unexpectedly matches")
upload = create_upload_with_metadata(
client,
alice,
chat,
content,
expected_sha256=wrong_digest,
)
put_to_s3(upload, content)
complete_upload(
client,
alice,
upload["upload_id"],
expected_status=409,
)
print("PASS: incorrect SHA-256 rejected with HTTP 409")
# 2. Incorrect expected size must be rejected.
upload = create_upload_with_metadata(
client,
alice,
chat,
content,
expected_size=len(content) + 1,
)
put_to_s3(upload, content)
complete_upload(
client,
alice,
upload["upload_id"],
expected_status=409,
)
print("PASS: incorrect file size rejected with HTTP 409")
# 3. Corrupted image must be rejected.
corrupt_content = b"This is not a valid PNG image"
upload = create_upload(
client,
alice,
chat,
corrupt_content,
)
put_to_s3(upload, corrupt_content)
complete_upload(
client,
alice,
upload["upload_id"],
expected_status=415,
)
print("PASS: corrupted image rejected with HTTP 415")
# 4. Another user must not complete Alice's upload.
upload = create_upload(
client,
alice,
chat,
content,
)
put_to_s3(upload, content)
complete_upload(
client,
eve,
upload["upload_id"],
expected_status=404,
)
print("PASS: foreign user cannot complete another user's upload")
# The legitimate owner must still be able to complete it.
owner_response = complete_upload(
client,
alice,
upload["upload_id"],
expected_status=200,
)
attachment = response_json(owner_response)
check(
attachment["status"] == "ready",
"owner's attachment is not ready",
)
check(
attachment["owner_id"] == alice["id"],
"attachment owner does not match",
)
check(
attachment["sha256"] == digest,
"attachment checksum mismatch",
)
print("PASS: legitimate owner can complete the upload")
# 5. Proxy upload must reject files larger than 10 MiB.
oversized_content = b"\x00" * (10 * 1024 * 1024 + 1)
client.call(
"POST",
f"/api/v1/chats/{chat['id']}/attachments?filename=oversized.png",
expected=413,
actor=alice,
raw=oversized_content,
headers={"Content-Type": "image/png"},
)
print("PASS: oversized proxy upload rejected with HTTP 413")
# 6. PNG content must not be accepted as JPEG.
client.call(
"POST",
f"/api/v1/chats/{chat['id']}/attachments?filename=mismatch.jpg",
expected=415,
actor=alice,
raw=content,
headers={"Content-Type": "image/jpeg"},
)
print("PASS: MIME mismatch rejected with HTTP 415")
# 7. Create a ready attachment in Alice and Bob's chat.
attachment, _ = client.attachment(alice, chat)
attachment_id = attachment["id"]
# Eve is not a member of Alice and Bob's chat.
client.call(
"GET",
f"/api/v1/attachments/{attachment_id}",
expected=404,
actor=eve,
)
print("PASS: non-member cannot access attachment metadata")
# 8. A non-member must not download the attachment.
client.call(
"GET",
f"/api/v1/attachments/{attachment_id}/content",
expected=404,
actor=eve,
)
print("PASS: non-member cannot download attachment")
# 9. Create another chat where Alice and Eve are members.
second_chat = client.call(
"POST",
"/api/v1/chats",
201,
actor=alice,
body={
"title": "Cross-chat attachment test",
"member_ids": [alice["id"], eve["id"]],
},
schema="Chat",
)["json"]
# Try to attach a file from the first chat to a message in the second.
client.call(
"POST",
f"/api/v1/chats/{second_chat['id']}/messages",
expected=404,
actor=alice,
body={
"text": "Cross-chat attachment test",
"attachment_ids": [attachment_id],
},
headers={"Idempotency-Key": uuid.uuid4().hex},
)
print("PASS: attachment from another chat rejected with HTTP 404")
print()
print("PASS: all presigned negative scenarios")
if __name__ == "__main__":
main()
+123
View File
@@ -0,0 +1,123 @@
import hashlib
import json
import sys
import urllib.request
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / "tests"))
from client import Client, check
BASE_URL = "http://localhost:8080"
IMAGE = ROOT / "tests" / "fixtures" / "sample.png"
def response_json(response):
"""Decode JSON from the raw response bytes."""
return json.loads(response["bytes"].decode("utf-8"))
def main():
content = IMAGE.read_bytes()
digest = hashlib.sha256(content).hexdigest()
client = Client(BASE_URL)
alice, bob, eve, chat = client.bootstrap()
# 1. Request a short-lived upload URL.
create_response = client.call(
"POST",
f"/api/v1/chats/{chat['id']}/attachments/presigned",
201,
actor=alice,
body={
"filename": "sample.png",
"content_type": "image/png",
"size_bytes": len(content),
"sha256": digest,
},
)
created = response_json(create_response)
check(created["method"] == "PUT", "unexpected upload method")
check(bool(created["upload_id"]), "missing upload_id")
check(created["expires_in"] > 0, "invalid upload expiration")
# 2. Upload directly to S3, without sending the file through the API.
request = urllib.request.Request(
created["upload_url"],
data=content,
headers=created["headers"],
method="PUT",
)
with urllib.request.urlopen(request, timeout=30) as response:
check(
response.status in (200, 201, 204),
"S3 upload failed",
)
# 3. Complete the upload.
complete_path = (
f"/api/v1/attachments/uploads/{created['upload_id']}/complete"
)
complete_response = client.call(
"POST",
complete_path,
200,
actor=alice,
)
attachment = response_json(complete_response)
check(attachment["status"] == "ready", "attachment is not ready")
check(attachment["chat_id"] == chat["id"], "wrong attachment chat")
check(attachment["owner_id"] == alice["id"], "wrong attachment owner")
check(attachment["sha256"] == digest, "attachment checksum mismatch")
check(
attachment["size_bytes"] == len(content),
"attachment size mismatch",
)
# 4. Complete again: must return the same attachment.
repeated_response = client.call(
"POST",
complete_path,
200,
actor=alice,
)
repeated = response_json(repeated_response)
check(
repeated["id"] == attachment["id"],
"repeated completion created a duplicate",
)
# 5. Download through the authorized API endpoint.
downloaded = client.call(
"GET",
f"/api/v1/attachments/{attachment['id']}/content",
200,
actor=alice,
)["bytes"]
check(downloaded == content, "downloaded bytes differ from original")
check(
hashlib.sha256(downloaded).hexdigest() == digest,
"downloaded checksum mismatch",
)
print("PASS: presigned upload, completion, idempotency and download")
print(f"Upload ID: {created['upload_id']}")
print(f"Attachment ID: {attachment['id']}")
print(f"File size: {len(content)} bytes")
print(f"SHA-256: {digest}")
if __name__ == "__main__":
main()
+127
View File
@@ -0,0 +1,127 @@
import sys
from pathlib import Path
from sqlalchemy import text
ROOT = Path("/app")
sys.path.insert(0, str(ROOT))
sys.path.insert(0, str(ROOT / "tests"))
from client import Client, check
from app.core.config import settings
from app.db.models import Attachment
from app.db.session import SessionLocal
from app.services.object_storage import client as s3_client
TRIGGER_NAME = "lab4_force_attachment_insert_failure"
FUNCTION_NAME = "lab4_fail_attachment_insert"
TEST_FILENAME = "lab4-forced-db-failure.png"
def install_failure_trigger():
table = Attachment.__table__
schema = table.schema or "public"
qualified_table = f'"{schema}"."{table.name}"'
with SessionLocal() as db:
db.execute(text(f"""
CREATE OR REPLACE FUNCTION public.{FUNCTION_NAME}()
RETURNS trigger
LANGUAGE plpgsql
AS $$
BEGIN
IF NEW.filename = '{TEST_FILENAME}' THEN
RAISE EXCEPTION 'Lab 4 controlled metadata failure'
USING ERRCODE = 'P0001';
END IF;
RETURN NEW;
END;
$$;
"""))
db.execute(text(
f'DROP TRIGGER IF EXISTS {TRIGGER_NAME} ON {qualified_table}'
))
db.execute(text(
f'CREATE TRIGGER {TRIGGER_NAME} '
f'BEFORE INSERT ON {qualified_table} '
f'FOR EACH ROW EXECUTE FUNCTION public.{FUNCTION_NAME}()'
))
db.commit()
def remove_failure_trigger():
table = Attachment.__table__
schema = table.schema or "public"
qualified_table = f'"{schema}"."{table.name}"'
with SessionLocal() as db:
db.execute(text(
f'DROP TRIGGER IF EXISTS {TRIGGER_NAME} ON {qualified_table}'
))
db.execute(text(
f'DROP FUNCTION IF EXISTS public.{FUNCTION_NAME}()'
))
db.commit()
def object_keys(s3, prefix):
result = s3.list_objects_v2(
Bucket=settings.s3_bucket,
Prefix=prefix,
)
return {
item["Key"]
for item in result.get("Contents", [])
}
def main():
client = Client("http://127.0.0.1:8080")
alice, bob, eve, chat = client.bootstrap()
content = (ROOT / "tests" / "fixtures" / "sample.png").read_bytes()
prefix = f"original/{chat['id']}/"
s3 = s3_client()
before = object_keys(s3, prefix)
install_failure_trigger()
print("Temporary PostgreSQL failure trigger installed.")
try:
response = client.call(
"POST",
f"/api/v1/chats/{chat['id']}/attachments"
f"?filename={TEST_FILENAME}",
expected=503,
actor=alice,
raw=content,
headers={"Content-Type": "image/png"},
)
check(
response["status"] == 503,
"Expected HTTP 503 after forced metadata failure",
)
print("PASS: S3 upload succeeded, PostgreSQL metadata insert failed")
after = object_keys(s3, prefix)
unexpected_objects = after - before
check(
not unexpected_objects,
"Compensation failed: unexpected S3 objects remain",
)
print("PASS: uploaded S3 object was deleted after DB failure")
print()
print("PASS: S3/PostgreSQL compensation scenario")
finally:
remove_failure_trigger()
print("Temporary PostgreSQL trigger removed.")
if __name__ == "__main__":
main()
+16
View File
@@ -0,0 +1,16 @@
#!/usr/bin/env python3
import sys
import time
import urllib.request
url = sys.argv[1] if len(sys.argv) > 1 else 'http://localhost:8080/health/ready'
timeout = int(sys.argv[2]) if len(sys.argv) > 2 else 30
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
try:
with urllib.request.urlopen(url, timeout=2) as response:
if response.status == 200:
raise SystemExit(0)
except Exception:
time.sleep(0.5)
raise SystemExit('Readiness timeout: ' + url)
+20 -6
View File
@@ -182,13 +182,27 @@ class Client:
headers = {'Idempotency-Key':key or uuid.uuid4().hex} if LAB>=3 else {}
return self.call('POST','/api/v1/chats/'+chat['id']+'/messages',201,actor=actor,body=body,headers=headers,schema='Message')
def attachment(self, actor, chat):
def attachment(self, actor, chat, include_instance=False):
content = (ROOT / 'tests/fixtures/sample.png').read_bytes()
att = self.call('POST','/api/v1/chats/'+chat['id']+'/attachments?filename=sample.png',201 if LAB==4 else 202,actor=actor,raw=content,headers={'Content-Type':'image/png'},schema='Attachment')['json']
check(att['status']==('ready' if LAB==4 else 'queued'),'incorrect upload status')
check(att['chat_id']==chat['id'] and att['owner_id']==actor['id'],'attachment ownership mismatch')
check(att['sha256']==hashlib.sha256(content).hexdigest() and att['size_bytes']==len(content),'original checksum/size mismatch')
return att,content
response = self.call(
'POST',
'/api/v1/chats/' + chat['id'] + '/attachments?filename=sample.png',
201 if LAB == 4 else 202,
actor=actor,
raw=content,
headers={'Content-Type': 'image/png'},
schema='Attachment'
)
att = response['json']
check(att['status'] == ('ready' if LAB == 4 else 'queued'), 'incorrect upload status')
check(att['chat_id'] == chat['id'] and att['owner_id'] == actor['id'], 'attachment ownership mismatch')
check(
att['sha256'] == hashlib.sha256(content).hexdigest()
and att['size_bytes'] == len(content),
'original checksum/size mismatch'
)
instance_id = response['headers'].get('X-Instance-Id', '')
return (att, content, instance_id) if include_instance else (att, content)
def wait_ready(self, actor, att, wait_seconds=60):
deadline = time.monotonic()+wait_seconds
BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 5.5 MiB

+15 -5
View File
@@ -40,12 +40,22 @@ def main():
alice,bob,eve,chat = c.bootstrap()
sent = c.message(alice,chat,'Сохранить при отказе')
original = sent['json']
target = sent['headers'].get('X-Instance-Id','')
target = sent['headers'].get('X-Instance-Id', '')
attachment = content = None
if LAB>=4:
attachment,content = c.attachment(alice,chat)
if LAB>=5:
attachment = c.wait_ready(alice,attachment)
if LAB >= 4:
attachment, content, upload_instance = c.attachment(alice, chat, include_instance=True)
if args.mode == 'failover':
check(bool(upload_instance), 'attachment upload response has no X-Instance-Id')
target = upload_instance
if LAB >= 5:
attachment = c.wait_ready(alice, attachment)
if args.mode == 'failover':
check(bool(target), 'target API instance ID is empty')
env = dict(os.environ)
env['TARGET_INSTANCE_ID'] = target
observed = set()