Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d96d4a2836 | ||
|
|
aaa834e412 |
@@ -0,0 +1,24 @@
|
||||
# Local Docker Compose configuration for Lab 4.
|
||||
# Copy this file to .env and replace every CHANGE_ME value locally.
|
||||
# Never commit .env or share its contents.
|
||||
POSTGRES_DB=messenger
|
||||
POSTGRES_USER=messenger
|
||||
POSTGRES_PASSWORD=CHANGE_ME_POSTGRES_PASSWORD
|
||||
MINIO_ROOT_USER=CHANGE_ME_S3_ADMIN_USER
|
||||
MINIO_ROOT_PASSWORD=CHANGE_ME_S3_ADMIN_PASSWORD
|
||||
S3_ACCESS_KEY_ID=CHANGE_ME_APP_ACCESS_KEY
|
||||
S3_SECRET_ACCESS_KEY=CHANGE_ME_APP_SECRET_KEY
|
||||
S3_REGION=us-east-1
|
||||
S3_BUCKET=messenger-attachments
|
||||
S3_PUBLIC_ENDPOINT_URL=http://localhost:9000
|
||||
SESSION_TTL_SECONDS=3600
|
||||
MESSAGE_CACHE_TTL_SECONDS=60
|
||||
IDEMPOTENCY_TTL_SECONDS=86400
|
||||
DB_POOL_SIZE=3
|
||||
DB_MAX_OVERFLOW=2
|
||||
DB_POOL_TIMEOUT=10
|
||||
MAX_UPLOAD_BYTES=10485760
|
||||
MAX_IMAGE_PIXELS=40000000
|
||||
ATTACHMENT_CLEANUP_INTERVAL_SECONDS=60
|
||||
ATTACHMENT_CLEANUP_GRACE_SECONDS=600
|
||||
ATTACHMENT_CLEANUP_BATCH_SIZE=100
|
||||
@@ -1,18 +1,48 @@
|
||||
## Лабораторная 4
|
||||
## Лабораторная 4 — S3-хранилище и вложения
|
||||
|
||||
Автор / группа: TODO
|
||||
Целевая оценка (3/4/5): TODO
|
||||
Трек 5, если заявлен: TODO
|
||||
**Автор / группа:** Перевалова Виктория, БПИ2302
|
||||
|
||||
Что реализовано и что перенесено: TODO
|
||||
Команды запуска из чистого clone: TODO
|
||||
Результаты проверок и ссылка на REPORT.md: TODO
|
||||
Ограничения / известные ошибки: TODO
|
||||
Источники и использование ИИ: TODO
|
||||
**Целевая оценка:** 5
|
||||
|
||||
- [ ] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
|
||||
- [ ] Контракт и публичные тесты не ослаблены.
|
||||
- [ ] make check и make test выполнены; приложен реальный вывод.
|
||||
- [ ] Сценарии отказов/безопасности своего уровня воспроизведены.
|
||||
- [ ] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
|
||||
- [ ] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
|
||||
**Трек:** Track 1 — Presigned Upload
|
||||
|
||||
### Что реализовано
|
||||
|
||||
- Перенесена основа предыдущей лабораторной; сохранены публичные API-сценарии и контракт.
|
||||
- Реализованы Proxy Upload и Presigned Upload с проверкой членства в чате, уникальными ключами, TTL, проверкой размера, SHA-256 и фактического формата изображения при complete.
|
||||
- Метаданные хранятся в PostgreSQL, бинарные данные — в приватном S3-compatible bucket.
|
||||
- Добавлены cleanup незавершённых presigned-загрузок, best-effort компенсация при ошибке записи метаданных, ограниченные S3 credentials и bootstrap bucket.
|
||||
- Стенд включает PostgreSQL, Redis, S3-compatible storage, миграции, Traefik и две API-реплики.
|
||||
|
||||
### Запуск
|
||||
|
||||
См. [`REPORT.md`](../REPORT.md), раздел «Запуск из чистого clone». Для локального Compose используется `.env.compose.example`; после копирования в `.env` необходимо заменить демонстрационные значения.
|
||||
|
||||
### Проверки и доказательства
|
||||
|
||||
- `scripts/check_contract.py` — PASS.
|
||||
- Python compileall — PASS.
|
||||
- Публичный smoke — PASS: повторно выполнен после исправления маршрутизации Traefik; результат сохранён в `evidence/smoke.txt`.
|
||||
- Presigned upload, completion, idempotency и download — PASS.
|
||||
- Все негативные сценарии Presigned Upload — PASS (409/415/413/404 согласно проверяемому сценарию).
|
||||
- Негативные сценарии, cleanup, компенсация S3/PostgreSQL, S3 unavailable/security, persistence, attachment failover и нагрузочные эксперименты описаны в `REPORT.md`; сводки находятся в `evidence/`.
|
||||
- Эксперимент Track 1 подтвердил сокращение тела запроса к API до 159 байт для файла 5,5 MiB; снижение RAM не подтверждено и не заявляется.
|
||||
|
||||
### Ограничения и отклонения
|
||||
|
||||
- Между PostgreSQL и S3 нет общей распределённой транзакции; компенсация Proxy Upload best-effort. Незарегистрированные orphan-объекты требуют отдельной сверки/lifecycle policy.
|
||||
- Пиковая RAM измерялась через периодический `docker stats`; короткие пики могли быть пропущены.
|
||||
- В Compose используется Silo с S3 API, хотя задание буквально перечисляет MinIO, SeaweedFS или RustFS. Это отклонение явно раскрыто в `REPORT.md`; требуется согласование с преподавателем.
|
||||
|
||||
### Источники и использование ИИ
|
||||
|
||||
Использованы исходный skeleton, контракт, документация зависимостей и материалы лабораторной. ChatGPT применялся для анализа требований, диагностики, подготовки отдельных тестов и редактирования документации. Результаты проверялись локальным запуском и тестами; ИИ не заменял выполнение проверок.
|
||||
|
||||
### Checklist
|
||||
|
||||
- [x] Контракт и публичные tests не ослаблялись ради прохождения.
|
||||
- [x] Сценарии отказов и безопасности уровня 4/5 воспроизведены; результаты приложены в `evidence/`.
|
||||
- [x] Локальные секреты исключены из Git через `.gitignore`; `.env` не должен добавляться в PR.
|
||||
- [x] Исходный README задания сохранён, инструкция конкретной реализации находится в `REPORT.md`.
|
||||
- [ ] Чистый запуск из clone и повторный `make test` после последних изменений подтверждены.
|
||||
- [ ] `make validate` выполнен с `requirements-dev.txt`; приложен вывод.
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
.env
|
||||
.env.*
|
||||
!.env.example
|
||||
!.env.compose.example
|
||||
secrets/
|
||||
certs/private/
|
||||
*.key
|
||||
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
FROM python:3.11.9-slim-bookworm
|
||||
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
PIP_NO_CACHE_DIR=1
|
||||
|
||||
WORKDIR /app
|
||||
RUN groupadd --system app && useradd --system --gid app --home-dir /app app
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
COPY app ./app
|
||||
COPY migrations ./migrations
|
||||
COPY scripts ./scripts
|
||||
COPY tests ./tests
|
||||
COPY contracts ./contracts
|
||||
COPY lab.json .
|
||||
COPY alembic.ini .
|
||||
RUN chown -R app:app /app
|
||||
USER app
|
||||
EXPOSE 8080
|
||||
HEALTHCHECK --interval=5s --timeout=3s --start-period=20s --retries=3 \
|
||||
CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=2)"
|
||||
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8080", "--timeout-keep-alive", "5", "--timeout-graceful-shutdown", "20"]
|
||||
@@ -1,37 +1,492 @@
|
||||
# Отчёт — лабораторная 4
|
||||
# Отчёт — лабораторная работа №4
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
- **Автор, группа:** Перевалова Виктория, БПИ2302.
|
||||
- **Целевая оценка:** 5.
|
||||
- **Выбранный трек:** Track 1 — Presigned Upload (прямая загрузка в S3).
|
||||
- **Основная гипотеза:** прямая загрузка файлов в S3-compatible хранилище по временной подписанной ссылке позволяет уменьшить объём данных, проходящих через API, и сократить время обработки больших файлов на стороне API.
|
||||
- **Commit/PR:** будет указан после итогового commit и публикации изменений в Gitea.
|
||||
- **Использованные источники, библиотеки и помощь ИИ:** исходный репозиторий лабораторной работы и материалы преподавателя; Python 3, FastAPI, SQLAlchemy, Alembic, PostgreSQL, Redis, boto3, Pillow, Docker Compose, S3-compatible object storage. ChatGPT использовался для анализа требований, проектирования отдельных компонентов, подготовки тестов, диагностики ошибок и оформления отчёта. Реализация и результаты проверялись запуском приложения и тестовых сценариев.
|
||||
|
||||
---
|
||||
|
||||
## Запуск из чистого clone
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
### 1. Требования к окружению
|
||||
|
||||
Для запуска проекта необходимы:
|
||||
|
||||
- Git;
|
||||
- Docker Desktop с поддержкой Docker Compose;
|
||||
- Python 3.13 для локального запуска вспомогательных тестовых скриптов;
|
||||
- PowerShell (для Windows) или совместимая командная оболочка;
|
||||
- доступ к Gitea и возможность клонировать репозиторий.
|
||||
|
||||
Основные сервисы запускаются в Docker Compose:
|
||||
|
||||
| Компонент | Используемая версия |
|
||||
|---|---|
|
||||
| PostgreSQL | 16.6-bookworm |
|
||||
| Redis | 7.4.2-bookworm |
|
||||
| S3-compatible storage (Silo) | RELEASE.2026-10-26T00-00-00Z |
|
||||
|
||||
| Traefik | 3.6.2 |
|
||||
| API | Docker image проекта |
|
||||
|
||||
### 2. Клонирование проекта
|
||||
|
||||
```powershell
|
||||
git clone https://gitea.privcloud.tech/perevalova/mtusi2026lab4.git
|
||||
cd mtusi2026lab4
|
||||
```
|
||||
|
||||
Для перехода на ветку с реализацией:
|
||||
|
||||
```powershell
|
||||
git checkout solution/lab4-implementation
|
||||
```
|
||||
|
||||
### 3. Подготовка конфигурации
|
||||
|
||||
Создать локальный файл `.env` на основе предоставленного примера:
|
||||
|
||||
```powershell
|
||||
Copy-Item .env.compose.example .env
|
||||
```
|
||||
|
||||
Перед запуском необходимо проверить соответствие переменных конфигурации требованиям `compose.yaml`.
|
||||
|
||||
Для работы инфраструктуры необходимо задать следующие переменные:
|
||||
|
||||
- `POSTGRES_PASSWORD` — пароль PostgreSQL;
|
||||
- `MINIO_ROOT_USER` — административный пользователь S3-хранилища;
|
||||
- `MINIO_ROOT_PASSWORD` — административный пароль S3-хранилища;
|
||||
- `S3_ACCESS_KEY_ID` — отдельный пользователь приложения для доступа к S3;
|
||||
- `S3_SECRET_ACCESS_KEY` — секретный ключ пользователя приложения;
|
||||
- `S3_REGION` — регион S3;
|
||||
- `S3_BUCKET` — имя bucket.
|
||||
|
||||
Дополнительные параметры ограничений загрузки и фоновой очистки имеют значения по умолчанию в Compose:
|
||||
|
||||
- `MAX_UPLOAD_BYTES` — 10 MiB;
|
||||
- `MAX_IMAGE_PIXELS` — 40 000 000;
|
||||
- `ATTACHMENT_CLEANUP_INTERVAL_SECONDS` — 60 секунд;
|
||||
- `ATTACHMENT_CLEANUP_GRACE_SECONDS` — 600 секунд;
|
||||
- `ATTACHMENT_CLEANUP_BATCH_SIZE` — 100 записей.
|
||||
|
||||
Секреты должны храниться только в локальном `.env`. Файл не должен попадать в Git.
|
||||
|
||||
Файл `.env.compose.example` содержит имена переменных, используемые непосредственно в `compose.yaml`. После копирования необходимо заменить все демонстрационные значения `CHANGE_ME` локальными секретами. Сам `.env` не добавляется в Git.
|
||||
|
||||
### 4. Проверка конфигурации и сборка
|
||||
|
||||
```powershell
|
||||
docker compose config --quiet
|
||||
docker compose build
|
||||
```
|
||||
|
||||
### 5. Запуск инфраструктуры и приложения
|
||||
|
||||
```powershell
|
||||
docker compose up -d --build --scale api=2
|
||||
```
|
||||
|
||||
В проекте предусмотрены следующие компоненты:
|
||||
|
||||
- PostgreSQL;
|
||||
- Redis;
|
||||
- S3-compatible storage;
|
||||
- сервис первоначальной настройки S3;
|
||||
- Alembic migrations;
|
||||
- две реплики API;
|
||||
- Traefik.
|
||||
|
||||
Миграции выполняются отдельным сервисом `migrate` перед запуском API.
|
||||
|
||||
### 6. Ожидание готовности
|
||||
|
||||
Проверить состояние контейнеров:
|
||||
|
||||
```powershell
|
||||
docker compose ps
|
||||
```
|
||||
|
||||
Проверить readiness API:
|
||||
|
||||
```powershell
|
||||
Invoke-RestMethod http://localhost:8080/health/ready
|
||||
```
|
||||
|
||||
Ожидаемый результат:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "ok"
|
||||
}
|
||||
```
|
||||
|
||||
### 7. Проверка приложения
|
||||
|
||||
Проверка структуры и контракта:
|
||||
|
||||
```powershell
|
||||
python -X utf8 scripts/check_contract.py
|
||||
python -X utf8 scripts/validate_openapi.py
|
||||
```
|
||||
|
||||
Для полной валидации OpenAPI установите dev-зависимости: `python -m pip install -r requirements-dev.txt`. Затем выполните `python -X utf8 scripts/validate_openapi.py`.
|
||||
|
||||
Проверка синтаксиса:
|
||||
|
||||
```powershell
|
||||
python -X utf8 -m compileall -q app migrations
|
||||
```
|
||||
|
||||
Публичный smoke-тест:
|
||||
|
||||
```powershell
|
||||
python -X utf8 tests/smoke.py --base-url http://localhost:8080
|
||||
```
|
||||
|
||||
Дополнительные проверки:
|
||||
|
||||
```powershell
|
||||
python -X utf8 scripts/test_presigned_upload.py
|
||||
python -X utf8 scripts/test_presigned_negative.py
|
||||
```
|
||||
|
||||
В Dockerfile вспомогательные каталоги `scripts/`, `tests/`, `contracts/` и файл `lab.json` должны копироваться в образ API. После пересборки тест очистки незавершённых загрузок выполняется внутри API-контейнера:
|
||||
|
||||
```powershell
|
||||
docker compose exec -T api python /app/scripts/test_attachment_cleanup.py
|
||||
```
|
||||
|
||||
Тест компенсационного удаления при отказе PostgreSQL выполняется внутри API-контейнера:
|
||||
|
||||
```powershell
|
||||
docker compose exec -T api python /app/scripts/test_s3_db_compensation.py
|
||||
```
|
||||
|
||||
### 8. Остановка проекта
|
||||
|
||||
Безопасная остановка контейнеров с сохранением данных:
|
||||
|
||||
```powershell
|
||||
docker compose down
|
||||
```
|
||||
|
||||
Повторный запуск:
|
||||
|
||||
```powershell
|
||||
docker compose up -d --build --scale api=2
|
||||
```
|
||||
|
||||
**Внимание:** намеренное удаление учебных данных выполняется отдельно:
|
||||
|
||||
```powershell
|
||||
docker compose down -v
|
||||
```
|
||||
|
||||
Эта команда удаляет Docker volumes, включая данные PostgreSQL, Redis и S3-хранилища. Использовать её следует только при осознанном сбросе учебного окружения.
|
||||
|
||||
---
|
||||
|
||||
## Архитектура и решения
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab3, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
### 1. Общая архитектура
|
||||
|
||||
Система состоит из нескольких взаимодействующих компонентов.
|
||||
|
||||
```text
|
||||
Клиент
|
||||
|
|
||||
v
|
||||
Traefik
|
||||
|
|
||||
----------------
|
||||
| |
|
||||
v v
|
||||
API-1 API-2
|
||||
| |
|
||||
---------------------------
|
||||
| | |
|
||||
v v v
|
||||
PostgreSQL Redis S3 Storage
|
||||
(метаданные) (кэш) (файлы)
|
||||
```
|
||||
|
||||
Дополнительно используются:
|
||||
|
||||
- Alembic для управления миграциями;
|
||||
- отдельный bootstrap-сервис для первоначальной настройки S3;
|
||||
- фоновая задача очистки незавершённых загрузок, работающая в API-репликах.
|
||||
|
||||
**Примечание о выборе S3-сервера.** В Compose закреплён образ `pgsty/silo:RELEASE.2026-10-26T00-00-00Z`, совместимый с S3 API. В формулировке задания прямо перечислены MinIO, SeaweedFS и RustFS; Silo в этом перечне не назван. Замена сервера не выполнялась, поскольку текущий стенд и доказательства собраны на Silo. Это потенциальное формальное отклонение, которое следует согласовать с преподавателем.
|
||||
|
||||
### 2. Назначение компонентов
|
||||
|
||||
**API (FastAPI)**
|
||||
|
||||
Обрабатывает запросы пользователей, проверяет авторизацию и членство в чатах, выполняет загрузку и скачивание вложений, проверяет метаданные и взаимодействует с хранилищами.
|
||||
|
||||
**PostgreSQL**
|
||||
|
||||
Является источником истины для метаданных вложений и незавершённых загрузок.
|
||||
|
||||
В базе хранятся:
|
||||
|
||||
- идентификатор вложения;
|
||||
- идентификатор чата;
|
||||
- идентификатор владельца;
|
||||
- имя файла;
|
||||
- MIME-тип;
|
||||
- размер;
|
||||
- SHA-256;
|
||||
- ключ объекта в S3;
|
||||
- статус обработки;
|
||||
- даты создания и изменения.
|
||||
|
||||
Для presigned-загрузок дополнительно сохраняются ожидаемые размер и контрольная сумма, срок действия и состояние загрузочной сессии.
|
||||
|
||||
**Redis**
|
||||
|
||||
Используется для кэширования истории сообщений и связанных с ней операций. Не является источником истины для вложений.
|
||||
|
||||
**S3-compatible storage**
|
||||
|
||||
Хранит бинарное содержимое файлов. Bucket `messenger-attachments` настроен как приватный.
|
||||
|
||||
API обращается к S3 по внутреннему адресу Docker-сети. Клиент использует публичный адрес S3 для прямой загрузки по подписанной ссылке.
|
||||
|
||||
**Traefik**
|
||||
|
||||
Распределяет запросы между двумя экземплярами API.
|
||||
|
||||
### 3. Источник истины и транзакционные границы
|
||||
|
||||
PostgreSQL хранит сведения о вложениях, а S3 — бинарные данные.
|
||||
|
||||
Между PostgreSQL и S3 нет общей атомарной транзакции. Поэтому согласованность обеспечивается последовательностью операций и компенсационными действиями.
|
||||
|
||||
Для Proxy Upload:
|
||||
|
||||
1. API принимает и проверяет файл.
|
||||
2. Загружает содержимое в S3.
|
||||
3. Создаёт запись метаданных в PostgreSQL.
|
||||
4. Если сохранение метаданных завершается ошибкой, API выполняет rollback и пытается удалить объект из S3.
|
||||
|
||||
Для Presigned Upload:
|
||||
|
||||
1. API проверяет пользователя и членство в чате.
|
||||
2. Создаёт уникальный ключ объекта и запись `pending` в PostgreSQL.
|
||||
3. Возвращает временную подписанную ссылку.
|
||||
4. Клиент загружает файл непосредственно в S3.
|
||||
5. Клиент вызывает endpoint завершения загрузки.
|
||||
6. API получает объект из S3, проверяет размер, SHA-256 и содержимое изображения.
|
||||
7. После успешной проверки создаёт запись вложения и переводит загрузку в состояние `completed`.
|
||||
|
||||
При ошибках проверки загрузка переводится в `failed`, а объект удаляется из S3 при успешном выполнении компенсационного действия.
|
||||
|
||||
Для незавершённых загрузок используется фоновая очистка с периодической проверкой просроченных записей. Для конкурентной работы нескольких API-реплик применяется блокировка PostgreSQL `FOR UPDATE SKIP LOCKED`.
|
||||
|
||||
### 4. Ограничения загрузки и безопасность
|
||||
|
||||
Поддерживаются PNG и JPEG.
|
||||
|
||||
Ограничения:
|
||||
|
||||
- максимальный размер файла — 10 MiB;
|
||||
- максимальное количество пикселей — 40 000 000;
|
||||
- проверяется фактический формат содержимого;
|
||||
- проверяется соответствие MIME-типа;
|
||||
- проверяется SHA-256;
|
||||
- используется безопасное имя файла;
|
||||
- тело запроса читается потоково;
|
||||
- для больших файлов применяется временный файл вместо неограниченного хранения в RAM;
|
||||
- скачивание доступно только авторизованным участникам соответствующего чата.
|
||||
|
||||
S3 bucket закрыт для анонимного доступа. Для приложения создан отдельный S3-пользователь с отдельной политикой доступа.
|
||||
|
||||
Административная консоль S3 не опубликована на хостовой машине.
|
||||
|
||||
---
|
||||
|
||||
## Доказательства критериев
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
В таблице приведены выполненные сценарии и фактические результаты.
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
| Критерий | Команда / сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
|---|---|---|---|
|
||||
| Проверка структуры проекта | `python -X utf8 scripts/check_contract.py` | PASS: структура, локальные ссылки и синтаксис | `evidence/check-contract.txt` (если сохранён; иначе вывод команды) |
|
||||
| Валидация OpenAPI | `python -X utf8 scripts/validate_openapi.py` | PASS: OpenAPI 3.1 и JSON Schema | Вывод команды |
|
||||
| Публичный smoke | `python -X utf8 tests/smoke.py --base-url http://localhost:8080` | PASS: публичный API-сценарий; повторно подтверждён после исправления маршрутизации Traefik | `evidence/smoke.txt` |
|
||||
| Готовность API | `Invoke-RestMethod http://localhost:8080/health/ready` | `status: ok` | Вывод команды |
|
||||
| Две API-реплики | `docker compose ps` | Обе реплики имеют статус healthy | `evidence/persistence-before.txt`, `evidence/failover-before.txt` |
|
||||
| Хранение бинарных данных в S3 | Proxy Upload и проверка скачивания | Файл сохраняется и возвращается с совпадающим SHA-256 | `evidence/track1-results.csv` |
|
||||
| Метаданные в PostgreSQL | Загрузка и получение вложения | Сохраняются владелец, чат, размер, MIME, SHA-256 и статус | `evidence/track1-results.csv` |
|
||||
| Ограничение размера | Загрузка файла больше 10 MiB | HTTP 413 | `evidence/negative-tests-results.txt` |
|
||||
| Повреждённое изображение | Загрузка некорректных байтов | HTTP 415 | `evidence/negative-tests-results.txt` |
|
||||
| Несовпадение MIME | PNG с `image/jpeg` | HTTP 415 | `evidence/negative-tests-results.txt` |
|
||||
| Проверка SHA-256 | Неверная ожидаемая контрольная сумма | HTTP 409 | `evidence/negative-tests-results.txt` |
|
||||
| Проверка размера | Неверный ожидаемый размер | HTTP 409 | `evidence/negative-tests-results.txt` |
|
||||
| Защита от чужой загрузки | Чужой пользователь вызывает complete | HTTP 404 | `evidence/negative-tests-results.txt` |
|
||||
| Защита метаданных | Неучастник чата запрашивает вложение | HTTP 404 | `evidence/negative-tests-results.txt` |
|
||||
| Защита скачивания | Неучастник чата скачивает вложение | HTTP 404 | `evidence/negative-tests-results.txt` |
|
||||
| Запрет межчатового прикрепления | Вложение из одного чата отправляется в другой | HTTP 404 | `evidence/negative-tests-results.txt` |
|
||||
| Идемпотентность завершения | Повторное завершение загрузки | Возвращается существующее вложение | Вывод `scripts/test_presigned_upload.py` |
|
||||
| Прямой Presigned Upload | Создание ссылки, PUT в S3, complete | Последний повторный запуск: PASS; загрузка завершена, SHA-256 совпадает | Вывод `scripts/test_presigned_upload.py` |
|
||||
| Очистка незавершённых загрузок | Ожидание фонового cleanup | Статус `expired`, объект удалён из S3 | `evidence/attachment-cleanup-results.txt` |
|
||||
| Компенсация Proxy Upload | Искусственная ошибка PostgreSQL после S3 PUT | HTTP 503, объект удалён из S3 | `evidence/s3-db-compensation-results.txt` |
|
||||
| Приватность bucket | `curl.exe -i http://localhost:9000/messenger-attachments/` без авторизации | HTTP 403 Forbidden (`AccessDenied`) | `evidence/s3-security-results.txt` |
|
||||
| Недоступность S3 Console | `docker compose port minio 9001` | Порт 9001 не опубликован на хост | `evidence/s3-security-results.txt` |
|
||||
| Сохранность данных | Перезапуск PostgreSQL, Redis, S3 и API без удаления volumes | Подтверждённые данные и существующая сессия сохранены | `evidence/persistence-results.txt` |
|
||||
| Отказоустойчивость API | Остановка одной API-реплики | 108 успешных записей, 1 неуспешная попытка, первое восстановление за 2,607 с (общий сценарий сообщений) | `evidence/failover-results.txt` |
|
||||
| Отказоустойчивость вложений | Остановка именно API-реплики, обслужившей Proxy Upload; скачивание через оставшуюся реплику | 2 реплики до отказа; 87 успешных записей, 4 неуспешные попытки; первое восстановление за 4,597 с; вложение скачано побайтно идентичным | `evidence/attachment-failover-results.txt` |
|
||||
| Недоступность S3 | Остановка S3-сервиса | Readiness сообщил `Required storage unavailable`; после запуска S3 обе API-реплики восстановили readiness | `evidence/s3-unavailable-results.txt` |
|
||||
| Эксперимент Track 1 | Сравнение Proxy и Presigned | Измерены полное время, время API и размер тела запросов | `evidence/track1-results.csv` |
|
||||
| Параллельная загрузка | 10 одновременных Proxy и 10 Presigned загрузок | 10/10 успешных в обоих режимах | `evidence/concurrent-upload-results.txt`, `evidence/concurrent-upload-results.csv` |
|
||||
|
||||
### Подтверждение инфраструктурных hook
|
||||
|
||||
Для проверки сохранности данных использовался сценарий перезапуска:
|
||||
|
||||
```sh
|
||||
docker compose restart postgres redis minio api
|
||||
```
|
||||
|
||||
Таким образом, перезапускались:
|
||||
|
||||
- PostgreSQL;
|
||||
- Redis;
|
||||
- S3-compatible storage;
|
||||
- API.
|
||||
|
||||
Docker volumes при этом не удалялись.
|
||||
|
||||
Для общего сценария отказоустойчивости API останавливался один из двух экземпляров API. Наблюдаемый результат: 108 успешных записей, 1 неуспешная попытка и первое восстановление за 2,607 секунды (`evidence/failover-results.txt`).
|
||||
|
||||
Отдельно проверялся отказ именно той API-реплики, которая обслужила загрузку вложения. Тест получил идентификатор обслужившей реплики из ответа API, остановил именно её, продолжил работу через оставшуюся реплику и скачал вложение. Результат: 87 успешных записей, 4 неуспешные попытки, первое восстановление за 4,597 секунды; содержимое вложения совпало побайтно (`evidence/attachment-failover-results.txt`).
|
||||
|
||||
---
|
||||
|
||||
## Эксперименты и отказы
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
### Исследовательский трек 1 — Presigned Upload
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
#### Гипотеза
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
Прямая загрузка файлов из клиента в S3-compatible хранилище по временной подписанной ссылке позволяет существенно сократить объём данных, передаваемых через API. Предполагается, что это уменьшит объём передаваемых через API данных и время обработки запроса на стороне API при работе с крупными вложениями.
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
#### Методика эксперимента
|
||||
|
||||
Сравнивались два способа загрузки:
|
||||
|
||||
1. **Proxy Upload** — клиент передаёт бинарное содержимое файла API, после чего API сохраняет его в S3.
|
||||
2. **Presigned Upload** — API выдаёт временную подписанную ссылку, клиент загружает файл непосредственно в S3, затем вызывает endpoint завершения загрузки.
|
||||
|
||||
Использовались два PNG-файла:
|
||||
|
||||
- `sample.png` — 111 817 байт (0,11 MiB);
|
||||
- `sample_large.png` — 5 769 330 байт (5,50 MiB).
|
||||
|
||||
Для каждого сочетания способа загрузки и размера файла выполнено по три повтора. Всего проведено 12 измерений.
|
||||
|
||||
Среда эксперимента: Docker Compose, два экземпляра API, PostgreSQL 16.6, Redis 7.4.2, Silo S3-compatible storage. API-контейнеры имели ограничение памяти 512 MiB.
|
||||
|
||||
#### Результаты
|
||||
|
||||
| Способ загрузки | Размер файла | Среднее полное время | Среднее время запросов к API | Тело запросов к API |
|
||||
|---|---:|---:|---:|---:|
|
||||
| Proxy | 111 817 Б | 0,0786 с | 0,0785 с | 111 817 Б |
|
||||
| Presigned | 111 817 Б | 0,1296 с | 0,0826 с | 158 Б |
|
||||
| Proxy | 5 769 330 Б | 0,2666 с | 0,2665 с | 5 769 330 Б |
|
||||
| Presigned | 5 769 330 Б | 0,2564 с | 0,1275 с | 159 Б |
|
||||
|
||||
При загрузке большого файла объём тела запросов клиента к API сократился примерно на 99,997%. При загрузке небольшого файла сокращение составило около 99,86%.
|
||||
|
||||
Полное время загрузки большого файла при Presigned Upload оказалось примерно на 3,8% меньше, чем при Proxy Upload. Суммарное время запросов к API сократилось примерно на 52,2%.
|
||||
|
||||
#### Параллельная загрузка
|
||||
|
||||
Дополнительно проведён эксперимент с десятью одновременными загрузками файла размером 5 769 330 байт.
|
||||
|
||||
Для каждого способа загрузки выполнялось по десять параллельных операций.
|
||||
|
||||
| Показатель | Proxy | Presigned |
|
||||
|---|---:|---:|
|
||||
| Количество загрузок | 10 | 10 |
|
||||
| Успешные загрузки | 10 | 10 |
|
||||
| Ошибки | 0 | 0 |
|
||||
| Общее время | 3,996 с | 4,941 с |
|
||||
| Пиковая суммарная RAM API | 220,2 MiB | 304,4 MiB |
|
||||
| Прирост RAM относительно базового уровня | 16,1 MiB | 84,2 MiB |
|
||||
|
||||
Параллельный эксперимент подтвердил, что оба способа справляются с десятью одновременными загрузками без ошибок.
|
||||
|
||||
При этом в данном запуске Presigned Upload показал более высокий пик памяти. Поэтому утверждать, что прямая загрузка снижает потребление RAM, нельзя.
|
||||
|
||||
#### Потребление памяти в последовательном эксперименте
|
||||
|
||||
В ходе последовательного эксперимента фиксировалась память API-контейнеров с помощью `docker stats`. Измерения выполнялись с интервалом около 100 мс.
|
||||
|
||||
Зафиксированные максимальные значения среди API-реплик:
|
||||
|
||||
- Proxy, маленький файл — 103,4 MiB;
|
||||
- Presigned, маленький файл — 112,7 MiB;
|
||||
- Proxy, большой файл — 109,2 MiB;
|
||||
- Presigned, большой файл — 128,9 MiB.
|
||||
|
||||
Результаты не подтверждают снижение RAM при Presigned Upload в текущем эксперименте. Измерения ограничены интервалом опроса Docker Stats и не позволяют исключить кратковременные пики между замерами.
|
||||
|
||||
#### Вывод
|
||||
|
||||
Гипотеза о сокращении объёма данных, передаваемых через API, подтверждена.
|
||||
|
||||
Presigned Upload позволяет передавать содержимое файла непосредственно в S3, оставляя API операции выдачи временной ссылки, проверки загрузки и сохранения метаданных.
|
||||
|
||||
Уменьшение времени обработки запросов API для большого файла также наблюдалось. Однако снижение общего времени загрузки небольших файлов не подтверждено, а снижение потребления RAM требует дополнительного исследования.
|
||||
|
||||
**Файлы с исходными измерениями:**
|
||||
|
||||
- `evidence/track1-results.csv`;
|
||||
- `evidence/concurrent-upload-results.csv`;
|
||||
- `evidence/concurrent-upload-results.txt`.
|
||||
|
||||
### Проверка отказов и восстановления
|
||||
|
||||
| Воздействие | Что ожидалось | Что наблюдалось | Время восстановления / потеря данных |
|
||||
|---|---|---|---|
|
||||
| Перезапуск PostgreSQL, Redis, S3 и API | Сохранение подтверждённых данных при сохранении volumes | Подтверждённые сообщения, вложения и существующая сессия сохранились | Данные не потеряны |
|
||||
| Остановка одной API-реплики (общий сценарий) | Продолжение работы через оставшуюся реплику | 108 успешных записей, 1 неуспешная попытка | Первое восстановление — 2,607 с |
|
||||
| Остановка реплики, обслужившей загрузку вложения | Доступность вложения через другую реплику | 87 успешных записей, 4 неуспешные попытки; вложение скачано побайтно идентичным | Первое восстановление — 4,597 с |
|
||||
| Незавершённая Presigned Upload | Удаление просроченного объекта и обновление статуса | Запись получила статус `expired`, объект удалён | Ожидание фоновой очистки |
|
||||
| Недоступность S3 | Остановка S3-сервиса | Readiness сообщил `Required storage unavailable`; после восстановления S3 readiness снова `ok` | Сервис восстановлен без пересоздания API |
|
||||
| Ошибка PostgreSQL после загрузки в S3 через Proxy | Компенсационное удаление объекта | HTTP 503, объект отсутствует в S3 | Объект удалён |
|
||||
| Ошибка проверки SHA-256 | Отклонение загрузки и удаление некорректного объекта | HTTP 409 | Ошибка обработана |
|
||||
| Несовпадение размера файла | Отклонение загрузки | HTTP 409 | Ошибка обработана |
|
||||
| Повреждённое изображение | Отклонение содержимого | HTTP 415 | Ошибка обработана |
|
||||
| Превышение лимита размера | Отклонение загрузки до неограниченного чтения | HTTP 413 | Ошибка обработана |
|
||||
| Несовпадение MIME-типа | Отклонение файла | HTTP 415 | Ошибка обработана |
|
||||
| Чужой пользователь завершает загрузку | Запрет доступа | HTTP 404 | Доступ запрещён |
|
||||
| Попытка прикрепить вложение из другого чата | Запрет межчатового доступа | HTTP 404 | Доступ запрещён |
|
||||
|
||||
### Известные ограничения
|
||||
|
||||
1. PostgreSQL и S3 не используют общую распределённую транзакцию. Согласованность обеспечивается компенсационными операциями и фоновой очисткой.
|
||||
2. Удаление объекта из S3 является best-effort операцией. При одновременном отказе хранилища и компенсационного удаления может потребоваться повторная проверка состояния.
|
||||
3. Фоновая очистка работает с зарегистрированными незавершёнными загрузками. Объекты, для которых вообще не была создана запись загрузки, требуют отдельного механизма сверки или политики жизненного цикла S3.
|
||||
4. Эксперимент по памяти проводился с периодическим опросом Docker Stats. В параллельном запуске мониторинг успел получить лишь 4–5 измерений, поэтому кратковременные пики могли быть пропущены.
|
||||
5. Эксперименты проведены в локальной Docker-среде, а не в распределённой производственной инфраструктуре. Результаты нельзя автоматически переносить на промышленную нагрузку.
|
||||
6. Для прямой загрузки необходима доступность S3 endpoint со стороны клиента. В локальном окружении используется опубликованный адрес S3 API.
|
||||
7. Фактическое снижение потребления RAM при Presigned Upload экспериментально не подтверждено.
|
||||
|
||||
### Итоговый вывод
|
||||
|
||||
В лабораторной работе реализовано S3-хранилище вложений для приложения обмена сообщениями. Бинарные данные хранятся отдельно от метаданных, которые сохраняются в PostgreSQL.
|
||||
|
||||
Реализованы два способа загрузки файлов: Proxy Upload и Presigned Upload. Для прямой загрузки разработан механизм выдачи временных подписанных ссылок и последующей проверки загруженного объекта.
|
||||
|
||||
Проведены проверки безопасности, негативные тесты, проверка очистки незавершённых загрузок, компенсации при отказе PostgreSQL, сохранности данных после перезапуска и отказоустойчивости двух API-реплик.
|
||||
|
||||
В исследовательском треке 1 подтверждено существенное сокращение объёма данных, передаваемых через API. Для файла размером 5,5 MiB тело запроса к API уменьшилось с 5 769 330 до 159 байт. При этом эксперимент не подтвердил снижение потребления оперативной памяти.
|
||||
|
||||
Таким образом, реализация демонстрирует преимущества прямой загрузки в S3, сохраняет совместимость с обязательным Proxy Upload и включает механизмы проверки и обработки ошибок при работе с двумя независимыми хранилищами.
|
||||
+117
@@ -0,0 +1,117 @@
|
||||
# A generic, single database configuration.
|
||||
|
||||
[alembic]
|
||||
# path to migration scripts
|
||||
# Use forward slashes (/) also on windows to provide an os agnostic path
|
||||
script_location = migrations
|
||||
|
||||
# template used to generate migration file names; The default value is %%(rev)s_%%(slug)s
|
||||
# Uncomment the line below if you want the files to be prepended with date and time
|
||||
# see https://alembic.sqlalchemy.org/en/latest/tutorial.html#editing-the-ini-file
|
||||
# for all available tokens
|
||||
# file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s
|
||||
|
||||
# sys.path path, will be prepended to sys.path if present.
|
||||
# defaults to the current working directory.
|
||||
prepend_sys_path = .
|
||||
|
||||
# timezone to use when rendering the date within the migration file
|
||||
# as well as the filename.
|
||||
# If specified, requires the python>=3.9 or backports.zoneinfo library.
|
||||
# Any required deps can installed by adding `alembic[tz]` to the pip requirements
|
||||
# string value is passed to ZoneInfo()
|
||||
# leave blank for localtime
|
||||
# timezone =
|
||||
|
||||
# max length of characters to apply to the "slug" field
|
||||
# truncate_slug_length = 40
|
||||
|
||||
# set to 'true' to run the environment during
|
||||
# the 'revision' command, regardless of autogenerate
|
||||
# revision_environment = false
|
||||
|
||||
# set to 'true' to allow .pyc and .pyo files without
|
||||
# a source .py file to be detected as revisions in the
|
||||
# versions/ directory
|
||||
# sourceless = false
|
||||
|
||||
# version location specification; This defaults
|
||||
# to migrations/versions. When using multiple version
|
||||
# directories, initial revisions must be specified with --version-path.
|
||||
# The path separator used here should be the separator specified by "version_path_separator" below.
|
||||
# version_locations = %(here)s/bar:%(here)s/bat:migrations/versions
|
||||
|
||||
# version path separator; As mentioned above, this is the character used to split
|
||||
# version_locations. The default within new alembic.ini files is "os", which uses os.pathsep.
|
||||
# If this key is omitted entirely, it falls back to the legacy behavior of splitting on spaces and/or commas.
|
||||
# Valid values for version_path_separator are:
|
||||
#
|
||||
# version_path_separator = :
|
||||
# version_path_separator = ;
|
||||
# version_path_separator = space
|
||||
# version_path_separator = newline
|
||||
version_path_separator = os # Use os.pathsep. Default configuration used for new projects.
|
||||
|
||||
# set to 'true' to search source files recursively
|
||||
# in each "version_locations" directory
|
||||
# new in Alembic version 1.10
|
||||
# recursive_version_locations = false
|
||||
|
||||
# the output encoding used when revision files
|
||||
# are written from script.py.mako
|
||||
# output_encoding = utf-8
|
||||
|
||||
sqlalchemy.url = driver://user:pass@localhost/dbname
|
||||
|
||||
|
||||
[post_write_hooks]
|
||||
# post_write_hooks defines scripts or Python functions that are run
|
||||
# on newly generated revision scripts. See the documentation for further
|
||||
# detail and examples
|
||||
|
||||
# format using "black" - use the console_scripts runner, against the "black" entrypoint
|
||||
# hooks = black
|
||||
# black.type = console_scripts
|
||||
# black.entrypoint = black
|
||||
# black.options = -l 79 REVISION_SCRIPT_FILENAME
|
||||
|
||||
# lint with attempts to fix using "ruff" - use the exec runner, execute a binary
|
||||
# hooks = ruff
|
||||
# ruff.type = exec
|
||||
# ruff.executable = %(here)s/.venv/bin/ruff
|
||||
# ruff.options = --fix REVISION_SCRIPT_FILENAME
|
||||
|
||||
# Logging configuration
|
||||
[loggers]
|
||||
keys = root,sqlalchemy,alembic
|
||||
|
||||
[handlers]
|
||||
keys = console
|
||||
|
||||
[formatters]
|
||||
keys = generic
|
||||
|
||||
[logger_root]
|
||||
level = WARNING
|
||||
handlers = console
|
||||
qualname =
|
||||
|
||||
[logger_sqlalchemy]
|
||||
level = WARNING
|
||||
handlers =
|
||||
qualname = sqlalchemy.engine
|
||||
|
||||
[logger_alembic]
|
||||
level = INFO
|
||||
handlers =
|
||||
qualname = alembic
|
||||
|
||||
[handler_console]
|
||||
class = StreamHandler
|
||||
args = (sys.stderr,)
|
||||
level = NOTSET
|
||||
formatter = generic
|
||||
|
||||
[formatter_generic]
|
||||
format = %(levelname)-5.5s [%(name)s] %(message)s
|
||||
datefmt = %H:%M:%S
|
||||
@@ -0,0 +1,422 @@
|
||||
import hashlib
|
||||
import tempfile
|
||||
import uuid
|
||||
from datetime import datetime, timezone, timedelta
|
||||
from urllib.parse import quote
|
||||
|
||||
from fastapi import APIRouter, Depends, Header, HTTPException, Query, Request, Response
|
||||
from fastapi.responses import StreamingResponse
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
from botocore.exceptions import BotoCoreError, ClientError
|
||||
|
||||
from app.api.deps import get_db, get_current_user
|
||||
from app.core.config import settings
|
||||
from app.models import CreateAttachmentUpload
|
||||
from app.db.models import Attachment, AttachmentUpload, User
|
||||
from app.services.object_storage import client, create_presigned_upload_url
|
||||
from app.services.image_validation import MAX_BYTES, validate_image_file
|
||||
from app.repositories.chat_repository import ChatRepository
|
||||
|
||||
router = APIRouter(prefix="/api/v1", tags=["Attachments"])
|
||||
|
||||
|
||||
def iso_utc(value):
|
||||
return value.astimezone(timezone.utc).isoformat(timespec="microseconds").replace("+00:00", "Z")
|
||||
|
||||
|
||||
def attachment_json(item):
|
||||
return {
|
||||
"id": str(item.id), "chat_id": str(item.chat_id), "owner_id": str(item.owner_id),
|
||||
"filename": item.filename, "content_type": item.content_type,
|
||||
"size_bytes": item.size_bytes, "sha256": item.sha256, "status": item.status,
|
||||
"variants": [], "error_code": item.error_code,
|
||||
"created_at": iso_utc(item.created_at), "updated_at": iso_utc(item.updated_at),
|
||||
}
|
||||
|
||||
|
||||
def member_attachment(db, attachment_id, user_id):
|
||||
item = db.get(Attachment, attachment_id)
|
||||
if item is None or not ChatRepository(db).is_member(item.chat_id, user_id):
|
||||
raise HTTPException(404, "Attachment not found")
|
||||
return item
|
||||
|
||||
|
||||
async def read_limited(request: Request):
|
||||
declared = request.headers.get("content-length")
|
||||
if declared and declared.isdigit() and int(declared) > MAX_BYTES:
|
||||
raise HTTPException(413, "Image exceeds 10 MiB")
|
||||
size = 0
|
||||
# Keep at most 1 MiB in RAM; larger bodies spill to a temporary file.
|
||||
spool = tempfile.SpooledTemporaryFile(max_size=1024 * 1024, mode="w+b")
|
||||
try:
|
||||
async for chunk in request.stream():
|
||||
size += len(chunk)
|
||||
if size > MAX_BYTES:
|
||||
raise HTTPException(413, "Image exceeds 10 MiB")
|
||||
spool.write(chunk)
|
||||
spool.seek(0)
|
||||
return spool, size
|
||||
except Exception:
|
||||
spool.close()
|
||||
raise
|
||||
|
||||
|
||||
def sha256_file(fileobj):
|
||||
digest = hashlib.sha256()
|
||||
fileobj.seek(0)
|
||||
while True:
|
||||
chunk = fileobj.read(1024 * 1024)
|
||||
if not chunk: break
|
||||
digest.update(chunk)
|
||||
fileobj.seek(0)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
@router.post("/chats/{chat_id}/attachments/presigned", status_code=201)
|
||||
def create_attachment_upload(
|
||||
chat_id: str,
|
||||
payload: CreateAttachmentUpload,
|
||||
db: Session = Depends(get_db),
|
||||
user: User = Depends(get_current_user),
|
||||
):
|
||||
try:
|
||||
chat_uuid = uuid.UUID(chat_id)
|
||||
except ValueError:
|
||||
raise HTTPException(400, "Invalid chat UUID") from None
|
||||
|
||||
if not ChatRepository(db).is_member(chat_uuid, user.id):
|
||||
raise HTTPException(404, "Chat not found")
|
||||
|
||||
if payload.size_bytes > settings.max_upload_bytes:
|
||||
raise HTTPException(413, "Image exceeds configured upload limit")
|
||||
|
||||
upload_id = uuid.uuid4()
|
||||
object_key = f"uploads/{chat_uuid}/{upload_id}"
|
||||
expires_at = datetime.now(timezone.utc) + timedelta(
|
||||
seconds=settings.presigned_upload_ttl_seconds
|
||||
)
|
||||
|
||||
try:
|
||||
upload_url = create_presigned_upload_url(
|
||||
object_key=object_key,
|
||||
content_type=payload.content_type,
|
||||
)
|
||||
except (BotoCoreError, ClientError):
|
||||
raise HTTPException(503, "Object storage unavailable") from None
|
||||
|
||||
upload = AttachmentUpload(
|
||||
id=upload_id,
|
||||
chat_id=chat_uuid,
|
||||
owner_id=user.id,
|
||||
filename=payload.filename,
|
||||
content_type=payload.content_type,
|
||||
expected_size_bytes=payload.size_bytes,
|
||||
expected_sha256=payload.sha256.lower(),
|
||||
object_key=object_key,
|
||||
status="pending",
|
||||
expires_at=expires_at,
|
||||
)
|
||||
|
||||
try:
|
||||
db.add(upload)
|
||||
db.commit()
|
||||
except Exception:
|
||||
db.rollback()
|
||||
raise HTTPException(503, "Could not save upload metadata") from None
|
||||
|
||||
return {
|
||||
"upload_id": str(upload_id),
|
||||
"upload_url": upload_url,
|
||||
"method": "PUT",
|
||||
"headers": {
|
||||
"Content-Type": payload.content_type,
|
||||
},
|
||||
"expires_in": settings.presigned_upload_ttl_seconds,
|
||||
"expires_at": iso_utc(expires_at),
|
||||
}
|
||||
|
||||
|
||||
@router.post("/attachments/uploads/{upload_id}/complete", status_code=200)
|
||||
def complete_attachment_upload(
|
||||
upload_id: str,
|
||||
db: Session = Depends(get_db),
|
||||
user: User = Depends(get_current_user),
|
||||
):
|
||||
try:
|
||||
parsed_id = uuid.UUID(upload_id)
|
||||
except ValueError:
|
||||
raise HTTPException(400, "Invalid upload UUID") from None
|
||||
|
||||
# Lock the upload row to prevent concurrent completion.
|
||||
upload = db.scalar(
|
||||
select(AttachmentUpload)
|
||||
.where(AttachmentUpload.id == parsed_id)
|
||||
.with_for_update()
|
||||
)
|
||||
|
||||
if upload is None:
|
||||
raise HTTPException(404, "Upload not found")
|
||||
|
||||
if (
|
||||
upload.owner_id != user.id
|
||||
or not ChatRepository(db).is_member(upload.chat_id, user.id)
|
||||
):
|
||||
raise HTTPException(404, "Upload not found")
|
||||
|
||||
# Idempotent completion: return the existing attachment.
|
||||
if upload.status == "completed":
|
||||
item = db.scalar(
|
||||
select(Attachment).where(
|
||||
Attachment.object_key == upload.object_key
|
||||
)
|
||||
)
|
||||
if item is None:
|
||||
raise HTTPException(409, "Completed attachment not found")
|
||||
return attachment_json(item)
|
||||
|
||||
if upload.status != "pending":
|
||||
raise HTTPException(409, "Upload is not pending")
|
||||
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
if upload.expires_at <= now:
|
||||
upload.status = "expired"
|
||||
db.commit()
|
||||
|
||||
try:
|
||||
client().delete_object(
|
||||
Bucket=settings.s3_bucket,
|
||||
Key=upload.object_key,
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
raise HTTPException(409, "Upload has expired")
|
||||
|
||||
s3 = client()
|
||||
|
||||
# Read the object from S3.
|
||||
try:
|
||||
result = s3.get_object(
|
||||
Bucket=settings.s3_bucket,
|
||||
Key=upload.object_key,
|
||||
)
|
||||
except ClientError as exc:
|
||||
code = exc.response.get("Error", {}).get("Code", "")
|
||||
|
||||
if code in {"NoSuchKey", "404", "NotFound"}:
|
||||
raise HTTPException(409, "Uploaded object not found") from None
|
||||
|
||||
raise HTTPException(503, "Object storage unavailable") from None
|
||||
except BotoCoreError:
|
||||
raise HTTPException(503, "Object storage unavailable") from None
|
||||
|
||||
body = result["Body"]
|
||||
spool = tempfile.SpooledTemporaryFile(
|
||||
max_size=1024 * 1024,
|
||||
mode="w+b",
|
||||
)
|
||||
|
||||
def reject_upload(status_code: int, detail: str):
|
||||
upload.status = "failed"
|
||||
db.commit()
|
||||
|
||||
try:
|
||||
s3.delete_object(
|
||||
Bucket=settings.s3_bucket,
|
||||
Key=upload.object_key,
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
raise HTTPException(status_code=status_code, detail=detail)
|
||||
|
||||
try:
|
||||
actual_size = result.get("ContentLength")
|
||||
|
||||
if actual_size != upload.expected_size_bytes:
|
||||
reject_upload(409, "Uploaded file size mismatch")
|
||||
|
||||
if actual_size > settings.max_upload_bytes:
|
||||
reject_upload(413, "Image exceeds configured upload limit")
|
||||
|
||||
total = 0
|
||||
digest = hashlib.sha256()
|
||||
|
||||
try:
|
||||
while True:
|
||||
chunk = body.read(64 * 1024)
|
||||
if not chunk:
|
||||
break
|
||||
|
||||
total += len(chunk)
|
||||
|
||||
if total > settings.max_upload_bytes:
|
||||
reject_upload(413, "Image exceeds configured upload limit")
|
||||
|
||||
if total > upload.expected_size_bytes:
|
||||
reject_upload(409, "Uploaded file size mismatch")
|
||||
|
||||
digest.update(chunk)
|
||||
spool.write(chunk)
|
||||
finally:
|
||||
body.close()
|
||||
|
||||
if total != upload.expected_size_bytes:
|
||||
reject_upload(409, "Uploaded file size mismatch")
|
||||
|
||||
if digest.hexdigest() != upload.expected_sha256:
|
||||
reject_upload(409, "Uploaded file checksum mismatch")
|
||||
|
||||
# Validate the actual image content, not only S3 metadata.
|
||||
try:
|
||||
actual_type, _, _ = validate_image_file(
|
||||
spool,
|
||||
upload.content_type,
|
||||
)
|
||||
except HTTPException as exc:
|
||||
reject_upload(exc.status_code, str(exc.detail))
|
||||
|
||||
attachment = Attachment(
|
||||
id=uuid.uuid4(),
|
||||
chat_id=upload.chat_id,
|
||||
owner_id=upload.owner_id,
|
||||
filename=upload.filename,
|
||||
content_type=actual_type,
|
||||
size_bytes=total,
|
||||
sha256=digest.hexdigest(),
|
||||
object_key=upload.object_key,
|
||||
status="ready",
|
||||
error_code=None,
|
||||
)
|
||||
|
||||
upload.status = "completed"
|
||||
upload.completed_at = now
|
||||
|
||||
db.add(attachment)
|
||||
db.commit()
|
||||
db.refresh(attachment)
|
||||
|
||||
return attachment_json(attachment)
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except (BotoCoreError, ClientError):
|
||||
db.rollback()
|
||||
raise HTTPException(503, "Object storage unavailable") from None
|
||||
except Exception:
|
||||
db.rollback()
|
||||
raise
|
||||
finally:
|
||||
spool.close()
|
||||
|
||||
|
||||
@router.post("/chats/{chat_id}/attachments", status_code=201)
|
||||
async def upload_attachment(
|
||||
chat_id: str, request: Request, filename: str = Query(..., min_length=1, max_length=255),
|
||||
db: Session = Depends(get_db), user: User = Depends(get_current_user),
|
||||
):
|
||||
try:
|
||||
chat_uuid = uuid.UUID(chat_id)
|
||||
except ValueError:
|
||||
raise HTTPException(400, "Invalid chat UUID") from None
|
||||
if not ChatRepository(db).is_member(chat_uuid, user.id):
|
||||
raise HTTPException(404, "Chat not found")
|
||||
if not filename.strip() or any(c in filename for c in ("/", "\\", "\x00")) or any(ord(c) < 32 or ord(c) == 127 for c in filename):
|
||||
raise HTTPException(400, "Unsafe filename")
|
||||
content_type = request.headers.get("content-type", "").split(";", 1)[0].strip().lower()
|
||||
if content_type not in {"image/png", "image/jpeg"}:
|
||||
raise HTTPException(415, "Only PNG and JPEG are supported")
|
||||
spool, size = await read_limited(request)
|
||||
try:
|
||||
actual_type, _, _ = validate_image_file(spool, content_type)
|
||||
digest = sha256_file(spool)
|
||||
attachment_id = uuid.uuid4()
|
||||
object_key = f"original/{chat_uuid}/{attachment_id}"
|
||||
s3 = client()
|
||||
try:
|
||||
s3.put_object(Bucket=settings.s3_bucket, Key=object_key, Body=spool, ContentType=actual_type)
|
||||
except (BotoCoreError, ClientError):
|
||||
raise HTTPException(503, "Object storage unavailable") from None
|
||||
item = Attachment(
|
||||
id=attachment_id, chat_id=chat_uuid, owner_id=user.id, filename=filename,
|
||||
content_type=actual_type, size_bytes=size, sha256=digest,
|
||||
object_key=object_key, status="ready", error_code=None,
|
||||
)
|
||||
try:
|
||||
db.add(item)
|
||||
db.commit()
|
||||
db.refresh(item)
|
||||
except Exception:
|
||||
db.rollback()
|
||||
try: s3.delete_object(Bucket=settings.s3_bucket, Key=object_key)
|
||||
except Exception: pass
|
||||
raise HTTPException(503, "Could not save attachment metadata") from None
|
||||
return attachment_json(item)
|
||||
finally:
|
||||
spool.close()
|
||||
|
||||
|
||||
@router.get("/attachments/{attachment_id}")
|
||||
def get_attachment(attachment_id: str, db: Session = Depends(get_db), user: User = Depends(get_current_user)):
|
||||
try: parsed = uuid.UUID(attachment_id)
|
||||
except ValueError: raise HTTPException(400, "Invalid attachment UUID") from None
|
||||
item = member_attachment(db, parsed, user.id)
|
||||
if item.status != "ready":
|
||||
raise HTTPException(409, "Attachment upload is not completed")
|
||||
return attachment_json(item)
|
||||
|
||||
|
||||
@router.get("/attachments/{attachment_id}/content")
|
||||
def download_attachment(attachment_id: str, variant: str = "original", db: Session = Depends(get_db), user: User = Depends(get_current_user)):
|
||||
if variant != "original":
|
||||
raise HTTPException(404, "Variant not found")
|
||||
try: parsed = uuid.UUID(attachment_id)
|
||||
except ValueError: raise HTTPException(400, "Invalid attachment UUID") from None
|
||||
item = member_attachment(db, parsed, user.id)
|
||||
if item.status != "ready":
|
||||
raise HTTPException(409, "Attachment is not ready")
|
||||
spool = tempfile.SpooledTemporaryFile(max_size=1024 * 1024, mode="w+b")
|
||||
try:
|
||||
result = client().get_object(Bucket=settings.s3_bucket, Key=item.object_key)
|
||||
body = result["Body"]
|
||||
total = 0
|
||||
digest = hashlib.sha256()
|
||||
try:
|
||||
while True:
|
||||
chunk = body.read(64 * 1024)
|
||||
if not chunk:
|
||||
break
|
||||
total += len(chunk)
|
||||
if total > MAX_BYTES:
|
||||
raise HTTPException(503, "Stored object exceeds configured limit")
|
||||
digest.update(chunk)
|
||||
spool.write(chunk)
|
||||
finally:
|
||||
body.close()
|
||||
if total != item.size_bytes or digest.hexdigest() != item.sha256:
|
||||
raise HTTPException(503, "Stored object checksum mismatch")
|
||||
spool.seek(0)
|
||||
except (BotoCoreError, ClientError):
|
||||
spool.close()
|
||||
raise HTTPException(503, "Object storage unavailable") from None
|
||||
except Exception:
|
||||
spool.close()
|
||||
raise
|
||||
|
||||
def stream_verified_file():
|
||||
try:
|
||||
while True:
|
||||
chunk = spool.read(64 * 1024)
|
||||
if not chunk:
|
||||
break
|
||||
yield chunk
|
||||
finally:
|
||||
spool.close()
|
||||
|
||||
return StreamingResponse(
|
||||
stream_verified_file(),
|
||||
media_type=item.content_type,
|
||||
headers={"Content-Disposition": f"inline; filename*=UTF-8''{quote(item.filename)}"},
|
||||
)
|
||||
@@ -0,0 +1,96 @@
|
||||
from datetime import timezone
|
||||
from fastapi import APIRouter, Depends, Header, HTTPException, status
|
||||
from fastapi.security import HTTPBasic, HTTPBasicCredentials
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.api.deps import get_db
|
||||
from app.api.users import user_to_response
|
||||
from app.services.auth_service import (
|
||||
AuthService,
|
||||
InvalidCredentialsError,
|
||||
InvalidSessionError,
|
||||
)
|
||||
|
||||
|
||||
router = APIRouter(prefix="/api/v1/auth", tags=["Authentication"])
|
||||
|
||||
basic_security = HTTPBasic(auto_error=False)
|
||||
|
||||
|
||||
|
||||
@router.post("/sessions", status_code=status.HTTP_201_CREATED)
|
||||
def create_session(
|
||||
credentials: HTTPBasicCredentials = Depends(basic_security),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
if credentials is None:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Basic authentication required",
|
||||
headers={
|
||||
"WWW-Authenticate": 'Basic realm="messenger", charset="UTF-8"'
|
||||
},
|
||||
)
|
||||
|
||||
try:
|
||||
result = AuthService(db).create_session(
|
||||
username=credentials.username,
|
||||
password=credentials.password,
|
||||
)
|
||||
|
||||
expires_at = result["expires_at"]
|
||||
|
||||
if expires_at.tzinfo is None:
|
||||
expires_at = expires_at.replace(tzinfo=timezone.utc)
|
||||
else:
|
||||
expires_at = expires_at.astimezone(timezone.utc)
|
||||
|
||||
return {
|
||||
"token": result["token"],
|
||||
"token_type": result["token_type"],
|
||||
"expires_at": expires_at.isoformat(
|
||||
timespec="microseconds"
|
||||
).replace("+00:00", "Z"),
|
||||
"user": user_to_response(result["user"]),
|
||||
}
|
||||
|
||||
except InvalidCredentialsError:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Invalid username or password",
|
||||
headers={
|
||||
"WWW-Authenticate": 'Basic realm="messenger", charset="UTF-8"'
|
||||
},
|
||||
) from None
|
||||
|
||||
|
||||
|
||||
@router.delete("/sessions/current", status_code=status.HTTP_204_NO_CONTENT)
|
||||
def delete_current_session(
|
||||
authorization: str | None = Header(default=None),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
if not authorization or not authorization.startswith("Bearer "):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Bearer token required",
|
||||
headers={"WWW-Authenticate": 'Bearer realm="messenger"'},
|
||||
)
|
||||
|
||||
token = authorization[7:].strip()
|
||||
|
||||
if not token:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Bearer token required",
|
||||
headers={"WWW-Authenticate": 'Bearer realm="messenger"'},
|
||||
)
|
||||
|
||||
try:
|
||||
AuthService(db).delete_session(token)
|
||||
except InvalidSessionError:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Invalid or expired session",
|
||||
headers={"WWW-Authenticate": 'Bearer realm="messenger"'},
|
||||
) from None
|
||||
@@ -0,0 +1,90 @@
|
||||
import uuid
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, status
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.api.deps import get_db, get_current_user
|
||||
from app.db.models import User as UserModel
|
||||
from app.models import Chat as ChatResponse, CreateChat
|
||||
from app.services.chat_service import (
|
||||
ChatService,
|
||||
ChatNotFoundError,
|
||||
InvalidChatDataError,
|
||||
)
|
||||
|
||||
|
||||
router = APIRouter(prefix="/api/v1/chats", tags=["Chats"])
|
||||
|
||||
|
||||
def chat_to_response(chat) -> ChatResponse:
|
||||
return ChatResponse(
|
||||
id=str(chat.id),
|
||||
title=chat.title,
|
||||
member_ids=[str(member.user_id) for member in chat.members],
|
||||
created_at=chat.created_at,
|
||||
)
|
||||
|
||||
|
||||
@router.post("", response_model=ChatResponse, status_code=status.HTTP_201_CREATED)
|
||||
def create_chat(
|
||||
body: CreateChat,
|
||||
db: Session = Depends(get_db),
|
||||
user: UserModel = Depends(get_current_user),
|
||||
):
|
||||
try:
|
||||
chat = ChatService(db).create_chat(
|
||||
title=body.title,
|
||||
member_ids=body.member_ids,
|
||||
creator_id=user.id,
|
||||
)
|
||||
return chat_to_response(chat)
|
||||
|
||||
except InvalidChatDataError as exc:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=str(exc),
|
||||
) from None
|
||||
|
||||
|
||||
@router.get("")
|
||||
def list_chats(
|
||||
limit: int = Query(default=50, ge=1, le=100),
|
||||
cursor: str | None = None,
|
||||
db: Session = Depends(get_db),
|
||||
user: UserModel = Depends(get_current_user),
|
||||
):
|
||||
if cursor is not None:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="Chat cursor pagination is not implemented yet",
|
||||
)
|
||||
|
||||
chats = ChatService(db).list_chats(user.id)
|
||||
chats = chats[:limit]
|
||||
|
||||
return {
|
||||
"items": [chat_to_response(chat) for chat in chats],
|
||||
"next_cursor": None,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/{chat_id}", response_model=ChatResponse)
|
||||
def get_chat(
|
||||
chat_id: str,
|
||||
db: Session = Depends(get_db),
|
||||
user: UserModel = Depends(get_current_user),
|
||||
):
|
||||
try:
|
||||
parsed_id = uuid.UUID(chat_id)
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=400, detail="Invalid chat UUID") from None
|
||||
|
||||
try:
|
||||
chat = ChatService(db).get_chat(parsed_id, user.id)
|
||||
return chat_to_response(chat)
|
||||
|
||||
except ChatNotFoundError:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail="Chat not found",
|
||||
) from None
|
||||
@@ -0,0 +1,40 @@
|
||||
from fastapi import Depends, HTTPException, status
|
||||
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db.session import SessionLocal
|
||||
from app.services.auth_service import AuthService, InvalidSessionError
|
||||
|
||||
|
||||
bearer_scheme = HTTPBearer(auto_error=False)
|
||||
|
||||
|
||||
def get_db():
|
||||
db = SessionLocal()
|
||||
try:
|
||||
yield db
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
def get_current_user(
|
||||
credentials: HTTPAuthorizationCredentials | None = Depends(bearer_scheme),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
if credentials is None or credentials.scheme.lower() != "bearer":
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Bearer token required",
|
||||
headers={"WWW-Authenticate": "Bearer"},
|
||||
)
|
||||
|
||||
try:
|
||||
user = AuthService(db).authenticate(credentials.credentials)
|
||||
return user
|
||||
|
||||
except InvalidSessionError:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Invalid or expired token",
|
||||
headers={"WWW-Authenticate": "Bearer"},
|
||||
) from None
|
||||
@@ -0,0 +1,73 @@
|
||||
import re
|
||||
import uuid
|
||||
|
||||
from fastapi import APIRouter, Depends, Header, HTTPException, Query, status
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.api.deps import get_db, get_current_user
|
||||
from app.db.models import User as UserModel
|
||||
from app.models import CreateMessage, Message as MessageResponse
|
||||
from app.services.message_service import (
|
||||
MessageService, MessageNotFoundError, InvalidMessageDataError,
|
||||
IdempotencyConflictError,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/api/v1/chats/{chat_id}/messages", tags=["Messages"])
|
||||
IDEMPOTENCY_KEY_RE = re.compile(r"^[A-Za-z0-9._:-]{1,128}$")
|
||||
|
||||
|
||||
def message_to_response(message) -> MessageResponse:
|
||||
return MessageResponse(
|
||||
id=str(message.id), chat_id=str(message.chat_id),
|
||||
sender_id=str(message.sender_id), text=message.text,
|
||||
created_at=message.created_at, attachment_ids=message.attachment_ids or [],
|
||||
)
|
||||
|
||||
|
||||
def parse_chat_id(chat_id: str) -> uuid.UUID:
|
||||
try:
|
||||
return uuid.UUID(chat_id)
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=400, detail="Invalid chat UUID") from None
|
||||
|
||||
|
||||
@router.post("", response_model=MessageResponse, status_code=status.HTTP_201_CREATED)
|
||||
def create_message(
|
||||
chat_id: str, body: CreateMessage,
|
||||
idempotency_key: str = Header(..., alias="Idempotency-Key", min_length=1, max_length=128),
|
||||
db: Session = Depends(get_db),
|
||||
user: UserModel = Depends(get_current_user),
|
||||
):
|
||||
if not IDEMPOTENCY_KEY_RE.fullmatch(idempotency_key):
|
||||
raise HTTPException(status_code=400, detail="Invalid Idempotency-Key")
|
||||
parsed_id = parse_chat_id(chat_id)
|
||||
try:
|
||||
message = MessageService(db).create_message(
|
||||
chat_id=parsed_id, sender_id=user.id, text=body.text,
|
||||
idempotency_key=idempotency_key, attachment_ids=body.attachment_ids,
|
||||
)
|
||||
return message_to_response(message)
|
||||
except MessageNotFoundError:
|
||||
raise HTTPException(status_code=404, detail="Chat not found") from None
|
||||
except InvalidMessageDataError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from None
|
||||
except IdempotencyConflictError:
|
||||
raise HTTPException(status_code=409, detail="Idempotency-Key was already used with a different request") from None
|
||||
|
||||
|
||||
@router.get("")
|
||||
def list_messages(
|
||||
chat_id: str, limit: int = Query(default=50, ge=1, le=100),
|
||||
cursor: str | None = None, db: Session = Depends(get_db),
|
||||
user: UserModel = Depends(get_current_user),
|
||||
):
|
||||
parsed_id = parse_chat_id(chat_id)
|
||||
try:
|
||||
messages, next_cursor = MessageService(db).get_history(
|
||||
chat_id=parsed_id, user_id=user.id, limit=limit, cursor=cursor
|
||||
)
|
||||
return {"items": [message_to_response(item) for item in messages], "next_cursor": next_cursor}
|
||||
except MessageNotFoundError:
|
||||
raise HTTPException(status_code=404, detail="Chat not found") from None
|
||||
except InvalidMessageDataError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from None
|
||||
@@ -0,0 +1,55 @@
|
||||
from fastapi import APIRouter, Depends, HTTPException, status
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.api.deps import get_db, get_current_user
|
||||
from app.db.models import User as UserModel
|
||||
from app.models import CreateUser, User
|
||||
from app.services.user_service import (
|
||||
UserService,
|
||||
UsernameAlreadyExistsError,
|
||||
)
|
||||
|
||||
|
||||
router = APIRouter(prefix="/api/v1/users", tags=["Users"])
|
||||
|
||||
|
||||
def user_to_response(user: UserModel) -> User:
|
||||
return User(
|
||||
id=str(user.id),
|
||||
username=user.username,
|
||||
display_name=user.display_name,
|
||||
created_at=user.created_at,
|
||||
)
|
||||
|
||||
|
||||
@router.post("", response_model=User, status_code=status.HTTP_201_CREATED)
|
||||
def register_user(
|
||||
body: CreateUser,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
try:
|
||||
user = UserService(db).register(
|
||||
username=body.username,
|
||||
display_name=body.display_name,
|
||||
password=body.password,
|
||||
)
|
||||
return user_to_response(user)
|
||||
|
||||
except UsernameAlreadyExistsError:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_409_CONFLICT,
|
||||
detail="Username already exists",
|
||||
) from None
|
||||
|
||||
except ValueError as exc:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail=str(exc),
|
||||
) from None
|
||||
|
||||
|
||||
@router.get("/me", response_model=User)
|
||||
def get_me(
|
||||
current_user: UserModel = Depends(get_current_user),
|
||||
):
|
||||
return user_to_response(current_user)
|
||||
@@ -0,0 +1,49 @@
|
||||
import os
|
||||
from pydantic import AliasChoices, Field
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
app_name: str = "Messenger API"
|
||||
app_env: str = "lab"
|
||||
database_url: str = "postgresql+psycopg://messenger:messenger@postgres:5432/messenger"
|
||||
db_pool_size: int = 5
|
||||
db_max_overflow: int = 5
|
||||
db_pool_timeout: int = 10
|
||||
session_ttl_seconds: int = 3600
|
||||
cache_url: str = "redis://redis:6379/0"
|
||||
message_cache_ttl_seconds: int = 60
|
||||
idempotency_ttl_seconds: int = 86400
|
||||
instance_id: str = os.getenv("INSTANCE_ID") or os.getenv("HOSTNAME") or "api-local"
|
||||
shutdown_grace_seconds: int = 20
|
||||
s3_endpoint_url: str = Field(
|
||||
default="http://minio:9000",
|
||||
validation_alias=AliasChoices("S3_ENDPOINT_URL", "S3_ENDPOINT"),
|
||||
)
|
||||
s3_public_endpoint_url: str = Field(
|
||||
default="http://localhost:9000",
|
||||
validation_alias=AliasChoices("S3_PUBLIC_ENDPOINT_URL"),
|
||||
)
|
||||
presigned_upload_ttl_seconds: int = 300
|
||||
attachment_cleanup_interval_seconds: int = Field(default=60, ge=5, le=3600)
|
||||
attachment_cleanup_grace_seconds: int = Field(default=600, ge=0, le=86400)
|
||||
attachment_cleanup_batch_size: int = Field(default=100, ge=1, le=1000)
|
||||
s3_region: str = Field(default="us-east-1", validation_alias=AliasChoices("S3_REGION"))
|
||||
s3_bucket: str = Field(default="messenger-attachments", validation_alias=AliasChoices("S3_BUCKET"))
|
||||
s3_access_key: str = Field(
|
||||
default="messenger-api",
|
||||
validation_alias=AliasChoices("S3_ACCESS_KEY", "S3_ACCESS_KEY_ID"),
|
||||
)
|
||||
s3_secret_key: str = Field(
|
||||
default="change-me-api-secret",
|
||||
validation_alias=AliasChoices("S3_SECRET_KEY", "S3_SECRET_ACCESS_KEY"),
|
||||
)
|
||||
max_upload_bytes: int = 10 * 1024 * 1024
|
||||
max_image_pixels: int = 40_000_000
|
||||
|
||||
model_config = SettingsConfigDict(
|
||||
env_file=".env", env_file_encoding="utf-8", extra="ignore"
|
||||
)
|
||||
|
||||
|
||||
settings = Settings()
|
||||
@@ -0,0 +1,5 @@
|
||||
from sqlalchemy.orm import DeclarativeBase
|
||||
|
||||
|
||||
class Base(DeclarativeBase):
|
||||
pass
|
||||
@@ -0,0 +1,298 @@
|
||||
import uuid
|
||||
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import (
|
||||
DateTime,
|
||||
JSON,
|
||||
ForeignKey,
|
||||
Index,
|
||||
String,
|
||||
Text,
|
||||
UniqueConstraint,
|
||||
)
|
||||
from sqlalchemy.dialects.postgresql import UUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from app.db.base import Base
|
||||
|
||||
|
||||
def utc_now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
class User(Base):
|
||||
__tablename__ = "users"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
primary_key=True,
|
||||
default=uuid.uuid4,
|
||||
)
|
||||
username: Mapped[str] = mapped_column(
|
||||
String(32),
|
||||
unique=True,
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
display_name: Mapped[str] = mapped_column(
|
||||
String(100),
|
||||
nullable=False,
|
||||
)
|
||||
password_hash: Mapped[str] = mapped_column(
|
||||
String(255),
|
||||
nullable=False,
|
||||
)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
default=utc_now,
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
memberships: Mapped[list["ChatMember"]] = relationship(
|
||||
back_populates="user",
|
||||
cascade="all, delete-orphan",
|
||||
)
|
||||
messages: Mapped[list["Message"]] = relationship(
|
||||
back_populates="sender",
|
||||
)
|
||||
|
||||
|
||||
class Chat(Base):
|
||||
__tablename__ = "chats"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
primary_key=True,
|
||||
default=uuid.uuid4,
|
||||
)
|
||||
title: Mapped[str] = mapped_column(
|
||||
String(100),
|
||||
nullable=False,
|
||||
)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
default=utc_now,
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
members: Mapped[list["ChatMember"]] = relationship(
|
||||
back_populates="chat",
|
||||
cascade="all, delete-orphan",
|
||||
)
|
||||
messages: Mapped[list["Message"]] = relationship(
|
||||
back_populates="chat",
|
||||
cascade="all, delete-orphan",
|
||||
)
|
||||
|
||||
|
||||
class ChatMember(Base):
|
||||
__tablename__ = "chat_members"
|
||||
|
||||
chat_id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
ForeignKey("chats.id", ondelete="CASCADE"),
|
||||
primary_key=True,
|
||||
)
|
||||
user_id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
ForeignKey("users.id", ondelete="CASCADE"),
|
||||
primary_key=True,
|
||||
)
|
||||
|
||||
chat: Mapped["Chat"] = relationship(
|
||||
back_populates="members",
|
||||
)
|
||||
user: Mapped["User"] = relationship(
|
||||
back_populates="memberships",
|
||||
)
|
||||
|
||||
|
||||
class Message(Base):
|
||||
__tablename__ = "messages"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
primary_key=True,
|
||||
default=uuid.uuid4,
|
||||
)
|
||||
chat_id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
ForeignKey("chats.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
)
|
||||
sender_id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
ForeignKey("users.id"),
|
||||
nullable=False,
|
||||
)
|
||||
text: Mapped[str] = mapped_column(
|
||||
Text,
|
||||
nullable=False,
|
||||
)
|
||||
attachment_ids: Mapped[list] = mapped_column(JSON, nullable=False, default=list)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
default=utc_now,
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
chat: Mapped["Chat"] = relationship(
|
||||
back_populates="messages",
|
||||
)
|
||||
sender: Mapped["User"] = relationship(
|
||||
back_populates="messages",
|
||||
)
|
||||
|
||||
__table_args__ = (
|
||||
Index(
|
||||
"ix_messages_chat_created_id",
|
||||
"chat_id",
|
||||
"created_at",
|
||||
"id",
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class Session(Base):
|
||||
__tablename__ = "sessions"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
primary_key=True,
|
||||
default=uuid.uuid4,
|
||||
)
|
||||
token_hash: Mapped[str] = mapped_column(
|
||||
String(64),
|
||||
unique=True,
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
user_id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
ForeignKey("users.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
)
|
||||
expires_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
nullable=False,
|
||||
)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
default=utc_now,
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
user: Mapped["User"] = relationship()
|
||||
|
||||
class IdempotencyRecord(Base):
|
||||
__tablename__ = "idempotency_records"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True), primary_key=True, default=uuid.uuid4
|
||||
)
|
||||
user_id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True), ForeignKey("users.id", ondelete="CASCADE"), nullable=False
|
||||
)
|
||||
method: Mapped[str] = mapped_column(String(10), nullable=False)
|
||||
path: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
idempotency_key: Mapped[str] = mapped_column(String(128), nullable=False)
|
||||
request_hash: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
# NULL is visible only inside the creating transaction; the message and
|
||||
# idempotency row are committed atomically.
|
||||
message_id: Mapped[uuid.UUID | None] = mapped_column(
|
||||
UUID(as_uuid=True), ForeignKey("messages.id", ondelete="CASCADE"), nullable=True
|
||||
)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), default=utc_now, nullable=False
|
||||
)
|
||||
expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"user_id", "method", "path", "idempotency_key",
|
||||
name="uq_idempotency_scope_key",
|
||||
),
|
||||
Index("ix_idempotency_records_expires_at", "expires_at"),
|
||||
)
|
||||
|
||||
|
||||
class Attachment(Base):
|
||||
__tablename__ = "attachments"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
chat_id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), ForeignKey("chats.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
owner_id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), ForeignKey("users.id", ondelete="CASCADE"), nullable=False)
|
||||
filename: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
content_type: Mapped[str] = mapped_column(String(32), nullable=False)
|
||||
size_bytes: Mapped[int] = mapped_column(nullable=False)
|
||||
sha256: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
object_key: Mapped[str] = mapped_column(String(512), unique=True, nullable=False)
|
||||
status: Mapped[str] = mapped_column(String(16), nullable=False, default="ready")
|
||||
error_code: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utc_now, nullable=False)
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utc_now, onupdate=utc_now, nullable=False)
|
||||
|
||||
|
||||
class AttachmentUpload(Base):
|
||||
__tablename__ = "attachment_uploads"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
primary_key=True,
|
||||
default=uuid.uuid4,
|
||||
)
|
||||
chat_id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
ForeignKey("chats.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
)
|
||||
owner_id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
ForeignKey("users.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
)
|
||||
filename: Mapped[str] = mapped_column(
|
||||
String(255),
|
||||
nullable=False,
|
||||
)
|
||||
content_type: Mapped[str] = mapped_column(
|
||||
String(32),
|
||||
nullable=False,
|
||||
)
|
||||
expected_size_bytes: Mapped[int] = mapped_column(
|
||||
nullable=False,
|
||||
)
|
||||
expected_sha256: Mapped[str] = mapped_column(
|
||||
String(64),
|
||||
nullable=False,
|
||||
)
|
||||
object_key: Mapped[str] = mapped_column(
|
||||
String(512),
|
||||
unique=True,
|
||||
nullable=False,
|
||||
)
|
||||
status: Mapped[str] = mapped_column(
|
||||
String(16),
|
||||
nullable=False,
|
||||
default="pending",
|
||||
)
|
||||
expires_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
nullable=False,
|
||||
)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
default=utc_now,
|
||||
nullable=False,
|
||||
)
|
||||
completed_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
nullable=True,
|
||||
)
|
||||
|
||||
__table_args__ = (
|
||||
Index("ix_attachment_uploads_chat_id", "chat_id"),
|
||||
Index("ix_attachment_uploads_owner_id", "owner_id"),
|
||||
Index("ix_attachment_uploads_expires_at", "expires_at"),
|
||||
)
|
||||
@@ -0,0 +1,15 @@
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from app.core.config import settings
|
||||
|
||||
engine = create_engine(
|
||||
settings.database_url,
|
||||
pool_pre_ping=True,
|
||||
pool_size=settings.db_pool_size,
|
||||
max_overflow=settings.db_max_overflow,
|
||||
pool_timeout=settings.db_pool_timeout,
|
||||
pool_recycle=1800,
|
||||
)
|
||||
|
||||
SessionLocal = sessionmaker(bind=engine, autoflush=False, autocommit=False)
|
||||
+106
@@ -0,0 +1,106 @@
|
||||
import asyncio
|
||||
import logging
|
||||
import uuid
|
||||
from contextlib import asynccontextmanager, suppress
|
||||
|
||||
from fastapi import FastAPI, HTTPException, Request
|
||||
from fastapi.exceptions import RequestValidationError
|
||||
from fastapi.responses import JSONResponse
|
||||
from pathlib import Path
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.exc import SQLAlchemyError
|
||||
|
||||
from app.api.auth import router as auth_router
|
||||
from app.api.users import router as users_router
|
||||
from app.api.chats import router as chats_router
|
||||
from app.api.messages import router as messages_router
|
||||
from app.api.attachments import router as attachments_router
|
||||
from app.core.config import settings
|
||||
from app.db.session import engine
|
||||
from app.services.object_storage import client as s3_client
|
||||
from app.services.attachment_cleanup import cleanup_expired_uploads
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s %(message)s")
|
||||
logger = logging.getLogger(__name__)
|
||||
async def attachment_cleanup_loop():
|
||||
while True:
|
||||
try:
|
||||
# Synchronous boto3/SQLAlchemy I/O must not block the event loop.
|
||||
await asyncio.to_thread(cleanup_expired_uploads)
|
||||
except Exception:
|
||||
logger.exception("Attachment cleanup worker iteration failed")
|
||||
await asyncio.sleep(settings.attachment_cleanup_interval_seconds)
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
logger.info("Application startup instance_id=%s", settings.instance_id)
|
||||
cleanup_task = asyncio.create_task(attachment_cleanup_loop())
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
cleanup_task.cancel()
|
||||
with suppress(asyncio.CancelledError):
|
||||
await cleanup_task
|
||||
engine.dispose()
|
||||
logger.info("Application shutdown instance_id=%s", settings.instance_id)
|
||||
|
||||
app = FastAPI(title=settings.app_name, version="1.4.0", lifespan=lifespan)
|
||||
app.include_router(users_router)
|
||||
app.include_router(auth_router)
|
||||
app.include_router(chats_router)
|
||||
app.include_router(messages_router)
|
||||
app.include_router(attachments_router)
|
||||
|
||||
@app.middleware("http")
|
||||
async def response_headers(request: Request, call_next):
|
||||
request_id = str(uuid.uuid4())
|
||||
request.state.request_id = request_id
|
||||
try:
|
||||
response = await call_next(request)
|
||||
except Exception:
|
||||
logger.exception("Unhandled request error; request_id=%s", request_id)
|
||||
response = JSONResponse(status_code=500, content={"error": {"code": "internal_error", "message": "Unexpected error", "request_id": request_id}})
|
||||
response.headers["X-Request-Id"] = request_id
|
||||
response.headers["X-Instance-Id"] = settings.instance_id
|
||||
response.headers["X-Content-Type-Options"] = "nosniff"
|
||||
return response
|
||||
|
||||
@app.exception_handler(HTTPException)
|
||||
async def http_exception_handler(request: Request, exc: HTTPException):
|
||||
request_id = getattr(request.state, "request_id", str(uuid.uuid4()))
|
||||
code = {400: "invalid_request", 401: "unauthorized", 404: "not_found", 409: "conflict", 413: "payload_too_large", 415: "unsupported_media_type", 429: "rate_limited", 500: "internal_error", 503: "unavailable"}.get(exc.status_code, "internal_error")
|
||||
headers = dict(exc.headers or {})
|
||||
headers["X-Request-Id"] = request_id
|
||||
headers["X-Instance-Id"] = settings.instance_id
|
||||
return JSONResponse(status_code=exc.status_code, content={"error": {"code": code, "message": exc.detail if isinstance(exc.detail, str) else "Request failed", "request_id": request_id}}, headers=headers)
|
||||
|
||||
@app.exception_handler(RequestValidationError)
|
||||
async def validation_exception_handler(request: Request, exc: RequestValidationError):
|
||||
request_id = getattr(request.state, "request_id", str(uuid.uuid4()))
|
||||
return JSONResponse(status_code=400, content={"error": {"code": "invalid_request", "message": "Validation error", "request_id": request_id}}, headers={"X-Request-Id": request_id, "X-Instance-Id": settings.instance_id})
|
||||
|
||||
@app.get("/health/live")
|
||||
def liveness():
|
||||
return {"status": "ok"}
|
||||
|
||||
DRAIN_FILE = Path("/tmp/api-draining")
|
||||
@app.get("/health/ready")
|
||||
def readiness():
|
||||
if DRAIN_FILE.exists():
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail="Instance is draining",
|
||||
)
|
||||
|
||||
try:
|
||||
with engine.connect() as connection:
|
||||
connection.execute(text("SELECT 1"))
|
||||
s3_client().head_bucket(Bucket=settings.s3_bucket)
|
||||
return {"status": "ok"}
|
||||
except Exception:
|
||||
logger.exception("PostgreSQL or S3 readiness check failed")
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail="Required storage unavailable",
|
||||
) from None
|
||||
+138
@@ -0,0 +1,138 @@
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from typing import List, Optional, Literal
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator
|
||||
|
||||
|
||||
class BaseConfig(BaseModel):
|
||||
model_config = ConfigDict(
|
||||
from_attributes=True,
|
||||
json_encoders={
|
||||
datetime: lambda value: (
|
||||
value.astimezone(timezone.utc)
|
||||
.isoformat(timespec="microseconds")
|
||||
.replace("+00:00", "Z")
|
||||
)
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
class CreateUser(BaseConfig):
|
||||
username: str = Field(..., pattern=r"^[a-z0-9_]{3,32}$")
|
||||
display_name: str = Field(..., min_length=1, max_length=100)
|
||||
password: str = Field(..., min_length=12, max_length=128)
|
||||
|
||||
@field_validator("display_name")
|
||||
@classmethod
|
||||
def validate_display_name(cls, value: str) -> str:
|
||||
value = value.strip()
|
||||
if not value:
|
||||
raise ValueError("display_name must not be blank")
|
||||
return value
|
||||
|
||||
|
||||
class User(BaseConfig):
|
||||
id: str
|
||||
username: str
|
||||
display_name: str
|
||||
created_at: datetime
|
||||
|
||||
|
||||
class CreateChat(BaseConfig):
|
||||
title: str = Field(..., min_length=1, max_length=100)
|
||||
member_ids: List[str] = Field(..., min_length=1, max_length=100)
|
||||
|
||||
@field_validator("title")
|
||||
@classmethod
|
||||
def validate_title(cls, value: str) -> str:
|
||||
value = value.strip()
|
||||
if not value:
|
||||
raise ValueError("title must not be blank")
|
||||
return value
|
||||
|
||||
@field_validator("member_ids")
|
||||
@classmethod
|
||||
def validate_unique_members(cls, value: List[str]) -> List[str]:
|
||||
if len(value) != len(set(value)):
|
||||
raise ValueError("member_ids must be unique")
|
||||
return value
|
||||
|
||||
|
||||
class Chat(BaseConfig):
|
||||
id: str
|
||||
title: str
|
||||
member_ids: List[str]
|
||||
created_at: datetime
|
||||
|
||||
|
||||
class CreateMessage(BaseConfig):
|
||||
text: str = Field(..., max_length=4000)
|
||||
attachment_ids: List[str] = Field(default_factory=list, max_length=10)
|
||||
|
||||
@field_validator("attachment_ids")
|
||||
@classmethod
|
||||
def validate_attachment_ids(cls, value):
|
||||
if len(value) != len(set(value)):
|
||||
raise ValueError("attachment_ids must be unique")
|
||||
return value
|
||||
|
||||
@field_validator("text")
|
||||
@classmethod
|
||||
def validate_text(cls, value: str) -> str:
|
||||
value = value.strip()
|
||||
return value
|
||||
|
||||
|
||||
class Message(BaseConfig):
|
||||
id: str
|
||||
chat_id: str
|
||||
sender_id: str
|
||||
text: str
|
||||
created_at: datetime
|
||||
attachment_ids: List[str] = Field(default_factory=list)
|
||||
|
||||
|
||||
class Health(BaseConfig):
|
||||
status: str = "ok"
|
||||
|
||||
|
||||
class ErrorDetail(BaseConfig):
|
||||
code: str
|
||||
message: str
|
||||
request_id: str
|
||||
|
||||
|
||||
class ErrorResponse(BaseConfig):
|
||||
error: ErrorDetail
|
||||
|
||||
|
||||
class ChatPage(BaseConfig):
|
||||
items: List[Chat]
|
||||
next_cursor: Optional[str] = None
|
||||
|
||||
|
||||
class MessagePage(BaseConfig):
|
||||
items: List[Message]
|
||||
next_cursor: Optional[str] = None
|
||||
|
||||
|
||||
class CreateAttachmentUpload(BaseConfig):
|
||||
filename: str = Field(..., min_length=1, max_length=255)
|
||||
content_type: Literal["image/png", "image/jpeg"]
|
||||
size_bytes: int = Field(..., gt=0, le=10 * 1024 * 1024)
|
||||
sha256: str = Field(..., pattern=r"^[a-fA-F0-9]{64}$")
|
||||
|
||||
@field_validator("filename")
|
||||
@classmethod
|
||||
def validate_filename(cls, value: str) -> str:
|
||||
value = value.strip()
|
||||
|
||||
if (
|
||||
not value
|
||||
or any(c in value for c in ("/", "\\", "\x00"))
|
||||
or any(ord(c) < 32 or ord(c) == 127 for c in value)
|
||||
):
|
||||
raise ValueError("Unsafe filename")
|
||||
|
||||
return value
|
||||
@@ -0,0 +1,71 @@
|
||||
import uuid
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session, selectinload
|
||||
|
||||
from app.db.models import Chat, ChatMember
|
||||
|
||||
|
||||
class ChatRepository:
|
||||
|
||||
def __init__(self, db: Session):
|
||||
self.db = db
|
||||
|
||||
def get_by_id(self, chat_id: uuid.UUID) -> Chat | None:
|
||||
statement = (
|
||||
select(Chat)
|
||||
.options(selectinload(Chat.members))
|
||||
.where(Chat.id == chat_id)
|
||||
)
|
||||
|
||||
return self.db.scalar(statement)
|
||||
|
||||
def get_by_user(
|
||||
self,
|
||||
user_id: uuid.UUID,
|
||||
limit: int = 50,
|
||||
) -> list[Chat]:
|
||||
statement = (
|
||||
select(Chat)
|
||||
.join(ChatMember, ChatMember.chat_id == Chat.id)
|
||||
.where(ChatMember.user_id == user_id)
|
||||
.order_by(Chat.created_at.asc(), Chat.id.asc())
|
||||
.limit(limit)
|
||||
)
|
||||
|
||||
return list(self.db.scalars(statement).all())
|
||||
|
||||
def is_member(
|
||||
self,
|
||||
chat_id: uuid.UUID,
|
||||
user_id: uuid.UUID,
|
||||
) -> bool:
|
||||
statement = select(ChatMember).where(
|
||||
ChatMember.chat_id == chat_id,
|
||||
ChatMember.user_id == user_id,
|
||||
)
|
||||
|
||||
return self.db.scalar(statement) is not None
|
||||
|
||||
def create(
|
||||
self,
|
||||
title: str,
|
||||
member_ids: list[uuid.UUID],
|
||||
) -> Chat:
|
||||
chat = Chat(title=title)
|
||||
|
||||
self.db.add(chat)
|
||||
self.db.flush()
|
||||
|
||||
memberships = [
|
||||
ChatMember(
|
||||
chat_id=chat.id,
|
||||
user_id=user_id,
|
||||
)
|
||||
for user_id in member_ids
|
||||
]
|
||||
|
||||
self.db.add_all(memberships)
|
||||
self.db.flush()
|
||||
|
||||
return chat
|
||||
@@ -0,0 +1,73 @@
|
||||
import uuid
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import and_, or_, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db.models import Message
|
||||
|
||||
|
||||
class MessageRepository:
|
||||
|
||||
def __init__(self, db: Session):
|
||||
self.db = db
|
||||
|
||||
def get_by_id(
|
||||
self,
|
||||
message_id: uuid.UUID,
|
||||
) -> Message | None:
|
||||
statement = select(Message).where(
|
||||
Message.id == message_id
|
||||
)
|
||||
|
||||
return self.db.scalar(statement)
|
||||
|
||||
def create(
|
||||
self,
|
||||
chat_id: uuid.UUID,
|
||||
sender_id: uuid.UUID,
|
||||
text: str,
|
||||
attachment_ids: list[str] | None = None,
|
||||
) -> Message:
|
||||
message = Message(
|
||||
chat_id=chat_id,
|
||||
sender_id=sender_id,
|
||||
text=text,
|
||||
attachment_ids=attachment_ids or [],
|
||||
)
|
||||
|
||||
self.db.add(message)
|
||||
self.db.flush()
|
||||
|
||||
return message
|
||||
|
||||
def get_history(
|
||||
self,
|
||||
chat_id: uuid.UUID,
|
||||
limit: int = 50,
|
||||
cursor_created_at: datetime | None = None,
|
||||
cursor_id: uuid.UUID | None = None,
|
||||
) -> list[Message]:
|
||||
statement = select(Message).where(
|
||||
Message.chat_id == chat_id
|
||||
)
|
||||
|
||||
if cursor_created_at is not None and cursor_id is not None:
|
||||
statement = statement.where(
|
||||
or_(
|
||||
Message.created_at > cursor_created_at,
|
||||
and_(
|
||||
Message.created_at == cursor_created_at,
|
||||
Message.id > cursor_id,
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
statement = (
|
||||
statement
|
||||
.order_by(Message.created_at.asc(), Message.id.asc())
|
||||
.limit(limit)
|
||||
)
|
||||
|
||||
return list(self.db.scalars(statement).all())
|
||||
@@ -0,0 +1,78 @@
|
||||
import uuid
|
||||
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import delete, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db.models import Session as UserSession
|
||||
|
||||
|
||||
class SessionRepository:
|
||||
|
||||
def __init__(self, db: Session):
|
||||
self.db = db
|
||||
|
||||
def create(
|
||||
self,
|
||||
user_id: uuid.UUID,
|
||||
token_hash: str,
|
||||
expires_at: datetime,
|
||||
) -> UserSession:
|
||||
session = UserSession(
|
||||
user_id=user_id,
|
||||
token_hash=token_hash,
|
||||
expires_at=expires_at,
|
||||
)
|
||||
|
||||
self.db.add(session)
|
||||
self.db.flush()
|
||||
|
||||
return session
|
||||
|
||||
def get_by_token_hash(
|
||||
self,
|
||||
token_hash: str,
|
||||
) -> UserSession | None:
|
||||
statement = select(UserSession).where(
|
||||
UserSession.token_hash == token_hash
|
||||
)
|
||||
|
||||
return self.db.scalar(statement)
|
||||
|
||||
def get_valid_session(
|
||||
self,
|
||||
token_hash: str,
|
||||
) -> UserSession | None:
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
statement = select(UserSession).where(
|
||||
UserSession.token_hash == token_hash,
|
||||
UserSession.expires_at > now,
|
||||
)
|
||||
|
||||
return self.db.scalar(statement)
|
||||
|
||||
def delete_by_token_hash(
|
||||
self,
|
||||
token_hash: str,
|
||||
) -> bool:
|
||||
statement = (
|
||||
delete(UserSession)
|
||||
.where(UserSession.token_hash == token_hash)
|
||||
.returning(UserSession.id)
|
||||
)
|
||||
|
||||
return self.db.scalar(statement) is not None
|
||||
|
||||
def delete_expired(self) -> int:
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
statement = (
|
||||
delete(UserSession)
|
||||
.where(UserSession.expires_at <= now)
|
||||
)
|
||||
|
||||
result = self.db.execute(statement)
|
||||
|
||||
return result.rowcount
|
||||
@@ -0,0 +1,41 @@
|
||||
import uuid
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db.models import User
|
||||
|
||||
|
||||
class UserRepository:
|
||||
|
||||
def __init__(self, db: Session):
|
||||
self.db = db
|
||||
|
||||
def get_by_id(self, user_id: uuid.UUID) -> User | None:
|
||||
statement = select(User).where(User.id == user_id)
|
||||
|
||||
return self.db.scalar(statement)
|
||||
|
||||
def get_by_username(self, username: str) -> User | None:
|
||||
statement = select(User).where(
|
||||
User.username == username
|
||||
)
|
||||
|
||||
return self.db.scalar(statement)
|
||||
|
||||
def create(
|
||||
self,
|
||||
username: str,
|
||||
display_name: str,
|
||||
password_hash: str,
|
||||
) -> User:
|
||||
user = User(
|
||||
username=username,
|
||||
display_name=display_name,
|
||||
password_hash=password_hash,
|
||||
)
|
||||
|
||||
self.db.add(user)
|
||||
self.db.flush()
|
||||
|
||||
return user
|
||||
@@ -0,0 +1,66 @@
|
||||
"""Best-effort cleanup for abandoned presigned uploads."""
|
||||
|
||||
import logging
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.core.config import settings
|
||||
from app.db.models import AttachmentUpload
|
||||
from app.db.session import SessionLocal
|
||||
from app.services.object_storage import client
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def cleanup_expired_uploads() -> int:
|
||||
"""Delete old pending objects and mark their upload records expired.
|
||||
|
||||
A grace period protects clients completing an upload near URL expiry.
|
||||
PostgreSQL SKIP LOCKED makes this safe across multiple API replicas.
|
||||
"""
|
||||
cutoff = datetime.now(timezone.utc) - timedelta(
|
||||
seconds=settings.attachment_cleanup_grace_seconds
|
||||
)
|
||||
deleted_count = 0
|
||||
|
||||
with SessionLocal() as db:
|
||||
uploads = db.scalars(
|
||||
select(AttachmentUpload)
|
||||
.where(
|
||||
AttachmentUpload.status == "pending",
|
||||
AttachmentUpload.expires_at <= cutoff,
|
||||
)
|
||||
.order_by(AttachmentUpload.expires_at.asc())
|
||||
.limit(settings.attachment_cleanup_batch_size)
|
||||
.with_for_update(skip_locked=True)
|
||||
).all()
|
||||
|
||||
if not uploads:
|
||||
return 0
|
||||
|
||||
s3 = client()
|
||||
try:
|
||||
for upload in uploads:
|
||||
# DeleteObject is idempotent: a missing object is also safe.
|
||||
s3.delete_object(
|
||||
Bucket=settings.s3_bucket,
|
||||
Key=upload.object_key,
|
||||
)
|
||||
upload.status = "expired"
|
||||
deleted_count += 1
|
||||
|
||||
db.commit()
|
||||
except Exception:
|
||||
db.rollback()
|
||||
logger.exception(
|
||||
"Attachment cleanup batch failed; pending records will be retried"
|
||||
)
|
||||
return 0
|
||||
|
||||
if deleted_count:
|
||||
logger.info(
|
||||
"Expired abandoned attachment uploads cleaned: count=%d",
|
||||
deleted_count,
|
||||
)
|
||||
return deleted_count
|
||||
@@ -0,0 +1,159 @@
|
||||
import hashlib
|
||||
import secrets
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from redis import Redis
|
||||
from redis.backoff import NoBackoff
|
||||
from redis.exceptions import RedisError
|
||||
from redis.retry import Retry
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.core.config import settings
|
||||
from app.db.models import User
|
||||
from app.repositories.session_repository import SessionRepository
|
||||
from app.repositories.user_repository import UserRepository
|
||||
from app.services.user_service import password_hasher
|
||||
|
||||
|
||||
class InvalidCredentialsError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class InvalidSessionError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class AuthService:
|
||||
def __init__(self, db: Session):
|
||||
self.db = db
|
||||
self.sessions = SessionRepository(db)
|
||||
self.users = UserRepository(db)
|
||||
|
||||
self._redis_available = True
|
||||
|
||||
self.redis = Redis.from_url(
|
||||
settings.cache_url,
|
||||
decode_responses=True,
|
||||
socket_connect_timeout=0.5,
|
||||
socket_timeout=0.5,
|
||||
retry_on_timeout=False,
|
||||
retry=Retry(NoBackoff(), 0),
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _hash_token(token: str) -> str:
|
||||
return hashlib.sha256(token.encode("utf-8")).hexdigest()
|
||||
|
||||
def create_session(self, username: str, password: str) -> dict:
|
||||
user = self.users.get_by_username(username)
|
||||
|
||||
if user is None or not password_hasher.verify(
|
||||
password, user.password_hash
|
||||
):
|
||||
raise InvalidCredentialsError("Invalid username or password")
|
||||
|
||||
token = secrets.token_urlsafe(32)
|
||||
token_hash = self._hash_token(token)
|
||||
|
||||
expires_at = datetime.now(timezone.utc) + timedelta(
|
||||
seconds=settings.session_ttl_seconds
|
||||
)
|
||||
|
||||
try:
|
||||
self.sessions.create(
|
||||
user_id=user.id,
|
||||
token_hash=token_hash,
|
||||
expires_at=expires_at,
|
||||
)
|
||||
self.db.commit()
|
||||
except Exception:
|
||||
self.db.rollback()
|
||||
raise
|
||||
|
||||
self._cache_session(token_hash, user.id, expires_at)
|
||||
|
||||
return {
|
||||
"token": token,
|
||||
"token_type": "Bearer",
|
||||
"expires_at": expires_at,
|
||||
"user": user,
|
||||
}
|
||||
|
||||
def _cache_session(
|
||||
self,
|
||||
token_hash: str,
|
||||
user_id,
|
||||
expires_at,
|
||||
) -> None:
|
||||
if not self._redis_available:
|
||||
return
|
||||
|
||||
ttl = max(
|
||||
1,
|
||||
int((expires_at - datetime.now(timezone.utc)).total_seconds()),
|
||||
)
|
||||
|
||||
try:
|
||||
self.redis.setex(
|
||||
f"session:{token_hash}",
|
||||
ttl,
|
||||
str(user_id),
|
||||
)
|
||||
except RedisError:
|
||||
self._redis_available = False
|
||||
|
||||
def authenticate(self, token: str) -> User:
|
||||
token_hash = self._hash_token(token)
|
||||
cached_user_id = None
|
||||
|
||||
if self._redis_available:
|
||||
try:
|
||||
cached_user_id = self.redis.get(
|
||||
f"session:{token_hash}"
|
||||
)
|
||||
except RedisError:
|
||||
self._redis_available = False
|
||||
|
||||
session = self.sessions.get_valid_session(token_hash)
|
||||
|
||||
if session is None:
|
||||
if self._redis_available:
|
||||
try:
|
||||
self.redis.delete(f"session:{token_hash}")
|
||||
except RedisError:
|
||||
self._redis_available = False
|
||||
|
||||
raise InvalidSessionError(
|
||||
"Session is invalid or expired"
|
||||
)
|
||||
|
||||
if cached_user_id is None:
|
||||
self._cache_session(
|
||||
token_hash,
|
||||
session.user_id,
|
||||
session.expires_at,
|
||||
)
|
||||
|
||||
return session.user
|
||||
|
||||
def delete_session(self, token: str) -> None:
|
||||
token_hash = self._hash_token(token)
|
||||
|
||||
try:
|
||||
deleted = self.sessions.delete_by_token_hash(token_hash)
|
||||
|
||||
if not deleted:
|
||||
raise InvalidSessionError(
|
||||
"Session is invalid or expired"
|
||||
)
|
||||
|
||||
self.db.commit()
|
||||
except Exception:
|
||||
self.db.rollback()
|
||||
raise
|
||||
|
||||
if self._redis_available:
|
||||
try:
|
||||
self.redis.delete(f"session:{token_hash}")
|
||||
except RedisError:
|
||||
self._redis_available = False
|
||||
@@ -0,0 +1,94 @@
|
||||
import uuid
|
||||
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db.models import Chat
|
||||
from app.repositories.chat_repository import ChatRepository
|
||||
from app.repositories.user_repository import UserRepository
|
||||
|
||||
|
||||
class ChatNotFoundError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class ChatAccessError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class InvalidChatDataError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class ChatService:
|
||||
def __init__(self, db: Session):
|
||||
self.db = db
|
||||
self.chats = ChatRepository(db)
|
||||
self.users = UserRepository(db)
|
||||
|
||||
def create_chat(
|
||||
self,
|
||||
title: str,
|
||||
member_ids: list[str],
|
||||
creator_id: uuid.UUID,
|
||||
) -> Chat:
|
||||
title = title.strip()
|
||||
|
||||
if not title or len(title) > 100:
|
||||
raise InvalidChatDataError("Invalid chat title")
|
||||
|
||||
if not member_ids or len(member_ids) > 100:
|
||||
raise InvalidChatDataError("Invalid members")
|
||||
|
||||
try:
|
||||
parsed_ids = [uuid.UUID(str(value)) for value in member_ids]
|
||||
except (ValueError, TypeError, AttributeError):
|
||||
raise InvalidChatDataError("Invalid member UUID") from None
|
||||
|
||||
if len(parsed_ids) != len(set(parsed_ids)):
|
||||
raise InvalidChatDataError("Duplicate members")
|
||||
|
||||
if creator_id not in parsed_ids:
|
||||
raise InvalidChatDataError("Creator must be a member")
|
||||
|
||||
for user_id in parsed_ids:
|
||||
if self.users.get_by_id(user_id) is None:
|
||||
raise InvalidChatDataError("Unknown member")
|
||||
|
||||
try:
|
||||
chat = self.chats.create(
|
||||
title=title,
|
||||
member_ids=parsed_ids,
|
||||
)
|
||||
self.db.commit()
|
||||
self.db.refresh(chat)
|
||||
return chat
|
||||
|
||||
except IntegrityError as exc:
|
||||
self.db.rollback()
|
||||
raise InvalidChatDataError("Unable to create chat") from exc
|
||||
|
||||
except Exception:
|
||||
self.db.rollback()
|
||||
raise
|
||||
|
||||
def get_chat(self, chat_id: uuid.UUID, user_id: uuid.UUID) -> Chat:
|
||||
chat = self.chats.get_by_id(chat_id)
|
||||
|
||||
if chat is None:
|
||||
raise ChatNotFoundError()
|
||||
|
||||
if not self.chats.is_member(chat_id, user_id):
|
||||
raise ChatNotFoundError()
|
||||
|
||||
return chat
|
||||
|
||||
def list_chats(self, user_id: uuid.UUID) -> list[Chat]:
|
||||
return self.chats.get_by_user(user_id)
|
||||
|
||||
def check_membership(
|
||||
self,
|
||||
chat_id: uuid.UUID,
|
||||
user_id: uuid.UUID,
|
||||
) -> bool:
|
||||
return self.chats.is_member(chat_id, user_id)
|
||||
@@ -0,0 +1,41 @@
|
||||
from io import BytesIO
|
||||
from PIL import Image, UnidentifiedImageError
|
||||
from fastapi import HTTPException
|
||||
|
||||
MAX_BYTES = 10 * 1024 * 1024
|
||||
MAX_PIXELS = 40_000_000
|
||||
ALLOWED = {"PNG": "image/png", "JPEG": "image/jpeg"}
|
||||
|
||||
|
||||
def validate_image_file(fileobj, declared_type: str):
|
||||
"""Validate a seekable, size-limited file without loading its full encoded bytes."""
|
||||
try:
|
||||
fileobj.seek(0)
|
||||
with Image.open(fileobj) as image:
|
||||
fmt = image.format
|
||||
width, height = image.size
|
||||
if fmt not in ALLOWED:
|
||||
raise HTTPException(415, "Only PNG and JPEG are supported")
|
||||
if width * height > MAX_PIXELS:
|
||||
raise HTTPException(413, "Image exceeds 40 million pixels")
|
||||
image.verify()
|
||||
fileobj.seek(0)
|
||||
with Image.open(fileobj) as image:
|
||||
image.load()
|
||||
except HTTPException:
|
||||
raise
|
||||
except (UnidentifiedImageError, OSError, ValueError, SyntaxError, Image.DecompressionBombError):
|
||||
raise HTTPException(415, "Invalid or corrupted image") from None
|
||||
actual_type = ALLOWED[fmt]
|
||||
if declared_type != actual_type:
|
||||
raise HTTPException(415, "Content-Type does not match image content")
|
||||
fileobj.seek(0)
|
||||
return actual_type, width, height
|
||||
|
||||
|
||||
def validate_image(data: bytes, declared_type: str):
|
||||
if not data:
|
||||
raise HTTPException(415, "Empty image")
|
||||
if len(data) > MAX_BYTES:
|
||||
raise HTTPException(413, "Image exceeds 10 MiB")
|
||||
return validate_image_file(BytesIO(data), declared_type)
|
||||
@@ -0,0 +1,435 @@
|
||||
import base64
|
||||
import hashlib
|
||||
import json
|
||||
import time
|
||||
import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from types import SimpleNamespace
|
||||
|
||||
from redis import Redis
|
||||
from redis.backoff import NoBackoff
|
||||
from redis.exceptions import RedisError
|
||||
from redis.retry import Retry
|
||||
from sqlalchemy import delete, select, update
|
||||
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.core.config import settings
|
||||
from app.db.models import Attachment, IdempotencyRecord, Message
|
||||
from app.repositories.chat_repository import ChatRepository
|
||||
from app.repositories.message_repository import MessageRepository
|
||||
|
||||
|
||||
class MessageNotFoundError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class MessageAccessError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class InvalidMessageDataError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class IdempotencyConflictError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class MessageService:
|
||||
def __init__(self, db: Session):
|
||||
self.db = db
|
||||
self.messages = MessageRepository(db)
|
||||
self.chats = ChatRepository(db)
|
||||
|
||||
self._redis_available = True
|
||||
|
||||
self.redis = Redis.from_url(
|
||||
settings.cache_url,
|
||||
decode_responses=True,
|
||||
socket_connect_timeout=0.5,
|
||||
socket_timeout=0.5,
|
||||
retry_on_timeout=False,
|
||||
retry=Retry(NoBackoff(), 0),
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def encode_cursor(message: Message) -> str:
|
||||
payload = {
|
||||
"created_at": message.created_at.isoformat(),
|
||||
"id": str(message.id),
|
||||
}
|
||||
|
||||
raw = json.dumps(payload).encode("utf-8")
|
||||
|
||||
return (
|
||||
base64.urlsafe_b64encode(raw)
|
||||
.decode("ascii")
|
||||
.rstrip("=")
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def decode_cursor(cursor: str | None):
|
||||
if not cursor:
|
||||
return None, None
|
||||
|
||||
try:
|
||||
padded = cursor + "=" * (-len(cursor) % 4)
|
||||
raw = base64.urlsafe_b64decode(padded.encode("ascii"))
|
||||
payload = json.loads(raw.decode("utf-8"))
|
||||
|
||||
created_at = datetime.fromisoformat(payload["created_at"])
|
||||
message_id = uuid.UUID(payload["id"])
|
||||
|
||||
if created_at.tzinfo is None:
|
||||
raise ValueError("Cursor timestamp must include timezone")
|
||||
|
||||
return created_at, message_id
|
||||
|
||||
except (ValueError, TypeError, KeyError, UnicodeError):
|
||||
raise InvalidMessageDataError("Invalid cursor") from None
|
||||
|
||||
def _version_key(self, chat_id: uuid.UUID) -> str:
|
||||
return f"messages:version:{chat_id}"
|
||||
|
||||
def _cache_key(
|
||||
self,
|
||||
chat_id: uuid.UUID,
|
||||
version: int,
|
||||
limit: int,
|
||||
cursor: str | None,
|
||||
) -> str:
|
||||
cursor_hash = hashlib.sha256(
|
||||
(cursor or "first-page").encode("utf-8")
|
||||
).hexdigest()[:16]
|
||||
|
||||
return (
|
||||
f"messages:{chat_id}:v{version}:"
|
||||
f"limit:{limit}:cursor:{cursor_hash}"
|
||||
)
|
||||
|
||||
def _get_version(self, chat_id: uuid.UUID) -> int:
|
||||
if not self._redis_available:
|
||||
return 0
|
||||
|
||||
try:
|
||||
value = self.redis.get(self._version_key(chat_id))
|
||||
return int(value or 0)
|
||||
except (RedisError, ValueError):
|
||||
self._redis_available = False
|
||||
return 0
|
||||
|
||||
def _bump_version(self, chat_id: uuid.UUID) -> None:
|
||||
if not self._redis_available:
|
||||
return
|
||||
|
||||
try:
|
||||
self.redis.incr(self._version_key(chat_id))
|
||||
except RedisError:
|
||||
self._redis_available = False
|
||||
|
||||
@staticmethod
|
||||
def _serialize(messages, next_cursor):
|
||||
return json.dumps({
|
||||
"items": [
|
||||
{
|
||||
"id": str(message.id),
|
||||
"chat_id": str(message.chat_id),
|
||||
"sender_id": str(message.sender_id),
|
||||
"text": message.text,
|
||||
"attachment_ids": message.attachment_ids or [],
|
||||
"created_at": message.created_at.isoformat(),
|
||||
}
|
||||
for message in messages
|
||||
],
|
||||
"next_cursor": next_cursor,
|
||||
})
|
||||
|
||||
@staticmethod
|
||||
def _deserialize(payload: str):
|
||||
data = json.loads(payload)
|
||||
|
||||
messages = [
|
||||
SimpleNamespace(
|
||||
id=uuid.UUID(item["id"]),
|
||||
chat_id=uuid.UUID(item["chat_id"]),
|
||||
sender_id=uuid.UUID(item["sender_id"]),
|
||||
text=item["text"],
|
||||
attachment_ids=item.get("attachment_ids", []),
|
||||
created_at=datetime.fromisoformat(item["created_at"]),
|
||||
)
|
||||
for item in data["items"]
|
||||
]
|
||||
|
||||
return messages, data["next_cursor"]
|
||||
|
||||
def _read_cache(self, key: str):
|
||||
if not self._redis_available:
|
||||
return None
|
||||
|
||||
try:
|
||||
payload = self.redis.get(key)
|
||||
|
||||
if payload is None:
|
||||
return None
|
||||
|
||||
return self._deserialize(payload)
|
||||
|
||||
except RedisError:
|
||||
self._redis_available = False
|
||||
return None
|
||||
except (ValueError, TypeError, KeyError):
|
||||
return None
|
||||
|
||||
def _load_from_database(
|
||||
self,
|
||||
chat_id,
|
||||
limit,
|
||||
cursor_created_at,
|
||||
cursor_id,
|
||||
):
|
||||
rows = self.messages.get_history(
|
||||
chat_id=chat_id,
|
||||
limit=limit + 1,
|
||||
cursor_created_at=cursor_created_at,
|
||||
cursor_id=cursor_id,
|
||||
)
|
||||
|
||||
has_more = len(rows) > limit
|
||||
items = rows[:limit]
|
||||
|
||||
next_cursor = (
|
||||
self.encode_cursor(items[-1])
|
||||
if has_more and items
|
||||
else None
|
||||
)
|
||||
|
||||
return items, next_cursor
|
||||
|
||||
def create_message(
|
||||
self,
|
||||
chat_id: uuid.UUID,
|
||||
sender_id: uuid.UUID,
|
||||
text: str,
|
||||
idempotency_key: str,
|
||||
attachment_ids: list[str] | None = None,
|
||||
) -> Message:
|
||||
text = text.strip()
|
||||
attachment_ids = attachment_ids or []
|
||||
if (not text and not attachment_ids) or len(text) > 4000 or len(attachment_ids) > 10:
|
||||
raise InvalidMessageDataError("Invalid message text")
|
||||
|
||||
# Access is checked for every retry, before consulting the key.
|
||||
if not self.chats.is_member(chat_id, sender_id):
|
||||
raise MessageNotFoundError()
|
||||
|
||||
if len(attachment_ids) != len(set(attachment_ids)):
|
||||
raise InvalidMessageDataError("Duplicate attachment IDs")
|
||||
if attachment_ids:
|
||||
try:
|
||||
parsed_attachment_ids = [uuid.UUID(value) for value in attachment_ids]
|
||||
except (ValueError, TypeError):
|
||||
raise InvalidMessageDataError("Invalid attachment UUID") from None
|
||||
attachments = list(self.db.scalars(select(Attachment).where(Attachment.id.in_(parsed_attachment_ids))).all())
|
||||
if len(attachments) != len(parsed_attachment_ids) or any(a.chat_id != chat_id for a in attachments):
|
||||
raise MessageNotFoundError()
|
||||
if any(a.status != "ready" for a in attachments):
|
||||
raise InvalidMessageDataError("Attachment is not ready")
|
||||
|
||||
method = "POST"
|
||||
path = f"/api/v1/chats/{chat_id}/messages"
|
||||
request_hash = hashlib.sha256(
|
||||
json.dumps({"text": text, "attachment_ids": attachment_ids}, sort_keys=True, separators=(",", ":")).encode("utf-8")
|
||||
).hexdigest()
|
||||
now = datetime.now(timezone.utc)
|
||||
expires_at = now + timedelta(seconds=max(settings.idempotency_ttl_seconds, 86400))
|
||||
scope = (
|
||||
IdempotencyRecord.user_id == sender_id,
|
||||
IdempotencyRecord.method == method,
|
||||
IdempotencyRecord.path == path,
|
||||
IdempotencyRecord.idempotency_key == idempotency_key,
|
||||
)
|
||||
|
||||
try:
|
||||
# An expired key may be reused. Deletion and claim are serialized by
|
||||
# PostgreSQL's unique constraint for the scope.
|
||||
self.db.execute(
|
||||
delete(IdempotencyRecord).where(*scope, IdempotencyRecord.expires_at <= now)
|
||||
)
|
||||
claim = (
|
||||
pg_insert(IdempotencyRecord)
|
||||
.values(
|
||||
id=uuid.uuid4(), user_id=sender_id, method=method, path=path,
|
||||
idempotency_key=idempotency_key, request_hash=request_hash,
|
||||
message_id=None, created_at=now, expires_at=expires_at,
|
||||
)
|
||||
.on_conflict_do_nothing(constraint="uq_idempotency_scope_key")
|
||||
.returning(IdempotencyRecord.id)
|
||||
)
|
||||
claimed_id = self.db.execute(claim).scalar_one_or_none()
|
||||
|
||||
if claimed_id is None:
|
||||
existing = self.db.execute(
|
||||
select(IdempotencyRecord).where(*scope).with_for_update()
|
||||
).scalar_one_or_none()
|
||||
if existing is None:
|
||||
# Rare expiration race: retry once in a fresh transaction.
|
||||
self.db.rollback()
|
||||
return self.create_message(chat_id, sender_id, text, idempotency_key, attachment_ids)
|
||||
if existing.request_hash != request_hash:
|
||||
self.db.rollback()
|
||||
raise IdempotencyConflictError()
|
||||
if existing.message_id is None:
|
||||
raise RuntimeError("Committed idempotency record has no message")
|
||||
message = self.db.get(Message, existing.message_id)
|
||||
if message is None:
|
||||
raise RuntimeError("Idempotency record references a missing message")
|
||||
self.db.commit()
|
||||
return message
|
||||
|
||||
message = self.messages.create(
|
||||
chat_id=chat_id, sender_id=sender_id, text=text, attachment_ids=attachment_ids
|
||||
)
|
||||
self.db.execute(
|
||||
update(IdempotencyRecord)
|
||||
.where(IdempotencyRecord.id == claimed_id)
|
||||
.values(message_id=message.id)
|
||||
)
|
||||
self.db.commit()
|
||||
self.db.refresh(message)
|
||||
self._bump_version(chat_id)
|
||||
return message
|
||||
except IdempotencyConflictError:
|
||||
raise
|
||||
except Exception:
|
||||
self.db.rollback()
|
||||
raise
|
||||
|
||||
def get_history(
|
||||
self,
|
||||
chat_id: uuid.UUID,
|
||||
user_id: uuid.UUID,
|
||||
limit: int = 50,
|
||||
cursor: str | None = None,
|
||||
):
|
||||
if not 1 <= limit <= 100:
|
||||
raise InvalidMessageDataError(
|
||||
"Limit must be between 1 and 100"
|
||||
)
|
||||
|
||||
if not self.chats.is_member(chat_id, user_id):
|
||||
raise MessageNotFoundError()
|
||||
|
||||
cursor_created_at, cursor_id = self.decode_cursor(cursor)
|
||||
|
||||
version = self._get_version(chat_id)
|
||||
|
||||
cache_key = self._cache_key(
|
||||
chat_id,
|
||||
version,
|
||||
limit,
|
||||
cursor,
|
||||
)
|
||||
|
||||
cached = self._read_cache(cache_key)
|
||||
|
||||
if cached is not None:
|
||||
return cached
|
||||
|
||||
if not self._redis_available:
|
||||
return self._load_from_database(
|
||||
chat_id,
|
||||
limit,
|
||||
cursor_created_at,
|
||||
cursor_id,
|
||||
)
|
||||
|
||||
lock_key = f"lock:{cache_key}"
|
||||
lock_token = str(uuid.uuid4())
|
||||
|
||||
try:
|
||||
acquired = self.redis.set(
|
||||
lock_key,
|
||||
lock_token,
|
||||
nx=True,
|
||||
ex=10,
|
||||
)
|
||||
except RedisError:
|
||||
self._redis_available = False
|
||||
return self._load_from_database(
|
||||
chat_id,
|
||||
limit,
|
||||
cursor_created_at,
|
||||
cursor_id,
|
||||
)
|
||||
|
||||
if not acquired:
|
||||
for _ in range(10):
|
||||
time.sleep(0.05)
|
||||
|
||||
cached = self._read_cache(cache_key)
|
||||
|
||||
if cached is not None:
|
||||
return cached
|
||||
|
||||
if not self._redis_available:
|
||||
break
|
||||
|
||||
return self._load_from_database(
|
||||
chat_id,
|
||||
limit,
|
||||
cursor_created_at,
|
||||
cursor_id,
|
||||
)
|
||||
|
||||
try:
|
||||
cached = self._read_cache(cache_key)
|
||||
|
||||
if cached is not None:
|
||||
return cached
|
||||
|
||||
if not self._redis_available:
|
||||
return self._load_from_database(
|
||||
chat_id,
|
||||
limit,
|
||||
cursor_created_at,
|
||||
cursor_id,
|
||||
)
|
||||
|
||||
items, next_cursor = self._load_from_database(
|
||||
chat_id,
|
||||
limit,
|
||||
cursor_created_at,
|
||||
cursor_id,
|
||||
)
|
||||
|
||||
if self._redis_available and self._get_version(chat_id) == version:
|
||||
try:
|
||||
self.redis.setex(
|
||||
cache_key,
|
||||
settings.message_cache_ttl_seconds,
|
||||
self._serialize(items, next_cursor),
|
||||
)
|
||||
except RedisError:
|
||||
self._redis_available = False
|
||||
|
||||
return items, next_cursor
|
||||
|
||||
finally:
|
||||
if self._redis_available:
|
||||
try:
|
||||
self.redis.eval(
|
||||
"""
|
||||
if redis.call('get', KEYS[1]) == ARGV[1]
|
||||
then
|
||||
return redis.call('del', KEYS[1])
|
||||
else
|
||||
return 0
|
||||
end
|
||||
""",
|
||||
1,
|
||||
lock_key,
|
||||
lock_token,
|
||||
)
|
||||
except RedisError:
|
||||
self._redis_available = False
|
||||
@@ -0,0 +1,39 @@
|
||||
import boto3
|
||||
from botocore.config import Config
|
||||
from app.core.config import settings
|
||||
|
||||
|
||||
def client(endpoint_url=None):
|
||||
return boto3.client(
|
||||
"s3", endpoint_url=endpoint_url or settings.s3_endpoint_url,
|
||||
aws_access_key_id=settings.s3_access_key,
|
||||
aws_secret_access_key=settings.s3_secret_key,
|
||||
region_name=settings.s3_region,
|
||||
config=Config(signature_version="s3v4", connect_timeout=3, read_timeout=15, retries={"max_attempts": 2}, s3={"addressing_style": "path"}),
|
||||
)
|
||||
|
||||
|
||||
def ensure_bucket() -> None:
|
||||
s3 = client()
|
||||
try:
|
||||
s3.head_bucket(Bucket=settings.s3_bucket)
|
||||
except Exception:
|
||||
s3.create_bucket(Bucket=settings.s3_bucket)
|
||||
|
||||
|
||||
def presigned_client():
|
||||
return client(endpoint_url=settings.s3_public_endpoint_url)
|
||||
|
||||
|
||||
def create_presigned_upload_url(object_key: str, content_type: str) -> str:
|
||||
s3 = presigned_client()
|
||||
|
||||
return s3.generate_presigned_url(
|
||||
ClientMethod="put_object",
|
||||
Params={
|
||||
"Bucket": settings.s3_bucket,
|
||||
"Key": object_key,
|
||||
"ContentType": content_type,
|
||||
},
|
||||
ExpiresIn=settings.presigned_upload_ttl_seconds,
|
||||
)
|
||||
@@ -0,0 +1,81 @@
|
||||
import re
|
||||
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.orm import Session
|
||||
from pwdlib import PasswordHash
|
||||
|
||||
from app.db.models import User
|
||||
from app.repositories.user_repository import UserRepository
|
||||
|
||||
|
||||
password_hasher = PasswordHash.recommended()
|
||||
|
||||
|
||||
class UsernameAlreadyExistsError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class UserService:
|
||||
|
||||
def __init__(self, db: Session):
|
||||
self.db = db
|
||||
self.users = UserRepository(db)
|
||||
|
||||
def register(
|
||||
self,
|
||||
username: str,
|
||||
display_name: str,
|
||||
password: str,
|
||||
) -> User:
|
||||
|
||||
if not re.fullmatch(r"[a-z0-9_]{3,32}", username):
|
||||
raise ValueError("Invalid username")
|
||||
|
||||
if not display_name or not display_name.strip():
|
||||
raise ValueError("Display name cannot be empty")
|
||||
|
||||
if len(display_name) > 100:
|
||||
raise ValueError("Display name is too long")
|
||||
|
||||
if not 12 <= len(password) <= 128:
|
||||
raise ValueError("Invalid password length")
|
||||
|
||||
if self.users.get_by_username(username) is not None:
|
||||
raise UsernameAlreadyExistsError(
|
||||
"Username already exists"
|
||||
)
|
||||
|
||||
password_hash = password_hasher.hash(password)
|
||||
|
||||
try:
|
||||
user = self.users.create(
|
||||
username=username,
|
||||
display_name=display_name,
|
||||
password_hash=password_hash,
|
||||
)
|
||||
|
||||
self.db.commit()
|
||||
self.db.refresh(user)
|
||||
|
||||
return user
|
||||
|
||||
except IntegrityError as exc:
|
||||
self.db.rollback()
|
||||
|
||||
raise UsernameAlreadyExistsError(
|
||||
"Username already exists"
|
||||
) from exc
|
||||
|
||||
except Exception:
|
||||
self.db.rollback()
|
||||
raise
|
||||
|
||||
def verify_password(
|
||||
self,
|
||||
user: User,
|
||||
password: str,
|
||||
) -> bool:
|
||||
return password_hasher.verify(
|
||||
password,
|
||||
user.password_hash,
|
||||
)
|
||||
+193
@@ -0,0 +1,193 @@
|
||||
import base64
|
||||
import bisect
|
||||
import binascii
|
||||
import json
|
||||
import os
|
||||
import threading
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from typing import Dict, List, Optional, Tuple
|
||||
|
||||
from .models import Chat, Message, User
|
||||
|
||||
|
||||
Key = Tuple[datetime, str]
|
||||
|
||||
|
||||
def _encode_cursor(kind: str, scope: str, item) -> str:
|
||||
payload = {
|
||||
"v": 1,
|
||||
"kind": kind,
|
||||
"scope": scope,
|
||||
"created_at": item.created_at.isoformat(),
|
||||
"id": item.id,
|
||||
}
|
||||
raw = json.dumps(payload, separators=(",", ":")).encode("utf-8")
|
||||
return base64.urlsafe_b64encode(raw).decode("ascii").rstrip("=")
|
||||
|
||||
|
||||
def _decode_cursor(cursor: str, kind: str, scope: str) -> Key:
|
||||
try:
|
||||
if not cursor or len(cursor) > 4096:
|
||||
raise ValueError
|
||||
padded = cursor + "=" * (-len(cursor) % 4)
|
||||
raw = base64.b64decode(padded, altchars=b"-_", validate=True)
|
||||
payload = json.loads(raw.decode("utf-8"))
|
||||
if not isinstance(payload, dict) or payload.get("v") != 1:
|
||||
raise ValueError
|
||||
if payload.get("kind") != kind or payload.get("scope") != scope:
|
||||
raise ValueError
|
||||
item_id = payload.get("id")
|
||||
timestamp = payload.get("created_at")
|
||||
if not isinstance(item_id, str) or not isinstance(timestamp, str):
|
||||
raise ValueError
|
||||
created_at = datetime.fromisoformat(timestamp)
|
||||
if created_at.tzinfo is None:
|
||||
raise ValueError
|
||||
created_at = created_at.astimezone(timezone.utc)
|
||||
return created_at, item_id
|
||||
except (
|
||||
ValueError,
|
||||
TypeError,
|
||||
KeyError,
|
||||
UnicodeDecodeError,
|
||||
json.JSONDecodeError,
|
||||
binascii.Error,
|
||||
):
|
||||
raise ValueError("invalid_request") from None
|
||||
|
||||
|
||||
class InMemoryStore:
|
||||
def __init__(self):
|
||||
self._lock = threading.RLock()
|
||||
self.users: Dict[str, User] = {}
|
||||
self.chats: Dict[str, Chat] = {}
|
||||
self.messages: Dict[str, List[Message]] = {}
|
||||
self.usernames: Dict[str, str] = {}
|
||||
self.message_keys: Dict[str, List[Key]] = {}
|
||||
self.message_index_enabled = os.getenv("MESSAGE_INDEX_ENABLED", "1") != "0"
|
||||
|
||||
def create_user(self, username: str, display_name: str) -> User:
|
||||
with self._lock:
|
||||
if username in self.usernames:
|
||||
raise ValueError("conflict")
|
||||
user_id = str(uuid.uuid4())
|
||||
user = User(
|
||||
id=user_id,
|
||||
username=username,
|
||||
display_name=display_name,
|
||||
created_at=datetime.now(timezone.utc),
|
||||
)
|
||||
self.users[user_id] = user
|
||||
self.usernames[username] = user_id
|
||||
return user
|
||||
|
||||
def get_user(self, user_id: str) -> Optional[User]:
|
||||
with self._lock:
|
||||
return self.users.get(user_id)
|
||||
|
||||
def create_chat(self, title: str, member_ids: List[str], creator_id: str) -> Chat:
|
||||
with self._lock:
|
||||
if creator_id not in member_ids or len(member_ids) != len(set(member_ids)):
|
||||
raise ValueError("invalid_request")
|
||||
if any(member_id not in self.users for member_id in member_ids):
|
||||
raise ValueError("invalid_request")
|
||||
|
||||
chat_id = str(uuid.uuid4())
|
||||
chat = Chat(
|
||||
id=chat_id,
|
||||
title=title,
|
||||
member_ids=list(member_ids),
|
||||
created_at=datetime.now(timezone.utc),
|
||||
)
|
||||
self.chats[chat_id] = chat
|
||||
self.messages[chat_id] = []
|
||||
self.message_keys[chat_id] = []
|
||||
return chat
|
||||
|
||||
def get_chat(self, chat_id: str, user_id: str) -> Optional[Chat]:
|
||||
with self._lock:
|
||||
chat = self.chats.get(chat_id)
|
||||
if chat is None or user_id not in chat.member_ids:
|
||||
return None
|
||||
return chat
|
||||
|
||||
def list_chats(
|
||||
self, user_id: str, limit: int, cursor: Optional[str]
|
||||
) -> Tuple[List[Chat], Optional[str]]:
|
||||
with self._lock:
|
||||
user_chats = [chat for chat in self.chats.values() if user_id in chat.member_ids]
|
||||
user_chats.sort(key=lambda chat: (chat.created_at, chat.id))
|
||||
keys = [(chat.created_at, chat.id) for chat in user_chats]
|
||||
|
||||
start_index = 0
|
||||
if cursor:
|
||||
target_key = _decode_cursor(cursor, kind="chats", scope=user_id)
|
||||
start_index = bisect.bisect_right(keys, target_key)
|
||||
|
||||
page_with_extra = user_chats[start_index : start_index + limit + 1]
|
||||
has_next = len(page_with_extra) > limit
|
||||
page = page_with_extra[:limit]
|
||||
next_cursor = None
|
||||
if has_next and page:
|
||||
next_cursor = _encode_cursor("chats", user_id, page[-1])
|
||||
return page, next_cursor
|
||||
|
||||
def create_message(self, chat_id: str, sender_id: str, text: str) -> Message:
|
||||
with self._lock:
|
||||
chat = self.chats.get(chat_id)
|
||||
if chat is None or sender_id not in chat.member_ids:
|
||||
raise ValueError("not_found")
|
||||
|
||||
message = Message(
|
||||
id=str(uuid.uuid4()),
|
||||
chat_id=chat_id,
|
||||
sender_id=sender_id,
|
||||
text=text,
|
||||
created_at=datetime.now(timezone.utc),
|
||||
)
|
||||
messages = self.messages[chat_id]
|
||||
keys = self.message_keys[chat_id]
|
||||
key = (message.created_at, message.id)
|
||||
|
||||
if self.message_index_enabled:
|
||||
index = bisect.bisect_right(keys, key)
|
||||
keys.insert(index, key)
|
||||
messages.insert(index, message)
|
||||
else:
|
||||
current_keys = [(item.created_at, item.id) for item in messages]
|
||||
index = bisect.bisect_right(current_keys, key)
|
||||
messages.insert(index, message)
|
||||
keys.insert(index, key)
|
||||
return message
|
||||
|
||||
def list_messages(
|
||||
self,
|
||||
chat_id: str,
|
||||
user_id: str,
|
||||
limit: int,
|
||||
cursor: Optional[str],
|
||||
) -> Tuple[List[Message], Optional[str]]:
|
||||
with self._lock:
|
||||
chat = self.chats.get(chat_id)
|
||||
if chat is None or user_id not in chat.member_ids:
|
||||
raise ValueError("not_found")
|
||||
|
||||
messages = self.messages[chat_id]
|
||||
if self.message_index_enabled:
|
||||
keys = self.message_keys[chat_id]
|
||||
else:
|
||||
keys = [(item.created_at, item.id) for item in messages]
|
||||
|
||||
start_index = 0
|
||||
if cursor:
|
||||
target_key = _decode_cursor(cursor, kind="messages", scope=chat_id)
|
||||
start_index = bisect.bisect_right(keys, target_key)
|
||||
|
||||
page_with_extra = messages[start_index : start_index + limit + 1]
|
||||
has_next = len(page_with_extra) > limit
|
||||
page = page_with_extra[:limit]
|
||||
next_cursor = None
|
||||
if has_next and page:
|
||||
next_cursor = _encode_cursor("messages", chat_id, page[-1])
|
||||
return page, next_cursor
|
||||
+183
@@ -0,0 +1,183 @@
|
||||
name: mtusi-lab4
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16.6-bookworm
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
POSTGRES_DB: ${POSTGRES_DB:-messenger}
|
||||
POSTGRES_USER: ${POSTGRES_USER:-messenger}
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U \"$$POSTGRES_USER\" -d \"$$POSTGRES_DB\""]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 15
|
||||
start_period: 10s
|
||||
networks: [backend]
|
||||
|
||||
redis:
|
||||
image: redis:7.4.2-bookworm
|
||||
restart: unless-stopped
|
||||
command: ["redis-server", "--appendonly", "yes", "--appendfsync", "everysec"]
|
||||
volumes:
|
||||
- redis_data:/data
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
networks: [backend]
|
||||
|
||||
minio:
|
||||
image: pgsty/silo:RELEASE.2026-10-26T00-00-00Z
|
||||
command: ["server", "/data", "--address", ":9000", "--console-address", ":9001"]
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:9000:9000"
|
||||
environment:
|
||||
MINIO_ROOT_USER: ${MINIO_ROOT_USER:?Set MINIO_ROOT_USER in .env}
|
||||
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:?Set MINIO_ROOT_PASSWORD in .env}
|
||||
volumes:
|
||||
- minio_data:/data
|
||||
# MinIO is intentionally internal: neither S3 API nor admin console is published on the host.
|
||||
networks: [backend]
|
||||
|
||||
minio-bootstrap:
|
||||
image: pgsty/mc:RELEASE.2026-10-26T00-00-00Z
|
||||
depends_on:
|
||||
minio:
|
||||
condition: service_started
|
||||
entrypoint: ["/bin/sh", "-ec"]
|
||||
command:
|
||||
- |
|
||||
mc --disable-pager alias set local http://minio:9000 "$${MINIO_ROOT_USER}" "$${MINIO_ROOT_PASSWORD}"
|
||||
until mc --disable-pager ready local; do echo 'Waiting for MinIO...'; sleep 2; done
|
||||
mc --disable-pager mb --ignore-existing local/messenger-attachments
|
||||
mc --disable-pager anonymous set none local/messenger-attachments
|
||||
if ! mc --disable-pager admin policy info local messenger-attachments-policy >/dev/null 2>&1; then
|
||||
mc --disable-pager admin policy create local messenger-attachments-policy /policy.json
|
||||
fi
|
||||
if ! mc --disable-pager admin user info local "$${S3_ACCESS_KEY_ID}" >/dev/null 2>&1; then
|
||||
mc --disable-pager admin user add local "$${S3_ACCESS_KEY_ID}" "$${S3_SECRET_ACCESS_KEY}"
|
||||
fi
|
||||
mc --disable-pager admin policy attach local messenger-attachments-policy --user "$${S3_ACCESS_KEY_ID}"
|
||||
environment:
|
||||
MINIO_ROOT_USER: ${MINIO_ROOT_USER:?Set MINIO_ROOT_USER in .env}
|
||||
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:?Set MINIO_ROOT_PASSWORD in .env}
|
||||
S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID:?Set S3_ACCESS_KEY_ID in .env}
|
||||
S3_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY:?Set S3_SECRET_ACCESS_KEY in .env}
|
||||
volumes:
|
||||
- ./infra/minio-policy.json:/policy.json:ro
|
||||
restart: "no"
|
||||
networks: [backend]
|
||||
|
||||
migrate:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
command: ["alembic", "upgrade", "head"]
|
||||
environment:
|
||||
DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-messenger}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB:-messenger}
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
restart: "no"
|
||||
networks: [backend]
|
||||
|
||||
api:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
APP_ENV: lab
|
||||
DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-messenger}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB:-messenger}
|
||||
CACHE_URL: redis://redis:6379/0
|
||||
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
|
||||
MESSAGE_CACHE_TTL_SECONDS: ${MESSAGE_CACHE_TTL_SECONDS:-60}
|
||||
IDEMPOTENCY_TTL_SECONDS: ${IDEMPOTENCY_TTL_SECONDS:-86400}
|
||||
DB_POOL_SIZE: ${DB_POOL_SIZE:-3}
|
||||
DB_MAX_OVERFLOW: ${DB_MAX_OVERFLOW:-2}
|
||||
DB_POOL_TIMEOUT: ${DB_POOL_TIMEOUT:-10}
|
||||
S3_ENDPOINT: http://minio:9000
|
||||
S3_PUBLIC_ENDPOINT_URL: ${S3_PUBLIC_ENDPOINT_URL:-http://localhost:9000}
|
||||
S3_REGION: ${S3_REGION:-us-east-1}
|
||||
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
|
||||
S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID:?Set S3_ACCESS_KEY_ID in .env}
|
||||
S3_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY:?Set S3_SECRET_ACCESS_KEY in .env}
|
||||
MAX_UPLOAD_BYTES: ${MAX_UPLOAD_BYTES:-10485760}
|
||||
MAX_IMAGE_PIXELS: ${MAX_IMAGE_PIXELS:-40000000}
|
||||
ATTACHMENT_CLEANUP_INTERVAL_SECONDS: ${ATTACHMENT_CLEANUP_INTERVAL_SECONDS:-60}
|
||||
ATTACHMENT_CLEANUP_GRACE_SECONDS: ${ATTACHMENT_CLEANUP_GRACE_SECONDS:-600}
|
||||
ATTACHMENT_CLEANUP_BATCH_SIZE: ${ATTACHMENT_CLEANUP_BATCH_SIZE:-100}
|
||||
depends_on:
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
minio-bootstrap:
|
||||
condition: service_completed_successfully
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=2)"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 3
|
||||
start_period: 20s
|
||||
deploy:
|
||||
replicas: 2
|
||||
resources:
|
||||
limits:
|
||||
cpus: "1.0"
|
||||
memory: 512M
|
||||
labels:
|
||||
- traefik.enable=true
|
||||
- traefik.docker.network=mtusi-lab4-edge
|
||||
- traefik.http.routers.messenger.rule=PathPrefix(`/`)
|
||||
- traefik.http.routers.messenger.entrypoints=web
|
||||
- traefik.http.services.messenger.loadbalancer.server.port=8080
|
||||
- traefik.http.services.messenger.loadbalancer.healthcheck.path=/health/ready
|
||||
- traefik.http.services.messenger.loadbalancer.healthcheck.interval=3s
|
||||
- traefik.http.services.messenger.loadbalancer.healthcheck.timeout=2s
|
||||
- traefik.http.services.messenger.loadbalancer.passhostheader=true
|
||||
networks: [backend, edge]
|
||||
|
||||
traefik:
|
||||
image: traefik:v3.6.2
|
||||
restart: unless-stopped
|
||||
command:
|
||||
- --providers.docker=true
|
||||
- --providers.docker.exposedbydefault=false
|
||||
- --providers.docker.network=mtusi-lab4-edge
|
||||
- --entrypoints.web.address=:8080
|
||||
- --ping=true
|
||||
- --ping.entrypoint=ping
|
||||
- --entrypoints.ping.address=:8082
|
||||
- --accesslog=true
|
||||
- --log.level=INFO
|
||||
ports:
|
||||
- "8080:8080"
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "-q", "--spider", "http://127.0.0.1:8082/ping"]
|
||||
interval: 10s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
networks: [edge]
|
||||
|
||||
networks:
|
||||
backend:
|
||||
driver: bridge
|
||||
edge:
|
||||
name: mtusi-lab4-edge
|
||||
driver: bridge
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
redis_data:
|
||||
minio_data:
|
||||
Binary file not shown.
@@ -0,0 +1,14 @@
|
||||
ATTACHMENT FAILOVER TEST
|
||||
|
||||
Mode: failover
|
||||
Observed API instances before failure: 2
|
||||
Successful writes: 87
|
||||
Failed attempts: 4
|
||||
First recovery: 4.597 seconds
|
||||
|
||||
The API replica that handled the attachment upload was stopped.
|
||||
The attachment remained accessible through the other API replica.
|
||||
Downloaded content matched the original bytes.
|
||||
Existing session and confirmed messages were preserved.
|
||||
|
||||
RESULT: PASS
|
||||
@@ -0,0 +1,3 @@
|
||||
method,concurrency,file_size_bytes,successful_uploads,failed_uploads,wall_time_s,throughput_uploads_per_s,peak_api_total_mib,baseline_api_total_mib,peak_delta_api_total_mib,api1_peak_mib,api2_peak_mib,memory_samples,memory_sampling_errors,avg_upload_time_s,max_upload_time_s,api_body_bytes_per_upload,s3_body_bytes_per_upload
|
||||
proxy,10,5769330,10,0,3.9963,2.502,220.2,204.1,16.1,110.2,110.0,4,0,2.3315,2.6514,5769330,0
|
||||
presigned,10,5769330,10,0,4.941,2.024,304.4,220.2,84.2,156.8,147.6,5,0,3.6495,3.9018,160,5769330
|
||||
|
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,15 @@
|
||||
S3 SECURITY TEST
|
||||
|
||||
1. Admin console port:
|
||||
Command: docker compose port minio 9001
|
||||
Result: no port 9001/tcp published.
|
||||
|
||||
2. Anonymous bucket access:
|
||||
Command: curl.exe -i http://localhost:9000/messenger-attachments/
|
||||
Result: HTTP 403 Forbidden
|
||||
S3 error: AccessDenied
|
||||
|
||||
3. Bucket:
|
||||
messenger-attachments
|
||||
|
||||
RESULT: PASS
|
||||
@@ -0,0 +1,25 @@
|
||||
S3 UNAVAILABLE TEST
|
||||
|
||||
1. Stopped S3:
|
||||
docker compose stop minio
|
||||
|
||||
2. Readiness response while S3 was unavailable:
|
||||
HTTP 503
|
||||
{
|
||||
"error": {
|
||||
"code": "unavailable",
|
||||
"message": "Required storage unavailable"
|
||||
}
|
||||
}
|
||||
|
||||
3. Restarted S3:
|
||||
docker compose start minio
|
||||
|
||||
4. Container status:
|
||||
Both API replicas healthy.
|
||||
S3 container running.
|
||||
|
||||
5. Readiness response after recovery:
|
||||
status: ok
|
||||
|
||||
RESULT: PASS
|
||||
Binary file not shown.
@@ -0,0 +1,13 @@
|
||||
method,file,file_size_bytes,repeat,elapsed_total_s,elapsed_api_s,client_to_api_body_bytes,client_to_s3_upload_bytes,api_peak_rss_mib,api_peak_delta_mib
|
||||
proxy,sample.png,111817,1,0.0838,0.0837,111817,0,103.4,0.0
|
||||
proxy,sample.png,111817,2,0.0773,0.0772,111817,0,103.4,0.0
|
||||
proxy,sample.png,111817,3,0.0746,0.0746,111817,0,103.4,0.0
|
||||
presigned,sample.png,111817,1,0.1103,0.084,158,111817,103.8,1.5
|
||||
presigned,sample.png,111817,2,0.1003,0.0556,158,111817,103.8,0.0
|
||||
presigned,sample.png,111817,3,0.1782,0.1082,158,111817,112.7,8.9
|
||||
proxy,sample_large.png,5769330,1,0.2416,0.2415,5769330,0,103.8,1.5
|
||||
proxy,sample_large.png,5769330,2,0.2777,0.2776,5769330,0,105.3,1.5
|
||||
proxy,sample_large.png,5769330,3,0.2806,0.2805,5769330,0,109.2,5.2
|
||||
presigned,sample_large.png,5769330,1,0.3062,0.1403,159,5769330,128.9,19.7
|
||||
presigned,sample_large.png,5769330,2,0.2171,0.1092,159,5769330,116.3,0.0
|
||||
presigned,sample_large.png,5769330,3,0.2459,0.1329,159,5769330,116.3,0.0
|
||||
|
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"s3:GetBucketLocation",
|
||||
"s3:ListBucket",
|
||||
"s3:GetObject",
|
||||
"s3:PutObject",
|
||||
"s3:DeleteObject"
|
||||
],
|
||||
"Resource": [
|
||||
"arn:aws:s3:::messenger-attachments",
|
||||
"arn:aws:s3:::messenger-attachments/*"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
Generic single-database configuration.
|
||||
@@ -0,0 +1,60 @@
|
||||
from logging.config import fileConfig
|
||||
|
||||
from alembic import context
|
||||
from sqlalchemy import engine_from_config, pool
|
||||
|
||||
from app.core.config import settings
|
||||
from app.db.base import Base
|
||||
from app.db import models
|
||||
|
||||
|
||||
config = context.config
|
||||
|
||||
if config.config_file_name is not None:
|
||||
fileConfig(config.config_file_name)
|
||||
|
||||
config.set_main_option(
|
||||
"sqlalchemy.url",
|
||||
settings.database_url.replace("%", "%%"),
|
||||
)
|
||||
|
||||
target_metadata = Base.metadata
|
||||
|
||||
|
||||
def run_migrations_offline() -> None:
|
||||
url = settings.database_url
|
||||
|
||||
context.configure(
|
||||
url=url,
|
||||
target_metadata=target_metadata,
|
||||
literal_binds=True,
|
||||
dialect_opts={"paramstyle": "named"},
|
||||
compare_type=True,
|
||||
)
|
||||
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
|
||||
|
||||
def run_migrations_online() -> None:
|
||||
connectable = engine_from_config(
|
||||
config.get_section(config.config_ini_section, {}),
|
||||
prefix="sqlalchemy.",
|
||||
poolclass=pool.NullPool,
|
||||
)
|
||||
|
||||
with connectable.connect() as connection:
|
||||
context.configure(
|
||||
connection=connection,
|
||||
target_metadata=target_metadata,
|
||||
compare_type=True,
|
||||
)
|
||||
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
|
||||
|
||||
if context.is_offline_mode():
|
||||
run_migrations_offline()
|
||||
else:
|
||||
run_migrations_online()
|
||||
@@ -0,0 +1,26 @@
|
||||
"""${message}
|
||||
|
||||
Revision ID: ${up_revision}
|
||||
Revises: ${down_revision | comma,n}
|
||||
Create Date: ${create_date}
|
||||
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
${imports if imports else ""}
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = ${repr(up_revision)}
|
||||
down_revision: Union[str, None] = ${repr(down_revision)}
|
||||
branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)}
|
||||
depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)}
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
${upgrades if upgrades else "pass"}
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
${downgrades if downgrades else "pass"}
|
||||
@@ -0,0 +1,79 @@
|
||||
"""create initial database schema
|
||||
|
||||
Revision ID: 21ad89656327
|
||||
Revises:
|
||||
Create Date: 2026-11-04 17:31:39.336368
|
||||
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = '21ad89656327'
|
||||
down_revision: Union[str, None] = None
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
op.create_table('chats',
|
||||
sa.Column('id', sa.UUID(), nullable=False),
|
||||
sa.Column('title', sa.String(length=100), nullable=False),
|
||||
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
|
||||
sa.PrimaryKeyConstraint('id')
|
||||
)
|
||||
op.create_table('users',
|
||||
sa.Column('id', sa.UUID(), nullable=False),
|
||||
sa.Column('username', sa.String(length=32), nullable=False),
|
||||
sa.Column('display_name', sa.String(length=100), nullable=False),
|
||||
sa.Column('password_hash', sa.String(length=255), nullable=False),
|
||||
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
|
||||
sa.PrimaryKeyConstraint('id')
|
||||
)
|
||||
op.create_index(op.f('ix_users_username'), 'users', ['username'], unique=True)
|
||||
op.create_table('chat_members',
|
||||
sa.Column('chat_id', sa.UUID(), nullable=False),
|
||||
sa.Column('user_id', sa.UUID(), nullable=False),
|
||||
sa.ForeignKeyConstraint(['chat_id'], ['chats.id'], ondelete='CASCADE'),
|
||||
sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'),
|
||||
sa.PrimaryKeyConstraint('chat_id', 'user_id')
|
||||
)
|
||||
op.create_table('messages',
|
||||
sa.Column('id', sa.UUID(), nullable=False),
|
||||
sa.Column('chat_id', sa.UUID(), nullable=False),
|
||||
sa.Column('sender_id', sa.UUID(), nullable=False),
|
||||
sa.Column('text', sa.Text(), nullable=False),
|
||||
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(['chat_id'], ['chats.id'], ondelete='CASCADE'),
|
||||
sa.ForeignKeyConstraint(['sender_id'], ['users.id'], ),
|
||||
sa.PrimaryKeyConstraint('id')
|
||||
)
|
||||
op.create_index('ix_messages_chat_created_id', 'messages', ['chat_id', 'created_at', 'id'], unique=False)
|
||||
op.create_table('sessions',
|
||||
sa.Column('id', sa.UUID(), nullable=False),
|
||||
sa.Column('token_hash', sa.String(length=64), nullable=False),
|
||||
sa.Column('user_id', sa.UUID(), nullable=False),
|
||||
sa.Column('expires_at', sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'),
|
||||
sa.PrimaryKeyConstraint('id')
|
||||
)
|
||||
op.create_index(op.f('ix_sessions_token_hash'), 'sessions', ['token_hash'], unique=True)
|
||||
# ### end Alembic commands ###
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
op.drop_index(op.f('ix_sessions_token_hash'), table_name='sessions')
|
||||
op.drop_table('sessions')
|
||||
op.drop_index('ix_messages_chat_created_id', table_name='messages')
|
||||
op.drop_table('messages')
|
||||
op.drop_table('chat_members')
|
||||
op.drop_index(op.f('ix_users_username'), table_name='users')
|
||||
op.drop_table('users')
|
||||
op.drop_table('chats')
|
||||
# ### end Alembic commands ###
|
||||
@@ -0,0 +1,37 @@
|
||||
"""add attachment metadata"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from typing import Sequence, Union
|
||||
|
||||
revision: str = "4d9a1f2c7b10"
|
||||
down_revision: Union[str, None] = "8c3d9f4a21b7"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column("messages", sa.Column("attachment_ids", sa.JSON(), nullable=False, server_default="[]"))
|
||||
op.create_table(
|
||||
"attachments",
|
||||
sa.Column("id", sa.UUID(), nullable=False),
|
||||
sa.Column("chat_id", sa.UUID(), nullable=False),
|
||||
sa.Column("owner_id", sa.UUID(), nullable=False),
|
||||
sa.Column("filename", sa.String(255), nullable=False),
|
||||
sa.Column("content_type", sa.String(32), nullable=False),
|
||||
sa.Column("size_bytes", sa.Integer(), nullable=False),
|
||||
sa.Column("sha256", sa.String(64), nullable=False),
|
||||
sa.Column("object_key", sa.String(512), nullable=False),
|
||||
sa.Column("status", sa.String(16), nullable=False),
|
||||
sa.Column("error_code", sa.String(64), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(["chat_id"], ["chats.id"], ondelete="CASCADE"),
|
||||
sa.ForeignKeyConstraint(["owner_id"], ["users.id"], ondelete="CASCADE"),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint("object_key"),
|
||||
)
|
||||
op.create_index("ix_attachments_chat_id", "attachments", ["chat_id"])
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("ix_attachments_chat_id", table_name="attachments")
|
||||
op.drop_table("attachments")
|
||||
op.drop_column("messages", "attachment_ids")
|
||||
@@ -0,0 +1,39 @@
|
||||
"""add shared PostgreSQL idempotency records
|
||||
|
||||
Revision ID: 8c3d9f4a21b7
|
||||
Revises: 21ad89656327
|
||||
Create Date: 2026-11-05
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision: str = "8c3d9f4a21b7"
|
||||
down_revision: Union[str, None] = "21ad89656327"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"idempotency_records",
|
||||
sa.Column("id", sa.UUID(), nullable=False),
|
||||
sa.Column("user_id", sa.UUID(), nullable=False),
|
||||
sa.Column("method", sa.String(length=10), nullable=False),
|
||||
sa.Column("path", sa.String(length=255), nullable=False),
|
||||
sa.Column("idempotency_key", sa.String(length=128), nullable=False),
|
||||
sa.Column("request_hash", sa.String(length=64), nullable=False),
|
||||
sa.Column("message_id", sa.UUID(), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(["message_id"], ["messages.id"], ondelete="CASCADE"),
|
||||
sa.ForeignKeyConstraint(["user_id"], ["users.id"], ondelete="CASCADE"),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint("user_id", "method", "path", "idempotency_key", name="uq_idempotency_scope_key"),
|
||||
)
|
||||
op.create_index("ix_idempotency_records_expires_at", "idempotency_records", ["expires_at"], unique=False)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("ix_idempotency_records_expires_at", table_name="idempotency_records")
|
||||
op.drop_table("idempotency_records")
|
||||
@@ -0,0 +1,78 @@
|
||||
"""add attachment uploads
|
||||
|
||||
Revision ID: a7c2e91d5f40
|
||||
Revises: 4d9a1f2c7b10
|
||||
"""
|
||||
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision: str = "a7c2e91d5f40"
|
||||
down_revision: Union[str, None] = "4d9a1f2c7b10"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"attachment_uploads",
|
||||
sa.Column("id", sa.UUID(), nullable=False),
|
||||
sa.Column("chat_id", sa.UUID(), nullable=False),
|
||||
sa.Column("owner_id", sa.UUID(), nullable=False),
|
||||
sa.Column("filename", sa.String(length=255), nullable=False),
|
||||
sa.Column("content_type", sa.String(length=32), nullable=False),
|
||||
sa.Column("expected_size_bytes", sa.Integer(), nullable=False),
|
||||
sa.Column("expected_sha256", sa.String(length=64), nullable=False),
|
||||
sa.Column("object_key", sa.String(length=512), nullable=False),
|
||||
sa.Column("status", sa.String(length=16), nullable=False),
|
||||
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("completed_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.ForeignKeyConstraint(
|
||||
["chat_id"],
|
||||
["chats.id"],
|
||||
ondelete="CASCADE",
|
||||
),
|
||||
sa.ForeignKeyConstraint(
|
||||
["owner_id"],
|
||||
["users.id"],
|
||||
ondelete="CASCADE",
|
||||
),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint("object_key"),
|
||||
)
|
||||
|
||||
op.create_index(
|
||||
"ix_attachment_uploads_chat_id",
|
||||
"attachment_uploads",
|
||||
["chat_id"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_attachment_uploads_owner_id",
|
||||
"attachment_uploads",
|
||||
["owner_id"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_attachment_uploads_expires_at",
|
||||
"attachment_uploads",
|
||||
["expires_at"],
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index(
|
||||
"ix_attachment_uploads_expires_at",
|
||||
table_name="attachment_uploads",
|
||||
)
|
||||
op.drop_index(
|
||||
"ix_attachment_uploads_owner_id",
|
||||
table_name="attachment_uploads",
|
||||
)
|
||||
op.drop_index(
|
||||
"ix_attachment_uploads_chat_id",
|
||||
table_name="attachment_uploads",
|
||||
)
|
||||
op.drop_table("attachment_uploads")
|
||||
@@ -0,0 +1,11 @@
|
||||
fastapi==0.115.6
|
||||
uvicorn[standard]==0.34.0
|
||||
pydantic==2.10.4
|
||||
pydantic-settings==2.7.1
|
||||
SQLAlchemy==2.0.36
|
||||
psycopg[binary]==3.2.3
|
||||
alembic==1.14.0
|
||||
redis==5.2.1
|
||||
pwdlib[argon2]==0.2.1
|
||||
boto3==1.35.99
|
||||
Pillow==11.0.0
|
||||
@@ -0,0 +1,30 @@
|
||||
import os
|
||||
import runpy
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
bash = r"C:\Program Files\Git\bin\bash.exe"
|
||||
original_popen = subprocess.Popen
|
||||
|
||||
def compatible_popen(args, **kwargs):
|
||||
if isinstance(args, (list, tuple)) and len(args) == 1:
|
||||
args = [bash, args[0]]
|
||||
return original_popen(args, **kwargs)
|
||||
|
||||
subprocess.Popen = compatible_popen
|
||||
sys.path.insert(0, os.path.abspath("tests"))
|
||||
|
||||
mode = sys.argv[1] if len(sys.argv) > 1 else "failover"
|
||||
|
||||
if mode not in ("persistence", "failover"):
|
||||
raise SystemExit("Mode must be persistence or failover")
|
||||
|
||||
sys.argv = [
|
||||
"tests/resilience.py",
|
||||
"--mode", mode,
|
||||
"--action-script",
|
||||
"scripts/restart.sh" if mode == "persistence" else "scripts/stop-one.sh",
|
||||
"--base-url", "http://localhost:8080",
|
||||
]
|
||||
|
||||
runpy.run_path("tests/resilience.py", run_name="__main__")
|
||||
@@ -0,0 +1,52 @@
|
||||
|
||||
#!/usr/bin/env python3
|
||||
"""Create charts from real Track 1 summary.csv data; does not invent measurements."""
|
||||
|
||||
import csv
|
||||
from pathlib import Path
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
root = Path(__file__).resolve().parents[1]
|
||||
source = root / 'evidence' / 'scaling' / 'summary.csv'
|
||||
|
||||
if not source.exists():
|
||||
raise SystemExit('Run scripts/run-scaling-experiment.py first')
|
||||
|
||||
with source.open(encoding='utf-8', newline='') as f:
|
||||
rows = list(csv.DictReader(f))
|
||||
|
||||
if not rows:
|
||||
raise SystemExit('summary.csv is empty')
|
||||
|
||||
out = source.parent
|
||||
|
||||
for key, title, ylabel, filename in [
|
||||
('successful_rps', 'Track 1: successful throughput',
|
||||
'Successful requests/sec', 'throughput.png'),
|
||||
('p95_ms', 'Track 1: p95 latency',
|
||||
'Milliseconds', 'p95-latency.png'),
|
||||
('error_ratio', 'Track 1: error ratio',
|
||||
'Error ratio', 'errors.png'),
|
||||
]:
|
||||
labels = [r['scenario'] for r in rows]
|
||||
values = [float(r[key]) if r[key] else 0 for r in rows]
|
||||
|
||||
plt.figure(figsize=(11, 5))
|
||||
plt.bar(labels, values)
|
||||
|
||||
if key == 'error_ratio':
|
||||
plt.ylim(0, 0.01)
|
||||
for i, value in enumerate(values):
|
||||
plt.text(
|
||||
i, 0.0002, f'{value:.1%}',
|
||||
ha='center', va='bottom'
|
||||
)
|
||||
|
||||
plt.title(title)
|
||||
plt.ylabel(ylabel)
|
||||
plt.xticks(rotation=25, ha='right')
|
||||
plt.tight_layout()
|
||||
plt.savefig(out / filename, dpi=160)
|
||||
plt.close()
|
||||
|
||||
print('Charts written to', out)
|
||||
@@ -0,0 +1,304 @@
|
||||
import csv
|
||||
import hashlib
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
import urllib.request
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
|
||||
from client import Client, check
|
||||
|
||||
BASE_URL = "http://localhost:8080"
|
||||
FIXTURE = ROOT / "tests" / "fixtures" / "sample_large.png"
|
||||
API_CONTAINERS = ("mtusi-lab4-api-1", "mtusi-lab4-api-2")
|
||||
CONCURRENCY = 10
|
||||
OUTPUT = ROOT / "evidence" / "concurrent-upload-results.csv"
|
||||
|
||||
|
||||
def parse_mib(value):
|
||||
units = {
|
||||
"B": 1 / (1024 * 1024),
|
||||
"KiB": 1 / 1024,
|
||||
"MiB": 1,
|
||||
"GiB": 1024,
|
||||
"Ki": 1 / 1024,
|
||||
"Mi": 1,
|
||||
"Gi": 1024,
|
||||
"kB": 1000 / (1024 * 1024),
|
||||
"MB": 1000 * 1000 / (1024 * 1024),
|
||||
"GB": 1000 * 1000 * 1000 / (1024 * 1024),
|
||||
}
|
||||
for unit in sorted(units, key=len, reverse=True):
|
||||
if value.endswith(unit):
|
||||
return float(value[:-len(unit)].strip()) * units[unit]
|
||||
raise ValueError(f"Unknown Docker memory unit: {value}")
|
||||
|
||||
|
||||
def docker_memory():
|
||||
result = subprocess.run(
|
||||
[
|
||||
"docker", "stats", "--no-stream",
|
||||
"--format", "{{.Name}}|{{.MemUsage}}",
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
values = {}
|
||||
for line in result.stdout.splitlines():
|
||||
if "|" not in line:
|
||||
continue
|
||||
name, usage = line.split("|", 1)
|
||||
if name in API_CONTAINERS:
|
||||
values[name] = parse_mib(usage.split("/", 1)[0].strip())
|
||||
|
||||
missing = set(API_CONTAINERS) - set(values)
|
||||
if missing:
|
||||
raise RuntimeError("Missing API containers: " + ", ".join(missing))
|
||||
return values
|
||||
|
||||
|
||||
class MemoryMonitor:
|
||||
def __init__(self):
|
||||
self.running = False
|
||||
self.samples = []
|
||||
self.thread = None
|
||||
self.errors = []
|
||||
|
||||
def sample(self):
|
||||
while self.running:
|
||||
try:
|
||||
values = docker_memory()
|
||||
self.samples.append(values)
|
||||
except Exception as exc:
|
||||
self.errors.append(type(exc).__name__)
|
||||
time.sleep(0.1)
|
||||
|
||||
def __enter__(self):
|
||||
self.baseline = docker_memory()
|
||||
self.samples = [dict(self.baseline)]
|
||||
self.running = True
|
||||
self.thread = threading.Thread(target=self.sample, daemon=True)
|
||||
self.thread.start()
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc_value, traceback):
|
||||
self.running = False
|
||||
self.thread.join(timeout=5)
|
||||
try:
|
||||
self.samples.append(docker_memory())
|
||||
except Exception as exc:
|
||||
self.errors.append(type(exc).__name__)
|
||||
|
||||
def summary(self):
|
||||
peak_by_container = {
|
||||
name: max(sample[name] for sample in self.samples)
|
||||
for name in API_CONTAINERS
|
||||
}
|
||||
total_samples = [
|
||||
sum(sample[name] for name in API_CONTAINERS)
|
||||
for sample in self.samples
|
||||
]
|
||||
baseline_total = sum(self.baseline.values())
|
||||
peak_total = max(total_samples)
|
||||
return {
|
||||
"baseline_total_mib": round(baseline_total, 2),
|
||||
"peak_total_mib": round(peak_total, 2),
|
||||
"peak_delta_total_mib": round(peak_total - baseline_total, 2),
|
||||
"api1_peak_mib": round(peak_by_container[API_CONTAINERS[0]], 2),
|
||||
"api2_peak_mib": round(peak_by_container[API_CONTAINERS[1]], 2),
|
||||
"samples": len(self.samples),
|
||||
"sampling_errors": len(self.errors),
|
||||
}
|
||||
|
||||
|
||||
def response_json(response):
|
||||
return json.loads(response["bytes"].decode("utf-8"))
|
||||
|
||||
|
||||
def upload_one(index, method, actor, chat, content, digest, barrier):
|
||||
client = Client(BASE_URL)
|
||||
barrier.wait(timeout=30)
|
||||
started = time.monotonic()
|
||||
|
||||
if method == "proxy":
|
||||
response = client.call(
|
||||
"POST",
|
||||
f"/api/v1/chats/{chat['id']}/attachments"
|
||||
f"?filename=parallel-{index}.png",
|
||||
201,
|
||||
actor=actor,
|
||||
raw=content,
|
||||
headers={"Content-Type": "image/png"},
|
||||
)
|
||||
attachment = response_json(response)
|
||||
api_elapsed = response["elapsed"]
|
||||
api_body_bytes = len(content)
|
||||
s3_body_bytes = 0
|
||||
|
||||
else:
|
||||
body = {
|
||||
"filename": f"parallel-{index}.png",
|
||||
"content_type": "image/png",
|
||||
"size_bytes": len(content),
|
||||
"sha256": digest,
|
||||
}
|
||||
create_response = client.call(
|
||||
"POST",
|
||||
f"/api/v1/chats/{chat['id']}/attachments/presigned",
|
||||
201,
|
||||
actor=actor,
|
||||
body=body,
|
||||
)
|
||||
created = response_json(create_response)
|
||||
|
||||
request = urllib.request.Request(
|
||||
created["upload_url"],
|
||||
data=content,
|
||||
headers=created["headers"],
|
||||
method="PUT",
|
||||
)
|
||||
with urllib.request.urlopen(request, timeout=90) as response:
|
||||
check(response.status in (200, 201, 204), "S3 PUT failed")
|
||||
|
||||
complete_response = client.call(
|
||||
"POST",
|
||||
f"/api/v1/attachments/uploads/{created['upload_id']}/complete",
|
||||
200,
|
||||
actor=actor,
|
||||
)
|
||||
attachment = response_json(complete_response)
|
||||
api_elapsed = create_response["elapsed"] + complete_response["elapsed"]
|
||||
api_body_bytes = len(json.dumps(body).encode("utf-8"))
|
||||
s3_body_bytes = len(content)
|
||||
|
||||
elapsed = time.monotonic() - started
|
||||
|
||||
check(attachment["status"] == "ready", "attachment is not ready")
|
||||
check(attachment["size_bytes"] == len(content), "wrong attachment size")
|
||||
check(attachment["sha256"] == digest, "wrong attachment checksum")
|
||||
|
||||
return {
|
||||
"worker": index,
|
||||
"elapsed_total_s": round(elapsed, 4),
|
||||
"elapsed_api_s": round(api_elapsed, 4),
|
||||
"client_to_api_body_bytes": api_body_bytes,
|
||||
"client_to_s3_upload_bytes": s3_body_bytes,
|
||||
"attachment_id": attachment["id"],
|
||||
}
|
||||
|
||||
|
||||
def run_method(method, actor, chat, content, digest):
|
||||
print(f"\nStarting {CONCURRENCY} simultaneous {method} uploads...")
|
||||
barrier = threading.Barrier(CONCURRENCY)
|
||||
results = []
|
||||
errors = []
|
||||
|
||||
with MemoryMonitor() as memory:
|
||||
started = time.monotonic()
|
||||
with ThreadPoolExecutor(max_workers=CONCURRENCY) as executor:
|
||||
futures = [
|
||||
executor.submit(
|
||||
upload_one, index, method, actor, chat,
|
||||
content, digest, barrier,
|
||||
)
|
||||
for index in range(1, CONCURRENCY + 1)
|
||||
]
|
||||
|
||||
for future in as_completed(futures):
|
||||
try:
|
||||
results.append(future.result())
|
||||
except Exception as exc:
|
||||
errors.append(f"{type(exc).__name__}: {exc}")
|
||||
|
||||
wall_time = time.monotonic() - started
|
||||
|
||||
mem = memory.summary()
|
||||
successful = len(results)
|
||||
print(
|
||||
f"{method}: success={successful}/{CONCURRENCY}, "
|
||||
f"errors={len(errors)}, wall={wall_time:.3f}s, "
|
||||
f"peak API RAM={mem['peak_total_mib']} MiB, "
|
||||
f"delta={mem['peak_delta_total_mib']} MiB, "
|
||||
f"samples={mem['samples']}"
|
||||
)
|
||||
|
||||
for error in errors:
|
||||
print("ERROR:", error)
|
||||
|
||||
return {
|
||||
"method": method,
|
||||
"concurrency": CONCURRENCY,
|
||||
"file_size_bytes": len(content),
|
||||
"successful_uploads": successful,
|
||||
"failed_uploads": len(errors),
|
||||
"wall_time_s": round(wall_time, 4),
|
||||
"throughput_uploads_per_s": round(successful / wall_time, 3),
|
||||
"peak_api_total_mib": mem["peak_total_mib"],
|
||||
"baseline_api_total_mib": mem["baseline_total_mib"],
|
||||
"peak_delta_api_total_mib": mem["peak_delta_total_mib"],
|
||||
"api1_peak_mib": mem["api1_peak_mib"],
|
||||
"api2_peak_mib": mem["api2_peak_mib"],
|
||||
"memory_samples": mem["samples"],
|
||||
"memory_sampling_errors": mem["sampling_errors"],
|
||||
"avg_upload_time_s": round(
|
||||
sum(item["elapsed_total_s"] for item in results) / successful, 4
|
||||
) if successful else None,
|
||||
"max_upload_time_s": max(
|
||||
(item["elapsed_total_s"] for item in results), default=None
|
||||
),
|
||||
"api_body_bytes_per_upload": (
|
||||
results[0]["client_to_api_body_bytes"] if results else None
|
||||
),
|
||||
"s3_body_bytes_per_upload": (
|
||||
results[0]["client_to_s3_upload_bytes"] if results else None
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
if not FIXTURE.exists():
|
||||
raise FileNotFoundError(FIXTURE)
|
||||
|
||||
content = FIXTURE.read_bytes()
|
||||
if len(content) > 10 * 1024 * 1024:
|
||||
raise RuntimeError("Fixture exceeds 10 MiB")
|
||||
|
||||
digest = hashlib.sha256(content).hexdigest()
|
||||
|
||||
with urllib.request.urlopen(BASE_URL + "/health/ready", timeout=10) as response:
|
||||
check(response.status == 200, "API is not ready")
|
||||
|
||||
# Prepare identities and chat before measuring concurrent uploads.
|
||||
setup = Client(BASE_URL)
|
||||
alice, _, _, chat = setup.bootstrap()
|
||||
|
||||
rows = []
|
||||
for method in ("proxy", "presigned"):
|
||||
rows.append(run_method(method, alice, chat, content, digest))
|
||||
|
||||
OUTPUT.parent.mkdir(parents=True, exist_ok=True)
|
||||
with OUTPUT.open("w", newline="", encoding="utf-8-sig") as file:
|
||||
writer = csv.DictWriter(file, fieldnames=list(rows[0].keys()))
|
||||
writer.writeheader()
|
||||
writer.writerows(rows)
|
||||
|
||||
print("\nResults:", OUTPUT)
|
||||
print("Fixture:", FIXTURE.name)
|
||||
print("File size:", len(content), "bytes")
|
||||
print("SHA-256:", digest)
|
||||
|
||||
if any(row["successful_uploads"] != CONCURRENCY for row in rows):
|
||||
raise SystemExit("FAIL: at least one concurrent upload failed")
|
||||
|
||||
print("PASS: 10 concurrent uploads completed for both methods")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,375 @@
|
||||
import csv
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
import urllib.request
|
||||
import hashlib
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
|
||||
from client import Client, check
|
||||
|
||||
|
||||
BASE_URL = "http://localhost:8080"
|
||||
FIXTURES = [
|
||||
ROOT / "tests" / "fixtures" / "sample.png",
|
||||
ROOT / "tests" / "fixtures" / "sample_large.png",
|
||||
]
|
||||
|
||||
API_CONTAINERS = [
|
||||
"mtusi-lab4-api-1",
|
||||
"mtusi-lab4-api-2",
|
||||
]
|
||||
|
||||
REPEATS = 3
|
||||
OUTPUT = ROOT / "evidence" / "track1-results.csv"
|
||||
|
||||
|
||||
def docker_memory():
|
||||
"""Return current memory usage in MiB for both API containers."""
|
||||
result = subprocess.run(
|
||||
[
|
||||
"docker",
|
||||
"stats",
|
||||
"--no-stream",
|
||||
"--format",
|
||||
"{{.Name}}|{{.MemUsage}}",
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
|
||||
values = {}
|
||||
|
||||
for line in result.stdout.splitlines():
|
||||
if "|" not in line:
|
||||
continue
|
||||
|
||||
name, usage = line.split("|", 1)
|
||||
|
||||
if name not in API_CONTAINERS:
|
||||
continue
|
||||
|
||||
memory_value = usage.split("/", 1)[0].strip()
|
||||
values[name] = parse_memory_mib(memory_value)
|
||||
|
||||
missing = set(API_CONTAINERS) - set(values)
|
||||
|
||||
if missing:
|
||||
raise RuntimeError(
|
||||
"Could not read memory for containers: "
|
||||
+ ", ".join(sorted(missing))
|
||||
)
|
||||
|
||||
return values
|
||||
|
||||
|
||||
|
||||
def parse_memory_mib(value):
|
||||
"""Convert Docker memory values to MiB."""
|
||||
value = value.strip()
|
||||
|
||||
units = {
|
||||
"B": 1 / (1024 * 1024),
|
||||
"KiB": 1 / 1024,
|
||||
"MiB": 1,
|
||||
"GiB": 1024,
|
||||
"kB": 1000 / (1024 * 1024),
|
||||
"MB": 1000 * 1000 / (1024 * 1024),
|
||||
"GB": 1000 * 1000 * 1000 / (1024 * 1024),
|
||||
"Ki": 1 / 1024,
|
||||
"Mi": 1,
|
||||
"Gi": 1024,
|
||||
}
|
||||
|
||||
for unit in sorted(units, key=len, reverse=True):
|
||||
if value.endswith(unit):
|
||||
number = value[:-len(unit)].strip()
|
||||
return float(number) * units[unit]
|
||||
|
||||
raise ValueError(f"Unknown Docker memory unit: {value}")
|
||||
|
||||
|
||||
|
||||
class MemoryMonitor:
|
||||
"""Sample API container memory while one upload is running."""
|
||||
|
||||
def __init__(self):
|
||||
self.baseline = {}
|
||||
self.peak = {}
|
||||
self.running = False
|
||||
self.thread = None
|
||||
|
||||
def _sample(self):
|
||||
while self.running:
|
||||
try:
|
||||
current = docker_memory()
|
||||
|
||||
for name, value in current.items():
|
||||
self.peak[name] = max(
|
||||
self.peak.get(name, value),
|
||||
value,
|
||||
)
|
||||
except Exception as exc:
|
||||
print(
|
||||
"WARNING: memory sampling failed:",
|
||||
type(exc).__name__,
|
||||
)
|
||||
|
||||
time.sleep(0.1)
|
||||
|
||||
def __enter__(self):
|
||||
self.baseline = docker_memory()
|
||||
self.peak = dict(self.baseline)
|
||||
self.running = True
|
||||
self.thread = threading.Thread(
|
||||
target=self._sample,
|
||||
daemon=True,
|
||||
)
|
||||
self.thread.start()
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc_value, traceback):
|
||||
self.running = False
|
||||
|
||||
if self.thread:
|
||||
self.thread.join(timeout=3)
|
||||
|
||||
try:
|
||||
current = docker_memory()
|
||||
|
||||
for name, value in current.items():
|
||||
self.peak[name] = max(
|
||||
self.peak.get(name, value),
|
||||
value,
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
@property
|
||||
def peak_rss_mib(self):
|
||||
return max(self.peak.values())
|
||||
|
||||
@property
|
||||
def peak_delta_mib(self):
|
||||
return max(
|
||||
self.peak[name] - self.baseline[name]
|
||||
for name in API_CONTAINERS
|
||||
)
|
||||
|
||||
|
||||
def response_json(response):
|
||||
return json.loads(response["bytes"].decode("utf-8"))
|
||||
|
||||
|
||||
def proxy_upload(client, actor, chat, content):
|
||||
"""Upload a file through the API proxy endpoint."""
|
||||
started = time.monotonic()
|
||||
|
||||
response = client.call(
|
||||
"POST",
|
||||
f"/api/v1/chats/{chat['id']}/attachments?filename=benchmark.png",
|
||||
201,
|
||||
actor=actor,
|
||||
raw=content,
|
||||
headers={"Content-Type": "image/png"},
|
||||
)
|
||||
|
||||
elapsed = time.monotonic() - started
|
||||
attachment = response_json(response)
|
||||
|
||||
check(attachment["status"] == "ready", "proxy attachment is not ready")
|
||||
check(
|
||||
attachment["size_bytes"] == len(content),
|
||||
"proxy attachment size mismatch",
|
||||
)
|
||||
check(
|
||||
attachment["sha256"] == hashlib.sha256(content).hexdigest(),
|
||||
"proxy attachment checksum mismatch",
|
||||
)
|
||||
|
||||
return {
|
||||
"elapsed_total_s": elapsed,
|
||||
"elapsed_api_s": response["elapsed"],
|
||||
"api_request_body_bytes": len(content),
|
||||
"s3_upload_bytes": 0,
|
||||
}
|
||||
|
||||
|
||||
def presigned_upload(client, actor, chat, content):
|
||||
"""Upload a file directly to S3 using a presigned URL."""
|
||||
digest = hashlib.sha256(content).hexdigest()
|
||||
|
||||
create_body = {
|
||||
"filename": "benchmark.png",
|
||||
"content_type": "image/png",
|
||||
"size_bytes": len(content),
|
||||
"sha256": digest,
|
||||
}
|
||||
|
||||
create_body_bytes = len(
|
||||
json.dumps(create_body, ensure_ascii=False).encode("utf-8")
|
||||
)
|
||||
|
||||
started = time.monotonic()
|
||||
|
||||
create_response = client.call(
|
||||
"POST",
|
||||
f"/api/v1/chats/{chat['id']}/attachments/presigned",
|
||||
201,
|
||||
actor=actor,
|
||||
body=create_body,
|
||||
)
|
||||
|
||||
created = response_json(create_response)
|
||||
|
||||
check(created["method"] == "PUT", "unexpected upload method")
|
||||
|
||||
s3_request = urllib.request.Request(
|
||||
created["upload_url"],
|
||||
data=content,
|
||||
headers=created["headers"],
|
||||
method="PUT",
|
||||
)
|
||||
|
||||
with urllib.request.urlopen(s3_request, timeout=60) as response:
|
||||
check(
|
||||
response.status in (200, 201, 204),
|
||||
"direct S3 upload failed",
|
||||
)
|
||||
|
||||
complete_response = client.call(
|
||||
"POST",
|
||||
f"/api/v1/attachments/uploads/{created['upload_id']}/complete",
|
||||
200,
|
||||
actor=actor,
|
||||
)
|
||||
|
||||
attachment = response_json(complete_response)
|
||||
|
||||
elapsed = time.monotonic() - started
|
||||
|
||||
check(attachment["status"] == "ready", "presigned attachment is not ready")
|
||||
check(
|
||||
attachment["size_bytes"] == len(content),
|
||||
"presigned attachment size mismatch",
|
||||
)
|
||||
check(
|
||||
attachment["sha256"] == digest,
|
||||
"presigned attachment checksum mismatch",
|
||||
)
|
||||
|
||||
return {
|
||||
"elapsed_total_s": elapsed,
|
||||
"elapsed_api_s": (
|
||||
create_response["elapsed"] + complete_response["elapsed"]
|
||||
),
|
||||
"api_request_body_bytes": create_body_bytes,
|
||||
"s3_upload_bytes": len(content),
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
OUTPUT.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
for path in FIXTURES:
|
||||
if not path.exists():
|
||||
raise FileNotFoundError(path)
|
||||
|
||||
if path.stat().st_size > 10 * 1024 * 1024:
|
||||
raise RuntimeError(f"Fixture exceeds 10 MiB: {path.name}")
|
||||
|
||||
print("Checking API readiness...")
|
||||
with urllib.request.urlopen(
|
||||
BASE_URL + "/health/ready",
|
||||
timeout=10,
|
||||
) as response:
|
||||
check(response.status == 200, "API is not ready")
|
||||
|
||||
client = Client(BASE_URL)
|
||||
alice, bob, eve, chat = client.bootstrap()
|
||||
|
||||
rows = []
|
||||
|
||||
for fixture in FIXTURES:
|
||||
content = fixture.read_bytes()
|
||||
size_bytes = len(content)
|
||||
|
||||
for method in ("proxy", "presigned"):
|
||||
for repeat in range(1, REPEATS + 1):
|
||||
print(
|
||||
f"Running {method}: {fixture.name}, "
|
||||
f"repeat {repeat}/{REPEATS}"
|
||||
)
|
||||
|
||||
with MemoryMonitor() as memory:
|
||||
if method == "proxy":
|
||||
metrics = proxy_upload(
|
||||
client,
|
||||
alice,
|
||||
chat,
|
||||
content,
|
||||
)
|
||||
else:
|
||||
metrics = presigned_upload(
|
||||
client,
|
||||
alice,
|
||||
chat,
|
||||
content,
|
||||
)
|
||||
|
||||
row = {
|
||||
"method": method,
|
||||
"file": fixture.name,
|
||||
"file_size_bytes": size_bytes,
|
||||
"repeat": repeat,
|
||||
"elapsed_total_s": round(
|
||||
metrics["elapsed_total_s"], 4
|
||||
),
|
||||
"elapsed_api_s": round(
|
||||
metrics["elapsed_api_s"], 4
|
||||
),
|
||||
"client_to_api_body_bytes": (
|
||||
metrics["api_request_body_bytes"]
|
||||
),
|
||||
"client_to_s3_upload_bytes": (
|
||||
metrics["s3_upload_bytes"]
|
||||
),
|
||||
"api_peak_rss_mib": round(
|
||||
memory.peak_rss_mib, 2
|
||||
),
|
||||
"api_peak_delta_mib": round(
|
||||
memory.peak_delta_mib, 2
|
||||
),
|
||||
}
|
||||
|
||||
rows.append(row)
|
||||
|
||||
print(
|
||||
f" total={row['elapsed_total_s']}s, "
|
||||
f"API={row['elapsed_api_s']}s, "
|
||||
f"API body={row['client_to_api_body_bytes']} bytes, "
|
||||
f"peak RAM={row['api_peak_rss_mib']} MiB, "
|
||||
f"RAM delta={row['api_peak_delta_mib']} MiB"
|
||||
)
|
||||
|
||||
fieldnames = list(rows[0].keys())
|
||||
|
||||
with OUTPUT.open("w", newline="", encoding="utf-8-sig") as file:
|
||||
writer = csv.DictWriter(file, fieldnames=fieldnames)
|
||||
writer.writeheader()
|
||||
writer.writerows(rows)
|
||||
|
||||
print()
|
||||
print("PASS: Track 1 benchmark completed")
|
||||
print(f"Results: {OUTPUT}")
|
||||
print(f"Measurements: {len(rows)}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,40 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Concurrent idempotency verification through the public load-balancer URL."""
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
import uuid
|
||||
import sys
|
||||
from pathlib import Path
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tests"))
|
||||
from client import Client, Failure, arguments, check, run
|
||||
|
||||
|
||||
def main():
|
||||
def configure(p):
|
||||
p.add_argument('--requests', type=int, default=20)
|
||||
args = arguments('Concurrent cross-replica idempotency test', configure)
|
||||
check(2 <= args.requests <= 100, 'requests must be 2..100')
|
||||
c = Client(args.base_url, args.ca_file, timeout=10)
|
||||
alice, bob, eve, chat = c.bootstrap()
|
||||
key = 'race-' + uuid.uuid4().hex
|
||||
body = 'same concurrent request ' + uuid.uuid4().hex
|
||||
|
||||
def send(_):
|
||||
return Client(args.base_url, args.ca_file, timeout=10).message(alice, chat, body, key=key)
|
||||
|
||||
with ThreadPoolExecutor(max_workers=args.requests) as pool:
|
||||
responses = list(pool.map(send, range(args.requests)))
|
||||
messages = [r['json'] for r in responses]
|
||||
check(all(m == messages[0] for m in messages), 'concurrent retries returned different messages')
|
||||
instances = {r['headers'].get('X-Instance-Id') for r in responses}
|
||||
check(None not in instances, 'missing X-Instance-Id')
|
||||
check(len(instances) >= 2, 'load balancer did not expose at least two API replicas')
|
||||
check(len({m['id'] for m in messages}) == 1, 'concurrent retries created duplicates')
|
||||
conflict = c.call('POST', '/api/v1/chats/' + chat['id'] + '/messages', 409,
|
||||
actor=alice, body={'text': body + ' changed'},
|
||||
headers={'Idempotency-Key': key})
|
||||
print({'requests': args.requests, 'unique_message_ids': 1,
|
||||
'observed_instances': len(instances), 'different_body_status': conflict['status']})
|
||||
print('PASS: concurrent same-key requests produced exactly one message')
|
||||
|
||||
if __name__ == '__main__':
|
||||
run(main)
|
||||
@@ -0,0 +1,116 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Diagnostic closed-loop load test. Does not modify public tests."""
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
import json
|
||||
import math
|
||||
import re
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
|
||||
from client import Client, Failure, LAB, arguments, check, run
|
||||
|
||||
|
||||
def main():
|
||||
def configure(p):
|
||||
p.add_argument('--duration', type=float, default=90)
|
||||
p.add_argument('--concurrency', type=int, default=8)
|
||||
p.add_argument('--output', default='evidence/rolling-update-diagnostic.json')
|
||||
|
||||
args = arguments('Diagnostic synthetic POST message load', configure)
|
||||
check(1 <= args.duration <= 3600, 'duration must be 1..3600 seconds')
|
||||
check(1 <= args.concurrency <= 128, 'concurrency must be 1..128')
|
||||
|
||||
c = Client(args.base_url, args.ca_file)
|
||||
alice, bob, eve, chat = c.bootstrap()
|
||||
barrier = threading.Barrier(args.concurrency)
|
||||
|
||||
def worker(worker_id):
|
||||
local = Client(args.base_url, args.ca_file, timeout=5)
|
||||
barrier.wait()
|
||||
deadline = time.monotonic() + args.duration
|
||||
latencies = []
|
||||
errors = Counter()
|
||||
examples = []
|
||||
index = 0
|
||||
|
||||
while time.monotonic() < deadline:
|
||||
start = time.monotonic()
|
||||
try:
|
||||
local.message(alice, chat, 'diagnostic-%s-%s' % (worker_id, index))
|
||||
latencies.append(time.monotonic() - start)
|
||||
except Failure as exc:
|
||||
message = str(exc)
|
||||
match = re.search(r'received (\d{3})', message)
|
||||
|
||||
if match:
|
||||
category = 'http_' + match.group(1)
|
||||
elif 'connection/TLS failure' in message:
|
||||
kind = re.search(r'\(([^)]+)\)', message)
|
||||
category = 'connection_' + (kind.group(1) if kind else 'unknown')
|
||||
else:
|
||||
category = 'other_failure'
|
||||
|
||||
errors[category] += 1
|
||||
if len(examples) < 10:
|
||||
examples.append(message[:300])
|
||||
|
||||
index += 1
|
||||
|
||||
return latencies, errors, examples
|
||||
|
||||
started = time.monotonic()
|
||||
with ThreadPoolExecutor(max_workers=args.concurrency) as pool:
|
||||
results = list(pool.map(worker, range(args.concurrency)))
|
||||
elapsed = time.monotonic() - started
|
||||
|
||||
latencies = sorted(x for values, _, _ in results for x in values)
|
||||
errors = Counter()
|
||||
examples = []
|
||||
|
||||
for _, counts, samples in results:
|
||||
errors.update(counts)
|
||||
examples.extend(samples)
|
||||
|
||||
total_errors = sum(errors.values())
|
||||
total = len(latencies) + total_errors
|
||||
|
||||
def percentile(p):
|
||||
if not latencies:
|
||||
return None
|
||||
return round(1000 * latencies[max(0, math.ceil(p * len(latencies)) - 1)], 3)
|
||||
|
||||
report = {
|
||||
'lab': LAB,
|
||||
'workload': 'closed-loop POST messages; no automatic retries',
|
||||
'duration_seconds': round(elapsed, 3),
|
||||
'concurrency': args.concurrency,
|
||||
'successful_requests': len(latencies),
|
||||
'failed_requests': total_errors,
|
||||
'error_ratio': round(total_errors / total, 6) if total else None,
|
||||
'errors_by_type': dict(errors),
|
||||
'successful_rps': round(len(latencies) / elapsed, 3),
|
||||
'successful_latency_ms': {
|
||||
'p50': percentile(.50),
|
||||
'p95': percentile(.95),
|
||||
'p99': percentile(.99)
|
||||
},
|
||||
'error_examples': examples[:20],
|
||||
'note': 'Latency percentiles cover successful requests only.'
|
||||
}
|
||||
|
||||
text = json.dumps(report, ensure_ascii=False, indent=2)
|
||||
dest = Path(args.output)
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
dest.write_text(text + '\n', encoding='utf-8')
|
||||
print(text)
|
||||
check(bool(latencies), 'no successful writes')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run(main)
|
||||
@@ -0,0 +1,45 @@
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from PIL import Image
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
FIXTURES = ROOT / "tests" / "fixtures"
|
||||
|
||||
WIDTH = 1600
|
||||
HEIGHT = 1200
|
||||
|
||||
|
||||
def main():
|
||||
FIXTURES.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
source = FIXTURES / "sample.png"
|
||||
large = FIXTURES / "sample_large.png"
|
||||
|
||||
if not source.exists():
|
||||
raise FileNotFoundError(
|
||||
f"Small fixture not found: {source}"
|
||||
)
|
||||
|
||||
pixels = os.urandom(WIDTH * HEIGHT * 3)
|
||||
|
||||
image = Image.frombytes(
|
||||
"RGB",
|
||||
(WIDTH, HEIGHT),
|
||||
pixels,
|
||||
)
|
||||
image.save(large, format="PNG", optimize=False)
|
||||
|
||||
for path in (source, large):
|
||||
size = path.stat().st_size
|
||||
print(f"{path.name}: {size} bytes ({size / 1024 / 1024:.2f} MiB)")
|
||||
|
||||
if large.stat().st_size > 10 * 1024 * 1024:
|
||||
raise RuntimeError("Large fixture exceeds the 10 MiB upload limit")
|
||||
|
||||
print("PASS: Track 1 fixtures prepared")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+4
@@ -0,0 +1,4 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
|
||||
docker compose restart postgres redis minio api
|
||||
@@ -0,0 +1,46 @@
|
||||
#!/usr/bin/env bash
|
||||
set -eu
|
||||
# Compose does not provide a native rolling-update primitive. This script
|
||||
# performs a surge replacement: add one new healthy replica, then remove one old.
|
||||
desired="${API_REPLICAS:-2}"
|
||||
load_duration="${ROLLING_LOAD_DURATION:-90}"
|
||||
load_script="${LOAD_SCRIPT:-tests/load.py}"
|
||||
load_output="${ROLLING_LOAD_OUTPUT:-evidence/rolling-update-load.json}"
|
||||
load_log="${ROLLING_LOAD_LOG:-evidence/rolling-update-load.log}"
|
||||
if [ "$desired" -lt 2 ]; then echo 'API_REPLICAS must be >= 2' >&2; exit 2; fi
|
||||
|
||||
docker compose build api
|
||||
python "$load_script" --base-url http://localhost:8080 --duration "$load_duration" --concurrency 8 --output "$load_output" >"$load_log" 2>&1 &
|
||||
load_pid=$!
|
||||
cleanup() { kill "$load_pid" 2>/dev/null || true; wait "$load_pid" 2>/dev/null || true; }
|
||||
trap cleanup EXIT INT TERM
|
||||
|
||||
# Capture the old replica ids. Each iteration adds a new container before removing an old one.
|
||||
old_ids=$(docker compose ps -q api)
|
||||
for old_id in $old_ids; do
|
||||
before=$(docker compose ps -q api | sort)
|
||||
docker compose up -d --no-deps --no-recreate --scale "api=$((desired + 1))" api
|
||||
new_id=''
|
||||
attempt=0
|
||||
while [ "$attempt" -lt 60 ]; do
|
||||
after=$(docker compose ps -q api | sort)
|
||||
new_id=$(comm -13 <(printf '%s\n' "$before") <(printf '%s\n' "$after") | head -n 1 || true)
|
||||
if [ -n "$new_id" ]; then
|
||||
health=$(docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' "$new_id")
|
||||
if [ "$health" = healthy ]; then break; fi
|
||||
fi
|
||||
attempt=$((attempt + 1)); sleep 2
|
||||
done
|
||||
if [ -z "$new_id" ] || [ "${health:-}" != healthy ]; then
|
||||
echo 'New replica did not become healthy; leaving old replicas running' >&2
|
||||
exit 1
|
||||
fi
|
||||
docker exec "$old_id" sh -c 'touch /tmp/api-draining'
|
||||
sleep 10
|
||||
docker stop --timeout 20 "$old_id" >/dev/null
|
||||
docker rm "$old_id" >/dev/null
|
||||
docker compose up -d --no-deps --no-recreate --scale "api=$desired" api
|
||||
python scripts/wait-ready.py http://localhost:8080/health/ready 30
|
||||
done
|
||||
wait "$load_pid" || { echo 'Load test failed; inspect evidence/rolling-update-load.log' >&2; exit 1; }
|
||||
trap - EXIT INT TERM
|
||||
@@ -0,0 +1,131 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run Track 1 scenarios on the user's Docker host; records only measured results."""
|
||||
import argparse
|
||||
import csv
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
OUT = ROOT / 'evidence' / 'scaling'
|
||||
BASE_ENV = ROOT / '.env'
|
||||
|
||||
|
||||
def run(cmd, env=None, check=True, capture=False):
|
||||
return subprocess.run(cmd, cwd=ROOT, env=env, check=check,
|
||||
text=True, stdout=subprocess.PIPE if capture else None,
|
||||
stderr=subprocess.STDOUT if capture else None)
|
||||
|
||||
|
||||
def stats():
|
||||
p = run(['docker', 'stats', '--no-stream', '--format', '{{.Name}};{{.CPUPerc}};{{.MemUsage}}'], capture=True)
|
||||
return p.stdout.strip().splitlines()
|
||||
|
||||
|
||||
def env_value(source, key, default):
|
||||
prefix = key + '='
|
||||
for line in source.splitlines():
|
||||
if line.startswith(prefix):
|
||||
return line[len(prefix):].strip().strip('\"').strip("'")
|
||||
return default
|
||||
|
||||
|
||||
def db_activity(env_path, db_user, db_name):
|
||||
query = "SELECT count(*) FILTER (WHERE state='active'), count(*), current_setting('max_connections') FROM pg_stat_activity"
|
||||
result = run(['docker', 'compose', '--env-file', str(env_path), 'exec', '-T', 'postgres',
|
||||
'psql', '-U', db_user, '-d', db_name, '-At', '-F', ',', '-c', query],
|
||||
capture=True, check=False)
|
||||
return result.stdout.strip() if result.returncode == 0 else 'unavailable'
|
||||
|
||||
|
||||
def scenario(name, replicas, total_cpu, total_mem, duration, concurrency):
|
||||
source = BASE_ENV.read_text(encoding='utf-8')
|
||||
db_user = env_value(source, 'POSTGRES_USER', 'messenger')
|
||||
db_name = env_value(source, 'POSTGRES_DB', 'messenger')
|
||||
api_cpu = total_cpu / replicas
|
||||
api_mem_mb = total_mem // replicas
|
||||
env_text = source.rstrip() + f'\nAPI_CPU_LIMIT={api_cpu:.3f}\nAPI_MEMORY_LIMIT={api_mem_mb}M\n'
|
||||
env_path = OUT / ('.env-' + name)
|
||||
env_path.write_text(env_text, encoding='utf-8')
|
||||
compose = ['docker', 'compose', '--env-file', str(env_path), 'up', '-d', '--build', '--scale', f'api={replicas}']
|
||||
run(compose)
|
||||
deadline = time.time() + 120
|
||||
while time.time() < deadline:
|
||||
result = run(['docker', 'compose', '--env-file', str(env_path), 'ps', '-q', 'api'], capture=True, check=False)
|
||||
ids = [line.strip() for line in result.stdout.splitlines() if line.strip()]
|
||||
healthy = 0
|
||||
for container_id in ids:
|
||||
inspected = run(['docker', 'inspect', '--format', '{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}', container_id], capture=True, check=False)
|
||||
if inspected.returncode == 0 and inspected.stdout.strip() == 'healthy':
|
||||
healthy += 1
|
||||
if len(ids) >= replicas and healthy >= replicas:
|
||||
break
|
||||
time.sleep(2)
|
||||
else:
|
||||
raise RuntimeError(f'{name}: API replicas did not become healthy')
|
||||
|
||||
output = OUT / (name + '.json')
|
||||
cmd = [sys.executable, 'tests/load.py', '--base-url', 'http://localhost:8080',
|
||||
'--duration', str(duration), '--concurrency', str(concurrency), '--output', str(output)]
|
||||
proc = subprocess.Popen(cmd, cwd=ROOT, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
|
||||
samples = []
|
||||
while proc.poll() is None:
|
||||
timestamp = time.time()
|
||||
try:
|
||||
samples.append({'timestamp_epoch': timestamp, 'containers': stats(),
|
||||
'postgres_active_total_max_connections': db_activity(env_path, db_user, db_name)})
|
||||
except Exception as exc:
|
||||
samples.append({'timestamp_epoch': timestamp, 'stats_error': type(exc).__name__})
|
||||
time.sleep(1)
|
||||
stdout = proc.stdout.read() if proc.stdout else ''
|
||||
if proc.returncode != 0:
|
||||
raise RuntimeError(f'load failed for {name}: {stdout}')
|
||||
metrics = json.loads(output.read_text(encoding='utf-8'))
|
||||
metrics.update({'scenario': name, 'api_replicas': replicas,
|
||||
'total_api_cpu_budget': total_cpu,
|
||||
'total_api_memory_budget_mb': total_mem,
|
||||
'per_replica_cpu_limit': api_cpu,
|
||||
'per_replica_memory_limit_mb': api_mem_mb,
|
||||
'container_stats_samples': samples})
|
||||
output.write_text(json.dumps(metrics, ensure_ascii=False, indent=2) + '\n', encoding='utf-8')
|
||||
run(['docker', 'compose', '--env-file', str(env_path), 'down'])
|
||||
return metrics
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser(description=__doc__)
|
||||
p.add_argument('--duration', type=int, default=30)
|
||||
p.add_argument('--concurrency', type=int, default=8)
|
||||
p.add_argument('--fixed-cpu', type=float, default=2.0)
|
||||
p.add_argument('--fixed-memory-mb', type=int, default=2048)
|
||||
p.add_argument('--expanded-cpu', type=float, nargs=3, default=[2.0, 4.0, 8.0])
|
||||
p.add_argument('--expanded-memory-mb', type=int, nargs=3, default=[2048, 4096, 8192])
|
||||
args = p.parse_args()
|
||||
if not BASE_ENV.exists():
|
||||
p.error('Create .env from .env.example before running')
|
||||
OUT.mkdir(parents=True, exist_ok=True)
|
||||
results = []
|
||||
for n in (1, 2, 4):
|
||||
results.append(scenario(f'fixed_{n}replicas', n, args.fixed_cpu,
|
||||
args.fixed_memory_mb, args.duration, args.concurrency))
|
||||
for cpu, mem in zip(args.expanded_cpu, args.expanded_memory_mb):
|
||||
results.append(scenario(f'expanded_4replicas_{cpu:g}cpu_{mem}mb', 4, cpu,
|
||||
mem, args.duration, args.concurrency))
|
||||
with (OUT / 'summary.csv').open('w', newline='', encoding='utf-8') as f:
|
||||
fields = ['scenario', 'api_replicas', 'total_api_cpu_budget', 'total_api_memory_budget_mb',
|
||||
'successful_rps', 'failed_requests', 'error_ratio', 'p50_ms', 'p95_ms', 'p99_ms']
|
||||
writer = csv.DictWriter(f, fieldnames=fields)
|
||||
writer.writeheader()
|
||||
for x in results:
|
||||
lat = x['successful_latency_ms']
|
||||
writer.writerow({'scenario': x['scenario'], 'api_replicas': x['api_replicas'],
|
||||
'total_api_cpu_budget': x['total_api_cpu_budget'],
|
||||
'total_api_memory_budget_mb': x['total_api_memory_budget_mb'],
|
||||
'successful_rps': x['successful_rps'], 'failed_requests': x['failed_requests'],
|
||||
'error_ratio': x['error_ratio'], 'p50_ms': lat['p50'], 'p95_ms': lat['p95'], 'p99_ms': lat['p99']})
|
||||
print(f'Wrote measured results to {OUT}')
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
Executable
+23
@@ -0,0 +1,23 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
|
||||
: "${TARGET_INSTANCE_ID:?TARGET_INSTANCE_ID is required}"
|
||||
|
||||
project="${COMPOSE_PROJECT_NAME:-mtusi-lab4}"
|
||||
found=""
|
||||
|
||||
for id in $(docker ps -q --filter "label=com.docker.compose.project=${project}" --filter "label=com.docker.compose.service=api"); do
|
||||
hostname=$(docker inspect --format '{{.Config.Hostname}}' "$id")
|
||||
|
||||
if [ "$hostname" = "$TARGET_INSTANCE_ID" ]; then
|
||||
found="$id"
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
if [ -z "$found" ]; then
|
||||
echo "No running API container matches TARGET_INSTANCE_ID" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
docker stop --time 3 "$found" >/dev/null
|
||||
@@ -0,0 +1,101 @@
|
||||
import hashlib
|
||||
import sys
|
||||
import time
|
||||
import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from botocore.exceptions import ClientError
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT))
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
|
||||
from client import Client
|
||||
from app.core.config import settings
|
||||
from app.db.models import AttachmentUpload
|
||||
from app.db.session import SessionLocal
|
||||
from app.services.object_storage import client as s3_client
|
||||
|
||||
|
||||
BASE_URL = "http://localhost:8080"
|
||||
FIXTURE = ROOT / "tests" / "fixtures" / "sample.png"
|
||||
|
||||
|
||||
def main():
|
||||
content = FIXTURE.read_bytes()
|
||||
digest = hashlib.sha256(content).hexdigest()
|
||||
|
||||
api = Client(BASE_URL)
|
||||
alice, _, _, chat = api.bootstrap()
|
||||
|
||||
upload_id = uuid.uuid4()
|
||||
object_key = f"cleanup-test/{chat['id']}/{upload_id}"
|
||||
|
||||
# Host-side test uses the published S3 endpoint.
|
||||
s3 = s3_client()
|
||||
|
||||
s3.put_object(
|
||||
Bucket=settings.s3_bucket,
|
||||
Key=object_key,
|
||||
Body=content,
|
||||
ContentType="image/png",
|
||||
)
|
||||
|
||||
with SessionLocal() as db:
|
||||
upload = AttachmentUpload(
|
||||
id=upload_id,
|
||||
chat_id=uuid.UUID(chat["id"]),
|
||||
owner_id=uuid.UUID(alice["id"]),
|
||||
filename="cleanup-test.png",
|
||||
content_type="image/png",
|
||||
expected_size_bytes=len(content),
|
||||
expected_sha256=digest,
|
||||
object_key=object_key,
|
||||
status="pending",
|
||||
# Older than expiry + 10-minute grace period.
|
||||
expires_at=datetime.now(timezone.utc) - timedelta(minutes=11),
|
||||
)
|
||||
db.add(upload)
|
||||
db.commit()
|
||||
|
||||
print(f"Created synthetic pending upload: {upload_id}")
|
||||
print("Waiting for background cleanup (up to 90 seconds)...")
|
||||
|
||||
deadline = time.monotonic() + 90
|
||||
final_status = None
|
||||
|
||||
while time.monotonic() < deadline:
|
||||
with SessionLocal() as db:
|
||||
row = db.get(AttachmentUpload, upload_id)
|
||||
final_status = row.status if row else None
|
||||
|
||||
if final_status == "expired":
|
||||
break
|
||||
|
||||
time.sleep(2)
|
||||
|
||||
if final_status != "expired":
|
||||
raise SystemExit(
|
||||
f"FAIL: expected status expired, got {final_status!r}"
|
||||
)
|
||||
|
||||
try:
|
||||
s3.head_object(Bucket=settings.s3_bucket, Key=object_key)
|
||||
except ClientError as exc:
|
||||
code = exc.response.get("Error", {}).get("Code", "")
|
||||
status = exc.response.get("ResponseMetadata", {}).get("HTTPStatusCode")
|
||||
if code not in ("404", "NoSuchKey", "NotFound") and status != 404:
|
||||
raise
|
||||
else:
|
||||
raise SystemExit("FAIL: S3 object still exists")
|
||||
|
||||
print("PASS: abandoned upload marked expired")
|
||||
print("PASS: corresponding S3 object deleted")
|
||||
print(f"Upload ID: {upload_id}")
|
||||
print(f"Object size: {len(content)} bytes")
|
||||
print(f"SHA-256: {digest}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,285 @@
|
||||
import uuid
|
||||
import hashlib
|
||||
import json
|
||||
import sys
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
|
||||
from client import Client, check
|
||||
|
||||
|
||||
BASE_URL = "http://localhost:8080"
|
||||
IMAGE = ROOT / "tests" / "fixtures" / "sample.png"
|
||||
|
||||
|
||||
def response_json(response):
|
||||
return json.loads(response["bytes"].decode("utf-8"))
|
||||
|
||||
|
||||
def create_upload(client, actor, chat, content, filename="sample.png"):
|
||||
"""Create a presigned upload and return its metadata."""
|
||||
response = client.call(
|
||||
"POST",
|
||||
f"/api/v1/chats/{chat['id']}/attachments/presigned",
|
||||
201,
|
||||
actor=actor,
|
||||
body={
|
||||
"filename": filename,
|
||||
"content_type": "image/png",
|
||||
"size_bytes": len(content),
|
||||
"sha256": hashlib.sha256(content).hexdigest(),
|
||||
},
|
||||
)
|
||||
return response_json(response)
|
||||
|
||||
|
||||
def create_upload_with_metadata(
|
||||
client,
|
||||
actor,
|
||||
chat,
|
||||
content,
|
||||
expected_size=None,
|
||||
expected_sha256=None,
|
||||
):
|
||||
"""Create an upload with deliberately supplied expected metadata."""
|
||||
response = client.call(
|
||||
"POST",
|
||||
f"/api/v1/chats/{chat['id']}/attachments/presigned",
|
||||
201,
|
||||
actor=actor,
|
||||
body={
|
||||
"filename": "sample.png",
|
||||
"content_type": "image/png",
|
||||
"size_bytes": (
|
||||
len(content) if expected_size is None else expected_size
|
||||
),
|
||||
"sha256": (
|
||||
hashlib.sha256(content).hexdigest()
|
||||
if expected_sha256 is None
|
||||
else expected_sha256
|
||||
),
|
||||
},
|
||||
)
|
||||
return response_json(response)
|
||||
|
||||
|
||||
def put_to_s3(upload, content):
|
||||
"""Upload bytes directly to S3 using the presigned URL."""
|
||||
request = urllib.request.Request(
|
||||
upload["upload_url"],
|
||||
data=content,
|
||||
headers=upload["headers"],
|
||||
method="PUT",
|
||||
)
|
||||
|
||||
with urllib.request.urlopen(request, timeout=30) as response:
|
||||
check(
|
||||
response.status in (200, 201, 204),
|
||||
"S3 upload failed",
|
||||
)
|
||||
|
||||
|
||||
def complete_upload(client, actor, upload_id, expected_status=200):
|
||||
return client.call(
|
||||
"POST",
|
||||
f"/api/v1/attachments/uploads/{upload_id}/complete",
|
||||
expected_status,
|
||||
actor=actor,
|
||||
)
|
||||
|
||||
|
||||
def main():
|
||||
content = IMAGE.read_bytes()
|
||||
digest = hashlib.sha256(content).hexdigest()
|
||||
|
||||
client = Client(BASE_URL)
|
||||
alice, bob, eve, chat = client.bootstrap()
|
||||
|
||||
# 1. Incorrect SHA-256 must be rejected.
|
||||
wrong_digest = "0" * 64
|
||||
check(wrong_digest != digest, "test checksum unexpectedly matches")
|
||||
|
||||
upload = create_upload_with_metadata(
|
||||
client,
|
||||
alice,
|
||||
chat,
|
||||
content,
|
||||
expected_sha256=wrong_digest,
|
||||
)
|
||||
put_to_s3(upload, content)
|
||||
|
||||
complete_upload(
|
||||
client,
|
||||
alice,
|
||||
upload["upload_id"],
|
||||
expected_status=409,
|
||||
)
|
||||
|
||||
print("PASS: incorrect SHA-256 rejected with HTTP 409")
|
||||
|
||||
# 2. Incorrect expected size must be rejected.
|
||||
upload = create_upload_with_metadata(
|
||||
client,
|
||||
alice,
|
||||
chat,
|
||||
content,
|
||||
expected_size=len(content) + 1,
|
||||
)
|
||||
put_to_s3(upload, content)
|
||||
|
||||
complete_upload(
|
||||
client,
|
||||
alice,
|
||||
upload["upload_id"],
|
||||
expected_status=409,
|
||||
)
|
||||
|
||||
print("PASS: incorrect file size rejected with HTTP 409")
|
||||
|
||||
# 3. Corrupted image must be rejected.
|
||||
corrupt_content = b"This is not a valid PNG image"
|
||||
|
||||
upload = create_upload(
|
||||
client,
|
||||
alice,
|
||||
chat,
|
||||
corrupt_content,
|
||||
)
|
||||
put_to_s3(upload, corrupt_content)
|
||||
|
||||
complete_upload(
|
||||
client,
|
||||
alice,
|
||||
upload["upload_id"],
|
||||
expected_status=415,
|
||||
)
|
||||
|
||||
print("PASS: corrupted image rejected with HTTP 415")
|
||||
|
||||
# 4. Another user must not complete Alice's upload.
|
||||
upload = create_upload(
|
||||
client,
|
||||
alice,
|
||||
chat,
|
||||
content,
|
||||
)
|
||||
put_to_s3(upload, content)
|
||||
|
||||
complete_upload(
|
||||
client,
|
||||
eve,
|
||||
upload["upload_id"],
|
||||
expected_status=404,
|
||||
)
|
||||
|
||||
print("PASS: foreign user cannot complete another user's upload")
|
||||
|
||||
# The legitimate owner must still be able to complete it.
|
||||
owner_response = complete_upload(
|
||||
client,
|
||||
alice,
|
||||
upload["upload_id"],
|
||||
expected_status=200,
|
||||
)
|
||||
attachment = response_json(owner_response)
|
||||
|
||||
check(
|
||||
attachment["status"] == "ready",
|
||||
"owner's attachment is not ready",
|
||||
)
|
||||
check(
|
||||
attachment["owner_id"] == alice["id"],
|
||||
"attachment owner does not match",
|
||||
)
|
||||
check(
|
||||
attachment["sha256"] == digest,
|
||||
"attachment checksum mismatch",
|
||||
)
|
||||
|
||||
print("PASS: legitimate owner can complete the upload")
|
||||
# 5. Proxy upload must reject files larger than 10 MiB.
|
||||
oversized_content = b"\x00" * (10 * 1024 * 1024 + 1)
|
||||
|
||||
client.call(
|
||||
"POST",
|
||||
f"/api/v1/chats/{chat['id']}/attachments?filename=oversized.png",
|
||||
expected=413,
|
||||
actor=alice,
|
||||
raw=oversized_content,
|
||||
headers={"Content-Type": "image/png"},
|
||||
)
|
||||
|
||||
print("PASS: oversized proxy upload rejected with HTTP 413")
|
||||
|
||||
# 6. PNG content must not be accepted as JPEG.
|
||||
client.call(
|
||||
"POST",
|
||||
f"/api/v1/chats/{chat['id']}/attachments?filename=mismatch.jpg",
|
||||
expected=415,
|
||||
actor=alice,
|
||||
raw=content,
|
||||
headers={"Content-Type": "image/jpeg"},
|
||||
)
|
||||
|
||||
print("PASS: MIME mismatch rejected with HTTP 415")
|
||||
|
||||
# 7. Create a ready attachment in Alice and Bob's chat.
|
||||
attachment, _ = client.attachment(alice, chat)
|
||||
attachment_id = attachment["id"]
|
||||
|
||||
# Eve is not a member of Alice and Bob's chat.
|
||||
client.call(
|
||||
"GET",
|
||||
f"/api/v1/attachments/{attachment_id}",
|
||||
expected=404,
|
||||
actor=eve,
|
||||
)
|
||||
|
||||
print("PASS: non-member cannot access attachment metadata")
|
||||
|
||||
# 8. A non-member must not download the attachment.
|
||||
client.call(
|
||||
"GET",
|
||||
f"/api/v1/attachments/{attachment_id}/content",
|
||||
expected=404,
|
||||
actor=eve,
|
||||
)
|
||||
|
||||
print("PASS: non-member cannot download attachment")
|
||||
|
||||
# 9. Create another chat where Alice and Eve are members.
|
||||
second_chat = client.call(
|
||||
"POST",
|
||||
"/api/v1/chats",
|
||||
201,
|
||||
actor=alice,
|
||||
body={
|
||||
"title": "Cross-chat attachment test",
|
||||
"member_ids": [alice["id"], eve["id"]],
|
||||
},
|
||||
schema="Chat",
|
||||
)["json"]
|
||||
|
||||
# Try to attach a file from the first chat to a message in the second.
|
||||
client.call(
|
||||
"POST",
|
||||
f"/api/v1/chats/{second_chat['id']}/messages",
|
||||
expected=404,
|
||||
actor=alice,
|
||||
body={
|
||||
"text": "Cross-chat attachment test",
|
||||
"attachment_ids": [attachment_id],
|
||||
},
|
||||
headers={"Idempotency-Key": uuid.uuid4().hex},
|
||||
)
|
||||
|
||||
print("PASS: attachment from another chat rejected with HTTP 404")
|
||||
print()
|
||||
print("PASS: all presigned negative scenarios")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,123 @@
|
||||
import hashlib
|
||||
import json
|
||||
import sys
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
|
||||
from client import Client, check
|
||||
|
||||
|
||||
BASE_URL = "http://localhost:8080"
|
||||
IMAGE = ROOT / "tests" / "fixtures" / "sample.png"
|
||||
|
||||
|
||||
def response_json(response):
|
||||
"""Decode JSON from the raw response bytes."""
|
||||
return json.loads(response["bytes"].decode("utf-8"))
|
||||
|
||||
|
||||
def main():
|
||||
content = IMAGE.read_bytes()
|
||||
digest = hashlib.sha256(content).hexdigest()
|
||||
|
||||
client = Client(BASE_URL)
|
||||
alice, bob, eve, chat = client.bootstrap()
|
||||
|
||||
# 1. Request a short-lived upload URL.
|
||||
create_response = client.call(
|
||||
"POST",
|
||||
f"/api/v1/chats/{chat['id']}/attachments/presigned",
|
||||
201,
|
||||
actor=alice,
|
||||
body={
|
||||
"filename": "sample.png",
|
||||
"content_type": "image/png",
|
||||
"size_bytes": len(content),
|
||||
"sha256": digest,
|
||||
},
|
||||
)
|
||||
|
||||
created = response_json(create_response)
|
||||
|
||||
check(created["method"] == "PUT", "unexpected upload method")
|
||||
check(bool(created["upload_id"]), "missing upload_id")
|
||||
check(created["expires_in"] > 0, "invalid upload expiration")
|
||||
|
||||
# 2. Upload directly to S3, without sending the file through the API.
|
||||
request = urllib.request.Request(
|
||||
created["upload_url"],
|
||||
data=content,
|
||||
headers=created["headers"],
|
||||
method="PUT",
|
||||
)
|
||||
|
||||
with urllib.request.urlopen(request, timeout=30) as response:
|
||||
check(
|
||||
response.status in (200, 201, 204),
|
||||
"S3 upload failed",
|
||||
)
|
||||
|
||||
# 3. Complete the upload.
|
||||
complete_path = (
|
||||
f"/api/v1/attachments/uploads/{created['upload_id']}/complete"
|
||||
)
|
||||
|
||||
complete_response = client.call(
|
||||
"POST",
|
||||
complete_path,
|
||||
200,
|
||||
actor=alice,
|
||||
)
|
||||
|
||||
attachment = response_json(complete_response)
|
||||
|
||||
check(attachment["status"] == "ready", "attachment is not ready")
|
||||
check(attachment["chat_id"] == chat["id"], "wrong attachment chat")
|
||||
check(attachment["owner_id"] == alice["id"], "wrong attachment owner")
|
||||
check(attachment["sha256"] == digest, "attachment checksum mismatch")
|
||||
check(
|
||||
attachment["size_bytes"] == len(content),
|
||||
"attachment size mismatch",
|
||||
)
|
||||
|
||||
# 4. Complete again: must return the same attachment.
|
||||
repeated_response = client.call(
|
||||
"POST",
|
||||
complete_path,
|
||||
200,
|
||||
actor=alice,
|
||||
)
|
||||
|
||||
repeated = response_json(repeated_response)
|
||||
|
||||
check(
|
||||
repeated["id"] == attachment["id"],
|
||||
"repeated completion created a duplicate",
|
||||
)
|
||||
|
||||
# 5. Download through the authorized API endpoint.
|
||||
downloaded = client.call(
|
||||
"GET",
|
||||
f"/api/v1/attachments/{attachment['id']}/content",
|
||||
200,
|
||||
actor=alice,
|
||||
)["bytes"]
|
||||
|
||||
check(downloaded == content, "downloaded bytes differ from original")
|
||||
check(
|
||||
hashlib.sha256(downloaded).hexdigest() == digest,
|
||||
"downloaded checksum mismatch",
|
||||
)
|
||||
|
||||
print("PASS: presigned upload, completion, idempotency and download")
|
||||
print(f"Upload ID: {created['upload_id']}")
|
||||
print(f"Attachment ID: {attachment['id']}")
|
||||
print(f"File size: {len(content)} bytes")
|
||||
print(f"SHA-256: {digest}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,127 @@
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy import text
|
||||
|
||||
ROOT = Path("/app")
|
||||
sys.path.insert(0, str(ROOT))
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
|
||||
from client import Client, check
|
||||
from app.core.config import settings
|
||||
from app.db.models import Attachment
|
||||
from app.db.session import SessionLocal
|
||||
from app.services.object_storage import client as s3_client
|
||||
|
||||
|
||||
TRIGGER_NAME = "lab4_force_attachment_insert_failure"
|
||||
FUNCTION_NAME = "lab4_fail_attachment_insert"
|
||||
TEST_FILENAME = "lab4-forced-db-failure.png"
|
||||
|
||||
|
||||
def install_failure_trigger():
|
||||
table = Attachment.__table__
|
||||
schema = table.schema or "public"
|
||||
qualified_table = f'"{schema}"."{table.name}"'
|
||||
|
||||
with SessionLocal() as db:
|
||||
db.execute(text(f"""
|
||||
CREATE OR REPLACE FUNCTION public.{FUNCTION_NAME}()
|
||||
RETURNS trigger
|
||||
LANGUAGE plpgsql
|
||||
AS $$
|
||||
BEGIN
|
||||
IF NEW.filename = '{TEST_FILENAME}' THEN
|
||||
RAISE EXCEPTION 'Lab 4 controlled metadata failure'
|
||||
USING ERRCODE = 'P0001';
|
||||
END IF;
|
||||
RETURN NEW;
|
||||
END;
|
||||
$$;
|
||||
"""))
|
||||
db.execute(text(
|
||||
f'DROP TRIGGER IF EXISTS {TRIGGER_NAME} ON {qualified_table}'
|
||||
))
|
||||
db.execute(text(
|
||||
f'CREATE TRIGGER {TRIGGER_NAME} '
|
||||
f'BEFORE INSERT ON {qualified_table} '
|
||||
f'FOR EACH ROW EXECUTE FUNCTION public.{FUNCTION_NAME}()'
|
||||
))
|
||||
db.commit()
|
||||
|
||||
|
||||
def remove_failure_trigger():
|
||||
table = Attachment.__table__
|
||||
schema = table.schema or "public"
|
||||
qualified_table = f'"{schema}"."{table.name}"'
|
||||
|
||||
with SessionLocal() as db:
|
||||
db.execute(text(
|
||||
f'DROP TRIGGER IF EXISTS {TRIGGER_NAME} ON {qualified_table}'
|
||||
))
|
||||
db.execute(text(
|
||||
f'DROP FUNCTION IF EXISTS public.{FUNCTION_NAME}()'
|
||||
))
|
||||
db.commit()
|
||||
|
||||
|
||||
def object_keys(s3, prefix):
|
||||
result = s3.list_objects_v2(
|
||||
Bucket=settings.s3_bucket,
|
||||
Prefix=prefix,
|
||||
)
|
||||
return {
|
||||
item["Key"]
|
||||
for item in result.get("Contents", [])
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
client = Client("http://127.0.0.1:8080")
|
||||
alice, bob, eve, chat = client.bootstrap()
|
||||
|
||||
content = (ROOT / "tests" / "fixtures" / "sample.png").read_bytes()
|
||||
prefix = f"original/{chat['id']}/"
|
||||
|
||||
s3 = s3_client()
|
||||
before = object_keys(s3, prefix)
|
||||
|
||||
install_failure_trigger()
|
||||
print("Temporary PostgreSQL failure trigger installed.")
|
||||
|
||||
try:
|
||||
response = client.call(
|
||||
"POST",
|
||||
f"/api/v1/chats/{chat['id']}/attachments"
|
||||
f"?filename={TEST_FILENAME}",
|
||||
expected=503,
|
||||
actor=alice,
|
||||
raw=content,
|
||||
headers={"Content-Type": "image/png"},
|
||||
)
|
||||
|
||||
check(
|
||||
response["status"] == 503,
|
||||
"Expected HTTP 503 after forced metadata failure",
|
||||
)
|
||||
print("PASS: S3 upload succeeded, PostgreSQL metadata insert failed")
|
||||
|
||||
after = object_keys(s3, prefix)
|
||||
unexpected_objects = after - before
|
||||
|
||||
check(
|
||||
not unexpected_objects,
|
||||
"Compensation failed: unexpected S3 objects remain",
|
||||
)
|
||||
print("PASS: uploaded S3 object was deleted after DB failure")
|
||||
|
||||
print()
|
||||
print("PASS: S3/PostgreSQL compensation scenario")
|
||||
|
||||
finally:
|
||||
remove_failure_trigger()
|
||||
print("Temporary PostgreSQL trigger removed.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,16 @@
|
||||
#!/usr/bin/env python3
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
|
||||
url = sys.argv[1] if len(sys.argv) > 1 else 'http://localhost:8080/health/ready'
|
||||
timeout = int(sys.argv[2]) if len(sys.argv) > 2 else 30
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
with urllib.request.urlopen(url, timeout=2) as response:
|
||||
if response.status == 200:
|
||||
raise SystemExit(0)
|
||||
except Exception:
|
||||
time.sleep(0.5)
|
||||
raise SystemExit('Readiness timeout: ' + url)
|
||||
+20
-6
@@ -182,13 +182,27 @@ class Client:
|
||||
headers = {'Idempotency-Key':key or uuid.uuid4().hex} if LAB>=3 else {}
|
||||
return self.call('POST','/api/v1/chats/'+chat['id']+'/messages',201,actor=actor,body=body,headers=headers,schema='Message')
|
||||
|
||||
def attachment(self, actor, chat):
|
||||
def attachment(self, actor, chat, include_instance=False):
|
||||
content = (ROOT / 'tests/fixtures/sample.png').read_bytes()
|
||||
att = self.call('POST','/api/v1/chats/'+chat['id']+'/attachments?filename=sample.png',201 if LAB==4 else 202,actor=actor,raw=content,headers={'Content-Type':'image/png'},schema='Attachment')['json']
|
||||
check(att['status']==('ready' if LAB==4 else 'queued'),'incorrect upload status')
|
||||
check(att['chat_id']==chat['id'] and att['owner_id']==actor['id'],'attachment ownership mismatch')
|
||||
check(att['sha256']==hashlib.sha256(content).hexdigest() and att['size_bytes']==len(content),'original checksum/size mismatch')
|
||||
return att,content
|
||||
response = self.call(
|
||||
'POST',
|
||||
'/api/v1/chats/' + chat['id'] + '/attachments?filename=sample.png',
|
||||
201 if LAB == 4 else 202,
|
||||
actor=actor,
|
||||
raw=content,
|
||||
headers={'Content-Type': 'image/png'},
|
||||
schema='Attachment'
|
||||
)
|
||||
att = response['json']
|
||||
check(att['status'] == ('ready' if LAB == 4 else 'queued'), 'incorrect upload status')
|
||||
check(att['chat_id'] == chat['id'] and att['owner_id'] == actor['id'], 'attachment ownership mismatch')
|
||||
check(
|
||||
att['sha256'] == hashlib.sha256(content).hexdigest()
|
||||
and att['size_bytes'] == len(content),
|
||||
'original checksum/size mismatch'
|
||||
)
|
||||
instance_id = response['headers'].get('X-Instance-Id', '')
|
||||
return (att, content, instance_id) if include_instance else (att, content)
|
||||
|
||||
def wait_ready(self, actor, att, wait_seconds=60):
|
||||
deadline = time.monotonic()+wait_seconds
|
||||
|
||||
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 5.5 MiB |
+15
-5
@@ -40,12 +40,22 @@ def main():
|
||||
alice,bob,eve,chat = c.bootstrap()
|
||||
sent = c.message(alice,chat,'Сохранить при отказе')
|
||||
original = sent['json']
|
||||
target = sent['headers'].get('X-Instance-Id','')
|
||||
|
||||
target = sent['headers'].get('X-Instance-Id', '')
|
||||
attachment = content = None
|
||||
if LAB>=4:
|
||||
attachment,content = c.attachment(alice,chat)
|
||||
if LAB>=5:
|
||||
attachment = c.wait_ready(alice,attachment)
|
||||
|
||||
if LAB >= 4:
|
||||
attachment, content, upload_instance = c.attachment(alice, chat, include_instance=True)
|
||||
|
||||
if args.mode == 'failover':
|
||||
check(bool(upload_instance), 'attachment upload response has no X-Instance-Id')
|
||||
target = upload_instance
|
||||
|
||||
if LAB >= 5:
|
||||
attachment = c.wait_ready(alice, attachment)
|
||||
|
||||
if args.mode == 'failover':
|
||||
check(bool(target), 'target API instance ID is empty')
|
||||
env = dict(os.environ)
|
||||
env['TARGET_INSTANCE_ID'] = target
|
||||
observed = set()
|
||||
|
||||
Reference in New Issue
Block a user