forked from mtusi2026/mtusi2026lab4
Compare commits
6
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
75cbe925d3 | ||
|
|
9af7b15809 | ||
|
|
64503fd0dd | ||
|
|
e594df18f9 | ||
|
|
c9e24a0b45 | ||
|
|
4b23b44eb8 |
@@ -18,3 +18,8 @@ S3_ACCESS_KEY_ID_FILE=/run/secrets/s3_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE=/run/secrets/s3_secret_access_key
|
||||
MAX_UPLOAD_BYTES=10485760
|
||||
MAX_IMAGE_PIXELS=40000000
|
||||
# Lab4: S3 is internal-only; Compose explicitly maps these to the API.
|
||||
S3_ENDPOINT=http://s3:9000
|
||||
S3_BUCKET=messenger-attachments
|
||||
S3_ACCESS_KEY=messengeradmin
|
||||
S3_SECRET_FILE=/run/secrets/s3_password
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
FROM python:3.11.11-slim-bookworm
|
||||
WORKDIR /app
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 APP_HOST=0.0.0.0 APP_PORT=8080 APP_ENV=lab
|
||||
COPY requirements.txt ./
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
COPY messenger ./messenger
|
||||
EXPOSE 8080
|
||||
COPY migrations ./migrations
|
||||
CMD ["python", "-m", "messenger"]
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
Разделить метаданные и бинарное содержимое, добавить изображения в сообщения и сохранить доступ к ним при переключении API-реплики.
|
||||
|
||||
Это **skeleton**, а не готовый сервер. Здесь есть задание, контракт и публичные проверки. Реализацию приложения, Dockerfile и требуемую инфраструктуру пишет студент. Язык и framework свободные; UI не обязателен.
|
||||
Этот архив содержит **готовую реализацию для оценки «3»** на базе лабораторной №3: Flask, PostgreSQL, HAProxy (две API-реплики), MinIO S3 и приватные вложения. Исходные задание, контракт и публичные тесты сохранены. Инструкции по запуску и решения/ограничения — в [REPORT.md](REPORT.md). UI не обязателен.
|
||||
|
||||
[Карта курса](COURSE.md) · [Процесс fork/PR](CONTRIBUTING.md) · [OpenAPI](contracts/openapi.json) · [Семантика API](contracts/README.md) · [Проверки](tests/README.md) · [Отчёт](REPORT.md) · [Примеры JSON](contracts/examples.json) · [Памятка преподавателю](MAINTAINERS.md)
|
||||
|
||||
|
||||
@@ -1,37 +1,87 @@
|
||||
# Отчёт — лабораторная 4
|
||||
# Отчёт — лабораторная работа №4
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
- Автор: **Сиренко Илья**.
|
||||
- Группа: **БПИ2304**.
|
||||
- Целевая оценка: **3**. Уровни 4/5 не заявлены.
|
||||
- Основа: лабораторная №3 (Flask + PostgreSQL + HAProxy, две API-реплики, сессии и идемпотентные сообщения).
|
||||
- Commit/PR: указать после публикации репозитория.
|
||||
- Стек: Python 3.9+, Flask 3.1.0, Waitress 3.0.2, PostgreSQL 16.6, psycopg 3.2.6, HAProxy 3.0.8, boto3 1.36.26, Pillow 11.1.0, S3-совместимое объектное хранилище.
|
||||
- Использованные источники и помощь ИИ: требования из Markdown-файлов лабораторной №4, `contracts/openapi.json`, публичные тесты и собственный проект лабораторной №3. ИИ использовался для разбора требований, проектирования структуры приложения, проверки соответствия контракту и подготовки отчёта.
|
||||
|
||||
## Запуск из чистого clone
|
||||
## Запуск из чистого clone / ZIP
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
Нужны Docker Engine с Compose v2, Python 3.9+ для публичных тестов и доступ к registry для загрузки контейнерных образов. Все команды выполняются в корне `mtusi2026lab4`. Опубликован только HAProxy на `127.0.0.1:8080`.
|
||||
|
||||
## Архитектура и решения
|
||||
```sh
|
||||
python3 scripts/init_local.py
|
||||
# Секреты создаются один раз в secrets/ и не входят в ZIP/Git.
|
||||
docker compose --env-file .env config --quiet
|
||||
docker compose --env-file .env up -d --build
|
||||
docker compose --env-file .env ps
|
||||
# дождаться: curl -fsS http://localhost:8080/health/ready
|
||||
make check
|
||||
make test
|
||||
```
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab3, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
Swagger UI в четвёртой лабораторной не заявлен; машинно-читаемый контракт доступен в `contracts/openapi.json`. Проверка через curl: `curl http://localhost:8080/health/live`. Выключение: `docker compose down` (данные остаются); намеренное уничтожение **всех учебных данных**: `docker compose down -v` (не делайте при проверке persistence). После изменения миграций запустить `docker compose up -d --build`, сервис `migrate` применит версии однократно. Если ранее был развёрнут lab3 со своими томами, сохраняйте его PostgreSQL volume и секреты при переносе; новая миграция `003_attachments.sql` добавляется к существующей схеме.
|
||||
|
||||
## Доказательства критериев
|
||||
## Архитектура
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
```
|
||||
REST-клиент -> HAProxy :8080 -> api1 / api2 (Flask + waitress)
|
||||
| |
|
||||
+-- PostgreSQL :5432 (таблицы/сессии/идемпотентность)
|
||||
+-- MinIO S3 :9000 (оригиналы PNG/JPEG)
|
||||
```
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
* `db`: `postgres:16.6-bookworm`, БД messenger, `postgres_data`, health `pg_isready`. `migrate` однократно создаёт/обновляет schema_migrations.
|
||||
* `s3`: `minio/minio:RELEASE.2025-02-07T23-21-09Z`, `s3_data`, без опубликованных портов, локальный health endpoint; `s3-bootstrap` идемпотентно создаёт приватный bucket `messenger-attachments`.
|
||||
* `api1` и `api2`: сборка из `Dockerfile` на `python:3.11.11-slim-bookworm`, readiness проверяет и БД, и S3; не имеют volume для бинарных вложений.
|
||||
* `lb`: `haproxy:3.0.8-bookworm`, опубликован только `127.0.0.1:8080`.
|
||||
* Секреты: `secrets/postgres_password`, `secrets/database_url`, `secrets/s3_password` создаются локально `scripts/init_local.py`; `DATABASE_URL_FILE`, `S3_SECRET_FILE` указывают на Docker secrets. `APP_ENV`, TTL и сетевые параметры заданы в Compose. API использует административные MinIO credentials ради простоты уровня 3; разграничение S3 permissions — задача уровня 4.
|
||||
|
||||
## Логика загрузки
|
||||
|
||||
1. Bearer-сессия и принадлежность чату проверяются перед чтением тела. `filename` валидируется, не используется в пути. Заголовок MIME и содержимое должны соответствовать PNG либо JPEG.
|
||||
2. Лимит чтения 10 MiB + 1 байт и пикселей 40 млн проверяется **до** полной распаковки; Pillow проверяет структуру и полное декодирование. Объекту назначается независимый случайный ключ `originals/<random>/<random>`.
|
||||
3. Бинарный оригинал записывается в **приватный S3**. После успешного PUT данные метаданных коммитятся в PostgreSQL, только затем отдаётся `201 ready` (`variants=[]`, `error_code=null`). SHA-256 и размер считаются по исходным байтам.
|
||||
4. Для создания сообщения допускается `attachment_ids` 0–10 уникальных UUID, включая пустой текст с непустым массивом. Все IDs должны принадлежать чату. `message_attachments` хранит порядок; идемпотентность учитывает текст и отсортированный по позиции список IDs (порядок важен).
|
||||
5. `GET /attachments/{id}` и `/content` разрешены только участникам чата. API скачивает байты напрямую из S3, без редиректа и без открытия bucket наружу. Чужому пользователю возвращается одинаковый `404` для несуществующего или недоступного объекта.
|
||||
|
||||
SQL-транзакция не охватывает S3: если PUT прошёл, а запись БД не состоялась, выполняется **best-effort** удаление ключа. При разрыве процесса либо при отказе удаления возможен orphan-объект; полноценную фоновую сверку на уровне 3 не заявляем. Приватность bucket обеспечивается отсутствием публичной policy и закрытым сетевым портом; авторизация реализована отдельно в API.
|
||||
|
||||
## Доказательства критериев оценки «3»
|
||||
|
||||
Результаты ниже отделяют **фактически выполненные локальные проверки** от контейнерных и сетевых сценариев, требующих работающего Docker-стенда. Для подтверждения на машине студента вывод команд можно сохранить в `evidence/`:
|
||||
|
||||
```sh
|
||||
mkdir -p evidence
|
||||
make check 2>&1 | tee evidence/contract-local.txt
|
||||
python3 -m unittest discover -s student-tests -p test_images.py -v 2>&1 | tee evidence/images-local.txt
|
||||
docker compose --env-file .env ps -a 2>&1 | tee evidence/docker-services.txt
|
||||
make test 2>&1 | tee evidence/public-smoke.txt
|
||||
```
|
||||
|
||||
|
||||
|
||||
| Проверка | Команда / метод | Ожидаемый критерий |
|
||||
| --- | --- | --- |
|
||||
| Контракт и исходники | `make check` | Корректная структура lab4/OpenAPI и синтаксис Python |
|
||||
| Локальная валидация файлов | `python -m unittest discover -s student-tests -p test_images.py -v` | Реальный PNG, неверный MIME, повреждение, лимит bytes/pixels |
|
||||
| Сквозной API | `make test` | Авторизация и lab3-функции, `201 ready`, SHA-256, бинарный download, чужой пользователь 404 |
|
||||
| Отказ API-реплики | `make test-failover ACTION_SCRIPT=scripts/stop-one.sh` | `TARGET_INSTANCE_ID` останавливается, другой API обслуживает запросы; тест включает S3 file check для lab4 |
|
||||
| Долговечность | `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | Перезапуск двух API, PostgreSQL и MinIO с прежними volumes, затем чтение с прежним токеном |
|
||||
| Приватность S3 | `docker compose exec -T api1 python -c 'from messenger.config import Settings; from messenger.storage import ObjectStore; s=ObjectStore(Settings.from_env()); print(s.s3.get_bucket_policy(Bucket=s.bucket))'` | Без заданной публичной policy запрос должен закончиться ошибкой `NoSuchBucketPolicy`; по умолчанию бакет приватен |
|
||||
|
||||
**Фактически выполнено в среде подготовки:** `python scripts/check_contract.py` — PASS; `python -m unittest discover -s student-tests -p test_images.py -v` — 4 OK; `python -m compileall -q messenger` — без ошибок.
|
||||
|
||||
## Эксперименты и отказы
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
| Сценарий | Ожидаемое поведение | Наблюдение / статус |
|
||||
| --- | --- | --- |
|
||||
| Неверный MIME или повреждённое изображение | Отказ без создания корректного вложения | Проверено локальными unit-тестами, 4 теста OK |
|
||||
| Файл, превышающий лимиты | Отказ в загрузке | Проверено локальными unit-тестами |
|
||||
| Чтение чужого вложения | 404 без раскрытия данных | Реализовано в API, end-to-end проверка на Docker-хосте требуется |
|
||||
| Перезапуск API и хранилищ | Сохранение метаданных в PostgreSQL и оригиналов в S3 volumes | Подготовлен тест `make test-persistence`, Docker-подтверждение не получено |
|
||||
| Остановка одной API-реплики | Вторая реплика продолжает обслуживать HTTP-запросы через HAProxy | Подготовлен тест `make test-failover`, Docker-подтверждение не получено |
|
||||
| Отказ между PUT в S3 и COMMIT в БД | Best-effort удаление S3-объекта | Реализована компенсация, гарантированной очистки orphan-объектов нет |
|
||||
|
||||
+109
@@ -0,0 +1,109 @@
|
||||
x-api: &api
|
||||
build: .
|
||||
environment: &app-environment
|
||||
APP_HOST: 0.0.0.0
|
||||
APP_PORT: 8080
|
||||
APP_ENV: ${APP_ENV:-lab}
|
||||
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
|
||||
IDEMPOTENCY_TTL_SECONDS: ${IDEMPOTENCY_TTL_SECONDS:-86400}
|
||||
SESSION_STORE: postgres
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
S3_ENDPOINT: http://s3:9000
|
||||
S3_BUCKET: messenger-attachments
|
||||
S3_ACCESS_KEY: messengeradmin
|
||||
S3_SECRET_FILE: /run/secrets/s3_password
|
||||
secrets:
|
||||
- database_url
|
||||
- s3_password
|
||||
depends_on:
|
||||
db:
|
||||
condition: service_healthy
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
s3-bootstrap:
|
||||
condition: service_completed_successfully
|
||||
stop_grace_period: 8s
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready',timeout=3)"]
|
||||
interval: 2s
|
||||
timeout: 4s
|
||||
retries: 20
|
||||
|
||||
services:
|
||||
db:
|
||||
image: postgres:16.6-bookworm
|
||||
environment:
|
||||
POSTGRES_DB: messenger
|
||||
POSTGRES_USER: messenger
|
||||
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
|
||||
secrets:
|
||||
- postgres_password
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U messenger -d messenger"]
|
||||
interval: 2s
|
||||
timeout: 3s
|
||||
retries: 30
|
||||
s3:
|
||||
image: minio/minio:RELEASE.2025-02-07T23-21-09Z
|
||||
command: server /data --address ':9000'
|
||||
environment:
|
||||
MINIO_ROOT_USER: messengeradmin
|
||||
MINIO_ROOT_PASSWORD_FILE: /run/secrets/s3_password
|
||||
secrets:
|
||||
- s3_password
|
||||
volumes:
|
||||
- s3_data:/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "curl -fsS http://127.0.0.1:9000/minio/health/live >/dev/null"]
|
||||
interval: 3s
|
||||
timeout: 5s
|
||||
retries: 25
|
||||
s3-bootstrap:
|
||||
build: .
|
||||
command: ["python", "-m", "messenger.bootstrap_s3"]
|
||||
environment: *app-environment
|
||||
secrets:
|
||||
- database_url
|
||||
- s3_password
|
||||
depends_on:
|
||||
s3:
|
||||
condition: service_started
|
||||
restart: "no"
|
||||
migrate:
|
||||
build: .
|
||||
command: ["python", "-m", "messenger.migrate"]
|
||||
environment: *app-environment
|
||||
secrets:
|
||||
- database_url
|
||||
- s3_password
|
||||
depends_on:
|
||||
db:
|
||||
condition: service_healthy
|
||||
restart: "no"
|
||||
api1:
|
||||
<<: *api
|
||||
api2:
|
||||
<<: *api
|
||||
lb:
|
||||
image: haproxy:3.0.8-bookworm
|
||||
ports:
|
||||
- "127.0.0.1:8080:8080"
|
||||
volumes:
|
||||
- ./infra/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
|
||||
depends_on:
|
||||
api1:
|
||||
condition: service_healthy
|
||||
api2:
|
||||
condition: service_healthy
|
||||
volumes:
|
||||
postgres_data:
|
||||
s3_data:
|
||||
secrets:
|
||||
postgres_password:
|
||||
file: ./secrets/postgres_password
|
||||
database_url:
|
||||
file: ./secrets/database_url
|
||||
s3_password:
|
||||
file: ./secrets/s3_password
|
||||
@@ -0,0 +1,29 @@
|
||||
# Схема и транзакционные границы
|
||||
|
||||
Проект схемы составлен перед реализацией SQL-адаптера. PostgreSQL — единый источник истины, включая сессии.
|
||||
|
||||
| Таблица | Ключи и ограничения | Индексы и назначение |
|
||||
| --- | --- | --- |
|
||||
| users | UUID PK, username UNIQUE, NOT NULL для имени/хеша/времени | UNIQUE username для регистрации и входа |
|
||||
| chats | UUID PK, title NOT NULL | (created_at, id) для списка |
|
||||
| chat_members | (chat_id, user_id) PK, два FK, position для порядка участников | (user_id, chat_id) для доступных чатов |
|
||||
| messages | UUID PK, chat_id FK, sender_id FK, составной FK на членство | (chat_id, created_at, id) для истории |
|
||||
| sessions | token_hash PK, user_id FK, expires_at NOT NULL | expiry для возможной очистки |
|
||||
| app_settings | name PK, value NOT NULL | постоянный случайный ключ подписи курсоров |
|
||||
| schema_migrations | version PK | только учёт применённых миграций |
|
||||
|
||||
Каждый метод записи репозитория — одна транзакция с COMMIT до HTTP-ответа. Создание чата: проверить существование всех пользователей, вставить чат и всех участников, COMMIT; ошибка откатывает всё. Регистрация защищена UNIQUE, конфликт возвращает 409. Сообщение вставляется после проверки членства; FK также не позволяет записать отправителя вне чата. Создание/отзыв сессии — отдельные транзакции. Сессия проверяется на каждом защищённом запросе по времени БД; рестарт не меняет expiry.
|
||||
|
||||
Миграции нумеруются в migrations/*.sql. Runner выполняет новые файлы и запись версии в одной транзакции под advisory lock. Повторный запуск не выполняет уже учтённую версию. Пароли — Argon2id; в sessions лежит SHA-256 случайного 256-битного токена, а не пароль. SHA-256 здесь служит индексом высокоэнтропийного токена и не заменяет password KDF.
|
||||
|
||||
Пагинация SQL keyset, LIMIT + 1; offset не используется. Курсор сохраняет область ресурса и подписан постоянным ключом из БД, поэтому продолжение работает после рестарта. Часы PostgreSQL задают created_at. Между страницами нет snapshot; поздно закоммиченная конкурентная запись с ключом до курсора может не попасть в уже начатый обход. Для неизменной истории гарантируется отсутствие повторов и пропусков.
|
||||
|
||||
## Lab3: общий результат отправки
|
||||
|
||||
Миграция 002 добавляет message_requests с PK `(user_id, chat_id, idempotency_key)`, FK на сообщение и expires_at. Таблица относится только к POST сообщений: HTTP-метод и тип пути заданы её назначением. В ключ не входят имя реплики или сессия, поэтому новый вход того же пользователя не меняет область идемпотентности.
|
||||
|
||||
Перед записью PostgreSQL advisory transaction lock блокирует только хеш области ключа. Все реплики вычисляют его одинаково через SHA-256; совпадение 64-битных хешей может лишь сериализовать независимые операции, но не объединяет их результаты: lookup всегда по полному составному ключу. После ожидания блокировки новый statement в READ COMMITTED видит результат победившей транзакции.
|
||||
|
||||
При существующей действующей записи сравнивается текст: тот же текст возвращает исходное Message с 201, другой — 409. Поля и типы валидируются до этого; в lab3 единственное поле CreateMessage — text. Создание Message и message_requests происходит в одной транзакции, поэтому 5xx/rollback не оставляет отдельного подтверждённого результата. После COMMIT потерянный HTTP-ответ можно восстановить повтором. Expiry не продлевается повтором; TTL не менее 24 часов. Просроченная запись ключа удаляется при повторном использовании этой области, старое сообщение остаётся в истории.
|
||||
|
||||
Реплики не запускают миграции при старте. Единственный Compose job migrate выполняет их перед API; advisory lock runner дополнительно защищает ручной повтор запуска. Существующие миграции 001 сохранены без изменения. В lab3 нет автоматического обновления работающего стенда с несовместимым DDL.
|
||||
@@ -0,0 +1,5 @@
|
||||
# S3 and DB boundaries
|
||||
|
||||
`minio/minio:RELEASE.2025-02-07T23-21-09Z` runs as `s3` on internal port 9000; it has a persistent Docker volume `s3_data`. Console is not exposed and S3 port is not published. `s3-bootstrap` creates private `messenger-attachments` bucket idempotently. The API uploads bytes to S3 before committing PostgreSQL metadata; 201 follows commit. A failed SQL transaction triggers best-effort S3 delete. This does **not** guarantee orphan-free operation if delete or host process fails; full reconciliation belongs to level 4/5.
|
||||
|
||||
Only server-generated random object keys are used; filenames never become paths. GET enforces chat membership before contacting object storage. PostgreSQL `attachments` contains metadata and an internal key, while `message_attachments` preserves ordering. `s3_data` and `postgres_data` must survive restart. For local demonstration credentials live in ignored `secrets/` files; this level-3 exercise uses the same S3 root credential for bootstrap and API (narrow scoped credentials are a level-4 task). Do not deploy to public internet without TLS and restricted credentials.
|
||||
@@ -0,0 +1,41 @@
|
||||
global
|
||||
log stdout format raw local0
|
||||
maxconn 256
|
||||
|
||||
defaults
|
||||
log global
|
||||
mode http
|
||||
option dontlognull
|
||||
timeout connect 1s
|
||||
timeout client 10s
|
||||
timeout server 10s
|
||||
timeout check 1s
|
||||
timeout http-request 10s
|
||||
timeout http-keep-alive 2s
|
||||
# No transparent replay of POSTs. The client owns retry + Idempotency-Key.
|
||||
retries 0
|
||||
|
||||
resolvers docker
|
||||
nameserver dns 127.0.0.11:53
|
||||
resolve_retries 3
|
||||
timeout resolve 1s
|
||||
timeout retry 1s
|
||||
hold valid 2s
|
||||
hold nx 1s
|
||||
hold other 1s
|
||||
hold refused 1s
|
||||
hold timeout 1s
|
||||
hold obsolete 1s
|
||||
|
||||
frontend public
|
||||
bind :8080
|
||||
default_backend messenger
|
||||
|
||||
backend messenger
|
||||
balance roundrobin
|
||||
option httpchk
|
||||
http-check send meth GET uri /health/ready ver HTTP/1.1 hdr Host messenger
|
||||
http-check expect status 200
|
||||
default-server inter 1s fastinter 500ms downinter 1s fall 2 rise 1
|
||||
server api1 api1:8080 check resolvers docker resolve-prefer ipv4 init-addr libc,none
|
||||
server api2 api2:8080 check resolvers docker resolve-prefer ipv4 init-addr libc,none
|
||||
@@ -0,0 +1 @@
|
||||
"""Messenger lab series."""
|
||||
@@ -0,0 +1,41 @@
|
||||
"""One process per replica; bounded drain on SIGTERM."""
|
||||
import logging
|
||||
import signal
|
||||
import time
|
||||
from waitress import create_server
|
||||
from .api import create_app
|
||||
from .config import Settings
|
||||
|
||||
|
||||
def main():
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
settings = Settings.from_env()
|
||||
app = create_app(settings=settings)
|
||||
server = create_server(app, host=settings.host, port=settings.port, threads=4)
|
||||
stopping = False
|
||||
|
||||
def stop(signum, frame):
|
||||
nonlocal stopping
|
||||
stopping = True
|
||||
|
||||
signal.signal(signal.SIGTERM, stop)
|
||||
signal.signal(signal.SIGINT, stop)
|
||||
logging.info('Starting messenger environment=%s', settings.environment)
|
||||
try:
|
||||
while not stopping:
|
||||
server.asyncore.loop(timeout=0.1, count=1, map=server._map)
|
||||
# Stop accepting new TCP connections. Keep the event loop alive briefly
|
||||
# so responses from existing worker tasks can still be flushed.
|
||||
app.config['DRAINING'] = True
|
||||
server.close()
|
||||
deadline = time.monotonic() + 5
|
||||
while server._map and time.monotonic() < deadline:
|
||||
server.asyncore.loop(timeout=0.1, count=1, map=server._map)
|
||||
finally:
|
||||
server.task_dispatcher.shutdown(timeout=1)
|
||||
server.asyncore.close_all(map=server._map)
|
||||
logging.info('Messenger stopped')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,190 @@
|
||||
"""HTTP adapter and composition root."""
|
||||
import json
|
||||
import hashlib
|
||||
import re
|
||||
import unicodedata
|
||||
from uuid import UUID
|
||||
from .images import MAX_BYTES, validate_image
|
||||
from .storage import ObjectStore
|
||||
from uuid import uuid4
|
||||
from flask import Flask, g, request, jsonify, Response, stream_with_context
|
||||
from werkzeug.exceptions import HTTPException
|
||||
from .domain import ApiError, invalid, uuid_value
|
||||
from .postgres import PostgresRepository
|
||||
from .database import Database
|
||||
from .config import Settings
|
||||
from .service import Messenger
|
||||
|
||||
|
||||
def create_app(repository=None, settings=None):
|
||||
app = Flask(__name__)
|
||||
app.json.ensure_ascii = False
|
||||
settings = settings or Settings.from_env()
|
||||
repository = repository if repository is not None else PostgresRepository(Database(settings.database_url))
|
||||
service = Messenger(repository, settings.session_ttl, settings.idempotency_ttl)
|
||||
instance_id = str(uuid4())
|
||||
store = None if app.config.get("TESTING_STORAGE_DISABLED") else (ObjectStore(settings) if repository is None or isinstance(repository, PostgresRepository) else None)
|
||||
|
||||
@app.before_request
|
||||
def identify():
|
||||
g.request_id = str(uuid4())
|
||||
if app.config.get('DRAINING'):
|
||||
raise ApiError(503, 'unavailable', 'Instance is shutting down')
|
||||
if request.path.startswith('/api/v1/') and not (
|
||||
request.path in ('/api/v1/users', '/api/v1/auth/sessions') and request.method == 'POST'
|
||||
):
|
||||
g.user, g.token_hash = service.auth.bearer(request.headers.get('Authorization'))
|
||||
|
||||
@app.after_request
|
||||
def request_id(response):
|
||||
response.headers['X-Request-Id'] = g.request_id
|
||||
response.headers['X-Instance-Id'] = instance_id
|
||||
return response
|
||||
|
||||
@app.errorhandler(Exception)
|
||||
def error(exc):
|
||||
if isinstance(exc, ApiError):
|
||||
status, code, message = exc.status, exc.code, exc.message
|
||||
elif isinstance(exc, HTTPException):
|
||||
status = exc.code
|
||||
code = {404: 'not_found', 413: 'payload_too_large', 415: 'unsupported_media_type'}.get(status, 'invalid_request')
|
||||
message = exc.name
|
||||
else:
|
||||
app.logger.error('Unhandled error; request_id=%s', g.request_id)
|
||||
status, code, message = 500, 'internal_error', 'Internal server error'
|
||||
response = jsonify(error={'code': code, 'message': message, 'request_id': g.request_id})
|
||||
response.status_code = status
|
||||
if status == 401:
|
||||
response.headers['WWW-Authenticate'] = (
|
||||
'Basic realm="messenger", charset="UTF-8"'
|
||||
if request.path == '/api/v1/auth/sessions' and request.method == 'POST'
|
||||
else 'Bearer realm="messenger"')
|
||||
return response
|
||||
|
||||
def body():
|
||||
if request.mimetype != 'application/json':
|
||||
raise ApiError(415, 'unsupported_media_type', 'Use application/json')
|
||||
try:
|
||||
def reject_constant(value):
|
||||
raise ValueError(value)
|
||||
return json.loads(request.get_data().decode('utf-8'), parse_constant=reject_constant)
|
||||
except (ValueError, UnicodeError):
|
||||
invalid('Malformed UTF-8 JSON')
|
||||
|
||||
def paging():
|
||||
for key in ('limit', 'cursor'):
|
||||
if len(request.args.getlist(key)) > 1:
|
||||
invalid('Duplicate query parameter')
|
||||
return request.args.get('limit', '50'), request.args.get('cursor')
|
||||
|
||||
@app.get('/health/live')
|
||||
def health():
|
||||
return jsonify(status='ok')
|
||||
|
||||
@app.get('/health/ready')
|
||||
def ready():
|
||||
repository.ready()
|
||||
if store is not None:
|
||||
store.ready()
|
||||
return jsonify(status='ok')
|
||||
|
||||
@app.post('/api/v1/auth/sessions')
|
||||
def login():
|
||||
# Session creation accepts no request body.
|
||||
if request.get_data():
|
||||
invalid('Session creation does not accept a body')
|
||||
return jsonify(service.auth.login(request.headers.get('Authorization'))), 201
|
||||
|
||||
@app.delete('/api/v1/auth/sessions/current')
|
||||
def logout():
|
||||
repository.revoke_session(g.token_hash)
|
||||
return '', 204
|
||||
|
||||
@app.post('/api/v1/users')
|
||||
def register():
|
||||
return jsonify(service.register(body())), 201
|
||||
|
||||
@app.get('/api/v1/users/me')
|
||||
def me():
|
||||
return jsonify(g.user)
|
||||
|
||||
@app.route('/api/v1/chats', methods=['GET', 'POST'])
|
||||
def chats():
|
||||
if request.method == 'POST':
|
||||
return jsonify(service.create_chat(g.user['id'], body())), 201
|
||||
return jsonify(service.chats(g.user['id'], *paging()))
|
||||
|
||||
@app.get('/api/v1/chats/<chat_id>')
|
||||
def chat(chat_id):
|
||||
return jsonify(service.chat(g.user['id'], chat_id))
|
||||
|
||||
@app.route('/api/v1/chats/<chat_id>/messages', methods=['GET', 'POST'])
|
||||
def messages(chat_id):
|
||||
if request.method == 'POST':
|
||||
service.chat(g.user['id'], chat_id)
|
||||
return jsonify(service.send(g.user['id'], chat_id, body(), request.headers.get('Idempotency-Key'))), 201
|
||||
return jsonify(service.history(g.user['id'], chat_id, *paging()))
|
||||
|
||||
@app.post('/api/v1/chats/<chat_id>/attachments')
|
||||
def upload(chat_id):
|
||||
chat_id = uuid_value(chat_id)
|
||||
service.chat(g.user['id'], chat_id)
|
||||
values = request.args.getlist('filename')
|
||||
if len(values) != 1:
|
||||
invalid('Exactly one filename is required')
|
||||
filename = values[0]
|
||||
if (not 1 <= len(filename) <= 255 or any(
|
||||
c in ('/', '\\') or unicodedata.category(c) == 'Cc' for c in filename)):
|
||||
invalid('Unsafe filename')
|
||||
if request.mimetype not in ('image/png', 'image/jpeg'):
|
||||
raise ApiError(415, 'unsupported_media_type', 'PNG or JPEG required')
|
||||
if request.content_length is not None and request.content_length > MAX_BYTES:
|
||||
raise ApiError(413, 'payload_too_large', 'Image exceeds 10 MiB')
|
||||
# Bounded read even if Content-Length is absent or misleading.
|
||||
content = request.stream.read(MAX_BYTES + 1)
|
||||
if len(content) > MAX_BYTES:
|
||||
raise ApiError(413, 'payload_too_large', 'Image exceeds 10 MiB')
|
||||
validate_image(content, request.mimetype)
|
||||
if store is None:
|
||||
raise ApiError(503, 'unavailable', 'Object storage unavailable')
|
||||
identifier = str(uuid4())
|
||||
key = 'originals/' + uuid4().hex + '/' + uuid4().hex
|
||||
metadata = {'id': identifier, 'filename': filename, 'content_type': request.mimetype,
|
||||
'size_bytes': len(content), 'sha256': hashlib.sha256(content).hexdigest(),
|
||||
'object_key': key}
|
||||
store.put(key, content, request.mimetype)
|
||||
try:
|
||||
result = repository.insert_attachment(g.user['id'], chat_id, metadata)
|
||||
except Exception:
|
||||
store.delete(key) # best effort compensation for DB commit failure
|
||||
raise
|
||||
return jsonify(result), 201
|
||||
|
||||
@app.get('/api/v1/attachments/<attachment_id>')
|
||||
def attachment(attachment_id):
|
||||
obj, _key = repository.attachment(g.user['id'], uuid_value(attachment_id))
|
||||
return jsonify(obj)
|
||||
|
||||
@app.get('/api/v1/attachments/<attachment_id>/content')
|
||||
def download(attachment_id):
|
||||
obj, key = repository.attachment(g.user['id'], uuid_value(attachment_id))
|
||||
variants = request.args.getlist('variant')
|
||||
if len(variants) > 1:
|
||||
invalid('Duplicate variant')
|
||||
if variants and variants[0] != 'original':
|
||||
raise ApiError(404, 'not_found', 'Variant not found')
|
||||
if store is None:
|
||||
raise ApiError(503, 'unavailable', 'Object storage unavailable')
|
||||
body = store.get(key)
|
||||
def chunks():
|
||||
try:
|
||||
for chunk in body.iter_chunks(chunk_size=65536):
|
||||
if chunk:
|
||||
yield chunk
|
||||
finally:
|
||||
body.close()
|
||||
return Response(stream_with_context(chunks()), mimetype=obj['content_type'],
|
||||
headers={'Content-Length': str(obj['size_bytes']),
|
||||
'Cache-Control': 'private, no-store'})
|
||||
|
||||
return app
|
||||
@@ -0,0 +1,27 @@
|
||||
"""Run after S3 readiness, repeatable. Default MinIO bucket is private."""
|
||||
import time
|
||||
from botocore.exceptions import ClientError, BotoCoreError
|
||||
from .config import Settings
|
||||
from .storage import ObjectStore
|
||||
|
||||
def main():
|
||||
storage = ObjectStore(Settings.from_env())
|
||||
for attempt in range(40):
|
||||
try:
|
||||
storage.s3.list_buckets()
|
||||
break
|
||||
except (ClientError, BotoCoreError, OSError):
|
||||
if attempt == 39:
|
||||
raise
|
||||
time.sleep(2)
|
||||
try:
|
||||
storage.s3.head_bucket(Bucket=storage.bucket)
|
||||
except ClientError as exc:
|
||||
if exc.response.get('ResponseMetadata', {}).get('HTTPStatusCode') not in (404,):
|
||||
raise
|
||||
storage.s3.create_bucket(Bucket=storage.bucket)
|
||||
# Public ACL/policy intentionally absent; MinIO bucket is private by default.
|
||||
print('Private S3 bucket exists')
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,40 @@
|
||||
"""Configuration is loaded only at process startup; secrets are never logged."""
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Settings:
|
||||
database_url: str
|
||||
host: str = '0.0.0.0'
|
||||
port: int = 8080
|
||||
environment: str = 'lab'
|
||||
session_ttl: int = 3600
|
||||
idempotency_ttl: int = 86400
|
||||
s3_endpoint: str = "http://s3:9000"
|
||||
s3_bucket: str = "messenger-attachments"
|
||||
s3_access_key: str = "messengeradmin"
|
||||
s3_secret_file: str = "/run/secrets/s3_password"
|
||||
|
||||
@classmethod
|
||||
def from_env(cls):
|
||||
source = os.getenv('DATABASE_URL_FILE')
|
||||
url = Path(source).read_text().strip() if source else os.getenv('DATABASE_URL', '')
|
||||
if not url:
|
||||
raise ValueError('Set DATABASE_URL_FILE or DATABASE_URL')
|
||||
if os.getenv('SESSION_STORE', 'postgres') != 'postgres':
|
||||
raise ValueError('This lab uses SESSION_STORE=postgres')
|
||||
port = int(os.getenv('APP_PORT', '8080'))
|
||||
ttl = int(os.getenv('SESSION_TTL_SECONDS', '3600'))
|
||||
idempotency_ttl = int(os.getenv('IDEMPOTENCY_TTL_SECONDS', '86400'))
|
||||
if idempotency_ttl < 86400:
|
||||
raise ValueError('IDEMPOTENCY_TTL_SECONDS must be at least 86400')
|
||||
if not 1 <= port <= 65535 or ttl <= 0:
|
||||
raise ValueError('Invalid APP_PORT or SESSION_TTL_SECONDS')
|
||||
return cls(url, os.getenv('APP_HOST', '0.0.0.0'), port,
|
||||
os.getenv('APP_ENV', 'lab'), ttl, idempotency_ttl,
|
||||
os.getenv("S3_ENDPOINT", "http://s3:9000"),
|
||||
os.getenv("S3_BUCKET", "messenger-attachments"),
|
||||
os.getenv("S3_ACCESS_KEY", "messengeradmin"),
|
||||
os.getenv("S3_SECRET_FILE", "/run/secrets/s3_password"))
|
||||
@@ -0,0 +1,20 @@
|
||||
"""Short-lived database connections, one transaction per operation."""
|
||||
from contextlib import contextmanager
|
||||
import psycopg
|
||||
from psycopg.rows import dict_row
|
||||
from .domain import ApiError
|
||||
|
||||
|
||||
class Database:
|
||||
def __init__(self, url):
|
||||
self.url = url
|
||||
|
||||
@contextmanager
|
||||
def transaction(self):
|
||||
try:
|
||||
with psycopg.connect(self.url, row_factory=dict_row, connect_timeout=3,
|
||||
options='-c statement_timeout=5000 -c lock_timeout=3000 -c timezone=UTC') as conn:
|
||||
yield conn
|
||||
except (psycopg.OperationalError, psycopg.InterfaceError,
|
||||
psycopg.errors.QueryCanceled, psycopg.errors.LockNotAvailable):
|
||||
raise ApiError(503, 'unavailable', 'Database temporarily unavailable') from None
|
||||
@@ -0,0 +1,33 @@
|
||||
"""Framework-independent validation and application errors."""
|
||||
import re
|
||||
from uuid import UUID
|
||||
|
||||
|
||||
class ApiError(Exception):
|
||||
def __init__(self, status, code, message):
|
||||
self.status, self.code, self.message = status, code, message
|
||||
|
||||
|
||||
def invalid(message='Invalid request'):
|
||||
raise ApiError(400, 'invalid_request', message)
|
||||
|
||||
|
||||
def uuid_value(value):
|
||||
if not isinstance(value, str) or not re.fullmatch(
|
||||
r'[0-9a-fA-F]{8}(?:-[0-9a-fA-F]{4}){3}-[0-9a-fA-F]{12}', value
|
||||
):
|
||||
invalid('Invalid UUID')
|
||||
return str(UUID(value))
|
||||
|
||||
|
||||
def fields(body, names):
|
||||
if not isinstance(body, dict) or set(body) != set(names):
|
||||
invalid('Required fields are missing or unknown fields supplied')
|
||||
|
||||
|
||||
def string(value, maximum):
|
||||
if (not isinstance(value, str) or not value.strip()
|
||||
or len(value) > maximum or "\x00" in value
|
||||
or any(0xD800 <= ord(c) <= 0xDFFF for c in value)):
|
||||
invalid('Invalid string')
|
||||
return value
|
||||
@@ -0,0 +1,30 @@
|
||||
"""Bounded image validation: magic+full decode, no trust in declared MIME."""
|
||||
import io
|
||||
from PIL import Image, ImageFile, UnidentifiedImageError
|
||||
from .domain import ApiError
|
||||
|
||||
MAX_BYTES = 10 * 1024 * 1024
|
||||
MAX_PIXELS = 40_000_000
|
||||
Image.MAX_IMAGE_PIXELS = MAX_PIXELS
|
||||
ImageFile.LOAD_TRUNCATED_IMAGES = False
|
||||
|
||||
|
||||
def validate_image(data, declared_type):
|
||||
if declared_type not in ('image/png', 'image/jpeg'):
|
||||
raise ApiError(415, 'unsupported_media_type', 'PNG or JPEG required')
|
||||
if len(data) > MAX_BYTES:
|
||||
raise ApiError(413, 'payload_too_large', 'Image exceeds 10 MiB')
|
||||
if not data:
|
||||
raise ApiError(415, 'unsupported_media_type', 'Empty image')
|
||||
try:
|
||||
with Image.open(io.BytesIO(data)) as im:
|
||||
real_type = {'PNG': 'image/png', 'JPEG': 'image/jpeg'}.get(im.format)
|
||||
if real_type is None or real_type != declared_type:
|
||||
raise ApiError(415, 'unsupported_media_type', 'Image content type mismatch')
|
||||
if im.width * im.height > MAX_PIXELS:
|
||||
raise ApiError(413, 'payload_too_large', 'Image pixel limit exceeded')
|
||||
im.load() # reject malformed/truncated payload before PUT
|
||||
except (UnidentifiedImageError, OSError, ValueError, SyntaxError):
|
||||
raise ApiError(415, 'unsupported_media_type', 'Invalid image') from None
|
||||
except Image.DecompressionBombError:
|
||||
raise ApiError(413, 'payload_too_large', 'Image pixel limit exceeded') from None
|
||||
@@ -0,0 +1,24 @@
|
||||
"""Versioned, transactional migrations; safe to run again."""
|
||||
from pathlib import Path
|
||||
import secrets
|
||||
from .config import Settings
|
||||
from .database import Database
|
||||
|
||||
|
||||
def migrate(database):
|
||||
directory = Path(__file__).resolve().parents[1] / 'migrations'
|
||||
with database.transaction() as conn:
|
||||
conn.execute('SELECT pg_advisory_xact_lock(20260201)')
|
||||
conn.execute('CREATE TABLE IF NOT EXISTS schema_migrations (version text PRIMARY KEY)')
|
||||
versions = {r['version'] for r in conn.execute('SELECT version FROM schema_migrations')}
|
||||
for path in sorted(directory.glob('*.sql')):
|
||||
if path.name not in versions:
|
||||
conn.execute(path.read_text(encoding='utf-8'))
|
||||
conn.execute('INSERT INTO schema_migrations(version) VALUES (%s)', (path.name,))
|
||||
conn.execute("INSERT INTO app_settings(name,value) VALUES ('cursor_key',%s) ON CONFLICT DO NOTHING",
|
||||
(secrets.token_hex(32),))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
migrate(Database(Settings.from_env().database_url))
|
||||
print('Migrations applied successfully (already applied versions skipped).')
|
||||
@@ -0,0 +1,43 @@
|
||||
"""Resource-scoped signed cursors; SQL fetches only limit+1 rows."""
|
||||
import base64
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import re
|
||||
from datetime import datetime
|
||||
from .domain import invalid, uuid_value
|
||||
|
||||
|
||||
class Pagination:
|
||||
def __init__(self, key):
|
||||
self._key = key
|
||||
|
||||
def decode(self, scope, limit, cursor):
|
||||
if not re.fullmatch(r'[0-9]{1,3}', limit) or not 1 <= int(limit) <= 100:
|
||||
invalid('Limit must be between 1 and 100')
|
||||
after = None
|
||||
if cursor is not None:
|
||||
try:
|
||||
if len(cursor) > 2048:
|
||||
raise ValueError()
|
||||
raw = base64.b64decode(cursor, altchars=b'-_', validate=True)
|
||||
payload, signature = raw[:-32], raw[-32:]
|
||||
if not hmac.compare_digest(signature, hmac.digest(self._key, payload, hashlib.sha256)):
|
||||
raise ValueError()
|
||||
saved_scope, timestamp, identifier = json.loads(payload)
|
||||
if saved_scope != scope or not isinstance(timestamp, str):
|
||||
raise ValueError()
|
||||
datetime.fromisoformat(timestamp.replace('Z', '+00:00'))
|
||||
after = (timestamp, uuid_value(identifier))
|
||||
except (ValueError, TypeError, UnicodeError):
|
||||
invalid('Invalid cursor for this resource')
|
||||
return int(limit), after
|
||||
|
||||
def page(self, rows, scope, limit):
|
||||
items = rows[:limit]
|
||||
next_cursor = None
|
||||
if len(rows) > limit:
|
||||
last = items[-1]
|
||||
payload = json.dumps([scope, last['created_at'], last['id']], separators=(',', ':')).encode()
|
||||
next_cursor = base64.urlsafe_b64encode(payload + hmac.digest(self._key, payload, hashlib.sha256)).decode()
|
||||
return {'items': items, 'next_cursor': next_cursor}
|
||||
@@ -0,0 +1,196 @@
|
||||
"""PostgreSQL repository: all SQL values are bound parameters."""
|
||||
from datetime import datetime, timezone
|
||||
import hashlib
|
||||
import json
|
||||
from uuid import UUID, uuid4
|
||||
import psycopg
|
||||
from .domain import ApiError, invalid
|
||||
|
||||
|
||||
def serialize(row):
|
||||
if row is None:
|
||||
return None
|
||||
result = {}
|
||||
for key, value in row.items():
|
||||
if isinstance(value, UUID):
|
||||
value = str(value)
|
||||
elif isinstance(value, datetime):
|
||||
value = value.astimezone(timezone.utc).isoformat(timespec='microseconds').replace('+00:00', 'Z')
|
||||
elif isinstance(value, list):
|
||||
value = [str(item) if isinstance(item, UUID) else item for item in value]
|
||||
result[key] = value
|
||||
return result
|
||||
|
||||
|
||||
USER_COLUMNS = 'id, username, display_name, created_at'
|
||||
# Members retain the request order, with no password/session fields in public objects.
|
||||
CHAT_COLUMNS = '''c.id, c.title, c.created_at,
|
||||
ARRAY(SELECT m.user_id FROM chat_members m WHERE m.chat_id=c.id ORDER BY m.position) AS member_ids'''
|
||||
|
||||
|
||||
class PostgresRepository:
|
||||
def __init__(self, database):
|
||||
self.db = database
|
||||
|
||||
def ready(self):
|
||||
with self.db.transaction() as conn:
|
||||
conn.execute('SELECT id FROM users LIMIT 0')
|
||||
conn.execute('SELECT token_hash FROM sessions LIMIT 0')
|
||||
|
||||
def cursor_key(self):
|
||||
with self.db.transaction() as conn:
|
||||
row = conn.execute("SELECT value FROM app_settings WHERE name='cursor_key'").fetchone()
|
||||
if row is None:
|
||||
raise ApiError(503, 'unavailable', 'Database is not initialized')
|
||||
return bytes.fromhex(row['value'])
|
||||
|
||||
def create_user(self, username, display_name, password_hash):
|
||||
try:
|
||||
with self.db.transaction() as conn:
|
||||
row = conn.execute('INSERT INTO users(id,username,display_name,password_hash) VALUES (%s,%s,%s,%s) RETURNING ' + USER_COLUMNS,
|
||||
(uuid4(), username, display_name, password_hash)).fetchone()
|
||||
return serialize(row)
|
||||
except psycopg.errors.UniqueViolation:
|
||||
raise ApiError(409, 'conflict', 'Username already exists') from None
|
||||
|
||||
def credentials(self, username):
|
||||
with self.db.transaction() as conn:
|
||||
return serialize(conn.execute('SELECT ' + USER_COLUMNS + ', password_hash FROM users WHERE username=%s',
|
||||
(username,)).fetchone())
|
||||
|
||||
def create_session(self, user_id, token_hash, ttl):
|
||||
with self.db.transaction() as conn:
|
||||
row = conn.execute('INSERT INTO sessions(token_hash,user_id,expires_at) VALUES (%s,%s,clock_timestamp() + %s * interval \'1 second\') RETURNING expires_at',
|
||||
(token_hash, user_id, ttl)).fetchone()
|
||||
return serialize(row)['expires_at']
|
||||
|
||||
def session_user(self, token_hash):
|
||||
with self.db.transaction() as conn:
|
||||
row = conn.execute('''SELECT u.id,u.username,u.display_name,u.created_at FROM users u
|
||||
JOIN sessions s ON s.user_id=u.id
|
||||
WHERE s.token_hash=%s AND s.expires_at>clock_timestamp()''', (token_hash,)).fetchone()
|
||||
if row is None:
|
||||
raise ApiError(401, 'unauthorized', 'Valid Bearer session required')
|
||||
return serialize(row)
|
||||
|
||||
def revoke_session(self, token_hash):
|
||||
with self.db.transaction() as conn:
|
||||
conn.execute('DELETE FROM sessions WHERE token_hash=%s', (token_hash,))
|
||||
|
||||
@staticmethod
|
||||
def _access(conn, actor, chat_id):
|
||||
if conn.execute('SELECT 1 FROM chat_members WHERE chat_id=%s AND user_id=%s', (chat_id, actor)).fetchone() is None:
|
||||
raise ApiError(404, 'not_found', 'Chat not found')
|
||||
|
||||
def create_chat(self, actor, title, members):
|
||||
with self.db.transaction() as conn:
|
||||
count = conn.execute('SELECT count(*) AS n FROM users WHERE id=ANY(%s::uuid[])', (members,)).fetchone()['n']
|
||||
if actor not in members or count != len(members):
|
||||
invalid('Members must exist and include the current user')
|
||||
identifier = uuid4()
|
||||
conn.execute('INSERT INTO chats(id,title) VALUES (%s,%s)', (identifier, title))
|
||||
with conn.cursor() as cur:
|
||||
cur.executemany('INSERT INTO chat_members(chat_id,user_id,position) VALUES (%s,%s,%s)',
|
||||
[(identifier, member, i) for i, member in enumerate(members)])
|
||||
row = conn.execute('SELECT ' + CHAT_COLUMNS + ' FROM chats c WHERE c.id=%s', (identifier,)).fetchone()
|
||||
return serialize(row)
|
||||
|
||||
def chat(self, actor, chat_id):
|
||||
with self.db.transaction() as conn:
|
||||
self._access(conn, actor, chat_id)
|
||||
row = conn.execute('SELECT ' + CHAT_COLUMNS + ' FROM chats c WHERE c.id=%s', (chat_id,)).fetchone()
|
||||
return serialize(row)
|
||||
|
||||
def chats(self, actor, limit, after):
|
||||
params = [actor]
|
||||
boundary = ''
|
||||
if after is not None:
|
||||
boundary = ' AND (c.created_at,c.id)>(%s::timestamptz,%s::uuid)'
|
||||
params.extend(after)
|
||||
params.append(limit)
|
||||
with self.db.transaction() as conn:
|
||||
rows = conn.execute('SELECT ' + CHAT_COLUMNS + ''' FROM chats c JOIN chat_members own
|
||||
ON own.chat_id=c.id WHERE own.user_id=%s''' + boundary +
|
||||
' ORDER BY c.created_at,c.id LIMIT %s', params).fetchall()
|
||||
return [serialize(r) for r in rows]
|
||||
|
||||
@staticmethod
|
||||
def _message(conn, message_id):
|
||||
row = conn.execute('SELECT * FROM messages WHERE id=%s', (message_id,)).fetchone()
|
||||
result = serialize(row)
|
||||
result['attachment_ids'] = [str(r['attachment_id']) for r in conn.execute(
|
||||
'SELECT attachment_id FROM message_attachments WHERE message_id=%s ORDER BY position',
|
||||
(message_id,)).fetchall()]
|
||||
return result
|
||||
|
||||
def create_message(self, actor, chat_id, text, attachments, key, ttl):
|
||||
scope = json.dumps([actor, chat_id, key], separators=(',', ':')).encode()
|
||||
lock_id = int.from_bytes(hashlib.sha256(scope).digest()[:8], 'big', signed=True)
|
||||
with self.db.transaction() as conn:
|
||||
self._access(conn, actor, chat_id)
|
||||
conn.execute('SELECT pg_advisory_xact_lock(%s)', (lock_id,))
|
||||
previous = conn.execute("""SELECT request_text, request_attachments, message_id
|
||||
FROM message_requests WHERE user_id=%s AND chat_id=%s AND idempotency_key=%s
|
||||
AND expires_at>clock_timestamp()""", (actor, chat_id, key)).fetchone()
|
||||
if previous:
|
||||
if previous['request_text'] != text or previous['request_attachments'] != attachments:
|
||||
raise ApiError(409, 'conflict', 'Idempotency key already used with another message')
|
||||
return self._message(conn, previous['message_id'])
|
||||
conn.execute('DELETE FROM message_requests WHERE user_id=%s AND chat_id=%s AND idempotency_key=%s',
|
||||
(actor, chat_id, key))
|
||||
if attachments:
|
||||
found = conn.execute('SELECT id FROM attachments WHERE chat_id=%s AND id=ANY(%s::uuid[])',
|
||||
(chat_id, attachments)).fetchall()
|
||||
if len(found) != len(attachments):
|
||||
raise ApiError(404, 'not_found', 'Attachment not found')
|
||||
message_id = uuid4()
|
||||
conn.execute('INSERT INTO messages(id,chat_id,sender_id,text) VALUES (%s,%s,%s,%s)',
|
||||
(message_id, chat_id, actor, text))
|
||||
for i, aid in enumerate(attachments):
|
||||
conn.execute('INSERT INTO message_attachments(message_id,attachment_id,position) VALUES (%s,%s,%s)',
|
||||
(message_id, aid, i))
|
||||
conn.execute("""INSERT INTO message_requests
|
||||
(user_id,chat_id,idempotency_key,request_text,request_attachments,message_id,expires_at)
|
||||
VALUES (%s,%s,%s,%s,%s::jsonb,%s,clock_timestamp() + %s * interval '1 second')""",
|
||||
(actor, chat_id, key, text, json.dumps(attachments), message_id, ttl))
|
||||
return self._message(conn, message_id)
|
||||
|
||||
def messages(self, actor, chat_id, limit, after):
|
||||
params = [chat_id]
|
||||
boundary = ''
|
||||
if after is not None:
|
||||
boundary = ' AND (created_at,id)>(%s::timestamptz,%s::uuid)'
|
||||
params.extend(after)
|
||||
params.append(limit)
|
||||
with self.db.transaction() as conn:
|
||||
self._access(conn, actor, chat_id)
|
||||
rows = conn.execute('SELECT id FROM messages WHERE chat_id=%s' + boundary +
|
||||
' ORDER BY created_at,id LIMIT %s', params).fetchall()
|
||||
return [self._message(conn, row['id']) for row in rows]
|
||||
|
||||
@staticmethod
|
||||
def _attachment(row):
|
||||
obj = serialize(row)
|
||||
obj.pop('object_key', None)
|
||||
obj['status'] = 'ready'
|
||||
obj['variants'] = []
|
||||
obj['error_code'] = None
|
||||
return obj
|
||||
|
||||
def insert_attachment(self, actor, chat_id, metadata):
|
||||
with self.db.transaction() as conn:
|
||||
self._access(conn, actor, chat_id)
|
||||
row = conn.execute("""INSERT INTO attachments
|
||||
(id,chat_id,owner_id,filename,content_type,size_bytes,sha256,object_key)
|
||||
VALUES (%s,%s,%s,%s,%s,%s,%s,%s) RETURNING *""",
|
||||
(metadata['id'], chat_id, actor, metadata['filename'], metadata['content_type'],
|
||||
metadata['size_bytes'], metadata['sha256'], metadata['object_key'])).fetchone()
|
||||
return self._attachment(row)
|
||||
|
||||
def attachment(self, actor, attachment_id):
|
||||
with self.db.transaction() as conn:
|
||||
row = conn.execute("""SELECT a.* FROM attachments a JOIN chat_members m ON m.chat_id=a.chat_id
|
||||
WHERE a.id=%s AND m.user_id=%s""", (attachment_id, actor)).fetchone()
|
||||
if row is None:
|
||||
raise ApiError(404, 'not_found', 'Attachment not found')
|
||||
return self._attachment(row), row['object_key']
|
||||
@@ -0,0 +1,17 @@
|
||||
"""Storage port. Concrete PostgreSQL adapter is wired in api.create_app."""
|
||||
from typing import Protocol
|
||||
|
||||
|
||||
class Repository(Protocol):
|
||||
def ready(self): ...
|
||||
def cursor_key(self): ...
|
||||
def create_user(self, username, display_name, password_hash): ...
|
||||
def credentials(self, username): ...
|
||||
def create_session(self, user_id, token_hash, ttl): ...
|
||||
def session_user(self, token_hash): ...
|
||||
def revoke_session(self, token_hash): ...
|
||||
def create_chat(self, actor, title, members): ...
|
||||
def chat(self, actor, chat_id): ...
|
||||
def chats(self, actor, limit, after): ...
|
||||
def create_message(self, actor, chat_id, text, key, ttl): ...
|
||||
def messages(self, actor, chat_id, limit, after): ...
|
||||
@@ -0,0 +1,60 @@
|
||||
"""Password KDF and HTTP credential parsing, without framework dependencies."""
|
||||
import base64
|
||||
import binascii
|
||||
import hashlib
|
||||
import re
|
||||
import secrets
|
||||
from argon2 import PasswordHasher, Type
|
||||
from argon2.exceptions import VerificationError, InvalidHashError
|
||||
from .domain import ApiError, invalid, string
|
||||
|
||||
|
||||
class Authentication:
|
||||
def __init__(self, repository, ttl):
|
||||
self.repository = repository
|
||||
self.ttl = ttl
|
||||
self.hasher = PasswordHasher(time_cost=3, memory_cost=65536, parallelism=2,
|
||||
hash_len=32, salt_len=16, type=Type.ID)
|
||||
self._dummy = self.hasher.hash(secrets.token_urlsafe(32))
|
||||
|
||||
@staticmethod
|
||||
def password(value):
|
||||
string(value, 128)
|
||||
if len(value) < 12:
|
||||
invalid('Password must contain 12 to 128 Unicode characters')
|
||||
return value
|
||||
|
||||
@staticmethod
|
||||
def digest(token):
|
||||
return hashlib.sha256(token.encode('ascii')).hexdigest()
|
||||
|
||||
def bearer(self, header):
|
||||
parts = (header or '').split()
|
||||
if len(parts) != 2 or parts[0].lower() != 'bearer' or not re.fullmatch(r'[A-Za-z0-9_-]{43}', parts[1]):
|
||||
raise ApiError(401, 'unauthorized', 'Valid Bearer session required')
|
||||
token_hash = self.digest(parts[1])
|
||||
return self.repository.session_user(token_hash), token_hash
|
||||
|
||||
def login(self, header):
|
||||
def denied():
|
||||
raise ApiError(401, 'unauthorized', 'Invalid credentials')
|
||||
try:
|
||||
parts = (header or '').split()
|
||||
if len(parts) != 2 or parts[0].lower() != 'basic':
|
||||
denied()
|
||||
username, password = base64.b64decode(parts[1], validate=True).decode('utf-8').split(':', 1)
|
||||
if not re.fullmatch(r'[a-z0-9_]{3,32}', username) or not 12 <= len(password) <= 128:
|
||||
denied()
|
||||
except (ValueError, UnicodeError, binascii.Error):
|
||||
denied()
|
||||
row = self.repository.credentials(username)
|
||||
try:
|
||||
self.hasher.verify(row['password_hash'] if row else self._dummy, password)
|
||||
except (VerificationError, InvalidHashError):
|
||||
denied()
|
||||
if row is None:
|
||||
denied()
|
||||
user = {key: value for key, value in row.items() if key != 'password_hash'}
|
||||
token = secrets.token_urlsafe(32)
|
||||
expires_at = self.repository.create_session(user['id'], self.digest(token), self.ttl)
|
||||
return {'token': token, 'token_type': 'Bearer', 'expires_at': expires_at, 'user': user}
|
||||
@@ -0,0 +1,68 @@
|
||||
"""Use cases; independent of Flask and the concrete storage adapter."""
|
||||
import re
|
||||
from .domain import fields, string, uuid_value, invalid, ApiError
|
||||
from .repository import Repository
|
||||
from .pagination import Pagination
|
||||
from .security import Authentication
|
||||
|
||||
|
||||
class Messenger:
|
||||
def __init__(self, repository: Repository, session_ttl=3600, idempotency_ttl=86400):
|
||||
self.repository = repository
|
||||
self.idempotency_ttl = idempotency_ttl
|
||||
self.pagination = Pagination(repository.cursor_key())
|
||||
self.auth = Authentication(repository, session_ttl)
|
||||
|
||||
def register(self, body):
|
||||
fields(body, ('username', 'display_name', 'password'))
|
||||
username = string(body['username'], 32)
|
||||
if not re.fullmatch(r'[a-z0-9_]{3,32}', username):
|
||||
invalid('Invalid username')
|
||||
display_name = string(body['display_name'], 100)
|
||||
password = self.auth.password(body['password'])
|
||||
return self.repository.create_user(username, display_name, self.auth.hasher.hash(password))
|
||||
|
||||
def create_chat(self, actor, body):
|
||||
fields(body, ('title', 'member_ids'))
|
||||
title = string(body['title'], 100)
|
||||
members = body['member_ids']
|
||||
if not isinstance(members, list) or not 1 <= len(members) <= 100:
|
||||
invalid('Invalid member list')
|
||||
members = [uuid_value(m) for m in members]
|
||||
if len(set(members)) != len(members):
|
||||
invalid('Duplicate members')
|
||||
return self.repository.create_chat(actor, title, members)
|
||||
|
||||
def chat(self, actor, chat_id):
|
||||
return self.repository.chat(actor, uuid_value(chat_id))
|
||||
|
||||
def send(self, actor, chat_id, body, key):
|
||||
self.chat(actor, chat_id)
|
||||
if not isinstance(key, str) or not re.fullmatch(r'[A-Za-z0-9._:-]{1,128}', key):
|
||||
invalid('Valid Idempotency-Key required')
|
||||
if not isinstance(body, dict) or set(body) not in ({'text'}, {'text', 'attachment_ids'}):
|
||||
invalid('Invalid message fields')
|
||||
if not isinstance(body['text'], str) or len(body['text']) > 4000 or '\x00' in body['text']:
|
||||
invalid('Invalid message text')
|
||||
attachments = body.get('attachment_ids', [])
|
||||
if not isinstance(attachments, list) or len(attachments) > 10:
|
||||
invalid('Invalid attachment IDs')
|
||||
attachments = [uuid_value(item) for item in attachments]
|
||||
if len(attachments) != len(set(attachments)):
|
||||
invalid('Duplicate attachment IDs')
|
||||
if not body['text'].strip() and not attachments:
|
||||
invalid('Message needs text or attachments')
|
||||
return self.repository.create_message(actor, uuid_value(chat_id), body['text'],
|
||||
attachments, key, self.idempotency_ttl)
|
||||
|
||||
def chats(self, actor, limit, cursor):
|
||||
scope = 'chats:' + actor
|
||||
limit, after = self.pagination.decode(scope, limit, cursor)
|
||||
return self.pagination.page(self.repository.chats(actor, limit + 1, after), scope, limit)
|
||||
|
||||
def history(self, actor, chat_id, limit, cursor):
|
||||
chat_id = uuid_value(chat_id)
|
||||
self.repository.chat(actor, chat_id)
|
||||
scope = 'messages:' + chat_id
|
||||
limit, after = self.pagination.decode(scope, limit, cursor)
|
||||
return self.pagination.page(self.repository.messages(actor, chat_id, limit + 1, after), scope, limit)
|
||||
@@ -0,0 +1,39 @@
|
||||
"""Private S3 object storage. No file bytes are stored on API disks."""
|
||||
from pathlib import Path
|
||||
import boto3
|
||||
from botocore.config import Config
|
||||
from botocore.exceptions import BotoCoreError, ClientError
|
||||
from .domain import ApiError
|
||||
|
||||
class ObjectStore:
|
||||
def __init__(self, settings):
|
||||
self.bucket = settings.s3_bucket
|
||||
self.s3 = boto3.client('s3', endpoint_url=settings.s3_endpoint,
|
||||
aws_access_key_id=settings.s3_access_key,
|
||||
aws_secret_access_key=Path(settings.s3_secret_file).read_text().strip(),
|
||||
region_name='us-east-1', config=Config(signature_version='s3v4',
|
||||
connect_timeout=3, read_timeout=10, retries={'max_attempts': 1}))
|
||||
|
||||
def ready(self):
|
||||
try:
|
||||
self.s3.head_bucket(Bucket=self.bucket)
|
||||
except (ClientError, BotoCoreError, OSError):
|
||||
raise ApiError(503, 'unavailable', 'Object storage unavailable') from None
|
||||
|
||||
def put(self, key, data, content_type):
|
||||
try:
|
||||
self.s3.put_object(Bucket=self.bucket, Key=key, Body=data, ContentType=content_type)
|
||||
except (ClientError, BotoCoreError, OSError):
|
||||
raise ApiError(503, 'unavailable', 'Object storage unavailable') from None
|
||||
|
||||
def get(self, key):
|
||||
try:
|
||||
return self.s3.get_object(Bucket=self.bucket, Key=key)['Body']
|
||||
except (ClientError, BotoCoreError, OSError):
|
||||
raise ApiError(503, 'unavailable', 'Object storage unavailable') from None
|
||||
|
||||
def delete(self, key):
|
||||
try:
|
||||
self.s3.delete_object(Bucket=self.bucket, Key=key)
|
||||
except (ClientError, BotoCoreError, OSError):
|
||||
pass # If compensation cannot reach S3, report it as an orphan limitation.
|
||||
@@ -0,0 +1,37 @@
|
||||
CREATE TABLE users (
|
||||
id uuid PRIMARY KEY,
|
||||
username text NOT NULL UNIQUE CHECK (username ~ '^[a-z0-9_]{3,32}$'),
|
||||
display_name text NOT NULL CHECK (char_length(display_name) BETWEEN 1 AND 100),
|
||||
password_hash text NOT NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
CREATE TABLE chats (
|
||||
id uuid PRIMARY KEY,
|
||||
title text NOT NULL CHECK (char_length(title) BETWEEN 1 AND 100),
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
CREATE INDEX chats_order_idx ON chats(created_at, id);
|
||||
CREATE TABLE chat_members (
|
||||
chat_id uuid NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||
user_id uuid NOT NULL REFERENCES users(id),
|
||||
position integer NOT NULL CHECK (position >= 0),
|
||||
PRIMARY KEY(chat_id, user_id),
|
||||
UNIQUE(chat_id, position)
|
||||
);
|
||||
CREATE INDEX chat_members_user_idx ON chat_members(user_id, chat_id);
|
||||
CREATE TABLE messages (
|
||||
id uuid PRIMARY KEY,
|
||||
chat_id uuid NOT NULL REFERENCES chats(id),
|
||||
sender_id uuid NOT NULL REFERENCES users(id),
|
||||
text text NOT NULL CHECK (char_length(text) BETWEEN 1 AND 4000),
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
FOREIGN KEY(chat_id, sender_id) REFERENCES chat_members(chat_id, user_id)
|
||||
);
|
||||
CREATE INDEX messages_history_idx ON messages(chat_id, created_at, id);
|
||||
CREATE TABLE sessions (
|
||||
token_hash text PRIMARY KEY CHECK (char_length(token_hash) = 64),
|
||||
user_id uuid NOT NULL REFERENCES users(id),
|
||||
expires_at timestamptz NOT NULL
|
||||
);
|
||||
CREATE INDEX sessions_expiry_idx ON sessions(expires_at);
|
||||
CREATE TABLE app_settings (name text PRIMARY KEY, value text NOT NULL);
|
||||
@@ -0,0 +1,10 @@
|
||||
CREATE TABLE message_requests (
|
||||
user_id uuid NOT NULL REFERENCES users(id),
|
||||
chat_id uuid NOT NULL REFERENCES chats(id),
|
||||
idempotency_key text NOT NULL CHECK (idempotency_key ~ '^[A-Za-z0-9._:-]{1,128}$'),
|
||||
request_text text NOT NULL,
|
||||
message_id uuid NOT NULL REFERENCES messages(id),
|
||||
expires_at timestamptz NOT NULL,
|
||||
PRIMARY KEY (user_id, chat_id, idempotency_key)
|
||||
);
|
||||
CREATE INDEX message_requests_expiry_idx ON message_requests(expires_at);
|
||||
@@ -0,0 +1,27 @@
|
||||
-- Lab4: only metadata and references live in PostgreSQL; bytes live in private S3.
|
||||
CREATE TABLE attachments (
|
||||
id uuid PRIMARY KEY,
|
||||
chat_id uuid NOT NULL REFERENCES chats(id),
|
||||
owner_id uuid NOT NULL REFERENCES users(id),
|
||||
filename text NOT NULL CHECK(char_length(filename) BETWEEN 1 AND 255),
|
||||
content_type text NOT NULL CHECK(content_type IN ('image/png','image/jpeg')),
|
||||
size_bytes bigint NOT NULL CHECK(size_bytes BETWEEN 1 AND 10485760),
|
||||
sha256 text NOT NULL CHECK(sha256 ~ '^[a-f0-9]{64}$'),
|
||||
object_key text NOT NULL UNIQUE,
|
||||
status text NOT NULL DEFAULT 'ready' CHECK(status='ready'),
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
updated_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
FOREIGN KEY(chat_id,owner_id) REFERENCES chat_members(chat_id,user_id)
|
||||
);
|
||||
CREATE INDEX attachments_chat_idx ON attachments(chat_id,id);
|
||||
ALTER TABLE messages ALTER COLUMN text DROP NOT NULL;
|
||||
ALTER TABLE messages DROP CONSTRAINT messages_text_check;
|
||||
ALTER TABLE messages ADD CONSTRAINT messages_text_check CHECK(char_length(text) BETWEEN 0 AND 4000);
|
||||
CREATE TABLE message_attachments (
|
||||
message_id uuid NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
|
||||
attachment_id uuid NOT NULL REFERENCES attachments(id),
|
||||
position integer NOT NULL CHECK(position BETWEEN 0 AND 9),
|
||||
PRIMARY KEY(message_id, attachment_id),
|
||||
UNIQUE(message_id, position)
|
||||
);
|
||||
ALTER TABLE message_requests ADD COLUMN request_attachments jsonb NOT NULL DEFAULT '[]'::jsonb;
|
||||
@@ -0,0 +1,15 @@
|
||||
Flask==3.1.0
|
||||
waitress==3.0.2
|
||||
Werkzeug==3.1.3
|
||||
Jinja2==3.1.6
|
||||
MarkupSafe==3.0.3
|
||||
itsdangerous==2.2.0
|
||||
click==8.1.8
|
||||
blinker==1.9.0
|
||||
psycopg[binary]==3.2.6
|
||||
argon2-cffi==23.1.0
|
||||
argon2-cffi-bindings==21.2.0
|
||||
cffi==1.17.1
|
||||
pycparser==2.22
|
||||
boto3==1.36.26
|
||||
Pillow==11.1.0
|
||||
@@ -0,0 +1,44 @@
|
||||
"""Concurrent HTTP contract check through LB, using one session on two replicas."""
|
||||
import argparse
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from pathlib import Path
|
||||
import sys
|
||||
from uuid import uuid4
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'tests'))
|
||||
from client import Client, check
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('--base-url', default='http://localhost:8080')
|
||||
args = parser.parse_args()
|
||||
c = Client(args.base_url)
|
||||
alice, bob, eve, chat = c.bootstrap()
|
||||
key = uuid4().hex
|
||||
path = '/api/v1/chats/' + chat['id'] + '/messages'
|
||||
def send(_):
|
||||
return Client(args.base_url).message(alice, chat, 'Одно сообщение 👋', key=key)
|
||||
with ThreadPoolExecutor(max_workers=8) as pool:
|
||||
responses = list(pool.map(send, range(24)))
|
||||
first = responses[0]['json']
|
||||
check(all(r['json'] == first and r['status'] == 201 for r in responses), 'Concurrent replay differs')
|
||||
instances = {r['headers']['X-Instance-Id'] for r in responses}
|
||||
check(len(instances) >= 2, 'Concurrent requests did not reach both replicas')
|
||||
page = c.call('GET', path, actor=bob, schema='MessagePage')['json']
|
||||
check(page['items'] == [first], 'Concurrent repeat created duplicate messages')
|
||||
c.call('POST', path, 409, actor=alice, body={'text': 'Different'}, headers={'Idempotency-Key': key})
|
||||
c.call('POST', path, 400, actor=alice, body={'text': 'Missing key'})
|
||||
c.call('POST', path, 400, actor=alice, body={'text': 'Bad key'}, headers={'Idempotency-Key': 'bad key'})
|
||||
c.call('POST', path, 404, actor=eve, body={'text': 'Одно сообщение 👋'}, headers={'Idempotency-Key': key})
|
||||
other = c.message(bob, chat, 'Same key, different user', key=key)['json']
|
||||
check(other['id'] != first['id'], 'User scope is not independent')
|
||||
second_chat = c.call('POST', '/api/v1/chats', 201, actor=alice,
|
||||
body={'title': 'Second', 'member_ids': [alice['id']]}, schema='Chat')['json']
|
||||
check(c.message(alice, second_chat, 'Other chat', key=key)['json']['id'] != first['id'], 'Chat scope is not independent')
|
||||
c.call('DELETE', '/api/v1/auth/sessions/current', 204, actor=alice)
|
||||
c.call('POST', path, 401, actor=alice, body={'text': 'Одно сообщение 👋'}, headers={'Idempotency-Key': key})
|
||||
print('PASS: 24 concurrent requests / 8 clients / %d replicas -> one message; conflict, scopes, access and logout verified.' % len(instances))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,201 @@
|
||||
"""Real local PostgreSQL + 2 API processes + HAProxy. Optional, not a Docker substitute.
|
||||
python scripts/check_local.py --pg-bin /path/to/bin --haproxy /path/to/haproxy
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import shlex
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from urllib.request import urlopen
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def pg(state, *args):
|
||||
subprocess.run([str(Path(state['bin']) / 'pg_ctl'), '-D', state['data'], *args],
|
||||
check=True, stdout=subprocess.DEVNULL)
|
||||
|
||||
|
||||
def wait_url(port):
|
||||
for _ in range(150):
|
||||
try:
|
||||
with urlopen('http://127.0.0.1:%s/health/ready' % port, timeout=1) as response:
|
||||
if response.status == 200:
|
||||
return
|
||||
except OSError:
|
||||
time.sleep(.1)
|
||||
raise RuntimeError('Service did not become ready on port %s' % port)
|
||||
|
||||
|
||||
def launch(state, name):
|
||||
temp = Path(state['data']).parent
|
||||
if name == 'lb':
|
||||
command = [state['haproxy'], '-db', '-f', str(temp / 'haproxy.cfg')]
|
||||
env = dict(os.environ)
|
||||
else:
|
||||
command = [sys.executable, '-m', 'messenger']
|
||||
env = dict(os.environ, APP_HOST='127.0.0.1', APP_PORT=str(state['ports'][name]))
|
||||
with (temp / (name + '.log')).open('a') as log:
|
||||
process = subprocess.Popen(command, env=env, cwd=ROOT, stdout=log, stderr=log, start_new_session=True)
|
||||
state['pids'][name] = process.pid
|
||||
return process
|
||||
|
||||
|
||||
def stop(pid):
|
||||
try:
|
||||
os.kill(pid, signal.SIGTERM)
|
||||
except ProcessLookupError:
|
||||
return
|
||||
for _ in range(70):
|
||||
try:
|
||||
# Zombie processes no longer own sockets and await their parent.
|
||||
status = Path('/proc/%s/stat' % pid)
|
||||
if not status.exists() or status.read_text().split()[2] == 'Z':
|
||||
return
|
||||
except FileNotFoundError:
|
||||
return
|
||||
time.sleep(.1)
|
||||
os.kill(pid, signal.SIGKILL)
|
||||
|
||||
|
||||
def hook(path, action):
|
||||
state = json.loads(path.read_text())
|
||||
if action == 'stop':
|
||||
target = os.environ['TARGET_INSTANCE_ID']
|
||||
for name, port in state['ports'].items():
|
||||
try:
|
||||
with urlopen('http://127.0.0.1:%s/health/live' % port, timeout=1) as r:
|
||||
matches = r.headers['X-Instance-Id'] == target
|
||||
except OSError:
|
||||
continue
|
||||
if matches:
|
||||
pid = state['pids'][name]
|
||||
start = time.monotonic()
|
||||
stop(pid)
|
||||
state['stopped'] = name
|
||||
path.write_text(json.dumps(state))
|
||||
with (ROOT / 'evidence/local-stops.txt').open('a') as f:
|
||||
f.write('%s PID=%s instance=%s stopped in %.3fs; remains stopped\n' % (name, pid, target, time.monotonic()-start))
|
||||
return
|
||||
raise RuntimeError('Target instance not found')
|
||||
import psycopg
|
||||
with psycopg.connect(os.environ['DATABASE_URL']) as conn:
|
||||
before = conn.execute('SELECT pg_postmaster_start_time()').fetchone()[0]
|
||||
old = dict(state['pids'])
|
||||
for pid in old.values():
|
||||
stop(pid)
|
||||
pg(state, '-m', 'fast', '-w', 'stop')
|
||||
pg(state, '-l', str(path.parent / 'postgres.log'), '-w', 'start')
|
||||
for name in ('api1', 'api2', 'lb'):
|
||||
launch(state, name)
|
||||
path.write_text(json.dumps(state))
|
||||
wait_url(18080)
|
||||
with psycopg.connect(os.environ['DATABASE_URL']) as conn:
|
||||
after = conn.execute('SELECT pg_postmaster_start_time()').fetchone()[0]
|
||||
assert before != after and all(old[k] != state['pids'][k] for k in old)
|
||||
(ROOT / 'evidence/local-restart.txt').write_text('Actual process PIDs before: %s\nAfter: %s\nPostgreSQL start: %s -> %s\nSame data directory, no reseed or migrations in hook.\n' %
|
||||
(old, state['pids'], before.isoformat(), after.isoformat()))
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('--pg-bin')
|
||||
parser.add_argument('--haproxy')
|
||||
parser.add_argument('--hook', choices=['stop', 'restart'])
|
||||
parser.add_argument('--state', type=Path)
|
||||
args = parser.parse_args()
|
||||
if args.hook:
|
||||
hook(args.state, args.hook)
|
||||
return
|
||||
if not args.pg_bin or not args.haproxy:
|
||||
parser.error('--pg-bin and --haproxy required')
|
||||
binary = Path(args.pg_bin).resolve()
|
||||
with tempfile.TemporaryDirectory(prefix='messenger-lab3-') as tmp:
|
||||
temp = Path(tmp)
|
||||
path = temp / 'state.json'
|
||||
state = {'bin': str(binary), 'haproxy': str(Path(args.haproxy).resolve()),
|
||||
'data': str(temp / 'data'), 'ports': {'api1': 18081, 'api2': 18082}, 'pids': {}}
|
||||
os.environ.update(DATABASE_URL='postgresql://127.0.0.1:15432/postgres',
|
||||
SESSION_STORE='postgres', SESSION_TTL_SECONDS='3600', IDEMPOTENCY_TTL_SECONDS='86400')
|
||||
os.environ.pop('DATABASE_URL_FILE', None)
|
||||
subprocess.run([str(binary / 'initdb'), '-D', state['data'], '-A', 'trust', '--no-locale', '-E', 'UTF8'],
|
||||
check=True, stdout=subprocess.DEVNULL)
|
||||
with (Path(state['data']) / 'postgresql.conf').open('a') as f:
|
||||
f.write("\nport=15432\nlisten_addresses='127.0.0.1'\nunix_socket_directories=''\ntimezone='UTC'\n")
|
||||
config = (ROOT / 'infra/haproxy.cfg').read_text().replace('bind :8080', 'bind 127.0.0.1:18080')
|
||||
for name, port in state['ports'].items():
|
||||
config = config.replace(name + ':8080', '127.0.0.1:' + str(port))
|
||||
# Local fixed loopback addresses have no Docker DNS; all LB behavior is unchanged.
|
||||
config = config.replace(' resolvers docker resolve-prefer ipv4 init-addr libc,none', '')
|
||||
(temp / 'haproxy.cfg').write_text(config)
|
||||
processes = []
|
||||
def run(command):
|
||||
print('$ ' + ' '.join(command), flush=True)
|
||||
subprocess.run(command, cwd=ROOT, check=True)
|
||||
try:
|
||||
pg(state, '-l', str(temp / 'postgres.log'), '-w', 'start')
|
||||
run([sys.executable, '-m', 'messenger.migrate'])
|
||||
run([sys.executable, '-m', 'messenger.migrate'])
|
||||
run([state['haproxy'], '-c', '-f', str(temp / 'haproxy.cfg')])
|
||||
for name in ('api1', 'api2', 'lb'):
|
||||
processes.append(launch(state, name))
|
||||
path.write_text(json.dumps(state))
|
||||
wait_url(18081); wait_url(18082); wait_url(18080)
|
||||
print('Distinct running processes:', state['pids'], flush=True)
|
||||
(ROOT / 'evidence/local-stops.txt').write_text('Actual local API stops (not Docker):\n')
|
||||
run(['make', 'check', 'validate', 'student-test', 'PYTHON=' + sys.executable])
|
||||
run(['make', 'test', 'test-idempotency', 'BASE_URL=http://127.0.0.1:18080', 'PYTHON=' + sys.executable])
|
||||
for action in ('restart', 'stop'):
|
||||
script = temp / (action + '.sh')
|
||||
script.write_text('#!/bin/sh\nexec ' + shlex.join([sys.executable, str(Path(__file__).resolve()),
|
||||
'--hook', action, '--state', str(path)]) + '\n')
|
||||
script.chmod(0o700)
|
||||
sys.path.insert(0, str(ROOT / 'tests'))
|
||||
from client import Client
|
||||
from uuid import uuid4
|
||||
client = Client('http://127.0.0.1:18080')
|
||||
actor, _, _, durable_chat = client.bootstrap()
|
||||
durable_key = uuid4().hex
|
||||
original = client.message(actor, durable_chat, 'Replay across restart', key=durable_key)['json']
|
||||
run(['make', 'test-persistence' , 'ACTION_SCRIPT=' + str(temp / 'restart.sh'),
|
||||
'BASE_URL=http://127.0.0.1:18080', 'PYTHON=' + sys.executable])
|
||||
assert client.message(actor, durable_chat, 'Replay across restart', key=durable_key)['json'] == original
|
||||
print('PASS: same key + old session after full restart returns original Message', flush=True)
|
||||
run(['make', 'test-idempotency', 'BASE_URL=http://127.0.0.1:18080', 'PYTHON=' + sys.executable])
|
||||
for attempt in range(2):
|
||||
if attempt:
|
||||
# Shift the round-robin position so the second public scenario
|
||||
# selects the other process, without changing the public test.
|
||||
with urlopen('http://127.0.0.1:18080/health/live'):
|
||||
pass
|
||||
run(['make', 'test-failover', 'ACTION_SCRIPT=' + str(temp / 'stop.sh'),
|
||||
'BASE_URL=http://127.0.0.1:18080', 'PYTHON=' + sys.executable])
|
||||
assert client.message(actor, durable_chat, 'Replay across restart', key=durable_key)['json'] == original
|
||||
print('PASS: original idempotent result survives selected replica stop', flush=True)
|
||||
state = json.loads(path.read_text())
|
||||
processes.append(launch(state, state.pop('stopped')))
|
||||
path.write_text(json.dumps(state))
|
||||
wait_url(18081); wait_url(18082)
|
||||
time.sleep(1.2)
|
||||
print((ROOT / 'evidence/local-restart.txt').read_text(), flush=True)
|
||||
print((ROOT / 'evidence/local-stops.txt').read_text(), flush=True)
|
||||
run([state['haproxy'], '-v'])
|
||||
run([str(binary / 'postgres'), '--version'])
|
||||
finally:
|
||||
if path.exists():
|
||||
state = json.loads(path.read_text())
|
||||
for pid in state['pids'].values():
|
||||
stop(pid)
|
||||
for process in processes:
|
||||
process.wait(timeout=10)
|
||||
subprocess.run([str(binary / 'pg_ctl'), '-D', state['data'], '-m', 'fast', '-w', 'stop'],
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,29 @@
|
||||
"""Create local secrets once. Never overwrite existing credentials."""
|
||||
from pathlib import Path
|
||||
import secrets
|
||||
import shutil
|
||||
|
||||
root = Path(__file__).resolve().parents[1]
|
||||
directory = root / 'secrets'
|
||||
directory.mkdir(mode=0o700, exist_ok=True)
|
||||
password_file = directory / 'postgres_password'
|
||||
url_file = directory / 'database_url'
|
||||
if password_file.exists() != url_file.exists():
|
||||
raise SystemExit('Incomplete secrets: restore the missing file; do not regenerate credentials for an existing volume.')
|
||||
if not password_file.exists():
|
||||
password = secrets.token_hex(32)
|
||||
for path, value in [(password_file, password),
|
||||
(url_file, 'postgresql://messenger:' + password + '@db:5432/messenger')]:
|
||||
with path.open('x', encoding='utf-8') as f:
|
||||
f.write(value + '\n')
|
||||
# Compose mounts secret files individually; PostgreSQL runs as a distinct UID.
|
||||
# The enclosing host directory is 0700, while this mounted file must be readable.
|
||||
path.chmod(0o644 if path == password_file else 0o600)
|
||||
s3_file = directory / 's3_password'
|
||||
if not s3_file.exists():
|
||||
with s3_file.open('x', encoding='utf-8') as f:
|
||||
f.write(secrets.token_hex(32) + '\n')
|
||||
s3_file.chmod(0o644)
|
||||
if not (root / '.env').exists():
|
||||
shutil.copyfile(root / '.env.example', root / '.env')
|
||||
print('Local configuration ready; existing credentials preserved.')
|
||||
Executable
+26
@@ -0,0 +1,26 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
mkdir -p evidence
|
||||
snapshot() {
|
||||
for service in db s3 api1 api2 lb; do
|
||||
id=$(docker compose ps -aq "$service")
|
||||
docker inspect --format '{{.Name}} {{.State.StartedAt}} {{.State.Status}}' "$id"
|
||||
done
|
||||
}
|
||||
snapshot > evidence/restart-before.txt
|
||||
docker compose stop -t 7 lb api1 api2
|
||||
docker compose restart -t 10 db s3
|
||||
i=0
|
||||
until docker compose exec -T db pg_isready -U messenger -d messenger >/dev/null 2>&1; do
|
||||
i=$((i + 1)); [ "$i" -lt 30 ] || exit 1
|
||||
sleep 1
|
||||
done
|
||||
i=0
|
||||
until docker compose exec -T s3 sh -c 'curl -fsS http://127.0.0.1:9000/minio/health/live >/dev/null' 2>/dev/null; do
|
||||
i=$((i+1)); [ "$i" -lt 30 ] || exit 1
|
||||
sleep 1
|
||||
done
|
||||
# Existing schema is retained; the restart hook does not migrate/reseed.
|
||||
docker compose start api1 api2 lb
|
||||
snapshot > evidence/restart-after.txt
|
||||
Executable
+18
@@ -0,0 +1,18 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
: "${TARGET_INSTANCE_ID:?Set TARGET_INSTANCE_ID to the observed API instance}"
|
||||
mkdir -p evidence
|
||||
for service in api1 api2; do
|
||||
instance=$(docker compose exec -T "$service" python -c 'import urllib.request; r=urllib.request.urlopen("http://127.0.0.1:8080/health/live",timeout=2); print(r.headers["X-Instance-Id"])' 2>/dev/null) || continue
|
||||
if [ "$instance" = "$TARGET_INSTANCE_ID" ]; then
|
||||
id=$(docker compose ps -q "$service")
|
||||
docker inspect --format '{{.Name}} {{.State.StartedAt}} {{.State.Status}}' "$id" > "evidence/stop-$service.txt"
|
||||
# SIGTERM, then bounded fallback. No automatic restart policy is enabled.
|
||||
docker compose stop -t 7 "$service"
|
||||
docker inspect --format '{{.Name}} {{.State.FinishedAt}} {{.State.Status}}' "$id" >> "evidence/stop-$service.txt"
|
||||
exit 0
|
||||
fi
|
||||
done
|
||||
printf '%s\n' 'No running replica matches TARGET_INSTANCE_ID' >&2
|
||||
exit 1
|
||||
@@ -0,0 +1,141 @@
|
||||
"""Integration tests. DATABASE_URL[_FILE] must point to a disposable PostgreSQL DB."""
|
||||
import base64
|
||||
import time
|
||||
import unittest
|
||||
from dataclasses import replace
|
||||
from uuid import uuid4
|
||||
from messenger.api import create_app
|
||||
from messenger.config import Settings
|
||||
from messenger.database import Database
|
||||
from messenger.migrate import migrate
|
||||
|
||||
|
||||
class ApiTests(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
cls.settings = Settings.from_env()
|
||||
cls.db = Database(cls.settings.database_url)
|
||||
migrate(cls.db)
|
||||
cls.app = create_app(settings=cls.settings)
|
||||
|
||||
def setUp(self):
|
||||
self.c = self.app.test_client()
|
||||
self.password = 'Учебный:пароль-12345'
|
||||
self.name = 'u_' + uuid4().hex[:16]
|
||||
self.registration = {'username': self.name, 'display_name': 'Имя 👋', 'password': self.password}
|
||||
r = self.c.post('/api/v1/users', json=self.registration)
|
||||
self.assertEqual(r.status_code, 201)
|
||||
self.user = r.json
|
||||
self.basic = {'Authorization': 'Basic ' + base64.b64encode((self.name + ':' + self.password).encode()).decode()}
|
||||
r = self.c.post('/api/v1/auth/sessions', headers=self.basic)
|
||||
self.assertEqual(r.status_code, 201)
|
||||
self.session = r.json
|
||||
self.headers = {'Authorization': 'Bearer ' + self.session['token'], 'Idempotency-Key': uuid4().hex}
|
||||
self.chat = self.c.post('/api/v1/chats', headers=self.headers, json={
|
||||
'title': 'Учебный чат', 'member_ids': [self.user['id']]}).json
|
||||
self.path = '/api/v1/chats/' + self.chat['id'] + '/messages'
|
||||
|
||||
def error(self, response, status):
|
||||
self.assertEqual(response.status_code, status)
|
||||
self.assertEqual(response.json['error']['request_id'], response.headers['X-Request-Id'])
|
||||
|
||||
def test_authentication_and_logout(self):
|
||||
for headers in [{}, self.basic, {'X-User-Id': self.user['id']}, {'Authorization': 'Bearer bad'}]:
|
||||
r = self.c.get('/api/v1/users/me', headers=headers)
|
||||
self.error(r, 401)
|
||||
self.assertEqual(r.headers['WWW-Authenticate'], 'Bearer realm="messenger"')
|
||||
for auth in ['Basic bad', 'Bearer bad', 'Basic ' + base64.b64encode((self.name + ':wrong-password').encode()).decode()]:
|
||||
r = self.c.post('/api/v1/auth/sessions', headers={'Authorization': auth})
|
||||
self.error(r, 401)
|
||||
self.assertEqual(r.headers['WWW-Authenticate'], 'Basic realm="messenger", charset="UTF-8"')
|
||||
r = self.c.get('/api/v1/users/me', headers=dict(self.headers, **{'X-User-Id': str(uuid4())}))
|
||||
self.assertEqual(r.json, self.user)
|
||||
self.error(self.c.post('/api/v1/auth/sessions', headers=self.basic, json={}), 400)
|
||||
r = self.c.delete('/api/v1/auth/sessions/current', headers=self.headers)
|
||||
self.assertEqual((r.status_code, r.data), (204, b''))
|
||||
self.error(self.c.get('/api/v1/users/me', headers=self.headers), 401)
|
||||
self.error(self.c.delete('/api/v1/auth/sessions/current', headers=self.headers), 401)
|
||||
|
||||
def test_expiry_and_restart_app(self):
|
||||
short = create_app(settings=replace(self.settings, session_ttl=2)).test_client()
|
||||
r = short.post('/api/v1/auth/sessions', headers=self.basic)
|
||||
self.assertEqual(r.status_code, 201)
|
||||
headers = {'Authorization': 'Bearer ' + r.json['token']}
|
||||
self.assertEqual(short.get('/api/v1/users/me', headers=headers).status_code, 200)
|
||||
# New app instance must respect the stored original expiry, not renew it.
|
||||
restarted = create_app(settings=replace(self.settings, session_ttl=3600)).test_client()
|
||||
self.assertEqual(restarted.get('/api/v1/users/me', headers=headers).status_code, 200)
|
||||
time.sleep(2.1)
|
||||
self.error(restarted.get('/api/v1/users/me', headers=headers), 401)
|
||||
|
||||
def test_strict_validation_and_private_chat(self):
|
||||
self.error(self.c.post('/api/v1/users', data='{', content_type='application/json'), 400)
|
||||
self.error(self.c.post('/api/v1/users', json=self.registration), 409)
|
||||
for patch in [{'password': 'short'}, {'password': 42}, {'display_name': ' '}, {'extra': 1}]:
|
||||
self.error(self.c.post('/api/v1/users', json=dict(self.registration, **patch)), 400)
|
||||
self.error(self.c.post(self.path, headers=self.headers, json={'text': ' \n'}), 400)
|
||||
self.error(self.c.post(self.path, headers=self.headers, json={'text': 'x', 'sender_id': self.user['id']}), 400)
|
||||
self.error(self.c.get('/api/v1/chats/' + str(uuid4()), headers=self.headers), 404)
|
||||
outsider = 'u_' + uuid4().hex[:16]
|
||||
self.c.post('/api/v1/users', json=dict(self.registration, username=outsider))
|
||||
basic = {'Authorization': 'Basic ' + base64.b64encode((outsider + ':' + self.password).encode()).decode()}
|
||||
token = self.c.post('/api/v1/auth/sessions', headers=basic).json['token']
|
||||
self.error(self.c.get(self.path, headers={'Authorization': 'Bearer ' + token}), 404)
|
||||
|
||||
def test_pagination_survives_new_app(self):
|
||||
expected = [self.c.post(self.path, headers=dict(self.headers, **{'Idempotency-Key': uuid4().hex}), json={'text': str(i)}).json for i in range(5)]
|
||||
first = self.c.get(self.path + '?limit=2', headers=self.headers).json
|
||||
cursor = first['next_cursor']
|
||||
new_client = create_app(settings=self.settings).test_client()
|
||||
second = new_client.get(self.path, query_string={'limit': 1, 'cursor': cursor}, headers=self.headers).json
|
||||
third = new_client.get(self.path, query_string={'limit': 100, 'cursor': second['next_cursor']}, headers=self.headers).json
|
||||
self.assertEqual(first['items'] + second['items'] + third['items'], expected)
|
||||
self.assertIsNone(third['next_cursor'])
|
||||
self.error(self.c.get('/api/v1/chats', query_string={'cursor': cursor}, headers=self.headers), 400)
|
||||
other = self.c.post('/api/v1/chats', headers=self.headers, json={'title': 'Other', 'member_ids': [self.user['id']]}).json
|
||||
self.error(self.c.get('/api/v1/chats/' + other['id'] + '/messages', query_string={'cursor': cursor}, headers=self.headers), 400)
|
||||
for limit in ['0', '101', '1.5', 'true']:
|
||||
self.error(self.c.get(self.path, query_string={'limit': limit}, headers=self.headers), 400)
|
||||
self.error(self.c.get(self.path, query_string={'cursor': 'invalid'}, headers=self.headers), 400)
|
||||
page = self.c.get('/api/v1/chats?limit=1', headers=self.headers).json
|
||||
last = self.c.get('/api/v1/chats', query_string={'cursor': page['next_cursor']}, headers=self.headers).json
|
||||
self.assertEqual(len(page['items'] + last['items']), 2)
|
||||
|
||||
def test_atomic_chat_and_migration_repeat(self):
|
||||
with self.db.transaction() as conn:
|
||||
before = conn.execute('SELECT count(*) AS n FROM chats').fetchone()['n']
|
||||
for members in [[self.user['id'], str(uuid4())], [self.user['id'], self.user['id']], []]:
|
||||
self.error(self.c.post('/api/v1/chats', headers=self.headers,
|
||||
json={'title': 'Invalid', 'member_ids': members}), 400)
|
||||
migrate(self.db)
|
||||
migrate(self.db)
|
||||
with self.db.transaction() as conn:
|
||||
self.assertEqual(conn.execute('SELECT count(*) AS n FROM chats').fetchone()['n'], before)
|
||||
self.assertEqual(conn.execute('SELECT count(*) AS n FROM schema_migrations').fetchone()['n'], 3)
|
||||
row = conn.execute('SELECT password_hash FROM users WHERE id=%s', (self.user['id'],)).fetchone()
|
||||
self.assertTrue(row['password_hash'].startswith('$argon2id$'))
|
||||
hashes = [r['token_hash'] for r in conn.execute('SELECT token_hash FROM sessions WHERE user_id=%s', (self.user['id'],))]
|
||||
self.assertNotIn(self.session['token'], hashes)
|
||||
self.assertEqual(self.c.get('/api/v1/users/me', headers=self.headers).json, self.user)
|
||||
|
||||
def test_idempotency_expiry_and_errors(self):
|
||||
first = self.c.post(self.path, headers=self.headers, json={'text': 'Persist me'})
|
||||
self.assertEqual(first.status_code, 201)
|
||||
replay = self.c.post(self.path, headers=self.headers, json={'text': 'Persist me'})
|
||||
self.assertEqual(replay.json, first.json)
|
||||
self.error(self.c.post(self.path, headers=self.headers, json={'text': 'Different'}), 409)
|
||||
for key in ['', 'bad key', 'x' * 129]:
|
||||
self.error(self.c.post(self.path, headers=dict(self.headers, **{'Idempotency-Key': key}), json={'text': 'Text'}), 400)
|
||||
# Expire only this test's synthetic record, without waiting 24 hours.
|
||||
with self.db.transaction() as conn:
|
||||
row = conn.execute("SELECT expires_at>clock_timestamp()+interval '23 hours' AS valid FROM message_requests WHERE user_id=%s", (self.user['id'],)).fetchone()
|
||||
self.assertTrue(row['valid'])
|
||||
conn.execute("UPDATE message_requests SET expires_at=clock_timestamp()-interval '1 second' WHERE user_id=%s", (self.user['id'],))
|
||||
new = self.c.post(self.path, headers=self.headers, json={'text': 'New after expiry'})
|
||||
self.assertEqual(new.status_code, 201)
|
||||
self.assertNotEqual(new.json['id'], first.json['id'])
|
||||
self.assertEqual(len(self.c.get(self.path, headers=self.headers).json['items']), 2)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -0,0 +1,39 @@
|
||||
"""Offline negative checks of image admission; execute: python -m unittest discover -s student-tests -p test_images.py."""
|
||||
import io
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from PIL import Image
|
||||
from messenger.images import validate_image, MAX_BYTES
|
||||
from messenger.domain import ApiError
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
class ImageAdmissionTests(unittest.TestCase):
|
||||
def test_fixture_and_byte_preservation(self):
|
||||
blob = (ROOT / 'tests/fixtures/sample.png').read_bytes()
|
||||
self.assertIsNone(validate_image(blob, 'image/png'))
|
||||
|
||||
def test_mismatch_or_invalid(self):
|
||||
data = (ROOT / 'tests/fixtures/sample.png').read_bytes()
|
||||
for blob, mime in [(data, 'image/jpeg'), (b'not-an-image', 'image/png'),
|
||||
(b'\xff\xd8broken', 'image/jpeg'), (data, 'application/octet-stream')]:
|
||||
with self.subTest(mime=mime):
|
||||
with self.assertRaises(ApiError) as error:
|
||||
validate_image(blob, mime)
|
||||
self.assertEqual(error.exception.status, 415)
|
||||
|
||||
def test_byte_limit(self):
|
||||
with self.assertRaises(ApiError) as error:
|
||||
validate_image(b'0' * (MAX_BYTES + 1), 'image/png')
|
||||
self.assertEqual(error.exception.status, 413)
|
||||
|
||||
def test_pixel_limit_before_decompression(self):
|
||||
img = Image.new('RGB', (8000, 5001))
|
||||
stream = io.BytesIO()
|
||||
img.save(stream, format='PNG')
|
||||
with self.assertRaises(ApiError) as error:
|
||||
validate_image(stream.getvalue(), 'image/png')
|
||||
self.assertEqual(error.exception.status, 413)
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user