Author SHA1 Message Date
nothingredmouth 75cbe925d3 Заполнить отчёт и README лабораторной работы
REPORT.md: запуск, архитектура, логика загрузки, ограничения уровня 3,
таблица доказательств. README: ссылка на отчёт вместо отсутствующего
START_HERE.md.
2026-11-17 03:37:45 +00:00
nothingredmouth 9af7b15809 Добавить собственные интеграционные тесты API и проверки изображений
test_api.py — сквозные сценарии на живой БД (auth, сессии, чаты,
идемпотентность, миграции, доступ к чужим чатам). test_images.py —
валидация PNG/JPEG: MIME, лимиты bytes/pixels, повреждённые файлы,
доступ только для участников чата. Указано 3 применённые миграции.
2026-11-17 03:37:42 +00:00
nothingredmouth 64503fd0dd Добавить сценарии рестарта, остановки реплики и локальных проверок
restart.sh (полный цикл стенда, без volumes/secrets), stop-one.sh (одна
реплика по TARGET_INSTANCE_ID, данные и секреты сохраняются),
check_local.py и check_idempotency.py — воспроизводимые локальные проверки.
2026-11-17 03:37:38 +00:00
nothingredmouth e594df18f9 Поднять стенд из двух реплик за HAProxy с приватным MinIO S3
Dockerfile (python:3.11, waitress), compose: api1/api2 + postgres +
migrate + s3 (pinned minio, без опубликованных портов) + s3-bootstrap +
lb (haproxy, round-robin, healthcheck на /health/ready). Секреты DATABASE_URL
и S3 — через Docker secrets, локальная инициализация — scripts/init_local.py.
2026-11-17 03:37:34 +00:00
nothingredmouth c9e24a0b45 Добавить приватные S3-вложения: валидация PNG/JPEG, загрузка и метаданные в БД
Байты хранятся в приватном bucket (случайный ключ, без публичного доступа),
метаданные и связи — в PostgreSQL (миграция 003_attachments). Лимиты
размера/пикселей проверяются до полной распаковки, идемпотентность учитывает
состав вложений. bootstrap_s3 идемпотентно создаёт bucket.
2026-11-17 03:37:25 +00:00
nothingredmouth 4b23b44eb8 Перенести приложение мессенджера с сессиями и идемпотентностью в PostgreSQL
Пользователи, аутентификация Argon2id, Bearer-сессии с TTL и отзывом,
транзакционный репозиторий, keyset-пагинация, идемпотентные отправки
(advisory lock + message_requests), версияемые миграции 001/002.
2026-11-17 03:37:07 +00:00
34 changed files with 1691 additions and 27 deletions
+5
View File
@@ -18,3 +18,8 @@ S3_ACCESS_KEY_ID_FILE=/run/secrets/s3_access_key_id
S3_SECRET_ACCESS_KEY_FILE=/run/secrets/s3_secret_access_key
MAX_UPLOAD_BYTES=10485760
MAX_IMAGE_PIXELS=40000000
# Lab4: S3 is internal-only; Compose explicitly maps these to the API.
S3_ENDPOINT=http://s3:9000
S3_BUCKET=messenger-attachments
S3_ACCESS_KEY=messengeradmin
S3_SECRET_FILE=/run/secrets/s3_password
+9
View File
@@ -0,0 +1,9 @@
FROM python:3.11.11-slim-bookworm
WORKDIR /app
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 APP_HOST=0.0.0.0 APP_PORT=8080 APP_ENV=lab
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
COPY messenger ./messenger
EXPOSE 8080
COPY migrations ./migrations
CMD ["python", "-m", "messenger"]
+1 -1
View File
@@ -2,7 +2,7 @@
Разделить метаданные и бинарное содержимое, добавить изображения в сообщения и сохранить доступ к ним при переключении API-реплики.
Это **skeleton**, а не готовый сервер. Здесь есть задание, контракт и публичные проверки. Реализацию приложения, Dockerfile и требуемую инфраструктуру пишет студент. Язык и framework свободные; UI не обязателен.
Этот архив содержит **готовую реализацию для оценки «3»** на базе лабораторной №3: Flask, PostgreSQL, HAProxy (две API-реплики), MinIO S3 и приватные вложения. Исходные задание, контракт и публичные тесты сохранены. Инструкции по запуску и решения/ограничения — в [REPORT.md](REPORT.md). UI не обязателен.
[Карта курса](COURSE.md) · [Процесс fork/PR](CONTRIBUTING.md) · [OpenAPI](contracts/openapi.json) · [Семантика API](contracts/README.md) · [Проверки](tests/README.md) · [Отчёт](REPORT.md) · [Примеры JSON](contracts/examples.json) · [Памятка преподавателю](MAINTAINERS.md)
+76 -26
View File
@@ -1,37 +1,87 @@
# Отчёт — лабораторная 4
# Отчёт — лабораторная работа №4
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
- Автор: **Сиренко Илья**.
- Группа: **БПИ2304**.
- Целевая оценка: **3**. Уровни 4/5 не заявлены.
- Основа: лабораторная №3 (Flask + PostgreSQL + HAProxy, две API-реплики, сессии и идемпотентные сообщения).
- Commit/PR: указать после публикации репозитория.
- Стек: Python 3.9+, Flask 3.1.0, Waitress 3.0.2, PostgreSQL 16.6, psycopg 3.2.6, HAProxy 3.0.8, boto3 1.36.26, Pillow 11.1.0, S3-совместимое объектное хранилище.
- Использованные источники и помощь ИИ: требования из Markdown-файлов лабораторной №4, `contracts/openapi.json`, публичные тесты и собственный проект лабораторной №3. ИИ использовался для разбора требований, проектирования структуры приложения, проверки соответствия контракту и подготовки отчёта.
## Запуск из чистого clone
## Запуск из чистого clone / ZIP
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
Нужны Docker Engine с Compose v2, Python 3.9+ для публичных тестов и доступ к registry для загрузки контейнерных образов. Все команды выполняются в корне `mtusi2026lab4`. Опубликован только HAProxy на `127.0.0.1:8080`.
## Архитектура и решения
```sh
python3 scripts/init_local.py
# Секреты создаются один раз в secrets/ и не входят в ZIP/Git.
docker compose --env-file .env config --quiet
docker compose --env-file .env up -d --build
docker compose --env-file .env ps
# дождаться: curl -fsS http://localhost:8080/health/ready
make check
make test
```
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab3, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
Swagger UI в четвёртой лабораторной не заявлен; машинно-читаемый контракт доступен в `contracts/openapi.json`. Проверка через curl: `curl http://localhost:8080/health/live`. Выключение: `docker compose down` (данные остаются); намеренное уничтожение **всех учебных данных**: `docker compose down -v` (не делайте при проверке persistence). После изменения миграций запустить `docker compose up -d --build`, сервис `migrate` применит версии однократно. Если ранее был развёрнут lab3 со своими томами, сохраняйте его PostgreSQL volume и секреты при переносе; новая миграция `003_attachments.sql` добавляется к существующей схеме.
## Доказательства критериев
## Архитектура
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
```
REST-клиент -> HAProxy :8080 -> api1 / api2 (Flask + waitress)
| |
+-- PostgreSQL :5432 (таблицы/сессии/идемпотентность)
+-- MinIO S3 :9000 (оригиналы PNG/JPEG)
```
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
* `db`: `postgres:16.6-bookworm`, БД messenger, `postgres_data`, health `pg_isready`. `migrate` однократно создаёт/обновляет schema_migrations.
* `s3`: `minio/minio:RELEASE.2025-02-07T23-21-09Z`, `s3_data`, без опубликованных портов, локальный health endpoint; `s3-bootstrap` идемпотентно создаёт приватный bucket `messenger-attachments`.
* `api1` и `api2`: сборка из `Dockerfile` на `python:3.11.11-slim-bookworm`, readiness проверяет и БД, и S3; не имеют volume для бинарных вложений.
* `lb`: `haproxy:3.0.8-bookworm`, опубликован только `127.0.0.1:8080`.
* Секреты: `secrets/postgres_password`, `secrets/database_url`, `secrets/s3_password` создаются локально `scripts/init_local.py`; `DATABASE_URL_FILE`, `S3_SECRET_FILE` указывают на Docker secrets. `APP_ENV`, TTL и сетевые параметры заданы в Compose. API использует административные MinIO credentials ради простоты уровня 3; разграничение S3 permissions — задача уровня 4.
## Логика загрузки
1. Bearer-сессия и принадлежность чату проверяются перед чтением тела. `filename` валидируется, не используется в пути. Заголовок MIME и содержимое должны соответствовать PNG либо JPEG.
2. Лимит чтения 10 MiB + 1 байт и пикселей 40 млн проверяется **до** полной распаковки; Pillow проверяет структуру и полное декодирование. Объекту назначается независимый случайный ключ `originals/<random>/<random>`.
3. Бинарный оригинал записывается в **приватный S3**. После успешного PUT данные метаданных коммитятся в PostgreSQL, только затем отдаётся `201 ready` (`variants=[]`, `error_code=null`). SHA-256 и размер считаются по исходным байтам.
4. Для создания сообщения допускается `attachment_ids` 0–10 уникальных UUID, включая пустой текст с непустым массивом. Все IDs должны принадлежать чату. `message_attachments` хранит порядок; идемпотентность учитывает текст и отсортированный по позиции список IDs (порядок важен).
5. `GET /attachments/{id}` и `/content` разрешены только участникам чата. API скачивает байты напрямую из S3, без редиректа и без открытия bucket наружу. Чужому пользователю возвращается одинаковый `404` для несуществующего или недоступного объекта.
SQL-транзакция не охватывает S3: если PUT прошёл, а запись БД не состоялась, выполняется **best-effort** удаление ключа. При разрыве процесса либо при отказе удаления возможен orphan-объект; полноценную фоновую сверку на уровне 3 не заявляем. Приватность bucket обеспечивается отсутствием публичной policy и закрытым сетевым портом; авторизация реализована отдельно в API.
## Доказательства критериев оценки «3»
Результаты ниже отделяют **фактически выполненные локальные проверки** от контейнерных и сетевых сценариев, требующих работающего Docker-стенда. Для подтверждения на машине студента вывод команд можно сохранить в `evidence/`:
```sh
mkdir -p evidence
make check 2>&1 | tee evidence/contract-local.txt
python3 -m unittest discover -s student-tests -p test_images.py -v 2>&1 | tee evidence/images-local.txt
docker compose --env-file .env ps -a 2>&1 | tee evidence/docker-services.txt
make test 2>&1 | tee evidence/public-smoke.txt
```
| Проверка | Команда / метод | Ожидаемый критерий |
| --- | --- | --- |
| Контракт и исходники | `make check` | Корректная структура lab4/OpenAPI и синтаксис Python |
| Локальная валидация файлов | `python -m unittest discover -s student-tests -p test_images.py -v` | Реальный PNG, неверный MIME, повреждение, лимит bytes/pixels |
| Сквозной API | `make test` | Авторизация и lab3-функции, `201 ready`, SHA-256, бинарный download, чужой пользователь 404 |
| Отказ API-реплики | `make test-failover ACTION_SCRIPT=scripts/stop-one.sh` | `TARGET_INSTANCE_ID` останавливается, другой API обслуживает запросы; тест включает S3 file check для lab4 |
| Долговечность | `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | Перезапуск двух API, PostgreSQL и MinIO с прежними volumes, затем чтение с прежним токеном |
| Приватность S3 | `docker compose exec -T api1 python -c 'from messenger.config import Settings; from messenger.storage import ObjectStore; s=ObjectStore(Settings.from_env()); print(s.s3.get_bucket_policy(Bucket=s.bucket))'` | Без заданной публичной policy запрос должен закончиться ошибкой `NoSuchBucketPolicy`; по умолчанию бакет приватен |
**Фактически выполнено в среде подготовки:** `python scripts/check_contract.py` — PASS; `python -m unittest discover -s student-tests -p test_images.py -v` — 4 OK; `python -m compileall -q messenger` — без ошибок.
## Эксперименты и отказы
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
## Ограничения и следующие шаги
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
| Сценарий | Ожидаемое поведение | Наблюдение / статус |
| --- | --- | --- |
| Неверный MIME или повреждённое изображение | Отказ без создания корректного вложения | Проверено локальными unit-тестами, 4 теста OK |
| Файл, превышающий лимиты | Отказ в загрузке | Проверено локальными unit-тестами |
| Чтение чужого вложения | 404 без раскрытия данных | Реализовано в API, end-to-end проверка на Docker-хосте требуется |
| Перезапуск API и хранилищ | Сохранение метаданных в PostgreSQL и оригиналов в S3 volumes | Подготовлен тест `make test-persistence`, Docker-подтверждение не получено |
| Остановка одной API-реплики | Вторая реплика продолжает обслуживать HTTP-запросы через HAProxy | Подготовлен тест `make test-failover`, Docker-подтверждение не получено |
| Отказ между PUT в S3 и COMMIT в БД | Best-effort удаление S3-объекта | Реализована компенсация, гарантированной очистки orphan-объектов нет |
+109
View File
@@ -0,0 +1,109 @@
x-api: &api
build: .
environment: &app-environment
APP_HOST: 0.0.0.0
APP_PORT: 8080
APP_ENV: ${APP_ENV:-lab}
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
IDEMPOTENCY_TTL_SECONDS: ${IDEMPOTENCY_TTL_SECONDS:-86400}
SESSION_STORE: postgres
DATABASE_URL_FILE: /run/secrets/database_url
S3_ENDPOINT: http://s3:9000
S3_BUCKET: messenger-attachments
S3_ACCESS_KEY: messengeradmin
S3_SECRET_FILE: /run/secrets/s3_password
secrets:
- database_url
- s3_password
depends_on:
db:
condition: service_healthy
migrate:
condition: service_completed_successfully
s3-bootstrap:
condition: service_completed_successfully
stop_grace_period: 8s
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready',timeout=3)"]
interval: 2s
timeout: 4s
retries: 20
services:
db:
image: postgres:16.6-bookworm
environment:
POSTGRES_DB: messenger
POSTGRES_USER: messenger
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
secrets:
- postgres_password
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U messenger -d messenger"]
interval: 2s
timeout: 3s
retries: 30
s3:
image: minio/minio:RELEASE.2025-02-07T23-21-09Z
command: server /data --address ':9000'
environment:
MINIO_ROOT_USER: messengeradmin
MINIO_ROOT_PASSWORD_FILE: /run/secrets/s3_password
secrets:
- s3_password
volumes:
- s3_data:/data
healthcheck:
test: ["CMD-SHELL", "curl -fsS http://127.0.0.1:9000/minio/health/live >/dev/null"]
interval: 3s
timeout: 5s
retries: 25
s3-bootstrap:
build: .
command: ["python", "-m", "messenger.bootstrap_s3"]
environment: *app-environment
secrets:
- database_url
- s3_password
depends_on:
s3:
condition: service_started
restart: "no"
migrate:
build: .
command: ["python", "-m", "messenger.migrate"]
environment: *app-environment
secrets:
- database_url
- s3_password
depends_on:
db:
condition: service_healthy
restart: "no"
api1:
<<: *api
api2:
<<: *api
lb:
image: haproxy:3.0.8-bookworm
ports:
- "127.0.0.1:8080:8080"
volumes:
- ./infra/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
depends_on:
api1:
condition: service_healthy
api2:
condition: service_healthy
volumes:
postgres_data:
s3_data:
secrets:
postgres_password:
file: ./secrets/postgres_password
database_url:
file: ./secrets/database_url
s3_password:
file: ./secrets/s3_password
+29
View File
@@ -0,0 +1,29 @@
# Схема и транзакционные границы
Проект схемы составлен перед реализацией SQL-адаптера. PostgreSQL — единый источник истины, включая сессии.
| Таблица | Ключи и ограничения | Индексы и назначение |
| --- | --- | --- |
| users | UUID PK, username UNIQUE, NOT NULL для имени/хеша/времени | UNIQUE username для регистрации и входа |
| chats | UUID PK, title NOT NULL | (created_at, id) для списка |
| chat_members | (chat_id, user_id) PK, два FK, position для порядка участников | (user_id, chat_id) для доступных чатов |
| messages | UUID PK, chat_id FK, sender_id FK, составной FK на членство | (chat_id, created_at, id) для истории |
| sessions | token_hash PK, user_id FK, expires_at NOT NULL | expiry для возможной очистки |
| app_settings | name PK, value NOT NULL | постоянный случайный ключ подписи курсоров |
| schema_migrations | version PK | только учёт применённых миграций |
Каждый метод записи репозитория — одна транзакция с COMMIT до HTTP-ответа. Создание чата: проверить существование всех пользователей, вставить чат и всех участников, COMMIT; ошибка откатывает всё. Регистрация защищена UNIQUE, конфликт возвращает 409. Сообщение вставляется после проверки членства; FK также не позволяет записать отправителя вне чата. Создание/отзыв сессии — отдельные транзакции. Сессия проверяется на каждом защищённом запросе по времени БД; рестарт не меняет expiry.
Миграции нумеруются в migrations/*.sql. Runner выполняет новые файлы и запись версии в одной транзакции под advisory lock. Повторный запуск не выполняет уже учтённую версию. Пароли — Argon2id; в sessions лежит SHA-256 случайного 256-битного токена, а не пароль. SHA-256 здесь служит индексом высокоэнтропийного токена и не заменяет password KDF.
Пагинация SQL keyset, LIMIT + 1; offset не используется. Курсор сохраняет область ресурса и подписан постоянным ключом из БД, поэтому продолжение работает после рестарта. Часы PostgreSQL задают created_at. Между страницами нет snapshot; поздно закоммиченная конкурентная запись с ключом до курсора может не попасть в уже начатый обход. Для неизменной истории гарантируется отсутствие повторов и пропусков.
## Lab3: общий результат отправки
Миграция 002 добавляет message_requests с PK `(user_id, chat_id, idempotency_key)`, FK на сообщение и expires_at. Таблица относится только к POST сообщений: HTTP-метод и тип пути заданы её назначением. В ключ не входят имя реплики или сессия, поэтому новый вход того же пользователя не меняет область идемпотентности.
Перед записью PostgreSQL advisory transaction lock блокирует только хеш области ключа. Все реплики вычисляют его одинаково через SHA-256; совпадение 64-битных хешей может лишь сериализовать независимые операции, но не объединяет их результаты: lookup всегда по полному составному ключу. После ожидания блокировки новый statement в READ COMMITTED видит результат победившей транзакции.
При существующей действующей записи сравнивается текст: тот же текст возвращает исходное Message с 201, другой — 409. Поля и типы валидируются до этого; в lab3 единственное поле CreateMessage — text. Создание Message и message_requests происходит в одной транзакции, поэтому 5xx/rollback не оставляет отдельного подтверждённого результата. После COMMIT потерянный HTTP-ответ можно восстановить повтором. Expiry не продлевается повтором; TTL не менее 24 часов. Просроченная запись ключа удаляется при повторном использовании этой области, старое сообщение остаётся в истории.
Реплики не запускают миграции при старте. Единственный Compose job migrate выполняет их перед API; advisory lock runner дополнительно защищает ручной повтор запуска. Существующие миграции 001 сохранены без изменения. В lab3 нет автоматического обновления работающего стенда с несовместимым DDL.
+5
View File
@@ -0,0 +1,5 @@
# S3 and DB boundaries
`minio/minio:RELEASE.2025-02-07T23-21-09Z` runs as `s3` on internal port 9000; it has a persistent Docker volume `s3_data`. Console is not exposed and S3 port is not published. `s3-bootstrap` creates private `messenger-attachments` bucket idempotently. The API uploads bytes to S3 before committing PostgreSQL metadata; 201 follows commit. A failed SQL transaction triggers best-effort S3 delete. This does **not** guarantee orphan-free operation if delete or host process fails; full reconciliation belongs to level 4/5.
Only server-generated random object keys are used; filenames never become paths. GET enforces chat membership before contacting object storage. PostgreSQL `attachments` contains metadata and an internal key, while `message_attachments` preserves ordering. `s3_data` and `postgres_data` must survive restart. For local demonstration credentials live in ignored `secrets/` files; this level-3 exercise uses the same S3 root credential for bootstrap and API (narrow scoped credentials are a level-4 task). Do not deploy to public internet without TLS and restricted credentials.
+41
View File
@@ -0,0 +1,41 @@
global
log stdout format raw local0
maxconn 256
defaults
log global
mode http
option dontlognull
timeout connect 1s
timeout client 10s
timeout server 10s
timeout check 1s
timeout http-request 10s
timeout http-keep-alive 2s
# No transparent replay of POSTs. The client owns retry + Idempotency-Key.
retries 0
resolvers docker
nameserver dns 127.0.0.11:53
resolve_retries 3
timeout resolve 1s
timeout retry 1s
hold valid 2s
hold nx 1s
hold other 1s
hold refused 1s
hold timeout 1s
hold obsolete 1s
frontend public
bind :8080
default_backend messenger
backend messenger
balance roundrobin
option httpchk
http-check send meth GET uri /health/ready ver HTTP/1.1 hdr Host messenger
http-check expect status 200
default-server inter 1s fastinter 500ms downinter 1s fall 2 rise 1
server api1 api1:8080 check resolvers docker resolve-prefer ipv4 init-addr libc,none
server api2 api2:8080 check resolvers docker resolve-prefer ipv4 init-addr libc,none
+1
View File
@@ -0,0 +1 @@
"""Messenger lab series."""
+41
View File
@@ -0,0 +1,41 @@
"""One process per replica; bounded drain on SIGTERM."""
import logging
import signal
import time
from waitress import create_server
from .api import create_app
from .config import Settings
def main():
logging.basicConfig(level=logging.INFO)
settings = Settings.from_env()
app = create_app(settings=settings)
server = create_server(app, host=settings.host, port=settings.port, threads=4)
stopping = False
def stop(signum, frame):
nonlocal stopping
stopping = True
signal.signal(signal.SIGTERM, stop)
signal.signal(signal.SIGINT, stop)
logging.info('Starting messenger environment=%s', settings.environment)
try:
while not stopping:
server.asyncore.loop(timeout=0.1, count=1, map=server._map)
# Stop accepting new TCP connections. Keep the event loop alive briefly
# so responses from existing worker tasks can still be flushed.
app.config['DRAINING'] = True
server.close()
deadline = time.monotonic() + 5
while server._map and time.monotonic() < deadline:
server.asyncore.loop(timeout=0.1, count=1, map=server._map)
finally:
server.task_dispatcher.shutdown(timeout=1)
server.asyncore.close_all(map=server._map)
logging.info('Messenger stopped')
if __name__ == '__main__':
main()
+190
View File
@@ -0,0 +1,190 @@
"""HTTP adapter and composition root."""
import json
import hashlib
import re
import unicodedata
from uuid import UUID
from .images import MAX_BYTES, validate_image
from .storage import ObjectStore
from uuid import uuid4
from flask import Flask, g, request, jsonify, Response, stream_with_context
from werkzeug.exceptions import HTTPException
from .domain import ApiError, invalid, uuid_value
from .postgres import PostgresRepository
from .database import Database
from .config import Settings
from .service import Messenger
def create_app(repository=None, settings=None):
app = Flask(__name__)
app.json.ensure_ascii = False
settings = settings or Settings.from_env()
repository = repository if repository is not None else PostgresRepository(Database(settings.database_url))
service = Messenger(repository, settings.session_ttl, settings.idempotency_ttl)
instance_id = str(uuid4())
store = None if app.config.get("TESTING_STORAGE_DISABLED") else (ObjectStore(settings) if repository is None or isinstance(repository, PostgresRepository) else None)
@app.before_request
def identify():
g.request_id = str(uuid4())
if app.config.get('DRAINING'):
raise ApiError(503, 'unavailable', 'Instance is shutting down')
if request.path.startswith('/api/v1/') and not (
request.path in ('/api/v1/users', '/api/v1/auth/sessions') and request.method == 'POST'
):
g.user, g.token_hash = service.auth.bearer(request.headers.get('Authorization'))
@app.after_request
def request_id(response):
response.headers['X-Request-Id'] = g.request_id
response.headers['X-Instance-Id'] = instance_id
return response
@app.errorhandler(Exception)
def error(exc):
if isinstance(exc, ApiError):
status, code, message = exc.status, exc.code, exc.message
elif isinstance(exc, HTTPException):
status = exc.code
code = {404: 'not_found', 413: 'payload_too_large', 415: 'unsupported_media_type'}.get(status, 'invalid_request')
message = exc.name
else:
app.logger.error('Unhandled error; request_id=%s', g.request_id)
status, code, message = 500, 'internal_error', 'Internal server error'
response = jsonify(error={'code': code, 'message': message, 'request_id': g.request_id})
response.status_code = status
if status == 401:
response.headers['WWW-Authenticate'] = (
'Basic realm="messenger", charset="UTF-8"'
if request.path == '/api/v1/auth/sessions' and request.method == 'POST'
else 'Bearer realm="messenger"')
return response
def body():
if request.mimetype != 'application/json':
raise ApiError(415, 'unsupported_media_type', 'Use application/json')
try:
def reject_constant(value):
raise ValueError(value)
return json.loads(request.get_data().decode('utf-8'), parse_constant=reject_constant)
except (ValueError, UnicodeError):
invalid('Malformed UTF-8 JSON')
def paging():
for key in ('limit', 'cursor'):
if len(request.args.getlist(key)) > 1:
invalid('Duplicate query parameter')
return request.args.get('limit', '50'), request.args.get('cursor')
@app.get('/health/live')
def health():
return jsonify(status='ok')
@app.get('/health/ready')
def ready():
repository.ready()
if store is not None:
store.ready()
return jsonify(status='ok')
@app.post('/api/v1/auth/sessions')
def login():
# Session creation accepts no request body.
if request.get_data():
invalid('Session creation does not accept a body')
return jsonify(service.auth.login(request.headers.get('Authorization'))), 201
@app.delete('/api/v1/auth/sessions/current')
def logout():
repository.revoke_session(g.token_hash)
return '', 204
@app.post('/api/v1/users')
def register():
return jsonify(service.register(body())), 201
@app.get('/api/v1/users/me')
def me():
return jsonify(g.user)
@app.route('/api/v1/chats', methods=['GET', 'POST'])
def chats():
if request.method == 'POST':
return jsonify(service.create_chat(g.user['id'], body())), 201
return jsonify(service.chats(g.user['id'], *paging()))
@app.get('/api/v1/chats/<chat_id>')
def chat(chat_id):
return jsonify(service.chat(g.user['id'], chat_id))
@app.route('/api/v1/chats/<chat_id>/messages', methods=['GET', 'POST'])
def messages(chat_id):
if request.method == 'POST':
service.chat(g.user['id'], chat_id)
return jsonify(service.send(g.user['id'], chat_id, body(), request.headers.get('Idempotency-Key'))), 201
return jsonify(service.history(g.user['id'], chat_id, *paging()))
@app.post('/api/v1/chats/<chat_id>/attachments')
def upload(chat_id):
chat_id = uuid_value(chat_id)
service.chat(g.user['id'], chat_id)
values = request.args.getlist('filename')
if len(values) != 1:
invalid('Exactly one filename is required')
filename = values[0]
if (not 1 <= len(filename) <= 255 or any(
c in ('/', '\\') or unicodedata.category(c) == 'Cc' for c in filename)):
invalid('Unsafe filename')
if request.mimetype not in ('image/png', 'image/jpeg'):
raise ApiError(415, 'unsupported_media_type', 'PNG or JPEG required')
if request.content_length is not None and request.content_length > MAX_BYTES:
raise ApiError(413, 'payload_too_large', 'Image exceeds 10 MiB')
# Bounded read even if Content-Length is absent or misleading.
content = request.stream.read(MAX_BYTES + 1)
if len(content) > MAX_BYTES:
raise ApiError(413, 'payload_too_large', 'Image exceeds 10 MiB')
validate_image(content, request.mimetype)
if store is None:
raise ApiError(503, 'unavailable', 'Object storage unavailable')
identifier = str(uuid4())
key = 'originals/' + uuid4().hex + '/' + uuid4().hex
metadata = {'id': identifier, 'filename': filename, 'content_type': request.mimetype,
'size_bytes': len(content), 'sha256': hashlib.sha256(content).hexdigest(),
'object_key': key}
store.put(key, content, request.mimetype)
try:
result = repository.insert_attachment(g.user['id'], chat_id, metadata)
except Exception:
store.delete(key) # best effort compensation for DB commit failure
raise
return jsonify(result), 201
@app.get('/api/v1/attachments/<attachment_id>')
def attachment(attachment_id):
obj, _key = repository.attachment(g.user['id'], uuid_value(attachment_id))
return jsonify(obj)
@app.get('/api/v1/attachments/<attachment_id>/content')
def download(attachment_id):
obj, key = repository.attachment(g.user['id'], uuid_value(attachment_id))
variants = request.args.getlist('variant')
if len(variants) > 1:
invalid('Duplicate variant')
if variants and variants[0] != 'original':
raise ApiError(404, 'not_found', 'Variant not found')
if store is None:
raise ApiError(503, 'unavailable', 'Object storage unavailable')
body = store.get(key)
def chunks():
try:
for chunk in body.iter_chunks(chunk_size=65536):
if chunk:
yield chunk
finally:
body.close()
return Response(stream_with_context(chunks()), mimetype=obj['content_type'],
headers={'Content-Length': str(obj['size_bytes']),
'Cache-Control': 'private, no-store'})
return app
+27
View File
@@ -0,0 +1,27 @@
"""Run after S3 readiness, repeatable. Default MinIO bucket is private."""
import time
from botocore.exceptions import ClientError, BotoCoreError
from .config import Settings
from .storage import ObjectStore
def main():
storage = ObjectStore(Settings.from_env())
for attempt in range(40):
try:
storage.s3.list_buckets()
break
except (ClientError, BotoCoreError, OSError):
if attempt == 39:
raise
time.sleep(2)
try:
storage.s3.head_bucket(Bucket=storage.bucket)
except ClientError as exc:
if exc.response.get('ResponseMetadata', {}).get('HTTPStatusCode') not in (404,):
raise
storage.s3.create_bucket(Bucket=storage.bucket)
# Public ACL/policy intentionally absent; MinIO bucket is private by default.
print('Private S3 bucket exists')
if __name__ == '__main__':
main()
+40
View File
@@ -0,0 +1,40 @@
"""Configuration is loaded only at process startup; secrets are never logged."""
import os
from dataclasses import dataclass
from pathlib import Path
@dataclass(frozen=True)
class Settings:
database_url: str
host: str = '0.0.0.0'
port: int = 8080
environment: str = 'lab'
session_ttl: int = 3600
idempotency_ttl: int = 86400
s3_endpoint: str = "http://s3:9000"
s3_bucket: str = "messenger-attachments"
s3_access_key: str = "messengeradmin"
s3_secret_file: str = "/run/secrets/s3_password"
@classmethod
def from_env(cls):
source = os.getenv('DATABASE_URL_FILE')
url = Path(source).read_text().strip() if source else os.getenv('DATABASE_URL', '')
if not url:
raise ValueError('Set DATABASE_URL_FILE or DATABASE_URL')
if os.getenv('SESSION_STORE', 'postgres') != 'postgres':
raise ValueError('This lab uses SESSION_STORE=postgres')
port = int(os.getenv('APP_PORT', '8080'))
ttl = int(os.getenv('SESSION_TTL_SECONDS', '3600'))
idempotency_ttl = int(os.getenv('IDEMPOTENCY_TTL_SECONDS', '86400'))
if idempotency_ttl < 86400:
raise ValueError('IDEMPOTENCY_TTL_SECONDS must be at least 86400')
if not 1 <= port <= 65535 or ttl <= 0:
raise ValueError('Invalid APP_PORT or SESSION_TTL_SECONDS')
return cls(url, os.getenv('APP_HOST', '0.0.0.0'), port,
os.getenv('APP_ENV', 'lab'), ttl, idempotency_ttl,
os.getenv("S3_ENDPOINT", "http://s3:9000"),
os.getenv("S3_BUCKET", "messenger-attachments"),
os.getenv("S3_ACCESS_KEY", "messengeradmin"),
os.getenv("S3_SECRET_FILE", "/run/secrets/s3_password"))
+20
View File
@@ -0,0 +1,20 @@
"""Short-lived database connections, one transaction per operation."""
from contextlib import contextmanager
import psycopg
from psycopg.rows import dict_row
from .domain import ApiError
class Database:
def __init__(self, url):
self.url = url
@contextmanager
def transaction(self):
try:
with psycopg.connect(self.url, row_factory=dict_row, connect_timeout=3,
options='-c statement_timeout=5000 -c lock_timeout=3000 -c timezone=UTC') as conn:
yield conn
except (psycopg.OperationalError, psycopg.InterfaceError,
psycopg.errors.QueryCanceled, psycopg.errors.LockNotAvailable):
raise ApiError(503, 'unavailable', 'Database temporarily unavailable') from None
+33
View File
@@ -0,0 +1,33 @@
"""Framework-independent validation and application errors."""
import re
from uuid import UUID
class ApiError(Exception):
def __init__(self, status, code, message):
self.status, self.code, self.message = status, code, message
def invalid(message='Invalid request'):
raise ApiError(400, 'invalid_request', message)
def uuid_value(value):
if not isinstance(value, str) or not re.fullmatch(
r'[0-9a-fA-F]{8}(?:-[0-9a-fA-F]{4}){3}-[0-9a-fA-F]{12}', value
):
invalid('Invalid UUID')
return str(UUID(value))
def fields(body, names):
if not isinstance(body, dict) or set(body) != set(names):
invalid('Required fields are missing or unknown fields supplied')
def string(value, maximum):
if (not isinstance(value, str) or not value.strip()
or len(value) > maximum or "\x00" in value
or any(0xD800 <= ord(c) <= 0xDFFF for c in value)):
invalid('Invalid string')
return value
+30
View File
@@ -0,0 +1,30 @@
"""Bounded image validation: magic+full decode, no trust in declared MIME."""
import io
from PIL import Image, ImageFile, UnidentifiedImageError
from .domain import ApiError
MAX_BYTES = 10 * 1024 * 1024
MAX_PIXELS = 40_000_000
Image.MAX_IMAGE_PIXELS = MAX_PIXELS
ImageFile.LOAD_TRUNCATED_IMAGES = False
def validate_image(data, declared_type):
if declared_type not in ('image/png', 'image/jpeg'):
raise ApiError(415, 'unsupported_media_type', 'PNG or JPEG required')
if len(data) > MAX_BYTES:
raise ApiError(413, 'payload_too_large', 'Image exceeds 10 MiB')
if not data:
raise ApiError(415, 'unsupported_media_type', 'Empty image')
try:
with Image.open(io.BytesIO(data)) as im:
real_type = {'PNG': 'image/png', 'JPEG': 'image/jpeg'}.get(im.format)
if real_type is None or real_type != declared_type:
raise ApiError(415, 'unsupported_media_type', 'Image content type mismatch')
if im.width * im.height > MAX_PIXELS:
raise ApiError(413, 'payload_too_large', 'Image pixel limit exceeded')
im.load() # reject malformed/truncated payload before PUT
except (UnidentifiedImageError, OSError, ValueError, SyntaxError):
raise ApiError(415, 'unsupported_media_type', 'Invalid image') from None
except Image.DecompressionBombError:
raise ApiError(413, 'payload_too_large', 'Image pixel limit exceeded') from None
+24
View File
@@ -0,0 +1,24 @@
"""Versioned, transactional migrations; safe to run again."""
from pathlib import Path
import secrets
from .config import Settings
from .database import Database
def migrate(database):
directory = Path(__file__).resolve().parents[1] / 'migrations'
with database.transaction() as conn:
conn.execute('SELECT pg_advisory_xact_lock(20260201)')
conn.execute('CREATE TABLE IF NOT EXISTS schema_migrations (version text PRIMARY KEY)')
versions = {r['version'] for r in conn.execute('SELECT version FROM schema_migrations')}
for path in sorted(directory.glob('*.sql')):
if path.name not in versions:
conn.execute(path.read_text(encoding='utf-8'))
conn.execute('INSERT INTO schema_migrations(version) VALUES (%s)', (path.name,))
conn.execute("INSERT INTO app_settings(name,value) VALUES ('cursor_key',%s) ON CONFLICT DO NOTHING",
(secrets.token_hex(32),))
if __name__ == '__main__':
migrate(Database(Settings.from_env().database_url))
print('Migrations applied successfully (already applied versions skipped).')
+43
View File
@@ -0,0 +1,43 @@
"""Resource-scoped signed cursors; SQL fetches only limit+1 rows."""
import base64
import hashlib
import hmac
import json
import re
from datetime import datetime
from .domain import invalid, uuid_value
class Pagination:
def __init__(self, key):
self._key = key
def decode(self, scope, limit, cursor):
if not re.fullmatch(r'[0-9]{1,3}', limit) or not 1 <= int(limit) <= 100:
invalid('Limit must be between 1 and 100')
after = None
if cursor is not None:
try:
if len(cursor) > 2048:
raise ValueError()
raw = base64.b64decode(cursor, altchars=b'-_', validate=True)
payload, signature = raw[:-32], raw[-32:]
if not hmac.compare_digest(signature, hmac.digest(self._key, payload, hashlib.sha256)):
raise ValueError()
saved_scope, timestamp, identifier = json.loads(payload)
if saved_scope != scope or not isinstance(timestamp, str):
raise ValueError()
datetime.fromisoformat(timestamp.replace('Z', '+00:00'))
after = (timestamp, uuid_value(identifier))
except (ValueError, TypeError, UnicodeError):
invalid('Invalid cursor for this resource')
return int(limit), after
def page(self, rows, scope, limit):
items = rows[:limit]
next_cursor = None
if len(rows) > limit:
last = items[-1]
payload = json.dumps([scope, last['created_at'], last['id']], separators=(',', ':')).encode()
next_cursor = base64.urlsafe_b64encode(payload + hmac.digest(self._key, payload, hashlib.sha256)).decode()
return {'items': items, 'next_cursor': next_cursor}
+196
View File
@@ -0,0 +1,196 @@
"""PostgreSQL repository: all SQL values are bound parameters."""
from datetime import datetime, timezone
import hashlib
import json
from uuid import UUID, uuid4
import psycopg
from .domain import ApiError, invalid
def serialize(row):
if row is None:
return None
result = {}
for key, value in row.items():
if isinstance(value, UUID):
value = str(value)
elif isinstance(value, datetime):
value = value.astimezone(timezone.utc).isoformat(timespec='microseconds').replace('+00:00', 'Z')
elif isinstance(value, list):
value = [str(item) if isinstance(item, UUID) else item for item in value]
result[key] = value
return result
USER_COLUMNS = 'id, username, display_name, created_at'
# Members retain the request order, with no password/session fields in public objects.
CHAT_COLUMNS = '''c.id, c.title, c.created_at,
ARRAY(SELECT m.user_id FROM chat_members m WHERE m.chat_id=c.id ORDER BY m.position) AS member_ids'''
class PostgresRepository:
def __init__(self, database):
self.db = database
def ready(self):
with self.db.transaction() as conn:
conn.execute('SELECT id FROM users LIMIT 0')
conn.execute('SELECT token_hash FROM sessions LIMIT 0')
def cursor_key(self):
with self.db.transaction() as conn:
row = conn.execute("SELECT value FROM app_settings WHERE name='cursor_key'").fetchone()
if row is None:
raise ApiError(503, 'unavailable', 'Database is not initialized')
return bytes.fromhex(row['value'])
def create_user(self, username, display_name, password_hash):
try:
with self.db.transaction() as conn:
row = conn.execute('INSERT INTO users(id,username,display_name,password_hash) VALUES (%s,%s,%s,%s) RETURNING ' + USER_COLUMNS,
(uuid4(), username, display_name, password_hash)).fetchone()
return serialize(row)
except psycopg.errors.UniqueViolation:
raise ApiError(409, 'conflict', 'Username already exists') from None
def credentials(self, username):
with self.db.transaction() as conn:
return serialize(conn.execute('SELECT ' + USER_COLUMNS + ', password_hash FROM users WHERE username=%s',
(username,)).fetchone())
def create_session(self, user_id, token_hash, ttl):
with self.db.transaction() as conn:
row = conn.execute('INSERT INTO sessions(token_hash,user_id,expires_at) VALUES (%s,%s,clock_timestamp() + %s * interval \'1 second\') RETURNING expires_at',
(token_hash, user_id, ttl)).fetchone()
return serialize(row)['expires_at']
def session_user(self, token_hash):
with self.db.transaction() as conn:
row = conn.execute('''SELECT u.id,u.username,u.display_name,u.created_at FROM users u
JOIN sessions s ON s.user_id=u.id
WHERE s.token_hash=%s AND s.expires_at>clock_timestamp()''', (token_hash,)).fetchone()
if row is None:
raise ApiError(401, 'unauthorized', 'Valid Bearer session required')
return serialize(row)
def revoke_session(self, token_hash):
with self.db.transaction() as conn:
conn.execute('DELETE FROM sessions WHERE token_hash=%s', (token_hash,))
@staticmethod
def _access(conn, actor, chat_id):
if conn.execute('SELECT 1 FROM chat_members WHERE chat_id=%s AND user_id=%s', (chat_id, actor)).fetchone() is None:
raise ApiError(404, 'not_found', 'Chat not found')
def create_chat(self, actor, title, members):
with self.db.transaction() as conn:
count = conn.execute('SELECT count(*) AS n FROM users WHERE id=ANY(%s::uuid[])', (members,)).fetchone()['n']
if actor not in members or count != len(members):
invalid('Members must exist and include the current user')
identifier = uuid4()
conn.execute('INSERT INTO chats(id,title) VALUES (%s,%s)', (identifier, title))
with conn.cursor() as cur:
cur.executemany('INSERT INTO chat_members(chat_id,user_id,position) VALUES (%s,%s,%s)',
[(identifier, member, i) for i, member in enumerate(members)])
row = conn.execute('SELECT ' + CHAT_COLUMNS + ' FROM chats c WHERE c.id=%s', (identifier,)).fetchone()
return serialize(row)
def chat(self, actor, chat_id):
with self.db.transaction() as conn:
self._access(conn, actor, chat_id)
row = conn.execute('SELECT ' + CHAT_COLUMNS + ' FROM chats c WHERE c.id=%s', (chat_id,)).fetchone()
return serialize(row)
def chats(self, actor, limit, after):
params = [actor]
boundary = ''
if after is not None:
boundary = ' AND (c.created_at,c.id)>(%s::timestamptz,%s::uuid)'
params.extend(after)
params.append(limit)
with self.db.transaction() as conn:
rows = conn.execute('SELECT ' + CHAT_COLUMNS + ''' FROM chats c JOIN chat_members own
ON own.chat_id=c.id WHERE own.user_id=%s''' + boundary +
' ORDER BY c.created_at,c.id LIMIT %s', params).fetchall()
return [serialize(r) for r in rows]
@staticmethod
def _message(conn, message_id):
row = conn.execute('SELECT * FROM messages WHERE id=%s', (message_id,)).fetchone()
result = serialize(row)
result['attachment_ids'] = [str(r['attachment_id']) for r in conn.execute(
'SELECT attachment_id FROM message_attachments WHERE message_id=%s ORDER BY position',
(message_id,)).fetchall()]
return result
def create_message(self, actor, chat_id, text, attachments, key, ttl):
scope = json.dumps([actor, chat_id, key], separators=(',', ':')).encode()
lock_id = int.from_bytes(hashlib.sha256(scope).digest()[:8], 'big', signed=True)
with self.db.transaction() as conn:
self._access(conn, actor, chat_id)
conn.execute('SELECT pg_advisory_xact_lock(%s)', (lock_id,))
previous = conn.execute("""SELECT request_text, request_attachments, message_id
FROM message_requests WHERE user_id=%s AND chat_id=%s AND idempotency_key=%s
AND expires_at>clock_timestamp()""", (actor, chat_id, key)).fetchone()
if previous:
if previous['request_text'] != text or previous['request_attachments'] != attachments:
raise ApiError(409, 'conflict', 'Idempotency key already used with another message')
return self._message(conn, previous['message_id'])
conn.execute('DELETE FROM message_requests WHERE user_id=%s AND chat_id=%s AND idempotency_key=%s',
(actor, chat_id, key))
if attachments:
found = conn.execute('SELECT id FROM attachments WHERE chat_id=%s AND id=ANY(%s::uuid[])',
(chat_id, attachments)).fetchall()
if len(found) != len(attachments):
raise ApiError(404, 'not_found', 'Attachment not found')
message_id = uuid4()
conn.execute('INSERT INTO messages(id,chat_id,sender_id,text) VALUES (%s,%s,%s,%s)',
(message_id, chat_id, actor, text))
for i, aid in enumerate(attachments):
conn.execute('INSERT INTO message_attachments(message_id,attachment_id,position) VALUES (%s,%s,%s)',
(message_id, aid, i))
conn.execute("""INSERT INTO message_requests
(user_id,chat_id,idempotency_key,request_text,request_attachments,message_id,expires_at)
VALUES (%s,%s,%s,%s,%s::jsonb,%s,clock_timestamp() + %s * interval '1 second')""",
(actor, chat_id, key, text, json.dumps(attachments), message_id, ttl))
return self._message(conn, message_id)
def messages(self, actor, chat_id, limit, after):
params = [chat_id]
boundary = ''
if after is not None:
boundary = ' AND (created_at,id)>(%s::timestamptz,%s::uuid)'
params.extend(after)
params.append(limit)
with self.db.transaction() as conn:
self._access(conn, actor, chat_id)
rows = conn.execute('SELECT id FROM messages WHERE chat_id=%s' + boundary +
' ORDER BY created_at,id LIMIT %s', params).fetchall()
return [self._message(conn, row['id']) for row in rows]
@staticmethod
def _attachment(row):
obj = serialize(row)
obj.pop('object_key', None)
obj['status'] = 'ready'
obj['variants'] = []
obj['error_code'] = None
return obj
def insert_attachment(self, actor, chat_id, metadata):
with self.db.transaction() as conn:
self._access(conn, actor, chat_id)
row = conn.execute("""INSERT INTO attachments
(id,chat_id,owner_id,filename,content_type,size_bytes,sha256,object_key)
VALUES (%s,%s,%s,%s,%s,%s,%s,%s) RETURNING *""",
(metadata['id'], chat_id, actor, metadata['filename'], metadata['content_type'],
metadata['size_bytes'], metadata['sha256'], metadata['object_key'])).fetchone()
return self._attachment(row)
def attachment(self, actor, attachment_id):
with self.db.transaction() as conn:
row = conn.execute("""SELECT a.* FROM attachments a JOIN chat_members m ON m.chat_id=a.chat_id
WHERE a.id=%s AND m.user_id=%s""", (attachment_id, actor)).fetchone()
if row is None:
raise ApiError(404, 'not_found', 'Attachment not found')
return self._attachment(row), row['object_key']
+17
View File
@@ -0,0 +1,17 @@
"""Storage port. Concrete PostgreSQL adapter is wired in api.create_app."""
from typing import Protocol
class Repository(Protocol):
def ready(self): ...
def cursor_key(self): ...
def create_user(self, username, display_name, password_hash): ...
def credentials(self, username): ...
def create_session(self, user_id, token_hash, ttl): ...
def session_user(self, token_hash): ...
def revoke_session(self, token_hash): ...
def create_chat(self, actor, title, members): ...
def chat(self, actor, chat_id): ...
def chats(self, actor, limit, after): ...
def create_message(self, actor, chat_id, text, key, ttl): ...
def messages(self, actor, chat_id, limit, after): ...
+60
View File
@@ -0,0 +1,60 @@
"""Password KDF and HTTP credential parsing, without framework dependencies."""
import base64
import binascii
import hashlib
import re
import secrets
from argon2 import PasswordHasher, Type
from argon2.exceptions import VerificationError, InvalidHashError
from .domain import ApiError, invalid, string
class Authentication:
def __init__(self, repository, ttl):
self.repository = repository
self.ttl = ttl
self.hasher = PasswordHasher(time_cost=3, memory_cost=65536, parallelism=2,
hash_len=32, salt_len=16, type=Type.ID)
self._dummy = self.hasher.hash(secrets.token_urlsafe(32))
@staticmethod
def password(value):
string(value, 128)
if len(value) < 12:
invalid('Password must contain 12 to 128 Unicode characters')
return value
@staticmethod
def digest(token):
return hashlib.sha256(token.encode('ascii')).hexdigest()
def bearer(self, header):
parts = (header or '').split()
if len(parts) != 2 or parts[0].lower() != 'bearer' or not re.fullmatch(r'[A-Za-z0-9_-]{43}', parts[1]):
raise ApiError(401, 'unauthorized', 'Valid Bearer session required')
token_hash = self.digest(parts[1])
return self.repository.session_user(token_hash), token_hash
def login(self, header):
def denied():
raise ApiError(401, 'unauthorized', 'Invalid credentials')
try:
parts = (header or '').split()
if len(parts) != 2 or parts[0].lower() != 'basic':
denied()
username, password = base64.b64decode(parts[1], validate=True).decode('utf-8').split(':', 1)
if not re.fullmatch(r'[a-z0-9_]{3,32}', username) or not 12 <= len(password) <= 128:
denied()
except (ValueError, UnicodeError, binascii.Error):
denied()
row = self.repository.credentials(username)
try:
self.hasher.verify(row['password_hash'] if row else self._dummy, password)
except (VerificationError, InvalidHashError):
denied()
if row is None:
denied()
user = {key: value for key, value in row.items() if key != 'password_hash'}
token = secrets.token_urlsafe(32)
expires_at = self.repository.create_session(user['id'], self.digest(token), self.ttl)
return {'token': token, 'token_type': 'Bearer', 'expires_at': expires_at, 'user': user}
+68
View File
@@ -0,0 +1,68 @@
"""Use cases; independent of Flask and the concrete storage adapter."""
import re
from .domain import fields, string, uuid_value, invalid, ApiError
from .repository import Repository
from .pagination import Pagination
from .security import Authentication
class Messenger:
def __init__(self, repository: Repository, session_ttl=3600, idempotency_ttl=86400):
self.repository = repository
self.idempotency_ttl = idempotency_ttl
self.pagination = Pagination(repository.cursor_key())
self.auth = Authentication(repository, session_ttl)
def register(self, body):
fields(body, ('username', 'display_name', 'password'))
username = string(body['username'], 32)
if not re.fullmatch(r'[a-z0-9_]{3,32}', username):
invalid('Invalid username')
display_name = string(body['display_name'], 100)
password = self.auth.password(body['password'])
return self.repository.create_user(username, display_name, self.auth.hasher.hash(password))
def create_chat(self, actor, body):
fields(body, ('title', 'member_ids'))
title = string(body['title'], 100)
members = body['member_ids']
if not isinstance(members, list) or not 1 <= len(members) <= 100:
invalid('Invalid member list')
members = [uuid_value(m) for m in members]
if len(set(members)) != len(members):
invalid('Duplicate members')
return self.repository.create_chat(actor, title, members)
def chat(self, actor, chat_id):
return self.repository.chat(actor, uuid_value(chat_id))
def send(self, actor, chat_id, body, key):
self.chat(actor, chat_id)
if not isinstance(key, str) or not re.fullmatch(r'[A-Za-z0-9._:-]{1,128}', key):
invalid('Valid Idempotency-Key required')
if not isinstance(body, dict) or set(body) not in ({'text'}, {'text', 'attachment_ids'}):
invalid('Invalid message fields')
if not isinstance(body['text'], str) or len(body['text']) > 4000 or '\x00' in body['text']:
invalid('Invalid message text')
attachments = body.get('attachment_ids', [])
if not isinstance(attachments, list) or len(attachments) > 10:
invalid('Invalid attachment IDs')
attachments = [uuid_value(item) for item in attachments]
if len(attachments) != len(set(attachments)):
invalid('Duplicate attachment IDs')
if not body['text'].strip() and not attachments:
invalid('Message needs text or attachments')
return self.repository.create_message(actor, uuid_value(chat_id), body['text'],
attachments, key, self.idempotency_ttl)
def chats(self, actor, limit, cursor):
scope = 'chats:' + actor
limit, after = self.pagination.decode(scope, limit, cursor)
return self.pagination.page(self.repository.chats(actor, limit + 1, after), scope, limit)
def history(self, actor, chat_id, limit, cursor):
chat_id = uuid_value(chat_id)
self.repository.chat(actor, chat_id)
scope = 'messages:' + chat_id
limit, after = self.pagination.decode(scope, limit, cursor)
return self.pagination.page(self.repository.messages(actor, chat_id, limit + 1, after), scope, limit)
+39
View File
@@ -0,0 +1,39 @@
"""Private S3 object storage. No file bytes are stored on API disks."""
from pathlib import Path
import boto3
from botocore.config import Config
from botocore.exceptions import BotoCoreError, ClientError
from .domain import ApiError
class ObjectStore:
def __init__(self, settings):
self.bucket = settings.s3_bucket
self.s3 = boto3.client('s3', endpoint_url=settings.s3_endpoint,
aws_access_key_id=settings.s3_access_key,
aws_secret_access_key=Path(settings.s3_secret_file).read_text().strip(),
region_name='us-east-1', config=Config(signature_version='s3v4',
connect_timeout=3, read_timeout=10, retries={'max_attempts': 1}))
def ready(self):
try:
self.s3.head_bucket(Bucket=self.bucket)
except (ClientError, BotoCoreError, OSError):
raise ApiError(503, 'unavailable', 'Object storage unavailable') from None
def put(self, key, data, content_type):
try:
self.s3.put_object(Bucket=self.bucket, Key=key, Body=data, ContentType=content_type)
except (ClientError, BotoCoreError, OSError):
raise ApiError(503, 'unavailable', 'Object storage unavailable') from None
def get(self, key):
try:
return self.s3.get_object(Bucket=self.bucket, Key=key)['Body']
except (ClientError, BotoCoreError, OSError):
raise ApiError(503, 'unavailable', 'Object storage unavailable') from None
def delete(self, key):
try:
self.s3.delete_object(Bucket=self.bucket, Key=key)
except (ClientError, BotoCoreError, OSError):
pass # If compensation cannot reach S3, report it as an orphan limitation.
+37
View File
@@ -0,0 +1,37 @@
CREATE TABLE users (
id uuid PRIMARY KEY,
username text NOT NULL UNIQUE CHECK (username ~ '^[a-z0-9_]{3,32}$'),
display_name text NOT NULL CHECK (char_length(display_name) BETWEEN 1 AND 100),
password_hash text NOT NULL,
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE TABLE chats (
id uuid PRIMARY KEY,
title text NOT NULL CHECK (char_length(title) BETWEEN 1 AND 100),
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE INDEX chats_order_idx ON chats(created_at, id);
CREATE TABLE chat_members (
chat_id uuid NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
user_id uuid NOT NULL REFERENCES users(id),
position integer NOT NULL CHECK (position >= 0),
PRIMARY KEY(chat_id, user_id),
UNIQUE(chat_id, position)
);
CREATE INDEX chat_members_user_idx ON chat_members(user_id, chat_id);
CREATE TABLE messages (
id uuid PRIMARY KEY,
chat_id uuid NOT NULL REFERENCES chats(id),
sender_id uuid NOT NULL REFERENCES users(id),
text text NOT NULL CHECK (char_length(text) BETWEEN 1 AND 4000),
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
FOREIGN KEY(chat_id, sender_id) REFERENCES chat_members(chat_id, user_id)
);
CREATE INDEX messages_history_idx ON messages(chat_id, created_at, id);
CREATE TABLE sessions (
token_hash text PRIMARY KEY CHECK (char_length(token_hash) = 64),
user_id uuid NOT NULL REFERENCES users(id),
expires_at timestamptz NOT NULL
);
CREATE INDEX sessions_expiry_idx ON sessions(expires_at);
CREATE TABLE app_settings (name text PRIMARY KEY, value text NOT NULL);
+10
View File
@@ -0,0 +1,10 @@
CREATE TABLE message_requests (
user_id uuid NOT NULL REFERENCES users(id),
chat_id uuid NOT NULL REFERENCES chats(id),
idempotency_key text NOT NULL CHECK (idempotency_key ~ '^[A-Za-z0-9._:-]{1,128}$'),
request_text text NOT NULL,
message_id uuid NOT NULL REFERENCES messages(id),
expires_at timestamptz NOT NULL,
PRIMARY KEY (user_id, chat_id, idempotency_key)
);
CREATE INDEX message_requests_expiry_idx ON message_requests(expires_at);
+27
View File
@@ -0,0 +1,27 @@
-- Lab4: only metadata and references live in PostgreSQL; bytes live in private S3.
CREATE TABLE attachments (
id uuid PRIMARY KEY,
chat_id uuid NOT NULL REFERENCES chats(id),
owner_id uuid NOT NULL REFERENCES users(id),
filename text NOT NULL CHECK(char_length(filename) BETWEEN 1 AND 255),
content_type text NOT NULL CHECK(content_type IN ('image/png','image/jpeg')),
size_bytes bigint NOT NULL CHECK(size_bytes BETWEEN 1 AND 10485760),
sha256 text NOT NULL CHECK(sha256 ~ '^[a-f0-9]{64}$'),
object_key text NOT NULL UNIQUE,
status text NOT NULL DEFAULT 'ready' CHECK(status='ready'),
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
updated_at timestamptz NOT NULL DEFAULT clock_timestamp(),
FOREIGN KEY(chat_id,owner_id) REFERENCES chat_members(chat_id,user_id)
);
CREATE INDEX attachments_chat_idx ON attachments(chat_id,id);
ALTER TABLE messages ALTER COLUMN text DROP NOT NULL;
ALTER TABLE messages DROP CONSTRAINT messages_text_check;
ALTER TABLE messages ADD CONSTRAINT messages_text_check CHECK(char_length(text) BETWEEN 0 AND 4000);
CREATE TABLE message_attachments (
message_id uuid NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
attachment_id uuid NOT NULL REFERENCES attachments(id),
position integer NOT NULL CHECK(position BETWEEN 0 AND 9),
PRIMARY KEY(message_id, attachment_id),
UNIQUE(message_id, position)
);
ALTER TABLE message_requests ADD COLUMN request_attachments jsonb NOT NULL DEFAULT '[]'::jsonb;
+15
View File
@@ -0,0 +1,15 @@
Flask==3.1.0
waitress==3.0.2
Werkzeug==3.1.3
Jinja2==3.1.6
MarkupSafe==3.0.3
itsdangerous==2.2.0
click==8.1.8
blinker==1.9.0
psycopg[binary]==3.2.6
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
cffi==1.17.1
pycparser==2.22
boto3==1.36.26
Pillow==11.1.0
+44
View File
@@ -0,0 +1,44 @@
"""Concurrent HTTP contract check through LB, using one session on two replicas."""
import argparse
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
import sys
from uuid import uuid4
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'tests'))
from client import Client, check
def main():
parser = argparse.ArgumentParser()
parser.add_argument('--base-url', default='http://localhost:8080')
args = parser.parse_args()
c = Client(args.base_url)
alice, bob, eve, chat = c.bootstrap()
key = uuid4().hex
path = '/api/v1/chats/' + chat['id'] + '/messages'
def send(_):
return Client(args.base_url).message(alice, chat, 'Одно сообщение 👋', key=key)
with ThreadPoolExecutor(max_workers=8) as pool:
responses = list(pool.map(send, range(24)))
first = responses[0]['json']
check(all(r['json'] == first and r['status'] == 201 for r in responses), 'Concurrent replay differs')
instances = {r['headers']['X-Instance-Id'] for r in responses}
check(len(instances) >= 2, 'Concurrent requests did not reach both replicas')
page = c.call('GET', path, actor=bob, schema='MessagePage')['json']
check(page['items'] == [first], 'Concurrent repeat created duplicate messages')
c.call('POST', path, 409, actor=alice, body={'text': 'Different'}, headers={'Idempotency-Key': key})
c.call('POST', path, 400, actor=alice, body={'text': 'Missing key'})
c.call('POST', path, 400, actor=alice, body={'text': 'Bad key'}, headers={'Idempotency-Key': 'bad key'})
c.call('POST', path, 404, actor=eve, body={'text': 'Одно сообщение 👋'}, headers={'Idempotency-Key': key})
other = c.message(bob, chat, 'Same key, different user', key=key)['json']
check(other['id'] != first['id'], 'User scope is not independent')
second_chat = c.call('POST', '/api/v1/chats', 201, actor=alice,
body={'title': 'Second', 'member_ids': [alice['id']]}, schema='Chat')['json']
check(c.message(alice, second_chat, 'Other chat', key=key)['json']['id'] != first['id'], 'Chat scope is not independent')
c.call('DELETE', '/api/v1/auth/sessions/current', 204, actor=alice)
c.call('POST', path, 401, actor=alice, body={'text': 'Одно сообщение 👋'}, headers={'Idempotency-Key': key})
print('PASS: 24 concurrent requests / 8 clients / %d replicas -> one message; conflict, scopes, access and logout verified.' % len(instances))
if __name__ == '__main__':
main()
+201
View File
@@ -0,0 +1,201 @@
"""Real local PostgreSQL + 2 API processes + HAProxy. Optional, not a Docker substitute.
python scripts/check_local.py --pg-bin /path/to/bin --haproxy /path/to/haproxy
"""
import argparse
import json
import os
from pathlib import Path
import shlex
import signal
import subprocess
import sys
import tempfile
import time
from urllib.request import urlopen
ROOT = Path(__file__).resolve().parents[1]
def pg(state, *args):
subprocess.run([str(Path(state['bin']) / 'pg_ctl'), '-D', state['data'], *args],
check=True, stdout=subprocess.DEVNULL)
def wait_url(port):
for _ in range(150):
try:
with urlopen('http://127.0.0.1:%s/health/ready' % port, timeout=1) as response:
if response.status == 200:
return
except OSError:
time.sleep(.1)
raise RuntimeError('Service did not become ready on port %s' % port)
def launch(state, name):
temp = Path(state['data']).parent
if name == 'lb':
command = [state['haproxy'], '-db', '-f', str(temp / 'haproxy.cfg')]
env = dict(os.environ)
else:
command = [sys.executable, '-m', 'messenger']
env = dict(os.environ, APP_HOST='127.0.0.1', APP_PORT=str(state['ports'][name]))
with (temp / (name + '.log')).open('a') as log:
process = subprocess.Popen(command, env=env, cwd=ROOT, stdout=log, stderr=log, start_new_session=True)
state['pids'][name] = process.pid
return process
def stop(pid):
try:
os.kill(pid, signal.SIGTERM)
except ProcessLookupError:
return
for _ in range(70):
try:
# Zombie processes no longer own sockets and await their parent.
status = Path('/proc/%s/stat' % pid)
if not status.exists() or status.read_text().split()[2] == 'Z':
return
except FileNotFoundError:
return
time.sleep(.1)
os.kill(pid, signal.SIGKILL)
def hook(path, action):
state = json.loads(path.read_text())
if action == 'stop':
target = os.environ['TARGET_INSTANCE_ID']
for name, port in state['ports'].items():
try:
with urlopen('http://127.0.0.1:%s/health/live' % port, timeout=1) as r:
matches = r.headers['X-Instance-Id'] == target
except OSError:
continue
if matches:
pid = state['pids'][name]
start = time.monotonic()
stop(pid)
state['stopped'] = name
path.write_text(json.dumps(state))
with (ROOT / 'evidence/local-stops.txt').open('a') as f:
f.write('%s PID=%s instance=%s stopped in %.3fs; remains stopped\n' % (name, pid, target, time.monotonic()-start))
return
raise RuntimeError('Target instance not found')
import psycopg
with psycopg.connect(os.environ['DATABASE_URL']) as conn:
before = conn.execute('SELECT pg_postmaster_start_time()').fetchone()[0]
old = dict(state['pids'])
for pid in old.values():
stop(pid)
pg(state, '-m', 'fast', '-w', 'stop')
pg(state, '-l', str(path.parent / 'postgres.log'), '-w', 'start')
for name in ('api1', 'api2', 'lb'):
launch(state, name)
path.write_text(json.dumps(state))
wait_url(18080)
with psycopg.connect(os.environ['DATABASE_URL']) as conn:
after = conn.execute('SELECT pg_postmaster_start_time()').fetchone()[0]
assert before != after and all(old[k] != state['pids'][k] for k in old)
(ROOT / 'evidence/local-restart.txt').write_text('Actual process PIDs before: %s\nAfter: %s\nPostgreSQL start: %s -> %s\nSame data directory, no reseed or migrations in hook.\n' %
(old, state['pids'], before.isoformat(), after.isoformat()))
def main():
parser = argparse.ArgumentParser()
parser.add_argument('--pg-bin')
parser.add_argument('--haproxy')
parser.add_argument('--hook', choices=['stop', 'restart'])
parser.add_argument('--state', type=Path)
args = parser.parse_args()
if args.hook:
hook(args.state, args.hook)
return
if not args.pg_bin or not args.haproxy:
parser.error('--pg-bin and --haproxy required')
binary = Path(args.pg_bin).resolve()
with tempfile.TemporaryDirectory(prefix='messenger-lab3-') as tmp:
temp = Path(tmp)
path = temp / 'state.json'
state = {'bin': str(binary), 'haproxy': str(Path(args.haproxy).resolve()),
'data': str(temp / 'data'), 'ports': {'api1': 18081, 'api2': 18082}, 'pids': {}}
os.environ.update(DATABASE_URL='postgresql://127.0.0.1:15432/postgres',
SESSION_STORE='postgres', SESSION_TTL_SECONDS='3600', IDEMPOTENCY_TTL_SECONDS='86400')
os.environ.pop('DATABASE_URL_FILE', None)
subprocess.run([str(binary / 'initdb'), '-D', state['data'], '-A', 'trust', '--no-locale', '-E', 'UTF8'],
check=True, stdout=subprocess.DEVNULL)
with (Path(state['data']) / 'postgresql.conf').open('a') as f:
f.write("\nport=15432\nlisten_addresses='127.0.0.1'\nunix_socket_directories=''\ntimezone='UTC'\n")
config = (ROOT / 'infra/haproxy.cfg').read_text().replace('bind :8080', 'bind 127.0.0.1:18080')
for name, port in state['ports'].items():
config = config.replace(name + ':8080', '127.0.0.1:' + str(port))
# Local fixed loopback addresses have no Docker DNS; all LB behavior is unchanged.
config = config.replace(' resolvers docker resolve-prefer ipv4 init-addr libc,none', '')
(temp / 'haproxy.cfg').write_text(config)
processes = []
def run(command):
print('$ ' + ' '.join(command), flush=True)
subprocess.run(command, cwd=ROOT, check=True)
try:
pg(state, '-l', str(temp / 'postgres.log'), '-w', 'start')
run([sys.executable, '-m', 'messenger.migrate'])
run([sys.executable, '-m', 'messenger.migrate'])
run([state['haproxy'], '-c', '-f', str(temp / 'haproxy.cfg')])
for name in ('api1', 'api2', 'lb'):
processes.append(launch(state, name))
path.write_text(json.dumps(state))
wait_url(18081); wait_url(18082); wait_url(18080)
print('Distinct running processes:', state['pids'], flush=True)
(ROOT / 'evidence/local-stops.txt').write_text('Actual local API stops (not Docker):\n')
run(['make', 'check', 'validate', 'student-test', 'PYTHON=' + sys.executable])
run(['make', 'test', 'test-idempotency', 'BASE_URL=http://127.0.0.1:18080', 'PYTHON=' + sys.executable])
for action in ('restart', 'stop'):
script = temp / (action + '.sh')
script.write_text('#!/bin/sh\nexec ' + shlex.join([sys.executable, str(Path(__file__).resolve()),
'--hook', action, '--state', str(path)]) + '\n')
script.chmod(0o700)
sys.path.insert(0, str(ROOT / 'tests'))
from client import Client
from uuid import uuid4
client = Client('http://127.0.0.1:18080')
actor, _, _, durable_chat = client.bootstrap()
durable_key = uuid4().hex
original = client.message(actor, durable_chat, 'Replay across restart', key=durable_key)['json']
run(['make', 'test-persistence' , 'ACTION_SCRIPT=' + str(temp / 'restart.sh'),
'BASE_URL=http://127.0.0.1:18080', 'PYTHON=' + sys.executable])
assert client.message(actor, durable_chat, 'Replay across restart', key=durable_key)['json'] == original
print('PASS: same key + old session after full restart returns original Message', flush=True)
run(['make', 'test-idempotency', 'BASE_URL=http://127.0.0.1:18080', 'PYTHON=' + sys.executable])
for attempt in range(2):
if attempt:
# Shift the round-robin position so the second public scenario
# selects the other process, without changing the public test.
with urlopen('http://127.0.0.1:18080/health/live'):
pass
run(['make', 'test-failover', 'ACTION_SCRIPT=' + str(temp / 'stop.sh'),
'BASE_URL=http://127.0.0.1:18080', 'PYTHON=' + sys.executable])
assert client.message(actor, durable_chat, 'Replay across restart', key=durable_key)['json'] == original
print('PASS: original idempotent result survives selected replica stop', flush=True)
state = json.loads(path.read_text())
processes.append(launch(state, state.pop('stopped')))
path.write_text(json.dumps(state))
wait_url(18081); wait_url(18082)
time.sleep(1.2)
print((ROOT / 'evidence/local-restart.txt').read_text(), flush=True)
print((ROOT / 'evidence/local-stops.txt').read_text(), flush=True)
run([state['haproxy'], '-v'])
run([str(binary / 'postgres'), '--version'])
finally:
if path.exists():
state = json.loads(path.read_text())
for pid in state['pids'].values():
stop(pid)
for process in processes:
process.wait(timeout=10)
subprocess.run([str(binary / 'pg_ctl'), '-D', state['data'], '-m', 'fast', '-w', 'stop'],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
if __name__ == '__main__':
main()
+29
View File
@@ -0,0 +1,29 @@
"""Create local secrets once. Never overwrite existing credentials."""
from pathlib import Path
import secrets
import shutil
root = Path(__file__).resolve().parents[1]
directory = root / 'secrets'
directory.mkdir(mode=0o700, exist_ok=True)
password_file = directory / 'postgres_password'
url_file = directory / 'database_url'
if password_file.exists() != url_file.exists():
raise SystemExit('Incomplete secrets: restore the missing file; do not regenerate credentials for an existing volume.')
if not password_file.exists():
password = secrets.token_hex(32)
for path, value in [(password_file, password),
(url_file, 'postgresql://messenger:' + password + '@db:5432/messenger')]:
with path.open('x', encoding='utf-8') as f:
f.write(value + '\n')
# Compose mounts secret files individually; PostgreSQL runs as a distinct UID.
# The enclosing host directory is 0700, while this mounted file must be readable.
path.chmod(0o644 if path == password_file else 0o600)
s3_file = directory / 's3_password'
if not s3_file.exists():
with s3_file.open('x', encoding='utf-8') as f:
f.write(secrets.token_hex(32) + '\n')
s3_file.chmod(0o644)
if not (root / '.env').exists():
shutil.copyfile(root / '.env.example', root / '.env')
print('Local configuration ready; existing credentials preserved.')
+26
View File
@@ -0,0 +1,26 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")/.."
mkdir -p evidence
snapshot() {
for service in db s3 api1 api2 lb; do
id=$(docker compose ps -aq "$service")
docker inspect --format '{{.Name}} {{.State.StartedAt}} {{.State.Status}}' "$id"
done
}
snapshot > evidence/restart-before.txt
docker compose stop -t 7 lb api1 api2
docker compose restart -t 10 db s3
i=0
until docker compose exec -T db pg_isready -U messenger -d messenger >/dev/null 2>&1; do
i=$((i + 1)); [ "$i" -lt 30 ] || exit 1
sleep 1
done
i=0
until docker compose exec -T s3 sh -c 'curl -fsS http://127.0.0.1:9000/minio/health/live >/dev/null' 2>/dev/null; do
i=$((i+1)); [ "$i" -lt 30 ] || exit 1
sleep 1
done
# Existing schema is retained; the restart hook does not migrate/reseed.
docker compose start api1 api2 lb
snapshot > evidence/restart-after.txt
+18
View File
@@ -0,0 +1,18 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")/.."
: "${TARGET_INSTANCE_ID:?Set TARGET_INSTANCE_ID to the observed API instance}"
mkdir -p evidence
for service in api1 api2; do
instance=$(docker compose exec -T "$service" python -c 'import urllib.request; r=urllib.request.urlopen("http://127.0.0.1:8080/health/live",timeout=2); print(r.headers["X-Instance-Id"])' 2>/dev/null) || continue
if [ "$instance" = "$TARGET_INSTANCE_ID" ]; then
id=$(docker compose ps -q "$service")
docker inspect --format '{{.Name}} {{.State.StartedAt}} {{.State.Status}}' "$id" > "evidence/stop-$service.txt"
# SIGTERM, then bounded fallback. No automatic restart policy is enabled.
docker compose stop -t 7 "$service"
docker inspect --format '{{.Name}} {{.State.FinishedAt}} {{.State.Status}}' "$id" >> "evidence/stop-$service.txt"
exit 0
fi
done
printf '%s\n' 'No running replica matches TARGET_INSTANCE_ID' >&2
exit 1
+141
View File
@@ -0,0 +1,141 @@
"""Integration tests. DATABASE_URL[_FILE] must point to a disposable PostgreSQL DB."""
import base64
import time
import unittest
from dataclasses import replace
from uuid import uuid4
from messenger.api import create_app
from messenger.config import Settings
from messenger.database import Database
from messenger.migrate import migrate
class ApiTests(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.settings = Settings.from_env()
cls.db = Database(cls.settings.database_url)
migrate(cls.db)
cls.app = create_app(settings=cls.settings)
def setUp(self):
self.c = self.app.test_client()
self.password = 'Учебный:пароль-12345'
self.name = 'u_' + uuid4().hex[:16]
self.registration = {'username': self.name, 'display_name': 'Имя 👋', 'password': self.password}
r = self.c.post('/api/v1/users', json=self.registration)
self.assertEqual(r.status_code, 201)
self.user = r.json
self.basic = {'Authorization': 'Basic ' + base64.b64encode((self.name + ':' + self.password).encode()).decode()}
r = self.c.post('/api/v1/auth/sessions', headers=self.basic)
self.assertEqual(r.status_code, 201)
self.session = r.json
self.headers = {'Authorization': 'Bearer ' + self.session['token'], 'Idempotency-Key': uuid4().hex}
self.chat = self.c.post('/api/v1/chats', headers=self.headers, json={
'title': 'Учебный чат', 'member_ids': [self.user['id']]}).json
self.path = '/api/v1/chats/' + self.chat['id'] + '/messages'
def error(self, response, status):
self.assertEqual(response.status_code, status)
self.assertEqual(response.json['error']['request_id'], response.headers['X-Request-Id'])
def test_authentication_and_logout(self):
for headers in [{}, self.basic, {'X-User-Id': self.user['id']}, {'Authorization': 'Bearer bad'}]:
r = self.c.get('/api/v1/users/me', headers=headers)
self.error(r, 401)
self.assertEqual(r.headers['WWW-Authenticate'], 'Bearer realm="messenger"')
for auth in ['Basic bad', 'Bearer bad', 'Basic ' + base64.b64encode((self.name + ':wrong-password').encode()).decode()]:
r = self.c.post('/api/v1/auth/sessions', headers={'Authorization': auth})
self.error(r, 401)
self.assertEqual(r.headers['WWW-Authenticate'], 'Basic realm="messenger", charset="UTF-8"')
r = self.c.get('/api/v1/users/me', headers=dict(self.headers, **{'X-User-Id': str(uuid4())}))
self.assertEqual(r.json, self.user)
self.error(self.c.post('/api/v1/auth/sessions', headers=self.basic, json={}), 400)
r = self.c.delete('/api/v1/auth/sessions/current', headers=self.headers)
self.assertEqual((r.status_code, r.data), (204, b''))
self.error(self.c.get('/api/v1/users/me', headers=self.headers), 401)
self.error(self.c.delete('/api/v1/auth/sessions/current', headers=self.headers), 401)
def test_expiry_and_restart_app(self):
short = create_app(settings=replace(self.settings, session_ttl=2)).test_client()
r = short.post('/api/v1/auth/sessions', headers=self.basic)
self.assertEqual(r.status_code, 201)
headers = {'Authorization': 'Bearer ' + r.json['token']}
self.assertEqual(short.get('/api/v1/users/me', headers=headers).status_code, 200)
# New app instance must respect the stored original expiry, not renew it.
restarted = create_app(settings=replace(self.settings, session_ttl=3600)).test_client()
self.assertEqual(restarted.get('/api/v1/users/me', headers=headers).status_code, 200)
time.sleep(2.1)
self.error(restarted.get('/api/v1/users/me', headers=headers), 401)
def test_strict_validation_and_private_chat(self):
self.error(self.c.post('/api/v1/users', data='{', content_type='application/json'), 400)
self.error(self.c.post('/api/v1/users', json=self.registration), 409)
for patch in [{'password': 'short'}, {'password': 42}, {'display_name': ' '}, {'extra': 1}]:
self.error(self.c.post('/api/v1/users', json=dict(self.registration, **patch)), 400)
self.error(self.c.post(self.path, headers=self.headers, json={'text': ' \n'}), 400)
self.error(self.c.post(self.path, headers=self.headers, json={'text': 'x', 'sender_id': self.user['id']}), 400)
self.error(self.c.get('/api/v1/chats/' + str(uuid4()), headers=self.headers), 404)
outsider = 'u_' + uuid4().hex[:16]
self.c.post('/api/v1/users', json=dict(self.registration, username=outsider))
basic = {'Authorization': 'Basic ' + base64.b64encode((outsider + ':' + self.password).encode()).decode()}
token = self.c.post('/api/v1/auth/sessions', headers=basic).json['token']
self.error(self.c.get(self.path, headers={'Authorization': 'Bearer ' + token}), 404)
def test_pagination_survives_new_app(self):
expected = [self.c.post(self.path, headers=dict(self.headers, **{'Idempotency-Key': uuid4().hex}), json={'text': str(i)}).json for i in range(5)]
first = self.c.get(self.path + '?limit=2', headers=self.headers).json
cursor = first['next_cursor']
new_client = create_app(settings=self.settings).test_client()
second = new_client.get(self.path, query_string={'limit': 1, 'cursor': cursor}, headers=self.headers).json
third = new_client.get(self.path, query_string={'limit': 100, 'cursor': second['next_cursor']}, headers=self.headers).json
self.assertEqual(first['items'] + second['items'] + third['items'], expected)
self.assertIsNone(third['next_cursor'])
self.error(self.c.get('/api/v1/chats', query_string={'cursor': cursor}, headers=self.headers), 400)
other = self.c.post('/api/v1/chats', headers=self.headers, json={'title': 'Other', 'member_ids': [self.user['id']]}).json
self.error(self.c.get('/api/v1/chats/' + other['id'] + '/messages', query_string={'cursor': cursor}, headers=self.headers), 400)
for limit in ['0', '101', '1.5', 'true']:
self.error(self.c.get(self.path, query_string={'limit': limit}, headers=self.headers), 400)
self.error(self.c.get(self.path, query_string={'cursor': 'invalid'}, headers=self.headers), 400)
page = self.c.get('/api/v1/chats?limit=1', headers=self.headers).json
last = self.c.get('/api/v1/chats', query_string={'cursor': page['next_cursor']}, headers=self.headers).json
self.assertEqual(len(page['items'] + last['items']), 2)
def test_atomic_chat_and_migration_repeat(self):
with self.db.transaction() as conn:
before = conn.execute('SELECT count(*) AS n FROM chats').fetchone()['n']
for members in [[self.user['id'], str(uuid4())], [self.user['id'], self.user['id']], []]:
self.error(self.c.post('/api/v1/chats', headers=self.headers,
json={'title': 'Invalid', 'member_ids': members}), 400)
migrate(self.db)
migrate(self.db)
with self.db.transaction() as conn:
self.assertEqual(conn.execute('SELECT count(*) AS n FROM chats').fetchone()['n'], before)
self.assertEqual(conn.execute('SELECT count(*) AS n FROM schema_migrations').fetchone()['n'], 3)
row = conn.execute('SELECT password_hash FROM users WHERE id=%s', (self.user['id'],)).fetchone()
self.assertTrue(row['password_hash'].startswith('$argon2id$'))
hashes = [r['token_hash'] for r in conn.execute('SELECT token_hash FROM sessions WHERE user_id=%s', (self.user['id'],))]
self.assertNotIn(self.session['token'], hashes)
self.assertEqual(self.c.get('/api/v1/users/me', headers=self.headers).json, self.user)
def test_idempotency_expiry_and_errors(self):
first = self.c.post(self.path, headers=self.headers, json={'text': 'Persist me'})
self.assertEqual(first.status_code, 201)
replay = self.c.post(self.path, headers=self.headers, json={'text': 'Persist me'})
self.assertEqual(replay.json, first.json)
self.error(self.c.post(self.path, headers=self.headers, json={'text': 'Different'}), 409)
for key in ['', 'bad key', 'x' * 129]:
self.error(self.c.post(self.path, headers=dict(self.headers, **{'Idempotency-Key': key}), json={'text': 'Text'}), 400)
# Expire only this test's synthetic record, without waiting 24 hours.
with self.db.transaction() as conn:
row = conn.execute("SELECT expires_at>clock_timestamp()+interval '23 hours' AS valid FROM message_requests WHERE user_id=%s", (self.user['id'],)).fetchone()
self.assertTrue(row['valid'])
conn.execute("UPDATE message_requests SET expires_at=clock_timestamp()-interval '1 second' WHERE user_id=%s", (self.user['id'],))
new = self.c.post(self.path, headers=self.headers, json={'text': 'New after expiry'})
self.assertEqual(new.status_code, 201)
self.assertNotEqual(new.json['id'], first.json['id'])
self.assertEqual(len(self.c.get(self.path, headers=self.headers).json['items']), 2)
if __name__ == '__main__':
unittest.main()
+39
View File
@@ -0,0 +1,39 @@
"""Offline negative checks of image admission; execute: python -m unittest discover -s student-tests -p test_images.py."""
import io
import unittest
from pathlib import Path
from PIL import Image
from messenger.images import validate_image, MAX_BYTES
from messenger.domain import ApiError
ROOT = Path(__file__).resolve().parents[1]
class ImageAdmissionTests(unittest.TestCase):
def test_fixture_and_byte_preservation(self):
blob = (ROOT / 'tests/fixtures/sample.png').read_bytes()
self.assertIsNone(validate_image(blob, 'image/png'))
def test_mismatch_or_invalid(self):
data = (ROOT / 'tests/fixtures/sample.png').read_bytes()
for blob, mime in [(data, 'image/jpeg'), (b'not-an-image', 'image/png'),
(b'\xff\xd8broken', 'image/jpeg'), (data, 'application/octet-stream')]:
with self.subTest(mime=mime):
with self.assertRaises(ApiError) as error:
validate_image(blob, mime)
self.assertEqual(error.exception.status, 415)
def test_byte_limit(self):
with self.assertRaises(ApiError) as error:
validate_image(b'0' * (MAX_BYTES + 1), 'image/png')
self.assertEqual(error.exception.status, 413)
def test_pixel_limit_before_decompression(self):
img = Image.new('RGB', (8000, 5001))
stream = io.BytesIO()
img.save(stream, format='PNG')
with self.assertRaises(ApiError) as error:
validate_image(stream.getvalue(), 'image/png')
self.assertEqual(error.exception.status, 413)
if __name__ == '__main__':
unittest.main()