ЛР 3 — БВТ2301 — Александров— на 4 #81
@@ -0,0 +1,18 @@
|
||||
# Shell scripts and configs must stay LF so they run in Linux containers.
|
||||
*.sh text eol=lf
|
||||
*.conf text eol=lf
|
||||
nginx.conf text eol=lf
|
||||
Dockerfile text eol=lf
|
||||
dockerignore text eol=lf
|
||||
compose.yaml text eol=lf
|
||||
compose.example.yaml text eol=lf
|
||||
entrypoint.sh text eol=lf
|
||||
|
||||
# Python and text files are fine with platform-native line endings.
|
||||
*.py text
|
||||
*.md text
|
||||
*.json text
|
||||
*.txt text
|
||||
*.sql text
|
||||
.env.example text
|
||||
Makefile text
|
||||
@@ -1,18 +1,51 @@
|
||||
## Лабораторная 3
|
||||
|
||||
Автор / группа: TODO
|
||||
Целевая оценка (3/4/5): TODO
|
||||
Трек 5, если заявлен: TODO
|
||||
Автор / группа: Тимофей Александров, БВТ2301
|
||||
Целевая оценка (3/4/5): 5
|
||||
Трек 5: трек 1 «Цена масштабирования»
|
||||
|
||||
Что реализовано и что перенесено: TODO
|
||||
Команды запуска из чистого clone: TODO
|
||||
Результаты проверок и ссылка на REPORT.md: TODO
|
||||
Ограничения / известные ошибки: TODO
|
||||
Источники и использование ИИ: TODO
|
||||
Что реализовано и что перенесено:
|
||||
- Перенесено из lab2: PostgreSQL источник истины, Valkey кэш сессий, Argon2id пароли, Bearer-сессии, транзакционное создание чата, keyset-пагинация, circuit breaker на Valkey.
|
||||
- Новое в lab3: 2 API-реплики (api-1, api-2) с уникальным INSTANCE_ID; Nginx load balancer с health-check и автоматическим failover; Idempotency-Key в заголовке (replay возвращает исходное сообщение); X-Instance-Id в каждом ответе; /version и /info эндпоинты; housekeeping-loop для очистки истёкших сессий и ключей; таблица idempotency_keys с миграциями 004–007.
|
||||
- Исправлено по ревью: scope idempotency — `(actor_id, chat_id, key)`; атомарное резервирование ключа не даёт 500 при 32 concurrent replays; Bearer-сессия проверяет PostgreSQL на каждой реплике, поэтому старый cache-hit не может воскресить отозванный токен.
|
||||
- Исправлены импорты config/models/api/store/sessions (убраны несуществующие `config`, добавлены `IDEMPOTENCY_KEY_PATTERN`, `__version__`).
|
||||
- Добавлены скрипты stop-one.sh/cmd (failover) и restart.sh/cmd (persistence).
|
||||
|
||||
- [ ] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
|
||||
- [ ] Контракт и публичные тесты не ослаблены.
|
||||
- [ ] make check и make test выполнены; приложен реальный вывод.
|
||||
- [ ] Сценарии отказов/безопасности своего уровня воспроизведены.
|
||||
- [ ] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
|
||||
- [ ] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
|
||||
Команды запуска из чистого clone:
|
||||
```sh
|
||||
sh scripts/bootstrap-secrets.sh
|
||||
docker compose --env-file .env config --quiet
|
||||
docker compose --env-file .env up -d --build --wait --wait-timeout 180
|
||||
python3 scripts/check_contract.py
|
||||
python3 scripts/validate_openapi.py
|
||||
python3 tests/smoke.py --base-url http://127.0.0.1:8081
|
||||
python3 tests/resilience.py --mode persistence --action-script scripts/restart.sh --base-url http://127.0.0.1:8081
|
||||
python3 tests/resilience.py --mode failover --action-script scripts/stop-one.sh --base-url http://127.0.0.1:8081
|
||||
cd student-tests && BASE_URL=http://127.0.0.1:8081 python3 -u -B -m unittest -v test_api test_race test_idempotency test_degradation
|
||||
cd .. && python3 scripts/research-scale.py --base-url http://127.0.0.1:8081 --duration 10 --concurrency 4
|
||||
```
|
||||
|
||||
Результаты проверок и ссылка на REPORT.md:
|
||||
- Smoke: PASS: lab3 public API scenario
|
||||
- check_contract: PASS: offline skeleton structure, local refs and Python syntax
|
||||
- validate_openapi: PASS: full OpenAPI 3.1 and JSON Schema validation
|
||||
- Student tests: 28/28 OK; idempotency 2/2 OK (32 concurrent replays); degradation 2/2 OK (durable cross-replica logout)
|
||||
- Failover: PASS — 79 успешных записей из 83, восстановление за 2.6 с
|
||||
- Persistence: PASS — данные и сессия сохранены после рестарта
|
||||
- Degradation: OK — cache_down_p50=47 мс, db_down_write=2.05 с, recovery=1.97 с
|
||||
- Трек 5: fixed total RPS 1/2/4 = 36.9 / 49.4 / 36.2; growing resources = 42.4 / 45.2 / 57.0. Raw JSON: `evidence/scale-*.json`.
|
||||
- Подробности: REPORT.md
|
||||
|
||||
Ограничения / известные ошибки:
|
||||
- На Windows .sh-скрипты не запускаются напрямую (WinError 193); созданы эквиваленты .cmd.
|
||||
- PostgreSQL и Nginx остаются одиночными узлами; трек не заявляет выживание их отказа.
|
||||
- Нагрузочный тест закрытого цикла содержит coordinated omission.
|
||||
|
||||
Источники и использование ИИ: README/COURSE/CONTRIBUTING этого skeleton, contracts/README.md, tests/README.md; документация PostgreSQL, Valkey, Nginx; RFC 7617; OWASP Password Storage Cheat Sheet. ИИ (Codex) использовался для изучения задания, проектирования, написания исходников и тестов, проверки и подготовки отчёта. Автор проверяет результат и должен уметь объяснить и изменить его на защите.
|
||||
|
||||
- [x] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
|
||||
- [x] Контракт и публичные тесты не ослаблены.
|
||||
- [x] make check и make test выполнены; приложен реальный вывод.
|
||||
- [x] Сценарии отказов/безопасности своего уровня воспроизведены.
|
||||
- [x] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
|
||||
- [x] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
|
||||
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
# syntax=docker/dockerfile:1
|
||||
# Runtime image is pinned by digest, dependencies by version in requirements.txt.
|
||||
FROM python:3.12.7-slim-bookworm@sha256:60d9996b6a8a3689d36db740b49f4327be3be09a21122bd02fb8895abb38b50d
|
||||
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
APP_HOST=0.0.0.0 \
|
||||
APP_PORT=8080
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Unprivileged runtime user; the API never needs to write to the image.
|
||||
RUN groupadd --system --gid 10001 messenger \
|
||||
&& useradd --system --uid 10001 --gid messenger --no-create-home messenger
|
||||
|
||||
COPY requirements.txt /app/requirements.txt
|
||||
RUN pip install --no-cache-dir --requirement /app/requirements.txt
|
||||
|
||||
COPY --chown=messenger:messenger src/ /app/src/
|
||||
COPY --chown=messenger:messenger contracts/ /app/contracts/
|
||||
COPY --chown=messenger:messenger --chmod=0555 entrypoint.sh /app/entrypoint.sh
|
||||
# Windows hosts may store CRLF; Python and /bin/sh need LF.
|
||||
RUN find /app/src -name '*.py' -print0 | xargs -0 sed -i 's/\r$//' \
|
||||
&& sed -i 's/\r$//' /app/entrypoint.sh
|
||||
|
||||
USER messenger
|
||||
EXPOSE 8080
|
||||
STOPSIGNAL SIGTERM
|
||||
|
||||
# Readiness (not just liveness) so an unhealthy container is visible to Compose.
|
||||
HEALTHCHECK --interval=5s --timeout=5s --start-period=20s --retries=3 \
|
||||
CMD ["python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=2).read()"]
|
||||
|
||||
# Migrations are idempotent; the entrypoint execs the server so it is PID 1.
|
||||
ENTRYPOINT ["/app/entrypoint.sh"]
|
||||
@@ -1,13 +1,20 @@
|
||||
PYTHON ?= python3
|
||||
BASE_URL ?= http://localhost:8080
|
||||
# 127.0.0.1, not localhost: compose publishes the API on IPv4 only, and a
|
||||
# dual-stack localhost would add a ~2 s IPv6 retry to every request.
|
||||
BASE_URL ?= http://127.0.0.1:8080
|
||||
CA_FILE ?=
|
||||
ACTION_SCRIPT ?=
|
||||
TLS_ARGS = $(if $(CA_FILE),--ca-file "$(CA_FILE)",)
|
||||
TARGET_INSTANCE_ID ?= api-1
|
||||
STUDENT_DIR = student-tests
|
||||
|
||||
.PHONY: help check validate test load test-persistence test-failover
|
||||
.PHONY: help check validate test load test-persistence test-failover run bootstrap up down test-student test-degradation explain research-scale
|
||||
|
||||
help:
|
||||
@echo "check: offline structure; validate: full OpenAPI (requirements-dev.txt); test: running API; load: synthetic writes"
|
||||
@echo "bootstrap/up/down: local Compose stand; run: API without Docker (needs DATABASE_URL/CACHE_URL)"
|
||||
@echo "test-student: own functional+race tests; test-persistence: restart stack; test-failover: stop one API replica"
|
||||
@echo "explain: seed 12k messages and capture EXPLAIN (ANALYZE, BUFFERS) into evidence/"
|
||||
@echo "research-scale: grade-5 1/2/4 replica experiment; set BASE_URL, DURATION and CONCURRENCY as needed"
|
||||
|
||||
check:
|
||||
$(PYTHON) scripts/check_contract.py
|
||||
@@ -27,4 +34,30 @@ test-persistence:
|
||||
|
||||
test-failover:
|
||||
@test -n "$(ACTION_SCRIPT)" || (echo "Set ACTION_SCRIPT to your implemented executable stop-one hook"; exit 2)
|
||||
$(PYTHON) tests/resilience.py --mode failover --action-script "$(ACTION_SCRIPT)" --base-url "$(BASE_URL)" $(TLS_ARGS)
|
||||
TARGET_INSTANCE_ID="$(TARGET_INSTANCE_ID)" $(PYTHON) tests/resilience.py --mode failover --action-script "$(ACTION_SCRIPT)" --base-url "$(BASE_URL)" $(TLS_ARGS)
|
||||
|
||||
# --- lab 3 ----------------------------------------------------------------
|
||||
run:
|
||||
$(PYTHON) -B src/app.py
|
||||
|
||||
bootstrap:
|
||||
sh scripts/bootstrap-secrets.sh
|
||||
|
||||
up: bootstrap
|
||||
docker compose --env-file .env up -d --build --wait --wait-timeout 180
|
||||
docker compose --env-file .env ps
|
||||
|
||||
down:
|
||||
docker compose --env-file .env down
|
||||
|
||||
test-student:
|
||||
cd $(STUDENT_DIR) && BASE_URL="$(BASE_URL)" $(PYTHON) -B -m unittest -v test_api test_race test_idempotency
|
||||
|
||||
test-degradation:
|
||||
cd $(STUDENT_DIR) && BASE_URL="$(BASE_URL)" STUDENT_DOCKER_TESTS=1 $(PYTHON) -B -m unittest -v test_degradation
|
||||
|
||||
explain:
|
||||
sh scripts/explain-history.sh
|
||||
|
||||
research-scale:
|
||||
$(PYTHON) scripts/research-scale.py --base-url "$(BASE_URL)"
|
||||
|
||||
@@ -1,37 +1,115 @@
|
||||
# Отчёт — лабораторная 3
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
- Автор, группа: Тимофей Александров, БВТ2301
|
||||
- Целевая оценка; выбранный трек 5, если нужен: **5**, трек 1 «Цена масштабирования»
|
||||
- Commit/PR: подготовлен к публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: README/COURSE/CONTRIBUTING этого skeleton, `contracts/README.md` и `tests/README.md`; документация PostgreSQL (transactions, EXPLAIN), Valkey (persistence), Nginx (upstream, health checks), RFC 7617 (Basic), OWASP Password Storage Cheat Sheet (Argon2id). Библиотеки: FastAPI/Starlette/uvicorn, psycopg 3 + psycopg-pool, redis-py, argon2-cffi, nginx 1.27. Лабораторная 3 построена на кодовой базе лабораторной 2.
|
||||
|
||||
## Запуск из чистого clone
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
Версии, на которых всё воспроизведено: Docker Engine **29.9.2**, Docker Compose **v5.6.1**, на хосте Python **3.11.5** (нужен только для скриптов и тестов), внутри контейнера Python **3.12.7** (`python:3.12.7-slim-bookworm`), PostgreSQL **17.6**, Valkey **8.1.3**, Nginx **1.27-alpine** — образы закреплены (postgres и valkey по digest, nginx без digest). Аппаратура стенда: Intel Pentium Silver N5030 @ 1.10 GHz, 4 логических ядра, ~7.6 GiB RAM, лимитов CPU/RAM для контейнеров не задано.
|
||||
|
||||
Нужны: Git с POSIX-shell (на Windows — Git Bash, скрипты `scripts/*.sh`) и Docker с Compose. `make` в этой среде отсутствует, поэтому ниже приведены те же команды напрямую (сам `Makefile` сохранён и использует те же пути).
|
||||
|
||||
```sh
|
||||
# 1) секреты и .env: генерирует ./secrets (mode 0600, в .gitignore) и копирует .env
|
||||
sh scripts/bootstrap-secrets.sh # на Windows: sh из Git Bash
|
||||
|
||||
# 2) проверка конфигурации и подъём стенда (миграции применяет entrypoint контейнера api)
|
||||
docker compose --env-file .env config --quiet
|
||||
docker compose --env-file .env up -d --build --wait --wait-timeout 180
|
||||
docker compose --env-file .env ps
|
||||
|
||||
# 3) проверки
|
||||
python3 scripts/check_contract.py # = make check
|
||||
python3 scripts/validate_openapi.py # = make validate (нужен requirements-dev.txt)
|
||||
python3 tests/smoke.py --base-url http://127.0.0.1:8081 # = make test (порт 8081)
|
||||
python3 tests/resilience.py --mode persistence \
|
||||
--action-script scripts/restart.sh --base-url http://127.0.0.1:8081 # = make test-persistence
|
||||
python3 tests/resilience.py --mode failover \
|
||||
--action-script scripts/stop-one.sh --base-url http://127.0.0.1:8081 # = make test-failover
|
||||
TARGET_INSTANCE_ID=api-1 sh scripts/stop-one.sh # остановить одну реплику
|
||||
cd student-tests && BASE_URL=http://127.0.0.1:8081 \
|
||||
python3 -u -B -m unittest -v test_api test_race test_degradation
|
||||
sh scripts/explain-history.sh # = make explain (EXPLAIN ANALYZE на 12 000 сообщений)
|
||||
python3 scripts/research-scale.py --base-url http://127.0.0.1:8081 --duration 10 --concurrency 4
|
||||
```
|
||||
|
||||
Безопасная остановка с сохранением данных: `docker compose --env-file .env down` (volumes `pgdata`/`valkeydata` не удаляются, сессии и сообщения переживают рестарт). Намеренное удаление учебных данных: `docker compose --env-file .env down -v`. На Windows эквивалент restart-hook для `tests/resilience.py` — `scripts/restart.cmd` (тот же сценарий, что и `scripts/restart.sh`: `down` без `-v`, затем `up -d --wait`; Python не умеет запускать `.sh` напрямую, ошибка `WinError 193`). Аналогично для failover — `scripts/stop-one.cmd`.
|
||||
|
||||
URL вида `http://127.0.0.1:8081`, а не `localhost`: Compose публикует только IPv4-порт, а на dual-stack хосте `localhost` сначала идёт в `::1` и добавляет ~2 с TCP-retry к каждому запросу.
|
||||
|
||||
## Архитектура и решения
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab2, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
**Процессы/хранилища.** Клиент → `nginx` (load balancer, публичный entrypoint) → `api-1` / `api-2` (FastAPI + uvicorn, два процесса) → `postgres` (источник истины) и `valkey` (кэш сессий). Сеть `backend` внутренняя, наружу публикуется только `127.0.0.1:${API_PUBLISH_PORT:-8081}:8080` через nginx. Логика в `src/`, SQL-миграции в `src/migrations/`.
|
||||
|
||||
**Что перенесено из lab2.** Тот же контракт и та же логика keyset-пагинации, но хранилище заменено с in-memory dict на PostgreSQL; `X-User-Id` больше не используется нигде (входящее поле игнорируется, проверено тестом); вместо Basic-входа на каждый запрос — Basic только на `POST /api/v1/auth/sessions` и Bearer-токен на остальных. Добавлены: `X-Instance-Id` в каждом ответе (идентификация реплики), `/version` и `/info` эндпоинты, housekeeping-loop (очистка истёкших сессий и idempotency-ключей), таблица `idempotency_keys` с миграцией 004 и индексами миграции 005.
|
||||
|
||||
**Балансировка и отказ API-реплики (новое в lab3).** Два API-инстанса (`api-1`, `api-2`) с разными `INSTANCE_ID`. Nginx upstream балансирует round-robin с health-check (`max_fails=3`, `fail_timeout=10s`): при остановке одной реплики трафик автоматически идёт на вторую. `proxy_next_upstream` перенаправляет запрос на следующую реплику при 502/503. Порты API-реплик не публикуются наружу — только через LB. Failover-тест: 79 успешных записей из 83 попыток за 15 с при остановленной реплике, восстановление за 2.6 с.
|
||||
|
||||
**Источник истины и кэш.** Сессии живут в таблице `sessions` (хранится только SHA-256 токена). Valkey хранит публичный снимок пользователя под `session:<sha256>` максимум `CACHE_TTL_SECONDS=300` с и одновременно не дольше остатка жизни сессии. Tombstone `revoked` остаётся полезной быстрой отрицательной записью, но PostgreSQL подтверждает действительность сессии на каждом Bearer-запросе. Поэтому cache-hit не может воскресить отзыв: при остановленном Valkey, рестарте одной/обеих API-реплик и возврате старого кэш-тома токен остаётся 401 на любой реплике. Если Valkey недоступен — запрос идёт в PostgreSQL; кэш-ошибка никогда не означает «авторизован». Плюс 5-секундный circuit breaker.
|
||||
|
||||
**Транзакционные границы.** Создание чата (`chats` + строки `chat_members`) — одна транзакция. Регистрация и отправка сообщения — по одной транзакции. Миграции — версионированный раннер `src/migrate.py`: одна транзакция + `pg_advisory_xact_lock`, таблица `schema_migrations`, повторный запуск печатает `migrations_up_to_date` и не дублирует данные.
|
||||
|
||||
**Idempotency-Key (новое в lab3).** Ключ передаётся в заголовке `Idempotency-Key` (обязательный, паттерн `^[A-Za-z0-9._:-]{1,128}$`). Scope — `(actor_id, chat_id, key)`, то есть пользователь + POST + путь чата + ключ: один ключ независим между пользователями и чатами. Хранится SHA-256 тела и `message_id`; повтор с тем же телом возвращает исходное сообщение (201), другой body в той же области → 409. Миграция 006 добавляет `message_id`, миграция 007 добавляет `actor_id` и составной primary key. Резервирование ключа использует `INSERT ... ON CONFLICT DO NOTHING`, поэтому конкурентный repeat не оставляет транзакцию в aborted state и все запросы получают один и тот же Message.
|
||||
|
||||
**Ограничения и таймауты.** Пул `DB_POOL_MIN=1 … DB_POOL_MAX=10` c `DB_POOL_TIMEOUT_SECONDS=2`, `DB_CONNECT_TIMEOUT_SECONDS=2`, серверные `statement_timeout=5000`, `lock_timeout=2000`, `idle_in_transaction_session_timeout=10000`, `timezone=UTC`. Все запросы параметризованные (psycopg3). Кэш-операции ограничены `CACHE_TIMEOUT_SECONDS=0.5`.
|
||||
|
||||
**Аутентификация.** Пароли — Argon2id через `argon2-cffi` (19 MiB, 2 итерации, 1 lane; для неизвестного пользователя проверяется dummy-hash). Токен — `secrets.token_urlsafe(48)` = 384 бита, в БД только его SHA-256. `SESSION_TTL_SECONDS` (3600 по умолчанию), отзыв через `DELETE /api/v1/auth/sessions/current`.
|
||||
|
||||
**Секреты.** `scripts/bootstrap-secrets.sh` генерирует `./secrets/*` (mode 0600, каталог в `.gitignore`), compose монтирует их как файлы (`DATABASE_URL_FILE`, `POSTGRES_PASSWORD_FILE`, …), значения не попадают в `compose.yaml`, `docker inspect`, логи или отчёт.
|
||||
|
||||
**Соответствие `.env.example` ↔ `src/config.py`.** `APP_HOST/APP_PORT` → `load_address()`; `SESSION_TTL_SECONDS`, `CACHE_TTL_SECONDS`, `DB_POOL_*`, `DB_*_TIMEOUT*`, `CACHE_TIMEOUT_SECONDS`, `IDEMPOTENCY_TTL_SECONDS` → одноимённые поля `Settings`; `DATABASE_URL_FILE`/`CACHE_URL_FILE` → `_secret()`, с падением на старт-апе при нечитаемом файле; `SESSION_STORE` допускает только `postgres`; `INSTANCE_ID` → `settings.instance_id`.
|
||||
|
||||
## Доказательства критериев
|
||||
|
||||
Все выводы сохранены в `evidence/` (без токенов и секретов); файлы сгенерированы 2026-10-30.
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
| Публичный smoke | `python3 tests/smoke.py --base-url http://127.0.0.1:8081` | `PASS: lab3 public API scenario` | `evidence/smoke.txt` |
|
||||
| make check: структура/контракт | `python3 scripts/check_contract.py` | `PASS: offline skeleton structure, local refs and Python syntax` | строка в PR |
|
||||
| Полная валидация OpenAPI | `python3 scripts/validate_openapi.py` | `PASS: full OpenAPI 3.1 and JSON Schema validation` | строка в PR |
|
||||
| Уровень 3: рестарт сохраняет данные и сессию | `python3 tests/resilience.py --mode persistence --action-script scripts/restart.sh …` | `PASS: persistence preserved confirmed data and existing session` | `evidence/persistence.txt` |
|
||||
| Уровень 3: failover — остановка одной реплики | `python3 tests/resilience.py --mode failover --action-script scripts/stop-one.sh …` | `PASS: failover preserved confirmed data and existing session`; 79 успешных записей из 83, восстановление за 2.6 с | `evidence/failover.txt` |
|
||||
| Миграции, транзакции, гонка username, откат создания чата | `python3 -m unittest -v test_api test_race` | **26/26 OK**, включая `duplicate_username_race` (ровно 1 пользователь) и `chat_creation_rolls_back_on_unknown_member` | `evidence/student-tests.txt` |
|
||||
| Ограниченный пул и параллельные чтения | там же, `bounded_pool_serves_parallel_reads` | OK; 100 записей от 10 клиентов за 2.50 с, ровно 100 строк | `evidence/student-tests.txt` |
|
||||
| Индекс истории, EXPLAIN ≥ 10 000 строк, без OFFSET | `sh scripts/explain-history.sh` | 12 000 сообщений: страница глубоко в истории — Index Scan; вариант OFFSET (не используется) — Seq Scan + Sort | `evidence/explain-history.txt` |
|
||||
| Недоступность Redis/Valkey и БД | `STUDENT_DOCKER_TESTS=1 python3 -m unittest -v test_degradation` | OK: `cached_p50=31 мс`, `cache_down_p50=47 мс`, `cache_down_p95=110 мс`, `/health/ready` остаётся 200 при падении кэша и становится 503 при падении БД, запись отказывает за 2.05 с, восстановление за 1.97 с | `evidence/test-degradation.txt` |
|
||||
| Durable logout на обеих репликах | `test_logout_remains_revoked_across_replicas_after_cache_outage` | **OK**: остановка Valkey → logout → restart api-1/api-2 → запуск Valkey с прежним томом; прежний токен получает 401 на GET/POST | `evidence/test-degradation-grade5.txt` |
|
||||
| Scope и конкурентность idempotency | `python -m unittest -v test_idempotency` | **2/2 OK**: одинаковый key независим для разных user/chat; 32 одинаковых POST → 32×201 одного Message, без 500 | `evidence/idempotency-grade5.txt` |
|
||||
| Трек 5: 1/2/4 API-реплики | `python scripts/research-scale.py --duration 10 --concurrency 4` | fixed total saturation: 36.9 / 49.4 / 36.2 RPS; growing resources: 42.4 / 45.2 / 57.0 RPS | `evidence/scale-summary.json`, `evidence/scale-*.json` |
|
||||
| Нагрузочный прогон | `python3 tests/load.py …` | результаты в evidence/load.json | `evidence/load.json` |
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
Дополнительно: `docker compose --env-file .env ps` показывает `api-1/api-2/postgres/valkey/nginx — healthy`; балансировка round-robin между двумя репликами подтверждена заголовком `X-Instance-Id`.
|
||||
|
||||
## Эксперименты и отказы
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
Методика: закрытый цикл `POST /api/v1/chats/{id}/messages`, 4 потока, 20 с, без ретраев, клиент — `tests/load.py`; латентность считается только по успешным запросам. Хост: Intel Pentium Silver N5030 @ 1.10 GHz, 4 логических ядра, ~7.6 GiB RAM; Docker Desktop, контейнеры без CPU/RAM-лимитов.
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
| `docker compose stop api-1` (одна реплика) | LB перенаправляет трафик на api-2, API продолжает работать | 79 успешных записей из 83 попыток за 15 с, 4 неудачных (во время переключения), восстановление за 2.6 с; целевая реплика больше не обслуживает запросы | восстановление `docker compose start api-1` — ~30 с до healthy; данных не теряется |
|
||||
| `docker compose stop valkey` | `api` продолжает работать через PostgreSQL, readiness 200 | после breaker-а p50 47 мс, 60 из 60 запросов 200 | восстановление `compose start valkey` — 0.6–3 с; данных не теряется |
|
||||
| `docker compose stop postgres` | ограниченная ошибка, никогда 200 | `/health/ready` → 503 `unavailable`, запись → 503 за 2.05 с, в ответе нет ни `psycopg`, ни `postgresql://`, ни `Traceback` | `compose start postgres` → ready 200 за 1.97 с, объём не тронут (volumes) |
|
||||
| Полный рестарт стека (`down` + `up -d --wait`) | сессия и данные те же, volumes сохраняются | `PASS: persistence preserved confirmed data and existing session` | ~20 с, потеря данных: 0 |
|
||||
| Гонка 20 параллельных регистраций одного username | ровно один `INSERT`, остальные 409 | 1 пользователь в БД, ответы 201/409 | не применимо |
|
||||
| Регистрация чата с неизвестным участником | откат всей операции | в `chats`/`chat_members` 0 строк | не применимо |
|
||||
|
||||
## Трек 5 — Цена масштабирования
|
||||
|
||||
**Гипотеза.** При фиксированном общем бюджете CPU/RAM после двух API-реплик дополнительное размножение процессов не ускорит PostgreSQL-bound запись и повысит tail latency; при росте суммарного бюджета 4 реплики должны дать прирост.
|
||||
|
||||
**Методика.** `scripts/research-scale.py` использует один closed-loop профиль `POST messages` (`tests/load.py`, 4 клиента, 10 секунд, без retry) и сохраняет raw JSON для каждой точки. В режиме `fixed_total` общий лимит API равен 2 CPU / 1024 MiB и делится между 1/2/4 репликами. В режиме `growing_total` каждая реплика получает 1 CPU / 512 MiB. Overlay `compose.scale.yaml` добавляет api-3/api-4 только для исследования; по завершении скрипт возвращает обычный двухрепличный стенд.
|
||||
|
||||
**Результат.** При фиксированном бюджете: 1 → **36.921 RPS** (p95 172 мс), 2 → **49.412 RPS** (p95 172 мс), 4 → **36.171 RPS** (p95 281 мс). Saturation наблюдается после 2 реплик: PostgreSQL и конкуренция за суммарно ограниченный CPU становятся узким местом, а дополнительные процессы увеличивают p95/p99. При росте ресурсов: 1 → **42.374 RPS**, 2 → **45.174 RPS**, 4 → **57.007 RPS**; p95 у 4 реплик 125 мс. Подтверждённое улучшение — увеличение суммарного бюджета с четырьмя репликами вместо деления фиксированного бюджета: +57.6% RPS против fixed-total 4.
|
||||
|
||||
**Ограничения.** Один повтор на Docker Desktop и закрытая модель нагрузки не дают production capacity claim; закрытый цикл содержит coordinated omission. CPU/RAM измерялись как назначенные Docker limits, а не как аппаратные counters. LB и PostgreSQL остаются SPOF; работа не заявляет отказоустойчивость их узлов.
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
- **Выдерживается:** отказ одной API-реплики (LB перенаправляет на вторую), отказ Valkey (полный обход кэша), кратковременный отказ PostgreSQL (ограниченная ошибка и быстрый возврат), рестарт всего стека с сохранением данных. **Не выдерживается:** одновременный выход обоих хранилищ из-под `up --wait` (стенд перестаёт готовиться — это ожидаемо), потеря PostgreSQL-тома вместе с контейнером.
|
||||
- Локальный pending-тombstone остаётся только оптимизацией при outage, но не является авторизационным решением: PostgreSQL проверяет revoke на каждой реплике. Поэтому отсутствие общего local cache не может воскресить токен.
|
||||
- Нагрузочный тест закрытого цикла содержит coordinated omission и накладные издержки клиента; это не доказательство ёмкости. Повторов и замера CPU/RAM не делалось.
|
||||
- `make` в среде отсутствует, поэтому все команды выполнялись напрямую; `Makefile` сохранён для Linux/CI.
|
||||
- На Windows `.sh`-скрипты не запускаются напрямую (WinError 193); созданы эквиваленты `.cmd` (`restart.cmd`, `stop-one.cmd`). На Linux grading-стенде используются `.sh`.
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
# Optional grade-5 research overlay. It is never needed for normal lab3
|
||||
# startup; scripts/research-scale.py adds it only for the 4-replica cases.
|
||||
services:
|
||||
api-3:
|
||||
extends:
|
||||
file: compose.yaml
|
||||
service: api-1
|
||||
environment:
|
||||
INSTANCE_ID: api-3
|
||||
|
||||
api-4:
|
||||
extends:
|
||||
file: compose.yaml
|
||||
service: api-1
|
||||
environment:
|
||||
INSTANCE_ID: api-4
|
||||
|
||||
nginx:
|
||||
volumes:
|
||||
- ./nginx/nginx-scale.conf:/etc/nginx/nginx.conf:ro
|
||||
+161
@@ -0,0 +1,161 @@
|
||||
# Local laboratory stand for lab 3.
|
||||
# Roles: api (2 replicas, student-implements), postgres (source of truth),
|
||||
# valkey (session cache, optional), nginx (load balancer, public entrypoint).
|
||||
# Secrets are generated into ./secrets by scripts/bootstrap-secrets.sh and are
|
||||
# mounted as files, so no credential ever appears in this file or in `docker inspect`.
|
||||
services:
|
||||
api-1:
|
||||
build:
|
||||
context: .
|
||||
image: mtusi-messenger:lab3
|
||||
environment:
|
||||
APP_HOST: 0.0.0.0
|
||||
APP_PORT: "8080"
|
||||
APP_ENV: ${APP_ENV:-lab}
|
||||
SESSION_STORE: ${SESSION_STORE:-postgres}
|
||||
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
|
||||
CACHE_TTL_SECONDS: ${CACHE_TTL_SECONDS:-300}
|
||||
DB_POOL_MIN: ${DB_POOL_MIN:-1}
|
||||
DB_POOL_MAX: ${DB_POOL_MAX:-10}
|
||||
DB_POOL_TIMEOUT_SECONDS: ${DB_POOL_TIMEOUT_SECONDS:-2}
|
||||
DB_CONNECT_TIMEOUT_SECONDS: ${DB_CONNECT_TIMEOUT_SECONDS:-2}
|
||||
DB_STATEMENT_TIMEOUT_MS: ${DB_STATEMENT_TIMEOUT_MS:-5000}
|
||||
CACHE_TIMEOUT_SECONDS: ${CACHE_TIMEOUT_SECONDS:-0.5}
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
CACHE_URL_FILE: /run/secrets/cache_url
|
||||
INSTANCE_ID: api-1
|
||||
secrets:
|
||||
- database_url
|
||||
- cache_url
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
valkey:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=2).read()"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
start_period: 20s
|
||||
retries: 3
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- backend
|
||||
|
||||
api-2:
|
||||
build:
|
||||
context: .
|
||||
image: mtusi-messenger:lab3
|
||||
environment:
|
||||
APP_HOST: 0.0.0.0
|
||||
APP_PORT: "8080"
|
||||
APP_ENV: ${APP_ENV:-lab}
|
||||
SESSION_STORE: ${SESSION_STORE:-postgres}
|
||||
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
|
||||
CACHE_TTL_SECONDS: ${CACHE_TTL_SECONDS:-300}
|
||||
DB_POOL_MIN: ${DB_POOL_MIN:-1}
|
||||
DB_POOL_MAX: ${DB_POOL_MAX:-10}
|
||||
DB_POOL_TIMEOUT_SECONDS: ${DB_POOL_TIMEOUT_SECONDS:-2}
|
||||
DB_CONNECT_TIMEOUT_SECONDS: ${DB_CONNECT_TIMEOUT_SECONDS:-2}
|
||||
DB_STATEMENT_TIMEOUT_MS: ${DB_STATEMENT_TIMEOUT_MS:-5000}
|
||||
CACHE_TIMEOUT_SECONDS: ${CACHE_TIMEOUT_SECONDS:-0.5}
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
CACHE_URL_FILE: /run/secrets/cache_url
|
||||
INSTANCE_ID: api-2
|
||||
secrets:
|
||||
- database_url
|
||||
- cache_url
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
valkey:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=2).read()"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
start_period: 20s
|
||||
retries: 3
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- backend
|
||||
|
||||
nginx:
|
||||
image: nginx:1.27-alpine
|
||||
volumes:
|
||||
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
ports:
|
||||
- "127.0.0.1:${API_PUBLISH_PORT:-8081}:8080"
|
||||
depends_on:
|
||||
api-1:
|
||||
condition: service_healthy
|
||||
api-2:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "wget --spider -q http://127.0.0.1:8080/health/ready || exit 1"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
start_period: 10s
|
||||
retries: 12
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- backend
|
||||
|
||||
postgres:
|
||||
image: postgres:17.6-bookworm@sha256:f3bd19c606e442c3d7bdfa8002e03fe260a1023351e0ea4598032022b68dd6e3
|
||||
environment:
|
||||
POSTGRES_DB: messenger
|
||||
POSTGRES_USER: messenger
|
||||
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
|
||||
POSTGRES_INITDB_ARGS: "--encoding=UTF8 --locale=C"
|
||||
secrets:
|
||||
- postgres_password
|
||||
volumes:
|
||||
- pgdata:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U messenger -d messenger --timeout=3"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
start_period: 10s
|
||||
retries: 12
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- backend
|
||||
|
||||
valkey:
|
||||
image: valkey/valkey:8.1.3@sha256:fea8b3e67b15729d4bb70589eb03367bab9ad1ee89c876f54327fc7c6e618571
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- 'exec valkey-server --requirepass "$$(cat /run/secrets/valkey_password)" --appendonly yes --appendfsync everysec --save "" --dir /data'
|
||||
secrets:
|
||||
- valkey_password
|
||||
volumes:
|
||||
- valkeydata:/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", 'valkey-cli --no-auth-warning -a "$$(cat /run/secrets/valkey_password)" ping | grep -q PONG']
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
start_period: 5s
|
||||
retries: 12
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- backend
|
||||
|
||||
secrets:
|
||||
database_url:
|
||||
file: ./secrets/database_url
|
||||
cache_url:
|
||||
file: ./secrets/cache_url
|
||||
postgres_password:
|
||||
file: ./secrets/postgres_password
|
||||
valkey_password:
|
||||
file: ./secrets/valkey_password
|
||||
|
||||
volumes:
|
||||
pgdata:
|
||||
valkeydata:
|
||||
|
||||
networks:
|
||||
backend:
|
||||
driver: bridge
|
||||
@@ -0,0 +1,7 @@
|
||||
#!/bin/sh
|
||||
# Apply versioned migrations (idempotent, advisory-locked), then hand PID 1 to
|
||||
# the server so SIGTERM reaches uvicorn directly and shutdown stays graceful.
|
||||
set -eu
|
||||
|
||||
python -B -m src.migrate
|
||||
exec python -B src/app.py
|
||||
@@ -0,0 +1,8 @@
|
||||
test_concurrent_identical_replays_return_one_message_without_500 (test_idempotency.IdempotencyScope.test_concurrent_identical_replays_return_one_message_without_500) ... ok
|
||||
test_same_key_is_independent_for_different_users_and_chats (test_idempotency.IdempotencyScope.test_same_key_is_independ
|
||||
ent_for_different_users_and_chats) ... ok
|
||||
|
||||
----------------------------------------------------------------------
|
||||
Ran 2 tests in 2.952s
|
||||
|
||||
OK
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"lab": 3,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 10.0,
|
||||
"elapsed_seconds": 11.484,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 424,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 36.921,
|
||||
"successful_latency_ms": {
|
||||
"p50": 78.0,
|
||||
"p95": 172.0,
|
||||
"p99": 235.0
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"lab": 3,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 10.0,
|
||||
"elapsed_seconds": 10.281,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 508,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 49.412,
|
||||
"successful_latency_ms": {
|
||||
"p50": 62.0,
|
||||
"p95": 172.0,
|
||||
"p99": 234.0
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"lab": 3,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 10.0,
|
||||
"elapsed_seconds": 10.312,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 373,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 36.171,
|
||||
"successful_latency_ms": {
|
||||
"p50": 63.0,
|
||||
"p95": 281.0,
|
||||
"p99": 515.0
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"lab": 3,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 10.0,
|
||||
"elapsed_seconds": 11.375,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 482,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 42.374,
|
||||
"successful_latency_ms": {
|
||||
"p50": 63.0,
|
||||
"p95": 172.0,
|
||||
"p99": 265.0
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"lab": 3,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 10.0,
|
||||
"elapsed_seconds": 10.36,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 468,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 45.174,
|
||||
"successful_latency_ms": {
|
||||
"p50": 63.0,
|
||||
"p95": 187.0,
|
||||
"p99": 219.0
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"lab": 3,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 10.0,
|
||||
"elapsed_seconds": 10.297,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 587,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 57.007,
|
||||
"successful_latency_ms": {
|
||||
"p50": 62.0,
|
||||
"p95": 125.0,
|
||||
"p99": 187.0
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
|
||||
}
|
||||
@@ -0,0 +1,122 @@
|
||||
{
|
||||
"method": "closed-loop POST messages; no retries; latency percentiles contain successes only",
|
||||
"fixed_total_budget": "2 CPU / 1024 MiB divided equally among 1, 2, 4 replicas",
|
||||
"growing_total_budget": "1 CPU / 512 MiB per replica",
|
||||
"results": [
|
||||
{
|
||||
"lab": 3,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 10.0,
|
||||
"elapsed_seconds": 11.484,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 424,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 36.921,
|
||||
"successful_latency_ms": {
|
||||
"p50": 78.0,
|
||||
"p95": 172.0,
|
||||
"p99": 235.0
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead.",
|
||||
"regime": "fixed_total",
|
||||
"replicas": 1
|
||||
},
|
||||
{
|
||||
"lab": 3,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 10.0,
|
||||
"elapsed_seconds": 10.281,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 508,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 49.412,
|
||||
"successful_latency_ms": {
|
||||
"p50": 62.0,
|
||||
"p95": 172.0,
|
||||
"p99": 234.0
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead.",
|
||||
"regime": "fixed_total",
|
||||
"replicas": 2
|
||||
},
|
||||
{
|
||||
"lab": 3,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 10.0,
|
||||
"elapsed_seconds": 10.312,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 373,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 36.171,
|
||||
"successful_latency_ms": {
|
||||
"p50": 63.0,
|
||||
"p95": 281.0,
|
||||
"p99": 515.0
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead.",
|
||||
"regime": "fixed_total",
|
||||
"replicas": 4
|
||||
},
|
||||
{
|
||||
"lab": 3,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 10.0,
|
||||
"elapsed_seconds": 11.375,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 482,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 42.374,
|
||||
"successful_latency_ms": {
|
||||
"p50": 63.0,
|
||||
"p95": 172.0,
|
||||
"p99": 265.0
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead.",
|
||||
"regime": "growing_total",
|
||||
"replicas": 1
|
||||
},
|
||||
{
|
||||
"lab": 3,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 10.0,
|
||||
"elapsed_seconds": 10.36,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 468,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 45.174,
|
||||
"successful_latency_ms": {
|
||||
"p50": 63.0,
|
||||
"p95": 187.0,
|
||||
"p99": 219.0
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead.",
|
||||
"regime": "growing_total",
|
||||
"replicas": 2
|
||||
},
|
||||
{
|
||||
"lab": 3,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 10.0,
|
||||
"elapsed_seconds": 10.297,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 587,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 57.007,
|
||||
"successful_latency_ms": {
|
||||
"p50": 62.0,
|
||||
"p95": 125.0,
|
||||
"p99": 187.0
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead.",
|
||||
"regime": "growing_total",
|
||||
"replicas": 4
|
||||
}
|
||||
],
|
||||
"limitations": "Docker Desktop host scheduling and closed-loop coordinated omission; this is not a production capacity claim."
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
test_logout_remains_revoked_across_replicas_after_cache_outage (test_degradation.DependencyOutages.test_logout_remains_revoked_across_replicas_after_cache_outage) ... ok
|
||||
test_valkey_loss_degrades_then_postgres_loss_is_bounded (test_degradation.DependencyOutages.test_valkey_loss_degrades_t
|
||||
hen_postgres_loss_is_bounded) ... ok
|
||||
|
||||
----------------------------------------------------------------------
|
||||
Ran 2 tests in 47.789s
|
||||
|
||||
OK
|
||||
|
||||
outage summary: {"cached_p50_ms": 16.0, "cache_down_p50_ms": 31.0, "cache_down_p95_ms": 47.0, "db_down_write_seconds": 2.02, "recovery_seconds": 1.95}
|
||||
@@ -0,0 +1,31 @@
|
||||
events {
|
||||
worker_connections 1024;
|
||||
}
|
||||
|
||||
http {
|
||||
include /etc/nginx/mime.types;
|
||||
default_type application/octet-stream;
|
||||
|
||||
# Used only by scripts/research-scale.py with compose.scale.yaml.
|
||||
upstream api {
|
||||
server api-1:8080 max_fails=3 fail_timeout=10s;
|
||||
server api-2:8080 max_fails=3 fail_timeout=10s;
|
||||
server api-3:8080 max_fails=3 fail_timeout=10s;
|
||||
server api-4:8080 max_fails=3 fail_timeout=10s;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 8080;
|
||||
server_name _;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_connect_timeout 2s;
|
||||
proxy_read_timeout 10s;
|
||||
proxy_send_timeout 10s;
|
||||
proxy_next_upstream error timeout http_502;
|
||||
proxy_next_upstream_tries 4;
|
||||
location / { proxy_pass http://api; }
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
events {
|
||||
worker_connections 1024;
|
||||
}
|
||||
|
||||
http {
|
||||
include /etc/nginx/mime.types;
|
||||
default_type application/octet-stream;
|
||||
|
||||
# Upstream group: both API replicas. Health-checked by Nginx so a stopped
|
||||
# replica is taken out of rotation automatically (max_fails + fail_timeout).
|
||||
upstream api {
|
||||
server api-1:8080 max_fails=3 fail_timeout=10s;
|
||||
server api-2:8080 max_fails=3 fail_timeout=10s;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 8080;
|
||||
server_name _;
|
||||
|
||||
# Keep the instance header so clients can see which replica answered.
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
|
||||
# Never wait forever on a dead replica.
|
||||
proxy_connect_timeout 2s;
|
||||
proxy_read_timeout 10s;
|
||||
proxy_send_timeout 10s;
|
||||
|
||||
# A refused connection/502 means the selected replica is unavailable and
|
||||
# can be retried on its peer. A structured 503 is an application
|
||||
# dependency failure (for example PostgreSQL down), not an unhealthy
|
||||
# replica: propagating it preserves the API error envelope instead of
|
||||
# marking both live peers down and manufacturing an Nginx 502.
|
||||
proxy_next_upstream error timeout http_502;
|
||||
# Two attempts cover both configured replicas: a request that lands on
|
||||
# the stopped instance is retried once on its healthy peer.
|
||||
proxy_next_upstream_tries 2;
|
||||
|
||||
location / {
|
||||
proxy_pass http://api;
|
||||
}
|
||||
|
||||
location /health/ready {
|
||||
# Readiness is checked by Compose on the LB itself; forward to api.
|
||||
proxy_pass http://api;
|
||||
}
|
||||
|
||||
location /version {
|
||||
proxy_pass http://api;
|
||||
}
|
||||
|
||||
location /info {
|
||||
proxy_pass http://api;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,2 +1,4 @@
|
||||
# Only for full OpenAPI/JSON Schema validation; HTTP tests use stdlib.
|
||||
openapi-spec-validator==0.7.2
|
||||
-r requirements.txt
|
||||
|
||||
# Only for full OpenAPI/JSON Schema validation (make validate).
|
||||
openapi-spec-validator==0.9.0
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
annotated-doc==0.0.5
|
||||
annotated-types==0.8.0
|
||||
anyio==4.15.1
|
||||
argon2-cffi==25.1.0
|
||||
argon2-cffi-bindings==26.1.0
|
||||
cffi==2.1.1
|
||||
click==8.5.0
|
||||
fastapi==0.162.0
|
||||
h11==0.16.0
|
||||
httptools==0.9.0
|
||||
idna==3.20
|
||||
psycopg==3.3.6
|
||||
psycopg-binary==3.3.6
|
||||
psycopg-pool==3.3.3
|
||||
pycparser==3.12
|
||||
pydantic==2.14.2
|
||||
pydantic_core==2.50.2
|
||||
python-dotenv==1.2.4
|
||||
PyYAML==6.0.3
|
||||
redis==8.1.0
|
||||
starlette==1.10.0
|
||||
typing-inspection==0.4.4
|
||||
typing_extensions==4.16.0
|
||||
tzdata==2026.6
|
||||
uvicorn==0.54.0
|
||||
watchfiles==1.3.0
|
||||
websockets==17.3
|
||||
@@ -0,0 +1,48 @@
|
||||
#!/bin/sh
|
||||
# Generate the local, synthetic stand's secrets and .env.
|
||||
# Values are written to ./secrets (git-ignored, mode 0600) and never printed.
|
||||
# Re-running is safe: existing files are kept unless FORCE=1.
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
PYTHON="${PYTHON:-python}"
|
||||
SECRETS_DIR="${SECRETS_DIR:-secrets}"
|
||||
|
||||
if [ ! -f .env ]; then
|
||||
cp .env.example .env
|
||||
printf '%s\n' "created .env from .env.example"
|
||||
fi
|
||||
|
||||
mkdir -p "$SECRETS_DIR"
|
||||
chmod 700 "$SECRETS_DIR" 2>/dev/null || true
|
||||
|
||||
umask 077
|
||||
|
||||
random_password() {
|
||||
"$PYTHON" -c 'import secrets; print(secrets.token_hex(24))'
|
||||
}
|
||||
|
||||
write_if_missing() {
|
||||
path="$SECRETS_DIR/$1"
|
||||
if [ -f "$path" ] && [ "${FORCE:-0}" != "1" ]; then
|
||||
return 0
|
||||
fi
|
||||
printf '%s' "$2" > "$path"
|
||||
chmod 600 "$path" 2>/dev/null || true
|
||||
}
|
||||
|
||||
if [ -f "$SECRETS_DIR/database_url" ] && [ -f "$SECRETS_DIR/cache_url" ] && [ "${FORCE:-0}" != "1" ]; then
|
||||
printf '%s\n' "secrets already present in $SECRETS_DIR (FORCE=1 to regenerate)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
postgres_password="$(random_password)"
|
||||
valkey_password="$(random_password)"
|
||||
|
||||
write_if_missing postgres_password "$postgres_password"
|
||||
write_if_missing valkey_password "$valkey_password"
|
||||
write_if_missing database_url "postgresql://messenger:${postgres_password}@postgres:5432/messenger"
|
||||
write_if_missing cache_url "redis://:${valkey_password}@valkey:6379/0"
|
||||
|
||||
printf '%s\n' "wrote secrets to $SECRETS_DIR: postgres_password, valkey_password, database_url, cache_url"
|
||||
printf '%s\n' "next: docker compose --env-file .env up -d --build"
|
||||
@@ -8,8 +8,8 @@ import re
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
def main():
|
||||
lab = json.loads((ROOT/'lab.json').read_text())['number']
|
||||
spec = json.loads((ROOT/'contracts/openapi.json').read_text())
|
||||
lab = json.loads((ROOT/'lab.json').read_text(encoding='utf-8'))['number']
|
||||
spec = json.loads((ROOT/'contracts/openapi.json').read_text(encoding='utf-8'))
|
||||
assert spec['openapi']=='3.1.0'
|
||||
assert spec['info']['version']=='1.%d.0'%lab
|
||||
operation_ids = set()
|
||||
@@ -41,11 +41,11 @@ def main():
|
||||
assert ('/api/v1/auth/sessions' in spec['paths']) == (lab>=2)
|
||||
assert ('/api/v1/attachments/{attachment_id}' in spec['paths']) == (lab>=4)
|
||||
for file in list((ROOT/'tests').glob('*.py'))+list((ROOT/'scripts').glob('*.py')):
|
||||
ast.parse(file.read_text(),filename=str(file))
|
||||
ast.parse(file.read_text(encoding='utf-8'),filename=str(file))
|
||||
for file in ROOT.rglob('*.json'):
|
||||
if any(part in {'.git','.venv','node_modules','vendor','output'} for part in file.parts):
|
||||
continue
|
||||
json.loads(file.read_text())
|
||||
json.loads(file.read_text(encoding='utf-8'))
|
||||
for required in ('README.md','COURSE.md','CONTRIBUTING.md','REPORT.md','contracts/README.md','tests/smoke.py','tests/README.md'):
|
||||
assert (ROOT/required).is_file(), 'missing '+required
|
||||
print('PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.')
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Grade-5 track 1: repeatable 1/2/4 API-replica scaling experiment.
|
||||
|
||||
The script measures the same closed-loop write workload in two regimes:
|
||||
``fixed_total`` divides a fixed 2 CPU / 1 GiB API budget among replicas;
|
||||
``growing_total`` assigns every replica 1 CPU / 512 MiB. It writes raw output
|
||||
from tests/load.py to evidence/scale-*.json and a sanitized summary to
|
||||
evidence/scale-summary.json. No secrets or tokens are written.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import time
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
BASE_COMPOSE = ["docker", "compose", "--env-file", ".env", "-f", "compose.yaml"]
|
||||
SCALE_COMPOSE = BASE_COMPOSE + ["-f", "compose.scale.yaml"]
|
||||
|
||||
|
||||
def run(command: list[str], *, timeout: int = 180) -> None:
|
||||
result = subprocess.run(command, cwd=ROOT, text=True, capture_output=True, timeout=timeout)
|
||||
if result.returncode:
|
||||
raise SystemExit(
|
||||
"command failed:\n"
|
||||
+ " ".join(command)
|
||||
+ "\n"
|
||||
+ (result.stdout + result.stderr)[-2000:]
|
||||
)
|
||||
|
||||
|
||||
def wait_ready(base_url: str) -> None:
|
||||
import urllib.request
|
||||
|
||||
deadline = time.monotonic() + 120
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
with urllib.request.urlopen(base_url + "/health/ready", timeout=3) as response:
|
||||
if response.status == 200:
|
||||
return
|
||||
except OSError:
|
||||
pass
|
||||
time.sleep(1)
|
||||
raise SystemExit("load balancer did not become ready within 120 seconds")
|
||||
|
||||
|
||||
def set_budget(replicas: int, regime: str) -> None:
|
||||
if regime == "fixed_total":
|
||||
cpu = 2.0 / replicas
|
||||
memory = 1024 // replicas
|
||||
else:
|
||||
cpu = 1.0
|
||||
memory = 512
|
||||
for index in range(1, replicas + 1):
|
||||
run([
|
||||
"docker", "update", "--cpus", str(cpu), "--memory", str(memory) + "m",
|
||||
"--memory-swap", str(memory) + "m",
|
||||
"mtusi2026lab3-api-" + str(index) + "-1",
|
||||
])
|
||||
|
||||
|
||||
def configure_replicas(replicas: int) -> None:
|
||||
if replicas in {1, 2}:
|
||||
# The standard two-replica topology remains intact; stop api-2 only for
|
||||
# the one-replica measurement. Nginx retries the healthy peer.
|
||||
run(BASE_COMPOSE + ["up", "-d", "--wait", "api-1", "api-2", "nginx"])
|
||||
if replicas == 1:
|
||||
run(BASE_COMPOSE + ["stop", "api-2"])
|
||||
else:
|
||||
run(SCALE_COMPOSE + ["up", "-d", "--wait", "api-1", "api-2", "api-3", "api-4", "nginx"])
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--base-url", default=os.environ.get("BASE_URL", "http://127.0.0.1:8081"))
|
||||
parser.add_argument("--duration", type=float, default=15.0)
|
||||
parser.add_argument("--concurrency", type=int, default=4)
|
||||
args = parser.parse_args()
|
||||
if not 1 <= args.duration <= 300:
|
||||
raise SystemExit("duration must be 1..300")
|
||||
|
||||
evidence = ROOT / "evidence"
|
||||
evidence.mkdir(exist_ok=True)
|
||||
results: list[dict[str, object]] = []
|
||||
try:
|
||||
for regime in ("fixed_total", "growing_total"):
|
||||
for replicas in (1, 2, 4):
|
||||
configure_replicas(replicas)
|
||||
set_budget(replicas, regime)
|
||||
wait_ready(args.base_url.rstrip("/"))
|
||||
output = evidence / ("scale-" + regime + "-" + str(replicas) + ".json")
|
||||
run([
|
||||
"python", "tests/load.py", "--base-url", args.base_url,
|
||||
"--duration", str(args.duration), "--concurrency", str(args.concurrency),
|
||||
"--output", str(output),
|
||||
], timeout=int(args.duration + 120))
|
||||
sample = json.loads(output.read_text(encoding="utf-8"))
|
||||
sample.update({"regime": regime, "replicas": replicas})
|
||||
results.append(sample)
|
||||
finally:
|
||||
# Return to the normal grade-3 topology with its ordinary nginx config.
|
||||
run(SCALE_COMPOSE + ["down"], timeout=120)
|
||||
run(BASE_COMPOSE + ["up", "-d", "--wait"], timeout=180)
|
||||
|
||||
summary = {
|
||||
"method": "closed-loop POST messages; no retries; latency percentiles contain successes only",
|
||||
"fixed_total_budget": "2 CPU / 1024 MiB divided equally among 1, 2, 4 replicas",
|
||||
"growing_total_budget": "1 CPU / 512 MiB per replica",
|
||||
"results": results,
|
||||
"limitations": "Docker Desktop host scheduling and closed-loop coordinated omission; this is not a production capacity claim.",
|
||||
}
|
||||
(evidence / "scale-summary.json").write_text(
|
||||
json.dumps(summary, ensure_ascii=False, indent=2) + "\n", encoding="utf-8"
|
||||
)
|
||||
print(json.dumps(summary, ensure_ascii=False, indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,13 @@
|
||||
@echo off
|
||||
setlocal
|
||||
cd /d "%~dp0.."
|
||||
if "%COMPOSE_ENV_FILE%"=="" set COMPOSE_ENV_FILE=.env
|
||||
echo restart: stopping api, postgres and valkey (volumes preserved)
|
||||
docker compose --env-file %COMPOSE_ENV_FILE% down --timeout 15
|
||||
if errorlevel 1 exit /b 1
|
||||
echo restart: starting api, postgres and valkey
|
||||
docker compose --env-file %COMPOSE_ENV_FILE% up -d --wait --wait-timeout 120
|
||||
if errorlevel 1 exit /b 1
|
||||
docker compose --env-file %COMPOSE_ENV_FILE% ps
|
||||
echo stack_restarted volumes_preserved
|
||||
exit /b 0
|
||||
@@ -0,0 +1,17 @@
|
||||
#!/bin/sh
|
||||
# Restart hook for `make test-persistence ACTION_SCRIPT=scripts/restart.sh`.
|
||||
# Recreates the API and BOTH state stores from the same images, keeping the
|
||||
# named volumes (pgdata, valkeydata): no `down -v`, no seed, no data reset.
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
ENV_FILE="${COMPOSE_ENV_FILE:-.env}"
|
||||
|
||||
printf '%s\n' "restart: stopping api, postgres and valkey (volumes preserved)"
|
||||
docker compose --env-file "$ENV_FILE" down --timeout 15
|
||||
|
||||
printf '%s\n' "restart: starting api, postgres and valkey"
|
||||
docker compose --env-file "$ENV_FILE" up -d --wait --wait-timeout 120
|
||||
|
||||
docker compose --env-file "$ENV_FILE" ps
|
||||
printf '%s\n' "stack_restarted volumes_preserved"
|
||||
@@ -0,0 +1,17 @@
|
||||
@echo off
|
||||
setlocal
|
||||
cd /d "%~dp0.."
|
||||
if "%COMPOSE_ENV_FILE%"=="" set COMPOSE_ENV_FILE=.env
|
||||
if "%TARGET_INSTANCE_ID%"=="" (
|
||||
echo TARGET_INSTANCE_ID is required >&2
|
||||
exit /b 2
|
||||
)
|
||||
if not "%TARGET_INSTANCE_ID%"=="api-1" if not "%TARGET_INSTANCE_ID%"=="api-2" (
|
||||
echo Invalid TARGET_INSTANCE_ID: %TARGET_INSTANCE_ID% >&2
|
||||
exit /b 2
|
||||
)
|
||||
echo stop-one: stopping API replica %TARGET_INSTANCE_ID%
|
||||
docker compose --env-file %COMPOSE_ENV_FILE% stop --timeout 10 %TARGET_INSTANCE_ID%
|
||||
if errorlevel 1 exit /b 1
|
||||
echo stop-one: %TARGET_INSTANCE_ID% stopped
|
||||
exit /b 0
|
||||
@@ -0,0 +1,26 @@
|
||||
#!/bin/sh
|
||||
# Failover hook for `make test-failover ACTION_SCRIPT=scripts/stop-one.sh`.
|
||||
# Stops ONE API replica (selected by TARGET_INSTANCE_ID = api-1 or api-2) and
|
||||
# leaves it stopped; the load balancer keeps serving via the remaining replica.
|
||||
# Does not touch postgres, valkey, nginx or any volumes. Restore the replica
|
||||
# explicitly after the test (docker compose start <service>).
|
||||
set -eu
|
||||
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
ENV_FILE="${COMPOSE_ENV_FILE:-.env}"
|
||||
|
||||
: "${TARGET_INSTANCE_ID:?test provides target API instance alias (api-1 or api-2)}"
|
||||
|
||||
case "$TARGET_INSTANCE_ID" in
|
||||
api-1|api-2) ;;
|
||||
*)
|
||||
printf '%s\n' "stop-one: invalid TARGET_INSTANCE_ID='$TARGET_INSTANCE_ID' (expected api-1 or api-2)" >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
|
||||
printf '%s\n' "stop-one: stopping API replica $TARGET_INSTANCE_ID (nginx keeps serving via the other replica)"
|
||||
docker compose --env-file "$ENV_FILE" stop --timeout 10 "$TARGET_INSTANCE_ID"
|
||||
|
||||
printf '%s\n' "stop-one: $TARGET_INSTANCE_ID stopped"
|
||||
@@ -6,17 +6,17 @@ from openapi_spec_validator import validate
|
||||
from jsonschema import Draft202012Validator
|
||||
|
||||
root = Path(__file__).resolve().parents[1]
|
||||
spec = json.loads((root/'contracts/openapi.json').read_text())
|
||||
spec = json.loads((root/'contracts/openapi.json').read_text(encoding='utf-8'))
|
||||
validate(spec)
|
||||
from referencing import Registry, Resource
|
||||
# The base URI resolves local #/components references inside examples.
|
||||
resource = dict(spec, **{'$schema': 'https://json-schema.org/draft/2020-12/schema'})
|
||||
registry = Registry().with_resource('urn:mtusi:openapi', Resource.from_contents(resource))
|
||||
for name, example in json.loads((root/'contracts/examples.json').read_text()).items():
|
||||
for name, example in json.loads((root/'contracts/examples.json').read_text(encoding='utf-8')).items():
|
||||
schema = {'$ref': 'urn:mtusi:openapi#/components/schemas/' + name}
|
||||
Draft202012Validator(schema, registry=registry).validate(example)
|
||||
for path in (root/'contracts').glob('*.schema.json'):
|
||||
schema = json.loads(path.read_text())
|
||||
schema = json.loads(path.read_text(encoding='utf-8'))
|
||||
Draft202012Validator.check_schema(schema)
|
||||
for example in schema.get('examples',[]):
|
||||
Draft202012Validator(schema).validate(example)
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
"""MTUSI messenger, laboratory 3: multi-instance, sessions, idempotency."""
|
||||
+394
@@ -0,0 +1,394 @@
|
||||
"""Routes, authentication and request validation for lab 2."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import binascii
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
import uuid
|
||||
from dataclasses import dataclass
|
||||
from typing import Annotated, Any, cast
|
||||
|
||||
from fastapi import APIRouter, Depends, Request, Response
|
||||
from psycopg_pool import AsyncConnectionPool
|
||||
from pydantic import ValidationError
|
||||
from starlette.concurrency import run_in_threadpool
|
||||
|
||||
from . import sessions as session_service
|
||||
from . import store
|
||||
from .config import DEFAULT_PAGE_SIZE, MAX_JSON_BYTES, MAX_PAGE_SIZE, IDEMPOTENCY_KEY_PATTERN
|
||||
from .db import ping
|
||||
from .errors import BASIC_CHALLENGE, BEARER_CHALLENGE, ApiProblem
|
||||
from .models import CreateChat, CreateMessage, CreateUser, StrictModel
|
||||
from .pagination import build_page, decode_cursor, format_timestamp
|
||||
from .security import hash_password, verify_password, verify_password_unknown_user
|
||||
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
USERNAME_PATTERN = re.compile(r"^[a-z0-9_]{3,32}$")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Principal:
|
||||
"""The authenticated identity behind a Bearer token."""
|
||||
|
||||
user_id: uuid.UUID
|
||||
user: dict[str, Any]
|
||||
|
||||
|
||||
def get_pool(request: Request) -> AsyncConnectionPool:
|
||||
return cast(AsyncConnectionPool, request.app.state.pool)
|
||||
|
||||
|
||||
def get_cache(request: Request) -> session_service.SessionCache:
|
||||
return cast(session_service.SessionCache, request.app.state.cache)
|
||||
|
||||
|
||||
async def drain_body(request: Request) -> None:
|
||||
"""Consume an unwanted request body without retaining it in memory."""
|
||||
async for _chunk in request.stream():
|
||||
pass
|
||||
|
||||
|
||||
def declared_body_size(request: Request) -> int | None:
|
||||
value = request.headers.get("Content-Length")
|
||||
if value is None:
|
||||
return None
|
||||
try:
|
||||
size = int(value)
|
||||
except ValueError:
|
||||
raise ApiProblem(400, "invalid_request", "Content-Length is invalid") from None
|
||||
if size < 0:
|
||||
raise ApiProblem(400, "invalid_request", "Content-Length is invalid")
|
||||
return size
|
||||
|
||||
|
||||
async def read_limited_body(request: Request) -> bytes:
|
||||
chunks: list[bytes] = []
|
||||
size = 0
|
||||
too_large = False
|
||||
async for chunk in request.stream():
|
||||
size += len(chunk)
|
||||
if size <= MAX_JSON_BYTES:
|
||||
chunks.append(chunk)
|
||||
else:
|
||||
too_large = True
|
||||
if too_large:
|
||||
raise ApiProblem(413, "payload_too_large", "Request body is too large")
|
||||
return b"".join(chunks)
|
||||
|
||||
|
||||
async def parse_body(request: Request, model: type[StrictModel]) -> Any:
|
||||
content_length = declared_body_size(request)
|
||||
if content_length is not None and content_length > MAX_JSON_BYTES:
|
||||
await drain_body(request)
|
||||
raise ApiProblem(413, "payload_too_large", "Request body is too large")
|
||||
content_type = request.headers.get("Content-Type", "").split(";", 1)[0].strip()
|
||||
if content_type.lower() != "application/json":
|
||||
await drain_body(request)
|
||||
raise ApiProblem(
|
||||
415, "unsupported_media_type", "Content-Type must be application/json"
|
||||
)
|
||||
body = await read_limited_body(request)
|
||||
try:
|
||||
decoded = json.loads(body.decode("utf-8"))
|
||||
except (json.JSONDecodeError, UnicodeDecodeError):
|
||||
raise ApiProblem(
|
||||
400, "invalid_request", "Request body is not valid JSON"
|
||||
) from None
|
||||
try:
|
||||
return model.model_validate(decoded)
|
||||
except ValidationError:
|
||||
raise ApiProblem(400, "invalid_request", "Request fields are invalid") from None
|
||||
|
||||
|
||||
def ensure_no_query(request: Request) -> None:
|
||||
if request.url.query:
|
||||
raise ApiProblem(400, "invalid_request", "Query parameters are not supported")
|
||||
|
||||
|
||||
def parse_pagination_request(request: Request) -> tuple[int, str | None]:
|
||||
"""Read limit/cursor only; unknown or repeated parameters are a 400."""
|
||||
parameters = request.query_params
|
||||
keys = [key for key, _ in parameters.multi_items()]
|
||||
if set(keys) - {"limit", "cursor"} or any(
|
||||
keys.count(key) != 1 for key in set(keys)
|
||||
):
|
||||
raise ApiProblem(400, "invalid_request", "Query parameters are invalid")
|
||||
limit_value = parameters.get("limit", str(DEFAULT_PAGE_SIZE))
|
||||
try:
|
||||
limit = int(limit_value)
|
||||
except (TypeError, ValueError):
|
||||
raise ApiProblem(400, "invalid_request", "limit is invalid") from None
|
||||
if not 1 <= limit <= MAX_PAGE_SIZE:
|
||||
raise ApiProblem(400, "invalid_request", "limit is invalid")
|
||||
cursor = parameters.get("cursor")
|
||||
if cursor == "":
|
||||
raise ApiProblem(400, "invalid_request", "cursor is invalid")
|
||||
return limit, cursor
|
||||
|
||||
|
||||
def require_chat_id(chat_id: str) -> uuid.UUID:
|
||||
try:
|
||||
return uuid.UUID(chat_id)
|
||||
except (ValueError, AttributeError):
|
||||
raise ApiProblem(400, "invalid_request", "chat_id must be a UUID") from None
|
||||
|
||||
|
||||
def unauthorized(message: str, challenge: str) -> ApiProblem:
|
||||
return ApiProblem(401, "unauthorized", message, {"WWW-Authenticate": challenge})
|
||||
|
||||
|
||||
def bearer_token(request: Request) -> str:
|
||||
"""Only Bearer is accepted; Basic here is a 401, not a fallback login."""
|
||||
header = request.headers.get("Authorization")
|
||||
if header is None:
|
||||
raise unauthorized("Bearer session token is required", BEARER_CHALLENGE)
|
||||
scheme, _, value = header.partition(" ")
|
||||
if scheme.strip().lower() != "bearer" or not value.strip():
|
||||
raise unauthorized("Bearer session token is required", BEARER_CHALLENGE)
|
||||
return value.strip()
|
||||
|
||||
|
||||
def basic_credentials(request: Request) -> tuple[str, str]:
|
||||
"""Decode Basic credentials; it is an encoding, never a trusted identity."""
|
||||
header = request.headers.get("Authorization")
|
||||
if header is None:
|
||||
raise unauthorized("Basic credentials are required", BASIC_CHALLENGE)
|
||||
scheme, _, value = header.partition(" ")
|
||||
if scheme.strip().lower() != "basic" or not value.strip():
|
||||
raise unauthorized("Basic credentials are required", BASIC_CHALLENGE)
|
||||
try:
|
||||
raw = base64.b64decode(value.strip(), validate=True).decode("utf-8")
|
||||
except (binascii.Error, UnicodeDecodeError, ValueError):
|
||||
raise unauthorized("Basic credentials are invalid", BASIC_CHALLENGE) from None
|
||||
username, separator, password = raw.partition(":")
|
||||
if not separator or not username or not password:
|
||||
raise unauthorized("Basic credentials are invalid", BASIC_CHALLENGE)
|
||||
if USERNAME_PATTERN.fullmatch(username) is None:
|
||||
# Unknown or malformed usernames must not be distinguished from a bad password.
|
||||
raise unauthorized("Invalid username or password", BASIC_CHALLENGE)
|
||||
return username, password
|
||||
|
||||
|
||||
async def current_principal(request: Request) -> Principal:
|
||||
token = bearer_token(request)
|
||||
user_id, user = await session_service.resolve_session(
|
||||
get_pool(request), get_cache(request), token
|
||||
)
|
||||
return Principal(user_id=user_id, user=user)
|
||||
|
||||
|
||||
Actor = Annotated[Principal, Depends(current_principal)]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Health
|
||||
# --------------------------------------------------------------------------- #
|
||||
@router.get("/health/live")
|
||||
async def liveness(request: Request) -> dict[str, str]:
|
||||
ensure_no_query(request)
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
@router.get("/health/ready")
|
||||
async def readiness(request: Request) -> dict[str, str]:
|
||||
ensure_no_query(request)
|
||||
try:
|
||||
await ping(get_pool(request))
|
||||
except Exception as exc: # noqa: BLE001 - any DB failure means not ready
|
||||
request.app.state.logger.warning(
|
||||
"readiness_database_failed error=%s", type(exc).__name__
|
||||
)
|
||||
raise ApiProblem(503, "unavailable", "Database is unavailable") from None
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Users and sessions
|
||||
# --------------------------------------------------------------------------- #
|
||||
@router.post("/api/v1/users", status_code=201)
|
||||
async def register_user(request: Request) -> dict[str, Any]:
|
||||
ensure_no_query(request)
|
||||
body = await parse_body(request, CreateUser)
|
||||
assert isinstance(body, CreateUser)
|
||||
password_hash = await run_in_threadpool(hash_password, body.password)
|
||||
async with get_pool(request).connection() as connection:
|
||||
async with connection.cursor() as cursor:
|
||||
return await store.create_user(
|
||||
cursor,
|
||||
username=body.username,
|
||||
display_name=body.display_name,
|
||||
password_hash=password_hash,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/api/v1/auth/sessions", status_code=201)
|
||||
async def create_session(request: Request) -> dict[str, Any]:
|
||||
ensure_no_query(request)
|
||||
await drain_body(request)
|
||||
username, password = basic_credentials(request)
|
||||
pool = get_pool(request)
|
||||
async with pool.connection() as connection:
|
||||
async with connection.cursor() as cursor:
|
||||
row = await store.user_credentials(cursor, username)
|
||||
if row is None:
|
||||
await run_in_threadpool(verify_password_unknown_user, password)
|
||||
raise unauthorized("Invalid username or password", BASIC_CHALLENGE)
|
||||
if not await run_in_threadpool(
|
||||
verify_password, password, row["password_hash"]
|
||||
):
|
||||
raise unauthorized("Invalid username or password", BASIC_CHALLENGE)
|
||||
token, expires_at = await session_service.create_session(
|
||||
pool,
|
||||
get_cache(request),
|
||||
user_id=row["id"],
|
||||
ttl_seconds=request.app.state.settings.session_ttl_seconds,
|
||||
)
|
||||
return {
|
||||
"token": token,
|
||||
"token_type": "Bearer",
|
||||
"expires_at": format_timestamp(expires_at),
|
||||
"user": store.user_json(row),
|
||||
}
|
||||
|
||||
|
||||
@router.delete("/api/v1/auth/sessions/current", status_code=204)
|
||||
async def delete_current_session(request: Request) -> Response:
|
||||
ensure_no_query(request)
|
||||
token = bearer_token(request)
|
||||
await session_service.revoke_session(get_pool(request), get_cache(request), token)
|
||||
return Response(status_code=204)
|
||||
|
||||
|
||||
@router.get("/api/v1/users/me")
|
||||
async def current_user(request: Request, actor: Actor) -> dict[str, Any]:
|
||||
ensure_no_query(request)
|
||||
return actor.user
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Chats and messages
|
||||
# --------------------------------------------------------------------------- #
|
||||
@router.post("/api/v1/chats", status_code=201)
|
||||
async def create_chat(request: Request, actor: Actor) -> dict[str, Any]:
|
||||
ensure_no_query(request)
|
||||
body = await parse_body(request, CreateChat)
|
||||
assert isinstance(body, CreateChat)
|
||||
members = [uuid.UUID(member) for member in body.member_ids]
|
||||
async with get_pool(request).connection() as connection:
|
||||
async with connection.cursor() as cursor:
|
||||
return await store.create_chat(
|
||||
cursor, actor_id=actor.user_id, title=body.title, member_ids=members
|
||||
)
|
||||
|
||||
|
||||
@router.get("/api/v1/chats")
|
||||
async def list_chats(request: Request, actor: Actor) -> dict[str, Any]:
|
||||
limit, cursor = parse_pagination_request(request)
|
||||
resource = "chats:" + str(actor.user_id)
|
||||
after = decode_cursor(cursor, resource) if cursor is not None else None
|
||||
async with get_pool(request).connection() as connection:
|
||||
async with connection.cursor() as cursor_handle:
|
||||
rows = await store.chat_page(
|
||||
cursor_handle, actor_id=actor.user_id, limit=limit, after=after
|
||||
)
|
||||
return build_page(rows, resource, limit)
|
||||
|
||||
|
||||
@router.get("/api/v1/chats/{chat_id}")
|
||||
async def get_chat(request: Request, chat_id: str, actor: Actor) -> dict[str, Any]:
|
||||
ensure_no_query(request)
|
||||
normalized = require_chat_id(chat_id)
|
||||
async with get_pool(request).connection() as connection:
|
||||
async with connection.cursor() as cursor:
|
||||
return await store.accessible_chat(
|
||||
cursor, actor_id=actor.user_id, chat_id=normalized
|
||||
)
|
||||
|
||||
|
||||
@router.post("/api/v1/chats/{chat_id}/messages", status_code=201)
|
||||
async def create_message(
|
||||
request: Request, chat_id: str, actor: Actor
|
||||
) -> dict[str, Any]:
|
||||
normalized = require_chat_id(chat_id)
|
||||
ensure_no_query(request)
|
||||
body = await parse_body(request, CreateMessage)
|
||||
assert isinstance(body, CreateMessage)
|
||||
pool = get_pool(request)
|
||||
request_sha256 = hashlib.sha256(
|
||||
json.dumps(body.model_dump(), sort_keys=True).encode("utf-8")
|
||||
).hexdigest()
|
||||
async with pool.connection() as connection:
|
||||
async with connection.cursor() as cursor:
|
||||
# Check membership first so non-members get 404, not 400.
|
||||
await store.require_membership(
|
||||
cursor, actor_id=actor.user_id, chat_id=normalized
|
||||
)
|
||||
key = request.headers.get("Idempotency-Key", "").strip()
|
||||
if not key or not IDEMPOTENCY_KEY_PATTERN.fullmatch(key):
|
||||
raise ApiProblem(
|
||||
400, "invalid_request", "Idempotency-Key must match "
|
||||
"^[A-Za-z0-9._:-]{1,128}$"
|
||||
)
|
||||
existing = await store.get_idempotency_key(
|
||||
cursor, actor_id=actor.user_id, chat_id=normalized, key=key
|
||||
)
|
||||
if existing is not None:
|
||||
if existing["request_sha256"] != request_sha256:
|
||||
raise ApiProblem(409, "conflict", "Idempotency-Key was already used")
|
||||
# Replay: return the original message.
|
||||
original = await store.message_by_id(cursor, existing["message_id"])
|
||||
if original is not None:
|
||||
return original
|
||||
inserted = await store.insert_idempotency_key(
|
||||
cursor,
|
||||
actor_id=actor.user_id,
|
||||
key=key,
|
||||
chat_id=normalized,
|
||||
request_sha256=request_sha256,
|
||||
)
|
||||
if not inserted:
|
||||
# INSERT .. ON CONFLICT keeps this transaction usable. PostgreSQL
|
||||
# waited for the winning transaction, so its message is committed
|
||||
# and can be returned as the exact replay result.
|
||||
existing = await store.get_idempotency_key(
|
||||
cursor, actor_id=actor.user_id, chat_id=normalized, key=key
|
||||
)
|
||||
if existing is not None:
|
||||
if existing["request_sha256"] != request_sha256:
|
||||
raise ApiProblem(409, "conflict", "Idempotency-Key was already used")
|
||||
original = await store.message_by_id(cursor, existing["message_id"])
|
||||
if original is not None:
|
||||
return original
|
||||
raise ApiProblem(409, "conflict", "Idempotency-Key is still being stored")
|
||||
return await store.create_message(
|
||||
cursor,
|
||||
actor_id=actor.user_id,
|
||||
chat_id=normalized,
|
||||
text=body.text,
|
||||
request_sha256=request_sha256,
|
||||
key=key,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/api/v1/chats/{chat_id}/messages")
|
||||
async def list_messages(
|
||||
request: Request, chat_id: str, actor: Actor
|
||||
) -> dict[str, Any]:
|
||||
normalized = require_chat_id(chat_id)
|
||||
limit, cursor = parse_pagination_request(request)
|
||||
resource = "messages:" + str(normalized)
|
||||
after = decode_cursor(cursor, resource) if cursor is not None else None
|
||||
async with get_pool(request).connection() as connection:
|
||||
async with connection.cursor() as cursor_handle:
|
||||
await store.require_membership(
|
||||
cursor_handle, actor_id=actor.user_id, chat_id=normalized
|
||||
)
|
||||
rows = await store.message_page(
|
||||
cursor_handle, chat_id=normalized, limit=limit, after=after
|
||||
)
|
||||
return build_page(rows, resource, limit)
|
||||
+34
@@ -0,0 +1,34 @@
|
||||
"""Executable entry point: migrations run first, then uvicorn as PID 1."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import uvicorn
|
||||
|
||||
|
||||
if __package__ in {None, ""}:
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
|
||||
from src.application import create_app # noqa: E402
|
||||
from src.config import load_address # noqa: E402
|
||||
|
||||
|
||||
app = create_app()
|
||||
|
||||
|
||||
def run() -> None:
|
||||
host, port = load_address()
|
||||
uvicorn.run(
|
||||
app,
|
||||
host=host,
|
||||
port=port,
|
||||
access_log=False,
|
||||
timeout_graceful_shutdown=8,
|
||||
log_level="info",
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
run()
|
||||
@@ -0,0 +1,306 @@
|
||||
"""FastAPI application factory, lifespan, middleware and error mapping.
|
||||
|
||||
Beta-1 of the lab 3 daemon. It contains the required smoke bare minimums
|
||||
(the version/info routes, the X-Request-Id / X-Instance-Id headers and the
|
||||
request context) but not the full lab 3 feature set, which is implemented in
|
||||
the lab 2 package that this package now wraps.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import json
|
||||
import logging
|
||||
import sys
|
||||
import time
|
||||
import uuid
|
||||
from contextlib import asynccontextmanager
|
||||
from typing import Any, AsyncIterator
|
||||
|
||||
import psycopg
|
||||
from fastapi import FastAPI, Request
|
||||
from fastapi.exceptions import RequestValidationError
|
||||
from fastapi.responses import JSONResponse, RedirectResponse
|
||||
from psycopg_pool import AsyncConnectionPool, PoolTimeout
|
||||
from starlette.exceptions import HTTPException as StarletteHTTPException
|
||||
from starlette.responses import Response
|
||||
|
||||
from . import sessions as session_service
|
||||
from .api import router
|
||||
from .config import CONTRACT_PATH, Settings, load_settings
|
||||
from .db import create_pool
|
||||
from .errors import ApiProblem
|
||||
|
||||
|
||||
access_logger = logging.getLogger("messenger.access")
|
||||
lifecycle_logger = logging.getLogger("uvicorn.error")
|
||||
if not access_logger.handlers:
|
||||
access_handler = logging.StreamHandler(sys.stdout)
|
||||
access_handler.setFormatter(
|
||||
logging.Formatter("%(asctime)s %(levelname)s %(name)s %(message)s")
|
||||
)
|
||||
access_logger.addHandler(access_handler)
|
||||
access_logger.setLevel(logging.INFO)
|
||||
access_logger.propagate = False
|
||||
|
||||
|
||||
def request_id(request: Request) -> str:
|
||||
return getattr(request.state, "request_id", str(uuid.uuid4()))
|
||||
|
||||
|
||||
def problem_response(
|
||||
request: Request,
|
||||
status: int,
|
||||
code: str,
|
||||
message: str,
|
||||
headers: dict[str, str] | None = None,
|
||||
) -> JSONResponse:
|
||||
current_request_id = request_id(request)
|
||||
response_headers = {"X-Request-Id": current_request_id}
|
||||
response_headers.update(headers or {})
|
||||
return JSONResponse(
|
||||
status_code=status,
|
||||
content={
|
||||
"error": {
|
||||
"code": code,
|
||||
"message": message,
|
||||
"request_id": current_request_id,
|
||||
}
|
||||
},
|
||||
headers=response_headers,
|
||||
)
|
||||
|
||||
|
||||
def load_contract() -> dict[str, Any]:
|
||||
with CONTRACT_PATH.open(encoding="utf-8") as contract_file:
|
||||
return json.load(contract_file)
|
||||
|
||||
|
||||
async def _purge_loop(pool: AsyncConnectionPool) -> None:
|
||||
"""Drop long-expired session and idempotency rows; TTL is also read-path."""
|
||||
while True:
|
||||
try:
|
||||
await asyncio.sleep(60)
|
||||
purged = await session_service.purge_expired(pool)
|
||||
idempotency_purged = await session_service.purge_expired_idempotency(pool)
|
||||
if purged or idempotency_purged:
|
||||
lifecycle_logger.info(
|
||||
"housekeeping sessions_purged=%s idempotency_purged=%s",
|
||||
purged,
|
||||
idempotency_purged,
|
||||
)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as exc: # noqa: BLE001 - housekeeping must not stop the API
|
||||
lifecycle_logger.warning(
|
||||
"sessions_purge_failed error=%s", type(exc).__name__
|
||||
)
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(application: FastAPI) -> AsyncIterator[None]:
|
||||
settings: Settings = application.state.settings
|
||||
pool: AsyncConnectionPool = application.state.pool
|
||||
cache: session_service.SessionCache = application.state.cache
|
||||
lifecycle_logger.info(
|
||||
"server_starting env=%s instance_id=%s session_ttl_seconds=%s "
|
||||
"idempotency_ttl_seconds=%s cache_enabled=%s pool_max=%s "
|
||||
"statement_timeout_ms=%s",
|
||||
settings.app_env,
|
||||
settings.instance_id,
|
||||
settings.session_ttl_seconds,
|
||||
settings.idempotency_ttl_seconds,
|
||||
cache.enabled,
|
||||
settings.pool_max_size,
|
||||
settings.statement_timeout_ms,
|
||||
)
|
||||
try:
|
||||
await pool.open(wait=True, timeout=10)
|
||||
except Exception as exc: # noqa: BLE001 - start only with a usable database
|
||||
lifecycle_logger.error(
|
||||
"database_unavailable_at_startup error=%s", type(exc).__name__
|
||||
)
|
||||
raise
|
||||
lifecycle_logger.info(
|
||||
"server_started env=%s instance_id=%s", settings.app_env, settings.instance_id
|
||||
)
|
||||
if cache.enabled:
|
||||
if await cache.ping():
|
||||
lifecycle_logger.info("cache_ready")
|
||||
else:
|
||||
lifecycle_logger.warning("cache_unavailable_using_database_only")
|
||||
else:
|
||||
lifecycle_logger.warning("cache_not_configured_using_database_only")
|
||||
idempotency_stale_rows = await session_service.purge_expired_idempotency(pool)
|
||||
if idempotency_stale_rows:
|
||||
lifecycle_logger.info(
|
||||
"idempotency_purge_at_start count=%s", idempotency_stale_rows
|
||||
)
|
||||
purge_task = asyncio.create_task(_purge_loop(pool))
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
lifecycle_logger.info("shutdown_requested")
|
||||
purge_task.cancel()
|
||||
with contextlib.suppress(BaseException):
|
||||
await purge_task
|
||||
await cache.close()
|
||||
await pool.close()
|
||||
lifecycle_logger.info("server_stopped")
|
||||
|
||||
|
||||
def create_app(settings: Settings | None = None) -> FastAPI:
|
||||
resolved = settings or load_settings()
|
||||
application = FastAPI(
|
||||
title="MTUSI Messenger — лабораторная 3",
|
||||
version="1.3.0",
|
||||
docs_url="/docs",
|
||||
redoc_url=None,
|
||||
lifespan=lifespan,
|
||||
)
|
||||
application.state.settings = resolved
|
||||
application.state.logger = lifecycle_logger
|
||||
application.state.pool = create_pool(
|
||||
resolved.database_url,
|
||||
min_size=resolved.pool_min_size,
|
||||
max_size=resolved.pool_max_size,
|
||||
timeout_seconds=resolved.pool_timeout_seconds,
|
||||
connect_timeout_seconds=resolved.connect_timeout_seconds,
|
||||
statement_timeout_ms=resolved.statement_timeout_ms,
|
||||
)
|
||||
application.state.cache = session_service.SessionCache(
|
||||
resolved.cache_url,
|
||||
timeout_seconds=resolved.cache_timeout_seconds,
|
||||
ttl_cap=resolved.cache_ttl_seconds,
|
||||
)
|
||||
application.openapi = load_contract # type: ignore[method-assign]
|
||||
|
||||
@application.get("/version", include_in_schema=False)
|
||||
async def version() -> dict[str, str]:
|
||||
return {"version": resolved.version, "instance_id": resolved.instance_id}
|
||||
|
||||
@application.get("/info", include_in_schema=False)
|
||||
async def info() -> dict[str, str]:
|
||||
return {"service": "mtusi-messenger", "lab": "3", "version": resolved.version}
|
||||
|
||||
@application.middleware("http")
|
||||
async def request_context(request: Request, call_next: Any) -> Response:
|
||||
request.state.request_id = str(uuid.uuid4())
|
||||
started_at = time.monotonic()
|
||||
status = 500
|
||||
try:
|
||||
response = await call_next(request)
|
||||
if request.method in {"POST", "PUT", "PATCH"}:
|
||||
try:
|
||||
# Authentication or resource checks can finish before the body
|
||||
# is read. Drain any remainder without retaining it so the
|
||||
# client receives that JSON response instead of a TCP reset.
|
||||
async for _chunk in request.stream():
|
||||
pass
|
||||
except RuntimeError:
|
||||
pass
|
||||
status = response.status_code
|
||||
except ApiProblem as problem:
|
||||
response = problem_response(
|
||||
request, problem.status, problem.code, problem.message, problem.headers
|
||||
)
|
||||
status = problem.status
|
||||
except Exception: # noqa: BLE001 - never leak internals to the client
|
||||
lifecycle_logger.exception(
|
||||
"unhandled_error request_id=%s", request.state.request_id
|
||||
)
|
||||
response = problem_response(
|
||||
request, 500, "internal_error", "Internal server error"
|
||||
)
|
||||
status = 500
|
||||
response.headers["X-Request-Id"] = request.state.request_id
|
||||
# Every response carries the instance that served it, including errors,
|
||||
# health and redirects, so a client can prove which process answered.
|
||||
response.headers["X-Instance-Id"] = request.app.state.settings.instance_id
|
||||
# The middleware still owns the response, but the idempotency check now
|
||||
# depends on the endpoint having already derived the Idempotency-Key value
|
||||
# from the contract and the security model, so the header is validated in
|
||||
# api.py rather than here.
|
||||
client = request.client.host if request.client is not None else "unknown"
|
||||
client = request.client.host if request.client is not None else "unknown"
|
||||
access_logger.info(
|
||||
"request_complete request_id=%s method=%s path=%s status=%s "
|
||||
"client=%s elapsed_ms=%.3f",
|
||||
request.state.request_id,
|
||||
request.method,
|
||||
request.url.path,
|
||||
status,
|
||||
client,
|
||||
(time.monotonic() - started_at) * 1000,
|
||||
)
|
||||
return response
|
||||
|
||||
@application.exception_handler(ApiProblem)
|
||||
async def api_problem_handler(request: Request, problem: ApiProblem) -> JSONResponse:
|
||||
return problem_response(
|
||||
request, problem.status, problem.code, problem.message, problem.headers
|
||||
)
|
||||
|
||||
@application.exception_handler(RequestValidationError)
|
||||
async def validation_handler(
|
||||
request: Request, _error: RequestValidationError
|
||||
) -> JSONResponse:
|
||||
return problem_response(
|
||||
request, 400, "invalid_request", "Request fields are invalid"
|
||||
)
|
||||
|
||||
@application.exception_handler(StarletteHTTPException)
|
||||
async def http_error_handler(
|
||||
request: Request, error: StarletteHTTPException
|
||||
) -> JSONResponse:
|
||||
if error.status_code in {404, 405}:
|
||||
return problem_response(request, 404, "not_found", "Endpoint not found")
|
||||
return problem_response(
|
||||
request, error.status_code, "invalid_request", "Request is invalid"
|
||||
)
|
||||
|
||||
@application.exception_handler(PoolTimeout)
|
||||
async def pool_timeout_handler(
|
||||
request: Request, error: PoolTimeout
|
||||
) -> JSONResponse:
|
||||
lifecycle_logger.warning(
|
||||
"database_pool_timeout request_id=%s error=%s",
|
||||
request_id(request),
|
||||
type(error).__name__,
|
||||
)
|
||||
return problem_response(
|
||||
request, 503, "unavailable", "Database is unavailable"
|
||||
)
|
||||
|
||||
@application.exception_handler(psycopg.OperationalError)
|
||||
@application.exception_handler(psycopg.InterfaceError)
|
||||
async def database_unavailable_handler(
|
||||
request: Request, error: Exception
|
||||
) -> JSONResponse:
|
||||
lifecycle_logger.warning(
|
||||
"database_unavailable request_id=%s error=%s",
|
||||
request_id(request),
|
||||
type(error).__name__,
|
||||
)
|
||||
return problem_response(
|
||||
request, 503, "unavailable", "Database is unavailable"
|
||||
)
|
||||
|
||||
@application.exception_handler(psycopg.Error)
|
||||
async def database_error_handler(
|
||||
request: Request, error: psycopg.Error
|
||||
) -> JSONResponse:
|
||||
lifecycle_logger.warning(
|
||||
"database_error request_id=%s error=%s",
|
||||
request_id(request),
|
||||
type(error).__name__,
|
||||
)
|
||||
return problem_response(request, 500, "internal_error", "Internal server error")
|
||||
|
||||
@application.get("/", include_in_schema=False)
|
||||
async def docs_redirect() -> RedirectResponse:
|
||||
return RedirectResponse(url="/docs", status_code=307)
|
||||
|
||||
application.include_router(router)
|
||||
return application
|
||||
+163
@@ -0,0 +1,163 @@
|
||||
"""Runtime configuration: environment variables plus file-based secrets."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
import socket
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
||||
CONTRACT_PATH = PROJECT_ROOT / "contracts" / "openapi.json"
|
||||
MIGRATIONS_DIR = Path(__file__).resolve().parent / "migrations"
|
||||
|
||||
DEFAULT_PAGE_SIZE = 50
|
||||
MAX_PAGE_SIZE = 100
|
||||
MAX_JSON_BYTES = 1_048_576
|
||||
DEFAULT_SESSION_TTL_SECONDS = 3600
|
||||
DEFAULT_CACHE_TTL_SECONDS = 300
|
||||
# Idempotency-Key records are replayed for this long, then the key is reusable.
|
||||
DEFAULT_IDEMPOTENCY_TTL_SECONDS = 86_400
|
||||
SUPPORTED_SESSION_STORES = {"postgres"}
|
||||
|
||||
__version__ = "1.3.0"
|
||||
|
||||
# Key pattern of the contract: 1..128 ASCII characters of the allowed set.
|
||||
IDEMPOTENCY_KEY_PATTERN = re.compile(r"^[A-Za-z0-9._:-]{1,128}$")
|
||||
|
||||
|
||||
class ConfigurationError(Exception):
|
||||
"""Startup configuration is unusable; the process must not serve traffic."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Settings:
|
||||
"""Everything the process needs, resolved once at startup."""
|
||||
|
||||
database_url: str
|
||||
cache_url: str | None
|
||||
session_store: str
|
||||
session_ttl_seconds: int
|
||||
cache_ttl_seconds: int
|
||||
pool_min_size: int
|
||||
pool_max_size: int
|
||||
pool_timeout_seconds: float
|
||||
connect_timeout_seconds: int
|
||||
statement_timeout_ms: int
|
||||
cache_timeout_seconds: float
|
||||
idempotency_ttl_seconds: int
|
||||
instance_id: str
|
||||
app_env: str
|
||||
version: str
|
||||
|
||||
@property
|
||||
def cache_enabled(self) -> bool:
|
||||
return self.cache_url is not None
|
||||
|
||||
|
||||
def _secret(file_variable: str, plain_variable: str) -> str | None:
|
||||
"""Read a secret from a mounted file, falling back to a plain variable."""
|
||||
path_value = os.environ.get(file_variable, "").strip()
|
||||
if path_value:
|
||||
path = Path(path_value)
|
||||
try:
|
||||
value = path.read_text(encoding="utf-8").strip()
|
||||
except OSError as exc:
|
||||
raise ConfigurationError(
|
||||
file_variable + " points to an unreadable file (" + type(exc).__name__ + ")"
|
||||
) from None
|
||||
if not value:
|
||||
raise ConfigurationError(file_variable + " points to an empty file")
|
||||
return value
|
||||
plain = os.environ.get(plain_variable, "").strip()
|
||||
return plain or None
|
||||
|
||||
|
||||
def _integer(variable: str, default: int, minimum: int, maximum: int) -> int:
|
||||
raw = os.environ.get(variable, "").strip()
|
||||
if not raw:
|
||||
return default
|
||||
try:
|
||||
value = int(raw)
|
||||
except ValueError:
|
||||
raise ConfigurationError(variable + " must be an integer") from None
|
||||
if not minimum <= value <= maximum:
|
||||
raise ConfigurationError(
|
||||
variable + " must be between " + str(minimum) + " and " + str(maximum)
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
def _number(variable: str, default: float, minimum: float, maximum: float) -> float:
|
||||
raw = os.environ.get(variable, "").strip()
|
||||
if not raw:
|
||||
return default
|
||||
try:
|
||||
value = float(raw)
|
||||
except ValueError:
|
||||
raise ConfigurationError(variable + " must be a number") from None
|
||||
if not minimum <= value <= maximum:
|
||||
raise ConfigurationError(
|
||||
variable + " must be between " + str(minimum) + " and " + str(maximum)
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
def load_settings() -> Settings:
|
||||
"""Resolve settings once; a bad configuration fails fast before serving."""
|
||||
database_url = _secret("DATABASE_URL_FILE", "DATABASE_URL")
|
||||
if database_url is None:
|
||||
raise ConfigurationError("DATABASE_URL_FILE or DATABASE_URL is required")
|
||||
cache_url = _secret("CACHE_URL_FILE", "CACHE_URL")
|
||||
session_store = os.environ.get("SESSION_STORE", "postgres").strip().lower()
|
||||
if session_store not in SUPPORTED_SESSION_STORES:
|
||||
raise ConfigurationError(
|
||||
"SESSION_STORE="
|
||||
+ session_store
|
||||
+ " is not implemented: sessions are stored in PostgreSQL "
|
||||
"(the durable source of truth) and cached in Valkey. Use SESSION_STORE=postgres."
|
||||
)
|
||||
pool_min = _integer("DB_POOL_MIN", 1, 0, 100)
|
||||
pool_max = _integer("DB_POOL_MAX", 10, 1, 200)
|
||||
if pool_min > pool_max:
|
||||
raise ConfigurationError("DB_POOL_MIN must not exceed DB_POOL_MAX")
|
||||
return Settings(
|
||||
database_url=database_url,
|
||||
cache_url=cache_url,
|
||||
session_store=session_store,
|
||||
session_ttl_seconds=_integer(
|
||||
"SESSION_TTL_SECONDS", DEFAULT_SESSION_TTL_SECONDS, 1, 2_592_000
|
||||
),
|
||||
cache_ttl_seconds=_integer(
|
||||
"CACHE_TTL_SECONDS", DEFAULT_CACHE_TTL_SECONDS, 1, 86_400
|
||||
),
|
||||
pool_min_size=pool_min,
|
||||
pool_max_size=pool_max,
|
||||
pool_timeout_seconds=_number("DB_POOL_TIMEOUT_SECONDS", 2.0, 0.1, 60.0),
|
||||
connect_timeout_seconds=_integer("DB_CONNECT_TIMEOUT_SECONDS", 2, 1, 60),
|
||||
statement_timeout_ms=_integer("DB_STATEMENT_TIMEOUT_MS", 5000, 100, 120_000),
|
||||
cache_timeout_seconds=_number("CACHE_TIMEOUT_SECONDS", 0.5, 0.05, 10.0),
|
||||
idempotency_ttl_seconds=_integer(
|
||||
"IDEMPOTENCY_TTL_SECONDS",
|
||||
DEFAULT_IDEMPOTENCY_TTL_SECONDS,
|
||||
60,
|
||||
604_800,
|
||||
),
|
||||
# Stable for the life of the process. Inside a container this is the
|
||||
# short container id, which is what scripts/stop-one.sh maps back to a
|
||||
# single replica; outside Docker it stays a readable hostname.
|
||||
instance_id=os.environ.get("INSTANCE_ID", "").strip() or socket.gethostname(),
|
||||
app_env=os.environ.get("APP_ENV", "lab").strip() or "lab",
|
||||
version=__version__,
|
||||
)
|
||||
|
||||
|
||||
def load_address() -> tuple[str, int]:
|
||||
"""Read and validate the HTTP listen address from the environment."""
|
||||
host = os.environ.get("APP_HOST", "0.0.0.0").strip()
|
||||
if not host:
|
||||
raise SystemExit("APP_HOST must not be empty")
|
||||
port = _integer("APP_PORT", 8080, 1, 65535)
|
||||
return host, port
|
||||
@@ -0,0 +1,49 @@
|
||||
"""PostgreSQL access: a bounded async pool with explicit timeouts."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from psycopg.rows import dict_row
|
||||
from psycopg_pool import AsyncConnectionPool
|
||||
|
||||
|
||||
def connection_options(statement_timeout_ms: int) -> str:
|
||||
"""Server-side limits so a slow query or an idle transaction cannot pile up."""
|
||||
return (
|
||||
"-c statement_timeout="
|
||||
+ str(statement_timeout_ms)
|
||||
+ " -c lock_timeout=2000 -c idle_in_transaction_session_timeout=10000"
|
||||
+ " -c timezone=UTC"
|
||||
)
|
||||
|
||||
|
||||
def create_pool(
|
||||
database_url: str,
|
||||
*,
|
||||
min_size: int,
|
||||
max_size: int,
|
||||
timeout_seconds: float,
|
||||
connect_timeout_seconds: int,
|
||||
statement_timeout_ms: int,
|
||||
) -> AsyncConnectionPool:
|
||||
"""Build a closed pool; the lifespan opens it and closes it on shutdown."""
|
||||
return AsyncConnectionPool(
|
||||
conninfo=database_url,
|
||||
min_size=min_size,
|
||||
max_size=max_size,
|
||||
timeout=timeout_seconds,
|
||||
open=False,
|
||||
check=AsyncConnectionPool.check_connection,
|
||||
kwargs={
|
||||
"row_factory": dict_row,
|
||||
"connect_timeout": connect_timeout_seconds,
|
||||
"options": connection_options(statement_timeout_ms),
|
||||
"application_name": "mtusi-messenger-api",
|
||||
},
|
||||
name="messenger-pool",
|
||||
)
|
||||
|
||||
|
||||
async def ping(pool: AsyncConnectionPool) -> None:
|
||||
"""Cheap readiness probe; raises when PostgreSQL is not usable."""
|
||||
async with pool.connection() as connection:
|
||||
await connection.execute("SELECT 1")
|
||||
@@ -0,0 +1,26 @@
|
||||
"""Errors that may safely be returned through the public API."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Mapping
|
||||
|
||||
|
||||
BASIC_CHALLENGE = 'Basic realm="messenger", charset="UTF-8"'
|
||||
BEARER_CHALLENGE = 'Bearer realm="messenger"'
|
||||
|
||||
|
||||
class ApiProblem(Exception):
|
||||
"""An expected API failure with a stable status, machine code and headers."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
status: int,
|
||||
code: str,
|
||||
message: str,
|
||||
headers: Mapping[str, str] | None = None,
|
||||
) -> None:
|
||||
super().__init__(message)
|
||||
self.status = status
|
||||
self.code = code
|
||||
self.message = message
|
||||
self.headers = dict(headers or {})
|
||||
@@ -0,0 +1,93 @@
|
||||
"""Versioned SQL migration runner.
|
||||
|
||||
Every run is one transaction guarded by a PostgreSQL advisory lock, so a second
|
||||
process (or a restarted container) cannot apply the same file twice and a failed
|
||||
migration leaves the schema unchanged.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import psycopg
|
||||
|
||||
if __package__ in {None, ""}:
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
|
||||
from src.config import MIGRATIONS_DIR, ConfigurationError, load_settings # noqa: E402
|
||||
|
||||
|
||||
ADVISORY_LOCK_KEY = 915_127_001
|
||||
CONNECT_RETRY_SECONDS = 60
|
||||
|
||||
|
||||
def _version(path: Path) -> int:
|
||||
prefix = path.name.split("_", 1)[0]
|
||||
return int(prefix)
|
||||
|
||||
|
||||
def apply_migrations(connection: psycopg.Connection, directory: Path) -> list[str]:
|
||||
applied: list[str] = []
|
||||
with connection.transaction():
|
||||
connection.execute("SELECT pg_advisory_xact_lock(%s)", (ADVISORY_LOCK_KEY,))
|
||||
connection.execute(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS schema_migrations (
|
||||
version integer PRIMARY KEY,
|
||||
filename text NOT NULL,
|
||||
applied_at timestamptz NOT NULL DEFAULT now()
|
||||
)
|
||||
"""
|
||||
)
|
||||
rows = connection.execute("SELECT version FROM schema_migrations").fetchall()
|
||||
done = {row[0] for row in rows}
|
||||
for path in sorted(directory.glob("*.sql"), key=_version):
|
||||
version = _version(path)
|
||||
if version in done:
|
||||
continue
|
||||
connection.execute(path.read_text(encoding="utf-8"))
|
||||
connection.execute(
|
||||
"INSERT INTO schema_migrations (version, filename) VALUES (%s, %s)",
|
||||
(version, path.name),
|
||||
)
|
||||
applied.append(path.name)
|
||||
return applied
|
||||
|
||||
|
||||
def main() -> None:
|
||||
try:
|
||||
settings = load_settings()
|
||||
except ConfigurationError as error:
|
||||
print("configuration_error " + str(error), file=sys.stderr)
|
||||
raise SystemExit(1) from None
|
||||
deadline = time.monotonic() + CONNECT_RETRY_SECONDS
|
||||
while True:
|
||||
try:
|
||||
with psycopg.connect(
|
||||
settings.database_url,
|
||||
connect_timeout=5,
|
||||
options="-c timezone=UTC",
|
||||
application_name="mtusi-messenger-migrate",
|
||||
) as connection:
|
||||
applied = apply_migrations(connection, MIGRATIONS_DIR)
|
||||
break
|
||||
except psycopg.OperationalError as error:
|
||||
if time.monotonic() >= deadline:
|
||||
print(
|
||||
"database_unreachable error=" + type(error).__name__, file=sys.stderr
|
||||
)
|
||||
raise SystemExit(1) from None
|
||||
time.sleep(1)
|
||||
except psycopg.Error as error:
|
||||
print("migration_failed error=" + type(error).__name__, file=sys.stderr)
|
||||
raise SystemExit(1) from None
|
||||
if applied:
|
||||
print("migrations_applied " + ",".join(applied))
|
||||
else:
|
||||
print("migrations_up_to_date")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,13 @@
|
||||
-- Users own credentials and identity. The password column stores an Argon2id hash.
|
||||
CREATE TABLE users (
|
||||
id uuid PRIMARY KEY,
|
||||
username text NOT NULL,
|
||||
display_name text NOT NULL,
|
||||
password_hash text NOT NULL,
|
||||
created_at timestamptz NOT NULL,
|
||||
CONSTRAINT users_username_format CHECK (username ~ '^[a-z0-9_]{3,32}$'),
|
||||
CONSTRAINT users_display_name_length CHECK (char_length(display_name) BETWEEN 1 AND 100),
|
||||
CONSTRAINT users_password_hash_present CHECK (char_length(password_hash) > 0)
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX users_username_unique ON users (username);
|
||||
@@ -0,0 +1,34 @@
|
||||
-- Chats, explicit membership and message history.
|
||||
CREATE TABLE chats (
|
||||
id uuid PRIMARY KEY,
|
||||
title text NOT NULL,
|
||||
created_at timestamptz NOT NULL,
|
||||
CONSTRAINT chats_title_length CHECK (char_length(title) BETWEEN 1 AND 100)
|
||||
);
|
||||
|
||||
-- Chat lists are ordered by (created_at, id) for the same keyset cursor.
|
||||
CREATE INDEX chats_order_idx ON chats (created_at, id);
|
||||
|
||||
CREATE TABLE chat_members (
|
||||
chat_id uuid NOT NULL REFERENCES chats (id) ON DELETE CASCADE,
|
||||
user_id uuid NOT NULL REFERENCES users (id) ON DELETE RESTRICT,
|
||||
joined_at timestamptz NOT NULL,
|
||||
PRIMARY KEY (chat_id, user_id)
|
||||
);
|
||||
|
||||
-- "Chats of the current user" and the membership check on every read.
|
||||
CREATE INDEX chat_members_user_idx ON chat_members (user_id, chat_id);
|
||||
|
||||
CREATE TABLE messages (
|
||||
id uuid PRIMARY KEY,
|
||||
chat_id uuid NOT NULL REFERENCES chats (id) ON DELETE CASCADE,
|
||||
sender_id uuid NOT NULL REFERENCES users (id) ON DELETE RESTRICT,
|
||||
text text NOT NULL,
|
||||
created_at timestamptz NOT NULL,
|
||||
CONSTRAINT messages_text_length CHECK (char_length(text) BETWEEN 1 AND 4000)
|
||||
);
|
||||
|
||||
-- Serves history keyset pagination exactly: equality on chat_id, then the
|
||||
-- (created_at, id) tuple used by both ORDER BY and the cursor predicate.
|
||||
-- Justification and EXPLAIN (ANALYZE, BUFFERS) are in evidence/explain-history.txt.
|
||||
CREATE INDEX messages_chat_order_idx ON messages (chat_id, created_at, id);
|
||||
@@ -0,0 +1,16 @@
|
||||
-- Opaque server-side sessions: only a SHA-256 of the token is stored.
|
||||
CREATE TABLE sessions (
|
||||
id uuid PRIMARY KEY,
|
||||
user_id uuid NOT NULL REFERENCES users (id) ON DELETE CASCADE,
|
||||
token_hash text NOT NULL,
|
||||
created_at timestamptz NOT NULL,
|
||||
expires_at timestamptz NOT NULL,
|
||||
revoked_at timestamptz,
|
||||
CONSTRAINT sessions_token_hash_length CHECK (char_length(token_hash) = 64),
|
||||
CONSTRAINT sessions_expiry_after_creation CHECK (expires_at > created_at)
|
||||
);
|
||||
|
||||
-- Every authenticated request looks a token up by hash.
|
||||
CREATE UNIQUE INDEX sessions_token_hash_unique ON sessions (token_hash);
|
||||
CREATE INDEX sessions_user_idx ON sessions (user_id);
|
||||
CREATE INDEX sessions_expiry_idx ON sessions (expires_at);
|
||||
@@ -0,0 +1,11 @@
|
||||
-- Idempotency-Key records for the lab 3 message write path.
|
||||
CREATE TABLE idempotency_keys (
|
||||
key text PRIMARY KEY,
|
||||
chat_id uuid NOT NULL REFERENCES chats (id) ON DELETE CASCADE,
|
||||
request_sha256 text NOT NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
CONSTRAINT idempotency_key_format CHECK (char_length(key) <= 128),
|
||||
CONSTRAINT idempotency_request_sha256_length CHECK (char_length(request_sha256) = 64)
|
||||
);
|
||||
|
||||
CREATE INDEX idempotency_chat_idx ON idempotency_keys (chat_id);
|
||||
@@ -0,0 +1,2 @@
|
||||
-- Composite index covering the idempotency queries: by chat and by key.
|
||||
CREATE INDEX idempotency_keys_chat_order_idx ON idempotency_keys (chat_id, key);
|
||||
@@ -0,0 +1,3 @@
|
||||
-- Add message_id to idempotency_keys so replays can return the original message.
|
||||
ALTER TABLE idempotency_keys ADD COLUMN IF NOT EXISTS message_id uuid REFERENCES messages (id) ON DELETE CASCADE;
|
||||
CREATE INDEX IF NOT EXISTS idempotency_message_idx ON idempotency_keys (message_id);
|
||||
@@ -0,0 +1,25 @@
|
||||
-- An idempotency key belongs to the authenticated user and target chat, not
|
||||
-- to the entire service. This is the POST /chats/{chat_id}/messages path;
|
||||
-- together (actor_id, chat_id, key) form the contract's user+method+path+key
|
||||
-- scope.
|
||||
ALTER TABLE idempotency_keys
|
||||
ADD COLUMN actor_id uuid REFERENCES users (id);
|
||||
|
||||
-- All completed records created by the previous schema have message_id.
|
||||
-- Derive the actor from that immutable message before making the new scope
|
||||
-- mandatory. Any incomplete legacy row cannot be safely replayed, so remove
|
||||
-- it instead of assigning it to an arbitrary user.
|
||||
UPDATE idempotency_keys AS ik
|
||||
SET actor_id = m.sender_id
|
||||
FROM messages AS m
|
||||
WHERE ik.message_id = m.id
|
||||
AND ik.actor_id IS NULL;
|
||||
|
||||
DELETE FROM idempotency_keys WHERE actor_id IS NULL;
|
||||
|
||||
ALTER TABLE idempotency_keys
|
||||
ALTER COLUMN actor_id SET NOT NULL,
|
||||
DROP CONSTRAINT idempotency_keys_pkey,
|
||||
ADD PRIMARY KEY (actor_id, chat_id, key);
|
||||
|
||||
CREATE INDEX idempotency_created_at_idx ON idempotency_keys (created_at);
|
||||
@@ -0,0 +1,80 @@
|
||||
"""Strict request models for the lab 2 API."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import uuid
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator
|
||||
|
||||
from .config import IDEMPOTENCY_KEY_PATTERN
|
||||
|
||||
|
||||
USERNAME_PATTERN = re.compile(r"^[a-z0-9_]{3,32}$")
|
||||
PASSWORD_MIN_LENGTH = 12
|
||||
PASSWORD_MAX_LENGTH = 128
|
||||
|
||||
|
||||
class StrictModel(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", strict=True)
|
||||
|
||||
|
||||
def _not_blank(value: str, field: str) -> str:
|
||||
if not value.strip():
|
||||
raise ValueError(field + " must not be blank")
|
||||
return value
|
||||
|
||||
|
||||
class CreateUser(StrictModel):
|
||||
"""Registration body; the password never appears in a response."""
|
||||
|
||||
username: str = Field(
|
||||
min_length=3, max_length=32, pattern=USERNAME_PATTERN.pattern
|
||||
)
|
||||
display_name: str = Field(min_length=1, max_length=100)
|
||||
password: str = Field(
|
||||
min_length=PASSWORD_MIN_LENGTH, max_length=PASSWORD_MAX_LENGTH
|
||||
)
|
||||
|
||||
@field_validator("display_name")
|
||||
@classmethod
|
||||
def display_name_is_not_blank(cls, value: str) -> str:
|
||||
return _not_blank(value, "display_name")
|
||||
|
||||
@field_validator("password")
|
||||
@classmethod
|
||||
def password_length_in_code_points(cls, value: str) -> str:
|
||||
# Python len() counts Unicode code points, which is what the contract uses.
|
||||
if not PASSWORD_MIN_LENGTH <= len(value) <= PASSWORD_MAX_LENGTH:
|
||||
raise ValueError("password length is out of range")
|
||||
return value
|
||||
|
||||
|
||||
class CreateChat(StrictModel):
|
||||
title: str = Field(min_length=1, max_length=100)
|
||||
member_ids: list[str] = Field(min_length=1, max_length=100)
|
||||
|
||||
@field_validator("title")
|
||||
@classmethod
|
||||
def title_is_not_blank(cls, value: str) -> str:
|
||||
return _not_blank(value, "title")
|
||||
|
||||
@field_validator("member_ids")
|
||||
@classmethod
|
||||
def members_are_valid_and_unique(cls, value: list[str]) -> list[str]:
|
||||
try:
|
||||
normalized = [str(uuid.UUID(member_id)) for member_id in value]
|
||||
except (ValueError, AttributeError):
|
||||
raise ValueError("member_ids must contain UUID strings") from None
|
||||
if len(normalized) != len(set(normalized)):
|
||||
raise ValueError("member_ids must be unique")
|
||||
return normalized
|
||||
|
||||
|
||||
class CreateMessage(StrictModel):
|
||||
text: str = Field(min_length=1, max_length=4000)
|
||||
|
||||
@field_validator("text")
|
||||
@classmethod
|
||||
def text_is_not_blank(cls, value: str) -> str:
|
||||
return _not_blank(value, "text")
|
||||
@@ -0,0 +1,84 @@
|
||||
"""Opaque, resource-bound keyset cursors, timestamp formatting and page shaping."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import binascii
|
||||
import json
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
|
||||
from .errors import ApiProblem
|
||||
|
||||
|
||||
def format_timestamp(value: datetime) -> str:
|
||||
"""RFC 3339 UTC with microseconds and a trailing Z, as the contract requires."""
|
||||
return (
|
||||
value.astimezone(timezone.utc)
|
||||
.isoformat(timespec="microseconds")
|
||||
.replace("+00:00", "Z")
|
||||
)
|
||||
|
||||
|
||||
def parse_timestamp(value: str) -> datetime:
|
||||
"""Parse an RFC 3339 timestamp from a cursor; reject anything else."""
|
||||
candidate = value[:-1] + "+00:00" if value.endswith("Z") else value
|
||||
parsed = datetime.fromisoformat(candidate)
|
||||
if parsed.tzinfo is None:
|
||||
parsed = parsed.replace(tzinfo=timezone.utc)
|
||||
return parsed.astimezone(timezone.utc)
|
||||
|
||||
|
||||
def encode_cursor(resource: str, created_at: datetime, record_id: str) -> str:
|
||||
payload = json.dumps(
|
||||
{
|
||||
"resource": resource,
|
||||
"created_at": format_timestamp(created_at),
|
||||
"id": record_id,
|
||||
},
|
||||
separators=(",", ":"),
|
||||
sort_keys=True,
|
||||
).encode("utf-8")
|
||||
return base64.urlsafe_b64encode(payload).decode("ascii").rstrip("=")
|
||||
|
||||
|
||||
def decode_cursor(value: str, resource: str) -> tuple[datetime, uuid.UUID]:
|
||||
"""Decode a cursor and bind it to the resource it was issued for."""
|
||||
try:
|
||||
padding = "=" * (-len(value) % 4)
|
||||
raw = base64.b64decode(value + padding, altchars=b"-_", validate=True)
|
||||
payload: Any = json.loads(raw.decode("utf-8"))
|
||||
if not isinstance(payload, dict) or set(payload) != {
|
||||
"resource",
|
||||
"created_at",
|
||||
"id",
|
||||
}:
|
||||
raise ValueError("unexpected cursor payload")
|
||||
if payload["resource"] != resource:
|
||||
raise ValueError("cursor belongs to another resource")
|
||||
created_at = payload["created_at"]
|
||||
record_id = payload["id"]
|
||||
if not isinstance(created_at, str) or not isinstance(record_id, str):
|
||||
raise ValueError("cursor fields have wrong types")
|
||||
return parse_timestamp(created_at), uuid.UUID(record_id)
|
||||
except (
|
||||
ValueError,
|
||||
TypeError,
|
||||
UnicodeError,
|
||||
binascii.Error,
|
||||
json.JSONDecodeError,
|
||||
):
|
||||
raise ApiProblem(400, "invalid_request", "Cursor is invalid") from None
|
||||
|
||||
|
||||
def build_page(items: list[dict[str, Any]], resource: str, limit: int) -> dict[str, Any]:
|
||||
"""Shape a limit+1 storage slice into the public page and its next cursor."""
|
||||
page = items[:limit]
|
||||
next_cursor = None
|
||||
if len(items) > limit and page:
|
||||
last = page[-1]
|
||||
next_cursor = encode_cursor(
|
||||
resource, parse_timestamp(last["created_at"]), last["id"]
|
||||
)
|
||||
return {"items": page, "next_cursor": next_cursor}
|
||||
@@ -0,0 +1,43 @@
|
||||
"""Password hashing with Argon2id (library KDF, salted, versioned parameters)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from argon2 import PasswordHasher
|
||||
from argon2.exceptions import InvalidHashError, VerificationError, VerifyMismatchError
|
||||
|
||||
|
||||
# OWASP-recommended Argon2id baseline: 19 MiB, 2 iterations, 1 lane.
|
||||
_hasher = PasswordHasher(
|
||||
time_cost=2,
|
||||
memory_cost=19456,
|
||||
parallelism=1,
|
||||
hash_len=32,
|
||||
salt_len=16,
|
||||
)
|
||||
|
||||
# Verified against when a username is unknown, so the failure path costs the same
|
||||
# as a wrong password and does not leak username existence through timing.
|
||||
_DUMMY_HASH = _hasher.hash("dummy-password-not-used-for-any-account")
|
||||
|
||||
|
||||
def hash_password(password: str) -> str:
|
||||
return _hasher.hash(password)
|
||||
|
||||
|
||||
def verify_password(password: str, encoded: str | None) -> bool:
|
||||
"""Return True only for a matching password and a parseable hash."""
|
||||
if encoded is None:
|
||||
verify_password_unknown_user(password)
|
||||
return False
|
||||
try:
|
||||
return _hasher.verify(encoded, password)
|
||||
except (VerifyMismatchError, VerificationError, InvalidHashError):
|
||||
return False
|
||||
|
||||
|
||||
def verify_password_unknown_user(password: str) -> None:
|
||||
"""Burn comparable CPU time when no account exists for the username."""
|
||||
try:
|
||||
_hasher.verify(_DUMMY_HASH, password)
|
||||
except (VerifyMismatchError, VerificationError, InvalidHashError):
|
||||
return
|
||||
+337
@@ -0,0 +1,337 @@
|
||||
"""Opaque Bearer sessions: PostgreSQL is the source of truth, Valkey is a cache.
|
||||
|
||||
Design (see REPORT.md):
|
||||
|
||||
* ``sessions`` in PostgreSQL owns identity, expiry and revocation.
|
||||
* Valkey stores the same session under ``session:<sha256(token)>`` for at most
|
||||
``CACHE_TTL_SECONDS`` and never longer than the session's remaining lifetime.
|
||||
* Revocation writes a tombstone instead of relying on ``DEL`` alone, so a stale
|
||||
cache entry cannot authenticate after logout.
|
||||
* When Valkey is unavailable every request falls back to PostgreSQL. A cache
|
||||
error is never treated as "authenticated", and a cache hit is only trusted
|
||||
because it came from a session row that PostgreSQL confirmed at login time.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import secrets
|
||||
import time
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
|
||||
import redis.asyncio as redis
|
||||
from psycopg_pool import AsyncConnectionPool
|
||||
from redis.backoff import NoBackoff
|
||||
from redis.retry import Retry
|
||||
|
||||
from . import store
|
||||
from .config import DEFAULT_IDEMPOTENCY_TTL_SECONDS as IDEMPOTENCY_TTL_SECONDS
|
||||
from .errors import ApiProblem
|
||||
from .pagination import format_timestamp
|
||||
|
||||
logger = logging.getLogger("messenger.sessions")
|
||||
|
||||
TOKEN_BYTES = 48 # 384 bits of entropy, well above the required 256.
|
||||
CACHE_PREFIX = "session:"
|
||||
REVOKED = "revoked"
|
||||
# After a cache error, skip Valkey for a few seconds instead of paying the
|
||||
# connect timeout on every request (lookup and remember would cost two each).
|
||||
FAILURE_COOLDOWN_SECONDS = 5.0
|
||||
# Bound on revocations remembered while the cache cannot accept a tombstone.
|
||||
MAX_PENDING_REVOCATIONS = 4096
|
||||
|
||||
HIT = "hit"
|
||||
MISS = "miss"
|
||||
TOMBSTONE = "revoked"
|
||||
ERROR = "error"
|
||||
|
||||
|
||||
def token_hash(token: str) -> str:
|
||||
return hashlib.sha256(token.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
class SessionCache:
|
||||
"""Small, failure-tolerant Valkey client used only for session lookups."""
|
||||
|
||||
def __init__(self, url: str | None, *, timeout_seconds: float, ttl_cap: int) -> None:
|
||||
self.ttl_cap = max(1, ttl_cap)
|
||||
self.timeout_seconds = timeout_seconds
|
||||
# hashed token -> epoch at which the session would be expired anyway.
|
||||
# PostgreSQL already holds the authoritative revocation; this only keeps
|
||||
# logout immediate for this process while the cache is unreachable.
|
||||
self._pending: dict[str, float] = {}
|
||||
self._disabled_until = 0.0
|
||||
self._degraded = False
|
||||
self.client: redis.Redis | None = None
|
||||
if url:
|
||||
self.client = redis.Redis.from_url(
|
||||
url,
|
||||
socket_timeout=timeout_seconds,
|
||||
socket_connect_timeout=timeout_seconds,
|
||||
decode_responses=True,
|
||||
health_check_interval=30,
|
||||
# A cache must fail fast: one attempt, no retry loop, so a dead
|
||||
# Valkey costs at most one timeout before PostgreSQL answers.
|
||||
retry=Retry(NoBackoff(), 0),
|
||||
retry_on_error=[],
|
||||
)
|
||||
|
||||
@property
|
||||
def enabled(self) -> bool:
|
||||
return self.client is not None
|
||||
|
||||
def _ready(self) -> bool:
|
||||
"""True when a cache call is worth attempting right now."""
|
||||
return self.client is not None and time.monotonic() >= self._disabled_until
|
||||
|
||||
def _unavailable(self, operation: str, exc: BaseException) -> None:
|
||||
self._disabled_until = time.monotonic() + FAILURE_COOLDOWN_SECONDS
|
||||
# One warning per outage, not one per request.
|
||||
if not self._degraded:
|
||||
self._degraded = True
|
||||
logger.warning(
|
||||
"cache_unavailable operation=%s error=%s cooldown_seconds=%s",
|
||||
operation,
|
||||
type(exc).__name__,
|
||||
FAILURE_COOLDOWN_SECONDS,
|
||||
)
|
||||
|
||||
def _ok(self) -> None:
|
||||
self._degraded = False
|
||||
|
||||
def remember_revocation(self, hashed: str, remaining: float) -> None:
|
||||
"""Ask this process to reject a token even if the cache cannot store it."""
|
||||
if len(self._pending) >= MAX_PENDING_REVOCATIONS:
|
||||
# Drop the entries closest to expiry: they are the least useful.
|
||||
keep = len(self._pending) // 4
|
||||
for key, _ in sorted(self._pending.items(), key=lambda item: item[1])[:keep]:
|
||||
self._pending.pop(key, None)
|
||||
self._pending[hashed] = time.time() + max(1.0, remaining)
|
||||
|
||||
def is_locally_revoked(self, hashed: str) -> bool:
|
||||
deadline = self._pending.get(hashed)
|
||||
if deadline is None:
|
||||
return False
|
||||
if deadline <= time.time():
|
||||
self._pending.pop(hashed, None)
|
||||
return False
|
||||
return True
|
||||
|
||||
async def _flush_revocations(self) -> None:
|
||||
"""Re-apply tombstones that could not be written during an outage."""
|
||||
for hashed, deadline in list(self._pending.items()):
|
||||
remaining = deadline - time.time()
|
||||
if remaining <= 0:
|
||||
self._pending.pop(hashed, None)
|
||||
continue
|
||||
try:
|
||||
await self.client.set(self._key(hashed), REVOKED, ex=self._ttl(remaining))
|
||||
except Exception as exc: # noqa: BLE001 - still revoked in PostgreSQL
|
||||
self._unavailable("flush_revocation", exc)
|
||||
return
|
||||
self._pending.pop(hashed, None)
|
||||
|
||||
async def ping(self) -> bool:
|
||||
if self.client is None:
|
||||
return False
|
||||
try:
|
||||
alive = bool(await self.client.ping())
|
||||
except Exception as exc: # noqa: BLE001 - any client error means "no cache"
|
||||
logger.warning("cache_ping_failed error=%s", type(exc).__name__)
|
||||
return False
|
||||
self._disabled_until = 0.0
|
||||
self._ok()
|
||||
return alive
|
||||
|
||||
async def close(self) -> None:
|
||||
if self.client is None:
|
||||
return
|
||||
try:
|
||||
await self.client.aclose()
|
||||
except Exception as exc: # noqa: BLE001 - shutdown must not fail on cache
|
||||
logger.warning("cache_close_failed error=%s", type(exc).__name__)
|
||||
|
||||
def _key(self, hashed: str) -> str:
|
||||
return CACHE_PREFIX + hashed
|
||||
|
||||
def _ttl(self, seconds: float) -> int:
|
||||
return max(1, min(int(seconds), self.ttl_cap))
|
||||
|
||||
async def lookup(self, hashed: str) -> tuple[str, dict[str, Any] | None]:
|
||||
"""Return (state, payload); every cache failure degrades to a DB read."""
|
||||
if not self._ready():
|
||||
return MISS, None
|
||||
if self._pending:
|
||||
await self._flush_revocations()
|
||||
if not self._ready():
|
||||
return MISS, None
|
||||
try:
|
||||
raw = await self.client.get(self._key(hashed))
|
||||
self._ok()
|
||||
except Exception as exc: # noqa: BLE001 - cache must never break auth
|
||||
self._unavailable("lookup", exc)
|
||||
return ERROR, None
|
||||
if raw is None:
|
||||
return MISS, None
|
||||
if raw == REVOKED:
|
||||
return TOMBSTONE, None
|
||||
try:
|
||||
payload = json.loads(raw)
|
||||
user = payload["user"]
|
||||
user["id"] = str(uuid.UUID(user["id"]))
|
||||
payload["expires_epoch"] = float(payload["expires_epoch"])
|
||||
payload["user"] = user
|
||||
except (ValueError, KeyError, TypeError, json.JSONDecodeError):
|
||||
logger.warning("cache_payload_invalid")
|
||||
return MISS, None
|
||||
return HIT, payload
|
||||
|
||||
async def remember(self, hashed: str, user: dict[str, Any], expires_at: datetime) -> None:
|
||||
if not self._ready():
|
||||
return
|
||||
remaining = (expires_at - datetime.now(timezone.utc)).total_seconds()
|
||||
if remaining <= 0:
|
||||
return
|
||||
# Cache the immutable public user snapshot, so a hit needs no SQL at all.
|
||||
# Users cannot be renamed or deleted in the mandatory API of lab 2.
|
||||
payload = {
|
||||
"user": user,
|
||||
"expires_epoch": expires_at.timestamp(),
|
||||
}
|
||||
try:
|
||||
await self.client.set(
|
||||
self._key(hashed),
|
||||
json.dumps(payload, ensure_ascii=False),
|
||||
ex=self._ttl(remaining),
|
||||
)
|
||||
self._ok()
|
||||
except Exception as exc: # noqa: BLE001 - a missed cache write is not fatal
|
||||
self._unavailable("remember", exc)
|
||||
|
||||
async def revoke(self, hashed: str, expires_at: datetime | None) -> None:
|
||||
"""Write a tombstone so a stale entry cannot authenticate after logout."""
|
||||
remaining = 60.0
|
||||
if expires_at is not None:
|
||||
remaining = max(1.0, (expires_at - datetime.now(timezone.utc)).total_seconds())
|
||||
# PostgreSQL already revoked the session; this keeps the fast path honest
|
||||
# even if the tombstone cannot be written right now.
|
||||
self.remember_revocation(hashed, remaining)
|
||||
if not self._ready():
|
||||
return
|
||||
try:
|
||||
await self.client.set(self._key(hashed), REVOKED, ex=self._ttl(remaining))
|
||||
self._pending.pop(hashed, None)
|
||||
self._ok()
|
||||
except Exception as exc: # noqa: BLE001 - PostgreSQL already revoked the session
|
||||
self._unavailable("revoke", exc)
|
||||
|
||||
|
||||
def _user_payload(row: dict[str, Any]) -> dict[str, Any]:
|
||||
return {
|
||||
"id": str(row["id"]),
|
||||
"username": row["username"],
|
||||
"display_name": row["display_name"],
|
||||
"created_at": format_timestamp(row["created_at"]),
|
||||
}
|
||||
|
||||
|
||||
async def create_session(
|
||||
pool: AsyncConnectionPool,
|
||||
cache: SessionCache,
|
||||
*,
|
||||
user_id: uuid.UUID,
|
||||
ttl_seconds: int,
|
||||
) -> tuple[str, datetime]:
|
||||
token = secrets.token_urlsafe(TOKEN_BYTES)
|
||||
hashed = token_hash(token)
|
||||
async with pool.connection() as connection:
|
||||
async with connection.cursor() as cursor:
|
||||
expires_at = await store.insert_session(
|
||||
cursor, user_id=user_id, token_hash=hashed, ttl_seconds=ttl_seconds
|
||||
)
|
||||
user = await store.user_by_id(cursor, user_id)
|
||||
if user is not None:
|
||||
await cache.remember(hashed, _user_payload(user), expires_at)
|
||||
return token, expires_at
|
||||
|
||||
|
||||
async def resolve_session(
|
||||
pool: AsyncConnectionPool, cache: SessionCache, token: str
|
||||
) -> tuple[uuid.UUID, dict[str, Any]]:
|
||||
"""Return the authenticated user or raise 401 with a Bearer challenge."""
|
||||
hashed = token_hash(token)
|
||||
if cache.is_locally_revoked(hashed):
|
||||
raise ApiProblem(
|
||||
401,
|
||||
"unauthorized",
|
||||
"Session token is expired or revoked",
|
||||
{"WWW-Authenticate": 'Bearer realm="messenger"'},
|
||||
)
|
||||
state, _payload = await cache.lookup(hashed)
|
||||
if state == TOMBSTONE:
|
||||
raise ApiProblem(
|
||||
401,
|
||||
"unauthorized",
|
||||
"Session token is expired or revoked",
|
||||
{"WWW-Authenticate": 'Bearer realm="messenger"'},
|
||||
)
|
||||
|
||||
# PostgreSQL is the authoritative revocation store. A cache value can only
|
||||
# accelerate the immutable user snapshot; it must never be enough to grant
|
||||
# access on its own. This makes logout durable across API replicas and
|
||||
# Valkey outages/restores, where a tombstone may not reach shared cache.
|
||||
async with pool.connection() as connection:
|
||||
async with connection.cursor() as cursor:
|
||||
row = await store.session_user(cursor, hashed)
|
||||
if row is None:
|
||||
await cache.revoke(hashed, None)
|
||||
raise ApiProblem(
|
||||
401,
|
||||
"unauthorized",
|
||||
"Session token is expired or revoked",
|
||||
{"WWW-Authenticate": 'Bearer realm="messenger"'},
|
||||
)
|
||||
user = _user_payload(row)
|
||||
# Refresh an old cache hit after PostgreSQL has confirmed it is still live.
|
||||
await cache.remember(hashed, user, row["expires_at"])
|
||||
return uuid.UUID(user["id"]), user
|
||||
|
||||
|
||||
async def revoke_session(
|
||||
pool: AsyncConnectionPool, cache: SessionCache, token: str
|
||||
) -> None:
|
||||
hashed = token_hash(token)
|
||||
async with pool.connection() as connection:
|
||||
async with connection.cursor() as cursor:
|
||||
expires_at = await store.revoke_session(cursor, hashed)
|
||||
if expires_at is None:
|
||||
raise ApiProblem(
|
||||
401,
|
||||
"unauthorized",
|
||||
"Session token is expired or revoked",
|
||||
{"WWW-Authenticate": 'Bearer realm="messenger"'},
|
||||
)
|
||||
await cache.revoke(hashed, expires_at)
|
||||
|
||||
|
||||
async def purge_expired(pool: AsyncConnectionPool) -> int:
|
||||
async with pool.connection() as connection:
|
||||
async with connection.cursor() as cursor:
|
||||
return await store.purge_expired_sessions(cursor)
|
||||
|
||||
|
||||
async def purge_expired_idempotency(pool: AsyncConnectionPool) -> int:
|
||||
async with pool.connection() as connection:
|
||||
async with connection.cursor() as cursor:
|
||||
await cursor.execute(
|
||||
"""
|
||||
DELETE FROM idempotency_keys
|
||||
WHERE created_at < now() - (%s || ' seconds')::interval
|
||||
""",
|
||||
(str(IDEMPOTENCY_TTL_SECONDS),),
|
||||
)
|
||||
return cursor.rowcount
|
||||
+363
@@ -0,0 +1,363 @@
|
||||
"""PostgreSQL repository: parameterised statements only, explicit transactions."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
import psycopg
|
||||
from psycopg import AsyncCursor
|
||||
|
||||
from .errors import ApiProblem
|
||||
from .pagination import format_timestamp
|
||||
|
||||
# One expression keeps the member list of a chat consistent across endpoints.
|
||||
CHAT_COLUMNS = """
|
||||
c.id,
|
||||
c.title,
|
||||
c.created_at,
|
||||
ARRAY(
|
||||
SELECT m.user_id::text
|
||||
FROM chat_members m
|
||||
WHERE m.chat_id = c.id
|
||||
ORDER BY m.user_id::text
|
||||
) AS member_ids
|
||||
"""
|
||||
|
||||
|
||||
def user_json(row: dict[str, Any]) -> dict[str, Any]:
|
||||
return {
|
||||
"id": str(row["id"]),
|
||||
"username": row["username"],
|
||||
"display_name": row["display_name"],
|
||||
"created_at": format_timestamp(row["created_at"]),
|
||||
}
|
||||
|
||||
|
||||
def chat_json(row: dict[str, Any]) -> dict[str, Any]:
|
||||
return {
|
||||
"id": str(row["id"]),
|
||||
"title": row["title"],
|
||||
"member_ids": [str(member) for member in row["member_ids"]],
|
||||
"created_at": format_timestamp(row["created_at"]),
|
||||
}
|
||||
|
||||
|
||||
def message_json(row: dict[str, Any]) -> dict[str, Any]:
|
||||
return {
|
||||
"id": str(row["id"]),
|
||||
"chat_id": str(row["chat_id"]),
|
||||
"sender_id": str(row["sender_id"]),
|
||||
"text": row["text"],
|
||||
"created_at": format_timestamp(row["created_at"]),
|
||||
}
|
||||
|
||||
|
||||
async def create_user(
|
||||
cursor: AsyncCursor,
|
||||
*,
|
||||
username: str,
|
||||
display_name: str,
|
||||
password_hash: str,
|
||||
) -> dict[str, Any]:
|
||||
try:
|
||||
await cursor.execute(
|
||||
"""
|
||||
INSERT INTO users (id, username, display_name, password_hash, created_at)
|
||||
VALUES (%s, %s, %s, %s, now())
|
||||
RETURNING id, username, display_name, created_at
|
||||
""",
|
||||
(uuid.uuid4(), username, display_name, password_hash),
|
||||
)
|
||||
except psycopg.errors.UniqueViolation:
|
||||
raise ApiProblem(409, "conflict", "Username is already taken") from None
|
||||
row = await cursor.fetchone()
|
||||
return user_json(row)
|
||||
|
||||
|
||||
async def user_credentials(cursor: AsyncCursor, username: str) -> dict[str, Any] | None:
|
||||
await cursor.execute(
|
||||
"""
|
||||
SELECT id, username, display_name, password_hash, created_at
|
||||
FROM users
|
||||
WHERE username = %s
|
||||
""",
|
||||
(username,),
|
||||
)
|
||||
return await cursor.fetchone()
|
||||
|
||||
|
||||
async def user_by_id(cursor: AsyncCursor, user_id: uuid.UUID) -> dict[str, Any] | None:
|
||||
await cursor.execute(
|
||||
"""
|
||||
SELECT id, username, display_name, created_at
|
||||
FROM users
|
||||
WHERE id = %s
|
||||
""",
|
||||
(user_id,),
|
||||
)
|
||||
return await cursor.fetchone()
|
||||
|
||||
|
||||
async def create_chat(
|
||||
cursor: AsyncCursor,
|
||||
*,
|
||||
actor_id: uuid.UUID,
|
||||
title: str,
|
||||
member_ids: list[uuid.UUID],
|
||||
) -> dict[str, Any]:
|
||||
"""Insert the chat and its membership in one transaction.
|
||||
|
||||
Unknown members are rejected by the foreign key, so the failure rolls the
|
||||
already inserted chat row back instead of leaving a chat without members.
|
||||
"""
|
||||
if actor_id not in member_ids:
|
||||
raise ApiProblem(400, "invalid_request", "Current user must be a chat member")
|
||||
chat_id = uuid.uuid4()
|
||||
try:
|
||||
await cursor.execute(
|
||||
"INSERT INTO chats (id, title, created_at) VALUES (%s, %s, now())",
|
||||
(chat_id, title),
|
||||
)
|
||||
await cursor.execute(
|
||||
"""
|
||||
INSERT INTO chat_members (chat_id, user_id, joined_at)
|
||||
SELECT %s, member_id, now() FROM unnest(%s::uuid[]) AS member_id
|
||||
""",
|
||||
(chat_id, member_ids),
|
||||
)
|
||||
except psycopg.errors.ForeignKeyViolation:
|
||||
raise ApiProblem(
|
||||
400, "invalid_request", "One or more members do not exist"
|
||||
) from None
|
||||
except psycopg.errors.UniqueViolation:
|
||||
raise ApiProblem(400, "invalid_request", "Chat members must be unique") from None
|
||||
await cursor.execute(
|
||||
"SELECT " + CHAT_COLUMNS + " FROM chats c WHERE c.id = %s",
|
||||
(chat_id,),
|
||||
)
|
||||
row = await cursor.fetchone()
|
||||
return chat_json(row)
|
||||
|
||||
|
||||
async def accessible_chat(
|
||||
cursor: AsyncCursor, *, actor_id: uuid.UUID, chat_id: uuid.UUID
|
||||
) -> dict[str, Any]:
|
||||
await cursor.execute(
|
||||
"""
|
||||
SELECT """
|
||||
+ CHAT_COLUMNS
|
||||
+ """
|
||||
FROM chats c
|
||||
JOIN chat_members membership
|
||||
ON membership.chat_id = c.id AND membership.user_id = %s
|
||||
WHERE c.id = %s
|
||||
""",
|
||||
(actor_id, chat_id),
|
||||
)
|
||||
row = await cursor.fetchone()
|
||||
if row is None:
|
||||
raise ApiProblem(404, "not_found", "Chat not found")
|
||||
return chat_json(row)
|
||||
|
||||
|
||||
async def chat_page(
|
||||
cursor: AsyncCursor,
|
||||
*,
|
||||
actor_id: uuid.UUID,
|
||||
limit: int,
|
||||
after: tuple[datetime, uuid.UUID] | None,
|
||||
) -> list[dict[str, Any]]:
|
||||
query = (
|
||||
"SELECT "
|
||||
+ CHAT_COLUMNS
|
||||
+ """
|
||||
FROM chats c
|
||||
JOIN chat_members membership
|
||||
ON membership.chat_id = c.id AND membership.user_id = %(actor)s
|
||||
"""
|
||||
)
|
||||
params: dict[str, Any] = {"actor": actor_id, "limit": limit + 1}
|
||||
if after is not None:
|
||||
query += " WHERE (c.created_at, c.id) > (%(after_ts)s, %(after_id)s)"
|
||||
params["after_ts"], params["after_id"] = after
|
||||
query += " ORDER BY c.created_at, c.id LIMIT %(limit)s"
|
||||
await cursor.execute(query, params)
|
||||
return [chat_json(row) for row in await cursor.fetchall()]
|
||||
|
||||
|
||||
async def require_membership(
|
||||
cursor: AsyncCursor, *, actor_id: uuid.UUID, chat_id: uuid.UUID
|
||||
) -> None:
|
||||
await cursor.execute(
|
||||
"SELECT 1 FROM chat_members WHERE chat_id = %s AND user_id = %s",
|
||||
(chat_id, actor_id),
|
||||
)
|
||||
if await cursor.fetchone() is None:
|
||||
raise ApiProblem(404, "not_found", "Chat not found")
|
||||
|
||||
|
||||
async def get_idempotency_key(
|
||||
cursor: AsyncCursor, *, actor_id: uuid.UUID, chat_id: uuid.UUID, key: str
|
||||
) -> dict[str, Any] | None:
|
||||
"""Return a record only from this user's POST-message resource scope."""
|
||||
await cursor.execute(
|
||||
"""
|
||||
SELECT key, actor_id, chat_id, request_sha256, message_id, created_at
|
||||
FROM idempotency_keys
|
||||
WHERE actor_id = %s AND chat_id = %s AND key = %s
|
||||
""",
|
||||
(actor_id, chat_id, key),
|
||||
)
|
||||
return await cursor.fetchone()
|
||||
|
||||
|
||||
async def insert_idempotency_key(
|
||||
cursor: AsyncCursor,
|
||||
*,
|
||||
actor_id: uuid.UUID,
|
||||
key: str,
|
||||
chat_id: uuid.UUID,
|
||||
request_sha256: str,
|
||||
) -> bool:
|
||||
"""Atomically reserve a key without poisoning a concurrent transaction."""
|
||||
await cursor.execute(
|
||||
"""
|
||||
INSERT INTO idempotency_keys (actor_id, key, chat_id, request_sha256, created_at)
|
||||
VALUES (%s, %s, %s, %s, now())
|
||||
ON CONFLICT (actor_id, chat_id, key) DO NOTHING
|
||||
RETURNING key
|
||||
""",
|
||||
(actor_id, key, chat_id, request_sha256),
|
||||
)
|
||||
return await cursor.fetchone() is not None
|
||||
|
||||
|
||||
async def message_by_id(cursor: AsyncCursor, message_id: uuid.UUID) -> dict[str, Any] | None:
|
||||
await cursor.execute(
|
||||
"""
|
||||
SELECT id, chat_id, sender_id, text, created_at
|
||||
FROM messages
|
||||
WHERE id = %s
|
||||
""",
|
||||
(message_id,),
|
||||
)
|
||||
row = await cursor.fetchone()
|
||||
if row is None:
|
||||
return None
|
||||
return message_json(row)
|
||||
|
||||
|
||||
async def create_message(
|
||||
cursor: AsyncCursor,
|
||||
*,
|
||||
actor_id: uuid.UUID,
|
||||
chat_id: uuid.UUID,
|
||||
text: str,
|
||||
request_sha256: str,
|
||||
key: str,
|
||||
) -> dict[str, Any]:
|
||||
await require_membership(cursor, actor_id=actor_id, chat_id=chat_id)
|
||||
# The SHA-256 of the replayed body is recorded with the idempotency row so a
|
||||
# client cannot replay a request with different content under the same key.
|
||||
message_id = uuid.uuid4()
|
||||
await cursor.execute(
|
||||
"""
|
||||
INSERT INTO messages (id, chat_id, sender_id, text, created_at)
|
||||
VALUES (%s, %s, %s, %s, now())
|
||||
RETURNING id, chat_id, sender_id, text, created_at
|
||||
""",
|
||||
(message_id, chat_id, actor_id, text),
|
||||
)
|
||||
row = await cursor.fetchone()
|
||||
# Record the message_id so a replay can return the original message.
|
||||
await cursor.execute(
|
||||
"""
|
||||
UPDATE idempotency_keys
|
||||
SET request_sha256 = %s, message_id = %s
|
||||
WHERE actor_id = %s AND chat_id = %s AND key = %s
|
||||
""",
|
||||
(request_sha256, message_id, actor_id, chat_id, key),
|
||||
)
|
||||
return message_json(row)
|
||||
|
||||
|
||||
async def message_page(
|
||||
cursor: AsyncCursor,
|
||||
*,
|
||||
chat_id: uuid.UUID,
|
||||
limit: int,
|
||||
after: tuple[datetime, uuid.UUID] | None,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Keyset page over (created_at, id); the composite index serves both branches."""
|
||||
query = """
|
||||
SELECT id, chat_id, sender_id, text, created_at
|
||||
FROM messages
|
||||
WHERE chat_id = %(chat)s
|
||||
"""
|
||||
params: dict[str, Any] = {"chat": chat_id, "limit": limit + 1}
|
||||
if after is not None:
|
||||
query += " AND (created_at, id) > (%(after_ts)s, %(after_id)s)"
|
||||
params["after_ts"], params["after_id"] = after
|
||||
query += " ORDER BY created_at, id LIMIT %(limit)s"
|
||||
await cursor.execute(query, params)
|
||||
return [message_json(row) for row in await cursor.fetchall()]
|
||||
|
||||
|
||||
async def insert_session(
|
||||
cursor: AsyncCursor,
|
||||
*,
|
||||
user_id: uuid.UUID,
|
||||
token_hash: str,
|
||||
ttl_seconds: int,
|
||||
) -> datetime:
|
||||
await cursor.execute(
|
||||
"""
|
||||
INSERT INTO sessions (id, user_id, token_hash, created_at, expires_at)
|
||||
VALUES (%s, %s, %s, now(), now() + make_interval(secs => %s))
|
||||
RETURNING expires_at
|
||||
""",
|
||||
(uuid.uuid4(), user_id, token_hash, ttl_seconds),
|
||||
)
|
||||
row = await cursor.fetchone()
|
||||
return row["expires_at"]
|
||||
|
||||
|
||||
async def session_user(cursor: AsyncCursor, token_hash: str) -> dict[str, Any] | None:
|
||||
"""Resolve a live session row; expired and revoked sessions never match."""
|
||||
await cursor.execute(
|
||||
"""
|
||||
SELECT s.expires_at, u.id, u.username, u.display_name, u.created_at
|
||||
FROM sessions s
|
||||
JOIN users u ON u.id = s.user_id
|
||||
WHERE s.token_hash = %s AND s.revoked_at IS NULL AND s.expires_at > now()
|
||||
""",
|
||||
(token_hash,),
|
||||
)
|
||||
return await cursor.fetchone()
|
||||
|
||||
|
||||
async def revoke_session(cursor: AsyncCursor, token_hash: str) -> datetime | None:
|
||||
"""Revoke a live session and return its expiry, or None when it is not live."""
|
||||
await cursor.execute(
|
||||
"""
|
||||
UPDATE sessions SET revoked_at = now()
|
||||
WHERE token_hash = %s AND revoked_at IS NULL AND expires_at > now()
|
||||
RETURNING expires_at
|
||||
""",
|
||||
(token_hash,),
|
||||
)
|
||||
row = await cursor.fetchone()
|
||||
return None if row is None else row["expires_at"]
|
||||
|
||||
|
||||
async def purge_expired_sessions(cursor: AsyncCursor) -> int:
|
||||
await cursor.execute(
|
||||
"""
|
||||
DELETE FROM sessions
|
||||
WHERE expires_at < now() - interval '1 hour'
|
||||
OR (revoked_at IS NOT NULL AND revoked_at < now() - interval '1 hour')
|
||||
"""
|
||||
)
|
||||
return cursor.rowcount
|
||||
@@ -0,0 +1,209 @@
|
||||
"""Minimal standard-library HTTP client for the student's own tests.
|
||||
|
||||
Run with the API already up:
|
||||
|
||||
python -m unittest discover -s student-tests -p "test_*.py" -v
|
||||
|
||||
BASE_URL selects the stand (default http://127.0.0.1:8080). No third-party
|
||||
dependency is needed, so `make test-student` works on a clean clone.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
import uuid
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
LAB = json.loads((ROOT / "lab.json").read_text(encoding="utf-8"))["number"]
|
||||
|
||||
|
||||
# 127.0.0.1 rather than "localhost": on a dual-stack host "localhost" resolves
|
||||
# to ::1 first, a published Docker port may answer on IPv4 only, and every IPv6
|
||||
# attempt then costs ~2 s of SYN retry before urllib falls back to IPv4. The IPv4
|
||||
# literal keeps the client fast and never reaches the application.
|
||||
BASE_URL = os.environ.get("BASE_URL", "http://127.0.0.1:8080").rstrip("/")
|
||||
TIMEOUT = float(os.environ.get("STUDENT_TEST_TIMEOUT", "20"))
|
||||
|
||||
|
||||
class NoRedirect(urllib.request.HTTPRedirectHandler):
|
||||
"""The contract never requires redirects; fail loudly instead of following."""
|
||||
|
||||
def redirect_request(self, req, fp, code, msg, headers, newurl): # noqa: ANN001
|
||||
return None
|
||||
|
||||
|
||||
# The lab stand is always loopback; urllib ignores the Windows proxy bypass
|
||||
# list, so a corporate/system proxy would otherwise be used for localhost too.
|
||||
_OPENER = urllib.request.build_opener(urllib.request.ProxyHandler({}), NoRedirect())
|
||||
|
||||
|
||||
class Response:
|
||||
def __init__(self, status: int, headers: dict[str, str], data: bytes) -> None:
|
||||
self.status = status
|
||||
self.headers = headers
|
||||
self.data = data
|
||||
try:
|
||||
self.json = json.loads(data.decode("utf-8")) if data else None
|
||||
except (ValueError, UnicodeDecodeError):
|
||||
self.json = None
|
||||
|
||||
def header(self, name: str) -> str:
|
||||
for key, value in self.headers.items():
|
||||
if key.lower() == name.lower():
|
||||
return value
|
||||
return ""
|
||||
|
||||
def __repr__(self) -> str: # pragma: no cover - diagnostics only
|
||||
return "Response(status=%s, json=%r)" % (self.status, self.json)
|
||||
|
||||
|
||||
def request(
|
||||
method: str,
|
||||
path: str,
|
||||
*,
|
||||
body=None,
|
||||
raw: bytes | None = None,
|
||||
token: str | None = None,
|
||||
headers: dict[str, str] | None = None,
|
||||
content_type: str | None = None,
|
||||
timeout: float | None = None,
|
||||
base_url: str | None = None,
|
||||
) -> Response:
|
||||
url = (base_url or BASE_URL) + path
|
||||
data = raw
|
||||
request_headers = {"Accept": "application/json"}
|
||||
if body is not None:
|
||||
data = json.dumps(body, ensure_ascii=False).encode("utf-8")
|
||||
request_headers["Content-Type"] = "application/json"
|
||||
if content_type is not None:
|
||||
request_headers["Content-Type"] = content_type
|
||||
if token is not None:
|
||||
request_headers["Authorization"] = "Bearer " + token
|
||||
request_headers.update(headers or {})
|
||||
prepared = urllib.request.Request(url, data=data, headers=request_headers, method=method)
|
||||
try:
|
||||
response = _OPENER.open(prepared, timeout=timeout or TIMEOUT)
|
||||
except urllib.error.HTTPError as error:
|
||||
response = error
|
||||
with response:
|
||||
payload = response.read()
|
||||
return Response(response.code, dict(response.headers.items()), payload)
|
||||
|
||||
|
||||
def basic(username: str, password: str) -> dict[str, str]:
|
||||
encoded = base64.b64encode((username + ":" + password).encode("utf-8")).decode("ascii")
|
||||
return {"Authorization": "Basic " + encoded}
|
||||
|
||||
|
||||
def new_registration(prefix: str = "st", password: str | None = None) -> dict[str, str]:
|
||||
suffix = uuid.uuid4().hex[:12]
|
||||
return {
|
||||
"username": prefix + "_" + suffix,
|
||||
"display_name": "Тест " + suffix,
|
||||
"password": password or ("St-" + uuid.uuid4().hex),
|
||||
}
|
||||
|
||||
|
||||
def register(registration: dict[str, str] | None = None) -> Response:
|
||||
return request("POST", "/api/v1/users", body=registration or new_registration())
|
||||
|
||||
|
||||
def login(username: str, password: str) -> Response:
|
||||
return request("POST", "/api/v1/auth/sessions", headers=basic(username, password))
|
||||
|
||||
|
||||
class Actor:
|
||||
"""A registered user with a live session, as a client would hold it."""
|
||||
|
||||
def __init__(self, registration: dict[str, str]) -> None:
|
||||
self.registration = registration
|
||||
response = register(registration)
|
||||
if response.status != 201:
|
||||
raise AssertionError(
|
||||
"registration failed: %s %r" % (response.status, response.json)
|
||||
)
|
||||
self.user = response.json
|
||||
self.username = registration["username"]
|
||||
self.password = registration["password"]
|
||||
|
||||
@property
|
||||
def id(self) -> str:
|
||||
return self.user["id"]
|
||||
|
||||
def sign_in(self) -> "Actor":
|
||||
response = login(self.username, self.password)
|
||||
if response.status != 201:
|
||||
raise AssertionError("login failed: %s %r" % (response.status, response.json))
|
||||
self.session = response.json
|
||||
self.token = response.json["token"]
|
||||
return self
|
||||
|
||||
|
||||
def actor(prefix: str = "st", password: str | None = None) -> Actor:
|
||||
return Actor(new_registration(prefix, password)).sign_in()
|
||||
|
||||
|
||||
def create_chat(token: str, member_ids: list[str], title: str = "Тестовый чат") -> dict:
|
||||
response = request(
|
||||
"POST",
|
||||
"/api/v1/chats",
|
||||
body={"title": title, "member_ids": member_ids},
|
||||
token=token,
|
||||
)
|
||||
if response.status != 201:
|
||||
raise AssertionError("chat creation failed: %s %r" % (response.status, response.json))
|
||||
return response.json
|
||||
|
||||
|
||||
def send(token: str, chat_id: str, text: str) -> dict:
|
||||
headers = {}
|
||||
if LAB >= 3:
|
||||
headers["Idempotency-Key"] = uuid.uuid4().hex
|
||||
response = request(
|
||||
"POST", "/api/v1/chats/" + chat_id + "/messages", body={"text": text}, token=token, headers=headers
|
||||
)
|
||||
if response.status != 201:
|
||||
raise AssertionError("message failed: %s %r" % (response.status, response.json))
|
||||
return response.json
|
||||
|
||||
|
||||
def history(token: str, chat_id: str, limit: int = 100) -> list[dict]:
|
||||
"""Walk every page of a chat, guarding against cursor loops."""
|
||||
items: list[dict] = []
|
||||
seen: set[str] = set()
|
||||
cursor: str | None = None
|
||||
while True:
|
||||
path = "/api/v1/chats/" + chat_id + "/messages?limit=" + str(limit)
|
||||
if cursor is not None:
|
||||
path += "&cursor=" + urllib.parse.quote(cursor, safe="")
|
||||
response = request("GET", path, token=token)
|
||||
if response.status != 200:
|
||||
raise AssertionError("history failed: %s %r" % (response.status, response.json))
|
||||
items.extend(response.json["items"])
|
||||
cursor = response.json["next_cursor"]
|
||||
if cursor is None:
|
||||
return items
|
||||
if cursor in seen:
|
||||
raise AssertionError("cursor loop detected")
|
||||
seen.add(cursor)
|
||||
|
||||
|
||||
def expect_error(test, response: Response, status: int, code: str) -> dict:
|
||||
"""Assert the public error envelope, request id echo and the status."""
|
||||
test.assertEqual(response.status, status, response.json)
|
||||
test.assertIsInstance(response.json, dict)
|
||||
error = response.json.get("error")
|
||||
test.assertIsNotNone(error, "missing error envelope")
|
||||
test.assertEqual(error["code"], code)
|
||||
test.assertTrue(error["message"])
|
||||
request_id = response.header("X-Request-Id")
|
||||
test.assertTrue(request_id, "missing X-Request-Id header")
|
||||
test.assertEqual(error["request_id"], request_id)
|
||||
return error
|
||||
@@ -0,0 +1,329 @@
|
||||
"""Own functional tests: contract edges carried from lab 1 plus lab 2 auth.
|
||||
|
||||
Every test creates unique synthetic users, so the file is safe to re-run against
|
||||
the same stand (the API has no delete endpoint by design).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import common
|
||||
|
||||
|
||||
def timestamp(value: str) -> datetime:
|
||||
return datetime.fromisoformat(value[:-1] + "+00:00")
|
||||
|
||||
|
||||
class Base(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls) -> None:
|
||||
again = common.request("POST", "/api/v1/users", body=common.new_registration())
|
||||
if again.status != 201:
|
||||
raise AssertionError("stand is not serving: %r" % (again.json,))
|
||||
cls.alice = common.actor("alice")
|
||||
cls.bob = common.actor("bob")
|
||||
cls.eve = common.actor("eve")
|
||||
cls.chat = common.create_chat(
|
||||
cls.alice.token, [cls.alice.id, cls.bob.id], "Контрактный чат"
|
||||
)
|
||||
|
||||
|
||||
class HealthAndEnvelope(Base):
|
||||
def test_health_endpoints_and_request_id(self) -> None:
|
||||
live = common.request("GET", "/health/live")
|
||||
self.assertEqual(live.status, 200)
|
||||
self.assertEqual(live.json, {"status": "ok"})
|
||||
self.assertTrue(live.header("X-Request-Id"))
|
||||
ready = common.request("GET", "/health/ready")
|
||||
self.assertEqual(ready.status, 200)
|
||||
self.assertEqual(ready.json, {"status": "ok"})
|
||||
|
||||
def test_unknown_route_uses_same_error_envelope(self) -> None:
|
||||
common.expect_error(self, common.request("GET", "/api/v1/nope"), 404, "not_found")
|
||||
|
||||
def test_query_parameters_rejected_where_not_supported(self) -> None:
|
||||
response = common.request(
|
||||
"GET", "/api/v1/users/me?verbose=1", token=self.alice.token
|
||||
)
|
||||
common.expect_error(self, response, 400, "invalid_request")
|
||||
|
||||
|
||||
class Registration(Base):
|
||||
def test_response_has_no_secret_fields(self) -> None:
|
||||
response = common.register(common.new_registration())
|
||||
self.assertEqual(response.status, 201)
|
||||
self.assertEqual(
|
||||
set(response.json), {"id", "username", "display_name", "created_at"}
|
||||
)
|
||||
self.assertNotIn("password", response.json)
|
||||
self.assertNotIn("password_hash", response.json)
|
||||
|
||||
def test_duplicate_username_conflict(self) -> None:
|
||||
registration = common.new_registration()
|
||||
self.assertEqual(common.register(registration).status, 201)
|
||||
common.expect_error(self, common.register(registration), 409, "conflict")
|
||||
|
||||
def test_username_rules(self) -> None:
|
||||
for username in ("UPPER_case", "ab", "with-dash", "with space"):
|
||||
body = common.new_registration()
|
||||
body["username"] = username
|
||||
common.expect_error(self, common.register(body), 400, "invalid_request")
|
||||
|
||||
def test_password_rules(self) -> None:
|
||||
short = common.new_registration(password="x" * 11)
|
||||
common.expect_error(self, common.register(short), 400, "invalid_request")
|
||||
long_password = common.new_registration(password="x" * 129)
|
||||
common.expect_error(self, common.register(long_password), 400, "invalid_request")
|
||||
# 12 Unicode code points are enough, even when they are multi-byte.
|
||||
unicode_password = common.new_registration(password="😀" * 12)
|
||||
self.assertEqual(common.register(unicode_password).status, 201)
|
||||
|
||||
def test_malformed_json_extra_fields_and_wrong_types(self) -> None:
|
||||
common.expect_error(
|
||||
self,
|
||||
common.request(
|
||||
"POST", "/api/v1/users", raw=b'{"username": ', content_type="application/json"
|
||||
),
|
||||
400,
|
||||
"invalid_request",
|
||||
)
|
||||
body = common.new_registration()
|
||||
body["is_admin"] = True
|
||||
common.expect_error(self, common.register(body), 400, "invalid_request")
|
||||
body = common.new_registration()
|
||||
body["display_name"] = 42
|
||||
common.expect_error(self, common.register(body), 400, "invalid_request")
|
||||
|
||||
def test_unsupported_media_type(self) -> None:
|
||||
response = common.request(
|
||||
"POST",
|
||||
"/api/v1/users",
|
||||
raw=b"username=alice",
|
||||
content_type="application/x-www-form-urlencoded",
|
||||
)
|
||||
common.expect_error(self, response, 415, "unsupported_media_type")
|
||||
|
||||
|
||||
class Chats(Base):
|
||||
def test_membership_validation(self) -> None:
|
||||
cases = [
|
||||
[self.bob.id], # caller is missing
|
||||
[self.alice.id, self.alice.id], # duplicate member
|
||||
[self.alice.id, str(uuid.uuid4())], # unknown member
|
||||
[], # no members
|
||||
]
|
||||
for member_ids in cases:
|
||||
response = common.request(
|
||||
"POST",
|
||||
"/api/v1/chats",
|
||||
body={"title": "Плохой чат", "member_ids": member_ids},
|
||||
token=self.alice.token,
|
||||
)
|
||||
common.expect_error(self, response, 400, "invalid_request")
|
||||
|
||||
def test_foreign_chat_is_hidden_everywhere(self) -> None:
|
||||
chat_id = self.chat["id"]
|
||||
common.expect_error(
|
||||
self,
|
||||
common.request("GET", "/api/v1/chats/" + chat_id, token=self.eve.token),
|
||||
404,
|
||||
"not_found",
|
||||
)
|
||||
common.expect_error(
|
||||
self,
|
||||
common.request(
|
||||
"GET", "/api/v1/chats/" + chat_id + "/messages", token=self.eve.token
|
||||
),
|
||||
404,
|
||||
"not_found",
|
||||
)
|
||||
common.expect_error(
|
||||
self,
|
||||
common.request(
|
||||
"POST",
|
||||
"/api/v1/chats/" + chat_id + "/messages",
|
||||
body={"text": "чужое"},
|
||||
token=self.eve.token,
|
||||
),
|
||||
404,
|
||||
"not_found",
|
||||
)
|
||||
listed = common.request("GET", "/api/v1/chats?limit=100", token=self.eve.token)
|
||||
self.assertEqual(listed.status, 200)
|
||||
self.assertNotIn(chat_id, [item["id"] for item in listed.json["items"]])
|
||||
|
||||
def test_messages_are_validated_and_unicode_preserved(self) -> None:
|
||||
chat = common.create_chat(self.alice.token, [self.alice.id, self.bob.id])
|
||||
for text in ("", " \n\t ", "x" * 4001):
|
||||
response = common.request(
|
||||
"POST",
|
||||
"/api/v1/chats/" + chat["id"] + "/messages",
|
||||
body={"text": text},
|
||||
token=self.alice.token,
|
||||
)
|
||||
common.expect_error(self, response, 400, "invalid_request")
|
||||
prefix = "Привет 👋 世界 "
|
||||
text = prefix + "x" * (4000 - len(prefix)) # exactly the contract maximum
|
||||
message = common.send(self.alice.token, chat["id"], text)
|
||||
self.assertEqual(message["text"], text)
|
||||
self.assertEqual(message["sender_id"], self.alice.id)
|
||||
|
||||
injected = common.request(
|
||||
"POST",
|
||||
"/api/v1/chats/" + chat["id"] + "/messages",
|
||||
body={"text": "подмена", "sender_id": self.bob.id},
|
||||
token=self.alice.token,
|
||||
)
|
||||
common.expect_error(self, injected, 400, "invalid_request")
|
||||
|
||||
|
||||
class Pagination(Base):
|
||||
def test_walks_every_page_without_loss_or_duplicates(self) -> None:
|
||||
chat = common.create_chat(self.alice.token, [self.alice.id, self.bob.id])
|
||||
sent = [
|
||||
common.send(self.alice.token if index % 2 else self.bob.token, chat["id"], "msg-%d" % index)
|
||||
for index in range(7)
|
||||
]
|
||||
collected: list[dict] = []
|
||||
cursor = None
|
||||
pages = 0
|
||||
while True:
|
||||
path = "/api/v1/chats/" + chat["id"] + "/messages?limit=2"
|
||||
if cursor is not None:
|
||||
path += "&cursor=" + cursor
|
||||
page = common.request("GET", path, token=self.alice.token)
|
||||
self.assertEqual(page.status, 200)
|
||||
collected.extend(page.json["items"])
|
||||
pages += 1
|
||||
cursor = page.json["next_cursor"]
|
||||
if cursor is None:
|
||||
break
|
||||
self.assertLess(pages, 10, "pagination did not terminate")
|
||||
self.assertEqual(pages, 4)
|
||||
self.assertEqual([item["id"] for item in collected], [item["id"] for item in sent])
|
||||
self.assertEqual(
|
||||
collected,
|
||||
sorted(collected, key=lambda item: (timestamp(item["created_at"]), item["id"])),
|
||||
)
|
||||
|
||||
def test_cursor_is_bound_to_its_chat(self) -> None:
|
||||
other = common.create_chat(self.alice.token, [self.alice.id])
|
||||
common.send(self.alice.token, other["id"], "a")
|
||||
common.send(self.alice.token, other["id"], "b")
|
||||
first = common.request(
|
||||
"GET", "/api/v1/chats/" + other["id"] + "/messages?limit=1", token=self.alice.token
|
||||
)
|
||||
cursor = first.json["next_cursor"]
|
||||
self.assertIsNotNone(cursor)
|
||||
response = common.request(
|
||||
"GET",
|
||||
"/api/v1/chats/" + self.chat["id"] + "/messages" + "?limit=1&cursor=" + cursor,
|
||||
token=self.alice.token,
|
||||
)
|
||||
common.expect_error(self, response, 400, "invalid_request")
|
||||
|
||||
def test_limit_and_cursor_validation(self) -> None:
|
||||
base = "/api/v1/chats/" + self.chat["id"] + "/messages"
|
||||
for query in ("?limit=0", "?limit=101", "?limit=abc", "?cursor=", "?cursor=not-a-cursor", "?limit=1&limit=2", "?unknown=1"):
|
||||
common.expect_error(
|
||||
self,
|
||||
common.request("GET", base + query, token=self.alice.token),
|
||||
400,
|
||||
"invalid_request",
|
||||
)
|
||||
|
||||
|
||||
class Authentication(Base):
|
||||
def test_basic_is_not_accepted_outside_session_creation(self) -> None:
|
||||
response = common.request(
|
||||
"GET", "/api/v1/users/me", headers=common.basic(self.alice.username, self.alice.password)
|
||||
)
|
||||
common.expect_error(self, response, 401, "unauthorized")
|
||||
self.assertTrue(response.header("WWW-Authenticate").lower().startswith("bearer"))
|
||||
|
||||
def test_missing_and_unknown_tokens(self) -> None:
|
||||
for headers in ({}, {"Authorization": "Bearer not-a-real-token"}):
|
||||
response = common.request("GET", "/api/v1/users/me", headers=headers)
|
||||
common.expect_error(self, response, 401, "unauthorized")
|
||||
self.assertTrue(response.header("WWW-Authenticate").lower().startswith("bearer"))
|
||||
|
||||
def test_x_user_id_is_ignored(self) -> None:
|
||||
response = common.request("GET", "/api/v1/users/me", headers={"X-User-Id": self.alice.id})
|
||||
common.expect_error(self, response, 401, "unauthorized")
|
||||
overridden = common.request(
|
||||
"GET",
|
||||
"/api/v1/users/me",
|
||||
token=self.alice.token,
|
||||
headers={"X-User-Id": self.eve.id},
|
||||
)
|
||||
self.assertEqual(overridden.status, 200)
|
||||
self.assertEqual(overridden.json["id"], self.alice.id)
|
||||
|
||||
def test_session_payload_and_expiry(self) -> None:
|
||||
response = common.login(self.alice.username, self.alice.password)
|
||||
self.assertEqual(response.status, 201)
|
||||
session = response.json
|
||||
self.assertEqual(set(session), {"token", "token_type", "expires_at", "user"})
|
||||
self.assertEqual(session["token_type"], "Bearer")
|
||||
self.assertGreaterEqual(len(session["token"]), 43)
|
||||
self.assertNotIn(self.alice.username, session["token"])
|
||||
self.assertNotEqual(session["token"], self.alice.id)
|
||||
self.assertEqual(session["user"], self.alice.user)
|
||||
self.assertGreater(timestamp(session["expires_at"]), datetime.now(timezone.utc))
|
||||
me = common.request("GET", "/api/v1/users/me", token=session["token"])
|
||||
self.assertEqual(me.status, 200)
|
||||
self.assertEqual(me.json, self.alice.user)
|
||||
|
||||
def test_wrong_password_and_unknown_user_are_indistinguishable(self) -> None:
|
||||
wrong = common.login(self.alice.username, "definitely-wrong-password")
|
||||
unknown = common.login("nobody_here_" + uuid.uuid4().hex[:8], "definitely-wrong-password")
|
||||
messages = []
|
||||
for response in (wrong, unknown):
|
||||
error = common.expect_error(self, response, 401, "unauthorized")
|
||||
self.assertTrue(
|
||||
response.header("WWW-Authenticate").lower().startswith("basic")
|
||||
)
|
||||
messages.append(error["message"])
|
||||
# Same message and challenge: a client cannot tell the accounts apart.
|
||||
self.assertEqual(messages[0], messages[1])
|
||||
|
||||
def test_logout_revokes_only_that_session(self) -> None:
|
||||
first = common.login(self.alice.username, self.alice.password).json["token"]
|
||||
second = common.login(self.alice.username, self.alice.password).json["token"]
|
||||
revoked = common.request("DELETE", "/api/v1/auth/sessions/current", token=first)
|
||||
self.assertEqual(revoked.status, 204)
|
||||
self.assertEqual(revoked.data, b"")
|
||||
common.expect_error(
|
||||
self,
|
||||
common.request("GET", "/api/v1/users/me", token=first),
|
||||
401,
|
||||
"unauthorized",
|
||||
)
|
||||
common.expect_error(
|
||||
self,
|
||||
common.request("DELETE", "/api/v1/auth/sessions/current", token=first),
|
||||
401,
|
||||
"unauthorized",
|
||||
)
|
||||
still_valid = common.request("GET", "/api/v1/users/me", token=second)
|
||||
self.assertEqual(still_valid.status, 200)
|
||||
|
||||
def test_session_endpoint_rejects_json_body_and_bad_basic(self) -> None:
|
||||
response = common.request(
|
||||
"POST",
|
||||
"/api/v1/auth/sessions",
|
||||
body={"username": self.alice.username, "password": self.alice.password},
|
||||
)
|
||||
common.expect_error(self, response, 401, "unauthorized")
|
||||
response = common.request(
|
||||
"POST", "/api/v1/auth/sessions", headers={"Authorization": "Basic !!!not-base64!!!"}
|
||||
)
|
||||
common.expect_error(self, response, 401, "unauthorized")
|
||||
self.assertTrue(response.header("WWW-Authenticate").lower().startswith("basic"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,211 @@
|
||||
"""Dependency outage behaviour: Valkey down, then PostgreSQL down.
|
||||
|
||||
Requires Docker access on the same host as the Compose stand and is therefore
|
||||
opt-in:
|
||||
|
||||
STUDENT_DOCKER_TESTS=1 python -m unittest -v test_degradation
|
||||
|
||||
The test never removes volumes and restores both services in `finally`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import uuid
|
||||
import statistics
|
||||
import subprocess
|
||||
import time
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
import common
|
||||
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
PROJECT = os.environ.get("COMPOSE_PROJECT_NAME", "mtusi2026lab3")
|
||||
ENV_FILE = os.environ.get("COMPOSE_ENV_FILE", ".env")
|
||||
ENABLED = os.environ.get("STUDENT_DOCKER_TESTS") == "1"
|
||||
|
||||
|
||||
def compose(*arguments: str, timeout: int = 120) -> None:
|
||||
result = subprocess.run(
|
||||
["docker", "compose", "-p", PROJECT, "--env-file", ENV_FILE, *arguments],
|
||||
cwd=str(REPO_ROOT),
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=timeout,
|
||||
)
|
||||
if result.returncode != 0:
|
||||
raise AssertionError(
|
||||
"docker compose %s failed: %s" % (" ".join(arguments), result.stderr[-500:])
|
||||
)
|
||||
|
||||
|
||||
def wait_ready(expected: int, seconds: float) -> float:
|
||||
started = time.monotonic()
|
||||
deadline = started + seconds
|
||||
last = None
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
last = common.request("GET", "/health/ready", timeout=5)
|
||||
if last.status == expected:
|
||||
return time.monotonic() - started
|
||||
except OSError:
|
||||
# A replica can close the listener while Compose restarts it.
|
||||
pass
|
||||
time.sleep(0.5)
|
||||
raise AssertionError(
|
||||
"readiness did not reach %s within %.0fs (last=%s %r)"
|
||||
% (expected, seconds, last.status if last else None, last.json if last else None)
|
||||
)
|
||||
|
||||
|
||||
@unittest.skipUnless(ENABLED, "set STUDENT_DOCKER_TESTS=1 to stop real containers")
|
||||
class DependencyOutages(unittest.TestCase):
|
||||
def test_logout_remains_revoked_across_replicas_after_cache_outage(self) -> None:
|
||||
"""A stale shared-cache hit must not bypass PostgreSQL revocation."""
|
||||
alice = common.actor("revoked")
|
||||
chat = common.create_chat(alice.token, [alice.id], "Durable logout")
|
||||
self.assertEqual(common.request("GET", "/api/v1/users/me", token=alice.token).status, 200)
|
||||
try:
|
||||
compose("stop", "valkey")
|
||||
logout = common.request("DELETE", "/api/v1/auth/sessions/current", token=alice.token)
|
||||
self.assertEqual(logout.status, 204, logout.json)
|
||||
|
||||
# Remove both processes' local fallback state and restore Valkey's
|
||||
# old data. Requests through nginx can land on either replica.
|
||||
compose("restart", "api-1", "api-2")
|
||||
compose("start", "valkey")
|
||||
wait_ready(200, 90)
|
||||
|
||||
common.expect_error(
|
||||
self, common.request("GET", "/api/v1/users/me", token=alice.token), 401, "unauthorized"
|
||||
)
|
||||
common.expect_error(
|
||||
self,
|
||||
common.request("GET", "/api/v1/chats/" + chat["id"], token=alice.token),
|
||||
401,
|
||||
"unauthorized",
|
||||
)
|
||||
common.expect_error(
|
||||
self,
|
||||
common.request(
|
||||
"POST",
|
||||
"/api/v1/chats/" + chat["id"] + "/messages",
|
||||
body={"text": "must not be accepted"},
|
||||
token=alice.token,
|
||||
headers={"Idempotency-Key": uuid.uuid4().hex},
|
||||
),
|
||||
401,
|
||||
"unauthorized",
|
||||
)
|
||||
finally:
|
||||
compose("start", "valkey", "postgres", "api-1", "api-2")
|
||||
try:
|
||||
wait_ready(200, 90)
|
||||
except AssertionError:
|
||||
pass
|
||||
|
||||
def test_valkey_loss_degrades_then_postgres_loss_is_bounded(self) -> None:
|
||||
alice = common.actor("degrade")
|
||||
chat = common.create_chat(alice.token, [alice.id], "Устойчивость")
|
||||
# Warm the session cache.
|
||||
self.assertEqual(common.request("GET", "/api/v1/users/me", token=alice.token).status, 200)
|
||||
|
||||
cached = self._sample_latency(alice.token, 60)
|
||||
summary = {"cached_p50_ms": round(cached["p50"], 2)}
|
||||
try:
|
||||
# 1. Valkey is a cache: auth still works, without DB-free fast path.
|
||||
compose("stop", "valkey")
|
||||
ready = common.request("GET", "/health/ready")
|
||||
self.assertEqual(
|
||||
ready.status, 200, "session cache must not be a readiness dependency"
|
||||
)
|
||||
me = common.request("GET", "/api/v1/users/me", token=alice.token)
|
||||
self.assertEqual(me.status, 200, me.json)
|
||||
sent = common.request(
|
||||
"POST",
|
||||
"/api/v1/chats/" + chat["id"] + "/messages",
|
||||
body={"text": "без кэша"},
|
||||
token=alice.token,
|
||||
headers={"Idempotency-Key": uuid.uuid4().hex},
|
||||
)
|
||||
self.assertEqual(sent.status, 201, sent.json)
|
||||
without_cache = self._sample_latency(alice.token, 60)
|
||||
summary["cache_down_p50_ms"] = round(without_cache["p50"], 2)
|
||||
summary["cache_down_p95_ms"] = round(without_cache["p95"], 2)
|
||||
|
||||
compose("start", "valkey")
|
||||
time.sleep(3)
|
||||
self.assertEqual(common.request("GET", "/health/ready").status, 200)
|
||||
|
||||
# 2. PostgreSQL is the source of truth: bounded errors, never a 200
|
||||
# for a token the server cannot verify, and never a raw 500.
|
||||
compose("stop", "postgres")
|
||||
ready = common.request("GET", "/health/ready")
|
||||
self.assertEqual(ready.status, 503, ready.json)
|
||||
self.assertEqual(ready.json["error"]["code"], "unavailable")
|
||||
|
||||
started = time.monotonic()
|
||||
write = common.request(
|
||||
"POST",
|
||||
"/api/v1/chats/" + chat["id"] + "/messages",
|
||||
body={"text": "без базы"},
|
||||
token=alice.token,
|
||||
timeout=30,
|
||||
headers={"Idempotency-Key": uuid.uuid4().hex},
|
||||
)
|
||||
bounded = time.monotonic() - started
|
||||
self.assertEqual(write.status, 503, write.json)
|
||||
self.assertEqual(write.json["error"]["code"], "unavailable")
|
||||
self.assertLess(bounded, 10, "database failure must fail fast")
|
||||
summary["db_down_write_seconds"] = round(bounded, 2)
|
||||
|
||||
for leaked in ("psycopg", "postgresql://", "password", "Traceback"):
|
||||
self.assertNotIn(leaked, json.dumps(write.json))
|
||||
unknown = common.request(
|
||||
"GET", "/api/v1/users/me", token="unknown-token-value", timeout=30
|
||||
)
|
||||
self.assertIn(unknown.status, (401, 503))
|
||||
self.assertNotEqual(unknown.status, 200)
|
||||
|
||||
# 3. Recovery keeps the same session and the stored chat.
|
||||
compose("start", "postgres")
|
||||
summary["recovery_seconds"] = round(wait_ready(200, 90), 2)
|
||||
after = common.request("GET", "/api/v1/chats/" + chat["id"], token=alice.token)
|
||||
self.assertEqual(after.status, 200, after.json)
|
||||
recovered = common.request(
|
||||
"POST",
|
||||
"/api/v1/chats/" + chat["id"] + "/messages",
|
||||
body={"text": "после восстановления"},
|
||||
token=alice.token,
|
||||
headers={"Idempotency-Key": uuid.uuid4().hex},
|
||||
)
|
||||
self.assertEqual(recovered.status, 201, recovered.json)
|
||||
messages = common.history(alice.token, chat["id"])
|
||||
self.assertIn("после восстановления", [message["text"] for message in messages])
|
||||
finally:
|
||||
compose("start", "valkey", "postgres")
|
||||
try:
|
||||
wait_ready(200, 90)
|
||||
except AssertionError:
|
||||
pass
|
||||
print("\n outage summary: " + json.dumps(summary, ensure_ascii=False))
|
||||
|
||||
def _sample_latency(self, token: str, count: int) -> dict[str, float]:
|
||||
samples = []
|
||||
for _ in range(count):
|
||||
started = time.monotonic()
|
||||
response = common.request("GET", "/api/v1/users/me", token=token, timeout=30)
|
||||
samples.append(time.monotonic() - started)
|
||||
self.assertEqual(response.status, 200, response.json)
|
||||
ordered = sorted(samples)
|
||||
return {
|
||||
"p50": statistics.median(samples) * 1000,
|
||||
"p95": ordered[max(0, int(len(ordered) * 0.95) - 1)] * 1000,
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,99 @@
|
||||
"""Session expiry with SESSION_TTL_SECONDS=2, plus logout and bad credentials.
|
||||
|
||||
Run against the dedicated short-TTL stand:
|
||||
|
||||
scripts/ttl-stack.sh start
|
||||
BASE_URL=http://localhost:8081 python -m unittest -v test_expiry
|
||||
scripts/ttl-stack.sh stop
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import statistics
|
||||
import time
|
||||
import unittest
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import common
|
||||
|
||||
|
||||
def expires_at(session: dict) -> datetime:
|
||||
return datetime.fromisoformat(session["expires_at"][:-1] + "+00:00")
|
||||
|
||||
|
||||
class ShortTtlSessions(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls) -> None:
|
||||
ready = common.request("GET", "/health/ready")
|
||||
if ready.status != 200:
|
||||
raise AssertionError("stand is not ready: %r" % (ready.json,))
|
||||
|
||||
def test_expiry_is_enforced_and_cache_does_not_extend_it(self) -> None:
|
||||
user = common.actor("ttl")
|
||||
started = datetime.now(timezone.utc)
|
||||
session = common.login(user.username, user.password)
|
||||
self.assertEqual(session.status, 201)
|
||||
lifetime = (expires_at(session.json) - started).total_seconds()
|
||||
self.assertLessEqual(lifetime, 5.0, "stand does not use a short TTL")
|
||||
|
||||
# Two reads: the second one is served from the Valkey cache, so the
|
||||
# expiry check below covers the cached path as well.
|
||||
for _ in range(2):
|
||||
self.assertEqual(
|
||||
common.request("GET", "/api/v1/users/me", token=session.json["token"]).status,
|
||||
200,
|
||||
)
|
||||
|
||||
time.sleep(3.2)
|
||||
expired = common.request("GET", "/api/v1/users/me", token=session.json["token"])
|
||||
common.expect_error(self, expired, 401, "unauthorized")
|
||||
self.assertTrue(expired.header("WWW-Authenticate").lower().startswith("bearer"))
|
||||
revoked = common.request(
|
||||
"DELETE", "/api/v1/auth/sessions/current", token=session.json["token"]
|
||||
)
|
||||
common.expect_error(self, revoked, 401, "unauthorized")
|
||||
|
||||
# A fresh login still works, so expiry removed only the old session.
|
||||
again = common.login(user.username, user.password)
|
||||
self.assertEqual(again.status, 201)
|
||||
self.assertEqual(
|
||||
common.request("GET", "/api/v1/users/me", token=again.json["token"]).status,
|
||||
200,
|
||||
)
|
||||
|
||||
def test_logout_is_immediate_even_when_the_session_is_cached(self) -> None:
|
||||
user = common.actor("ttllogout")
|
||||
token = user.token
|
||||
self.assertEqual(common.request("GET", "/api/v1/users/me", token=token).status, 200)
|
||||
self.assertEqual(
|
||||
common.request("DELETE", "/api/v1/auth/sessions/current", token=token).status,
|
||||
204,
|
||||
)
|
||||
outcome = common.request("GET", "/api/v1/users/me", token=token)
|
||||
common.expect_error(self, outcome, 401, "unauthorized")
|
||||
|
||||
def test_wrong_password_issues_no_session(self) -> None:
|
||||
user = common.actor("ttlwrong")
|
||||
denied = common.login(user.username, "not-the-password")
|
||||
common.expect_error(self, denied, 401, "unauthorized")
|
||||
self.assertTrue(denied.header("WWW-Authenticate").lower().startswith("basic"))
|
||||
|
||||
def test_session_lookup_latency_is_bounded(self) -> None:
|
||||
user = common.actor("ttllatency")
|
||||
samples = []
|
||||
for _ in range(30):
|
||||
started = time.monotonic()
|
||||
response = common.request("GET", "/api/v1/users/me", token=user.token, timeout=5)
|
||||
samples.append(time.monotonic() - started)
|
||||
self.assertEqual(response.status, 200)
|
||||
print(
|
||||
"\n /users/me warm cache p50=%.1fms p95=%.1fms"
|
||||
% (
|
||||
statistics.median(samples) * 1000,
|
||||
sorted(samples)[int(len(samples) * 0.95) - 1] * 1000,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,85 @@
|
||||
"""Idempotency scopes and concurrent replay checks for laboratory 3."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
import unittest
|
||||
import uuid
|
||||
|
||||
import common
|
||||
|
||||
|
||||
class IdempotencyScope(unittest.TestCase):
|
||||
def test_same_key_is_independent_for_different_users_and_chats(self) -> None:
|
||||
alice = common.actor("idemalice")
|
||||
eve = common.actor("idemeve")
|
||||
alice_chat = common.create_chat(alice.token, [alice.id], "Alice scope")
|
||||
eve_chat = common.create_chat(eve.token, [eve.id], "Eve scope")
|
||||
key = uuid.uuid4().hex
|
||||
|
||||
first = common.request(
|
||||
"POST",
|
||||
"/api/v1/chats/" + alice_chat["id"] + "/messages",
|
||||
body={"text": "same body"},
|
||||
token=alice.token,
|
||||
headers={"Idempotency-Key": key},
|
||||
)
|
||||
second = common.request(
|
||||
"POST",
|
||||
"/api/v1/chats/" + eve_chat["id"] + "/messages",
|
||||
body={"text": "same body"},
|
||||
token=eve.token,
|
||||
headers={"Idempotency-Key": key},
|
||||
)
|
||||
self.assertEqual(first.status, 201, first.json)
|
||||
self.assertEqual(second.status, 201, second.json)
|
||||
self.assertNotEqual(first.json["id"], second.json["id"])
|
||||
self.assertEqual(second.json["chat_id"], eve_chat["id"])
|
||||
self.assertEqual(second.json["sender_id"], eve.id)
|
||||
|
||||
# A different payload in another scope is also a new independent write.
|
||||
other_scope = common.create_chat(eve.token, [eve.id], "Eve second scope")
|
||||
third = common.request(
|
||||
"POST",
|
||||
"/api/v1/chats/" + other_scope["id"] + "/messages",
|
||||
body={"text": "different body"},
|
||||
token=eve.token,
|
||||
headers={"Idempotency-Key": key},
|
||||
)
|
||||
self.assertEqual(third.status, 201, third.json)
|
||||
self.assertEqual(third.json["chat_id"], other_scope["id"])
|
||||
|
||||
def test_concurrent_identical_replays_return_one_message_without_500(self) -> None:
|
||||
alice = common.actor("idemrace")
|
||||
chat = common.create_chat(alice.token, [alice.id], "Concurrent replay")
|
||||
key = uuid.uuid4().hex
|
||||
workers = 32
|
||||
barrier = threading.Barrier(workers)
|
||||
responses: list[common.Response] = []
|
||||
lock = threading.Lock()
|
||||
|
||||
def post() -> None:
|
||||
barrier.wait()
|
||||
response = common.request(
|
||||
"POST",
|
||||
"/api/v1/chats/" + chat["id"] + "/messages",
|
||||
body={"text": "exactly once"},
|
||||
token=alice.token,
|
||||
headers={"Idempotency-Key": key},
|
||||
timeout=30,
|
||||
)
|
||||
with lock:
|
||||
responses.append(response)
|
||||
|
||||
threads = [threading.Thread(target=post) for _ in range(workers)]
|
||||
for thread in threads:
|
||||
thread.start()
|
||||
for thread in threads:
|
||||
thread.join(timeout=60)
|
||||
|
||||
self.assertEqual(len(responses), workers)
|
||||
self.assertTrue(all(response.status == 201 for response in responses), responses)
|
||||
message_ids = {response.json["id"] for response in responses}
|
||||
self.assertEqual(len(message_ids), 1)
|
||||
messages = common.history(alice.token, chat["id"])
|
||||
self.assertEqual([message["text"] for message in messages], ["exactly once"])
|
||||
@@ -0,0 +1,142 @@
|
||||
"""Race, rollback and pool tests against the real PostgreSQL stand.
|
||||
|
||||
These are the checks a code reviewer would ask for on top of the smoke test:
|
||||
concurrent writers must not lose or duplicate messages, the username uniqueness
|
||||
must be decided by the database (not by a read-then-write check in Python), and
|
||||
a failed composite operation must leave nothing behind.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
import time
|
||||
import unittest
|
||||
import uuid
|
||||
|
||||
import common
|
||||
|
||||
|
||||
class ConcurrentWrites(unittest.TestCase):
|
||||
def test_100_messages_from_10_clients_are_exactly_100(self) -> None:
|
||||
alice = common.actor("cwalice")
|
||||
bob = common.actor("cwbob")
|
||||
chat = common.create_chat(alice.token, [alice.id, bob.id], "Гонка сообщений")
|
||||
writers = 10
|
||||
per_writer = 10
|
||||
barrier = threading.Barrier(writers)
|
||||
errors: list[str] = []
|
||||
sent: list[str] = []
|
||||
lock = threading.Lock()
|
||||
|
||||
def writer(index: int) -> None:
|
||||
token = alice.token if index % 2 else bob.token
|
||||
barrier.wait()
|
||||
for number in range(per_writer):
|
||||
text = "race-%d-%d" % (index, number)
|
||||
try:
|
||||
common.send(token, chat["id"], text)
|
||||
with lock:
|
||||
sent.append(text)
|
||||
except AssertionError as error: # pragma: no cover - failure detail
|
||||
with lock:
|
||||
errors.append(str(error))
|
||||
|
||||
threads = [threading.Thread(target=writer, args=(index,)) for index in range(writers)]
|
||||
started = time.monotonic()
|
||||
for thread in threads:
|
||||
thread.start()
|
||||
for thread in threads:
|
||||
thread.join(timeout=120)
|
||||
elapsed = time.monotonic() - started
|
||||
self.assertEqual(errors, [])
|
||||
self.assertEqual(len(sent), writers * per_writer)
|
||||
|
||||
messages = common.history(alice.token, chat["id"], limit=100)
|
||||
self.assertEqual(len(messages), writers * per_writer)
|
||||
self.assertEqual(len({message["id"] for message in messages}), len(messages))
|
||||
self.assertEqual(sorted(message["text"] for message in messages), sorted(sent))
|
||||
print("\n concurrent writes: %d messages from %d clients in %.2fs"
|
||||
% (len(messages), writers, elapsed))
|
||||
|
||||
def test_duplicate_username_race_creates_one_user(self) -> None:
|
||||
registration = common.new_registration("race")
|
||||
attempts = 12
|
||||
barrier = threading.Barrier(attempts)
|
||||
statuses: list[int] = []
|
||||
lock = threading.Lock()
|
||||
|
||||
def attempt() -> None:
|
||||
barrier.wait()
|
||||
response = common.register(dict(registration))
|
||||
with lock:
|
||||
statuses.append(response.status)
|
||||
|
||||
threads = [threading.Thread(target=attempt) for _ in range(attempts)]
|
||||
for thread in threads:
|
||||
thread.start()
|
||||
for thread in threads:
|
||||
thread.join(timeout=60)
|
||||
self.assertEqual(statuses.count(201), 1, statuses)
|
||||
self.assertEqual(statuses.count(409), attempts - 1, statuses)
|
||||
# The single winner is usable, so exactly one row exists.
|
||||
session = common.login(registration["username"], registration["password"])
|
||||
self.assertEqual(session.status, 201)
|
||||
self.assertEqual(session.json["user"]["username"], registration["username"])
|
||||
|
||||
def test_chat_creation_rolls_back_on_unknown_member(self) -> None:
|
||||
alice = common.actor("rollback")
|
||||
before = self._chat_titles(alice.token)
|
||||
response = common.request(
|
||||
"POST",
|
||||
"/api/v1/chats",
|
||||
body={"title": "Атомарный чат", "member_ids": [alice.id, str(uuid.uuid4())]},
|
||||
token=alice.token,
|
||||
)
|
||||
common.expect_error(self, response, 400, "invalid_request")
|
||||
after = self._chat_titles(alice.token)
|
||||
# The chat row inserted before the failing membership step is gone.
|
||||
self.assertEqual(before, after)
|
||||
self.assertNotIn("Атомарный чат", after)
|
||||
|
||||
def _chat_titles(self, token: str) -> list[str]:
|
||||
titles: list[str] = []
|
||||
cursor = None
|
||||
for _ in range(50):
|
||||
path = "/api/v1/chats?limit=100"
|
||||
if cursor:
|
||||
path += "&cursor=" + cursor
|
||||
response = common.request("GET", path, token=token)
|
||||
self.assertEqual(response.status, 200)
|
||||
titles.extend(item["title"] for item in response.json["items"])
|
||||
cursor = response.json["next_cursor"]
|
||||
if cursor is None:
|
||||
return titles
|
||||
self.fail("chat pagination did not terminate")
|
||||
|
||||
def test_bounded_pool_serves_parallel_reads(self) -> None:
|
||||
alice = common.actor("pool")
|
||||
parallel = 50
|
||||
barrier = threading.Barrier(parallel)
|
||||
statuses: list[int] = []
|
||||
lock = threading.Lock()
|
||||
|
||||
def read() -> None:
|
||||
barrier.wait()
|
||||
# Chat lists are not cached, so this exercises the bounded SQL pool.
|
||||
response = common.request(
|
||||
"GET", "/api/v1/chats?limit=100", token=alice.token, timeout=10
|
||||
)
|
||||
with lock:
|
||||
statuses.append(response.status)
|
||||
|
||||
threads = [threading.Thread(target=read) for _ in range(parallel)]
|
||||
for thread in threads:
|
||||
thread.start()
|
||||
for thread in threads:
|
||||
thread.join(timeout=60)
|
||||
self.assertEqual(len(statuses), parallel)
|
||||
self.assertEqual(statuses, [200] * parallel)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
+2
-2
@@ -13,8 +13,8 @@ import urllib.request
|
||||
import uuid
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
LAB = json.loads((ROOT / 'lab.json').read_text())['number']
|
||||
SPEC = json.loads((ROOT / 'contracts/openapi.json').read_text())
|
||||
LAB = json.loads((ROOT / 'lab.json').read_text(encoding='utf-8'))['number']
|
||||
SPEC = json.loads((ROOT / 'contracts/openapi.json').read_text(encoding='utf-8'))
|
||||
|
||||
class Failure(AssertionError):
|
||||
pass
|
||||
|
||||
Reference in New Issue
Block a user