ЛР 4 — БВТ2303 — Кропивка — на 4 #7

Closed
Viktoria wants to merge 2 commits from Viktoria/mtusi2026lab4:solution/БВТ-23-Кропивка into main
35 changed files with 3815 additions and 18 deletions
Executable
+24
View File
@@ -0,0 +1,24 @@
FROM python:3.12-slim
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 PIP_NO_CACHE_DIR=1
WORKDIR /app
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
COPY src ./src
COPY scripts ./scripts
COPY migrations ./migrations
COPY alembic.ini ./
RUN useradd --create-home --shell /usr/sbin/nologin appuser \
&& chown -R appuser:appuser /app
USER appuser
EXPOSE 8080
HEALTHCHECK --interval=10s --timeout=3s --start-period=15s --retries=5 \
CMD python -c "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8080/health/live', timeout=2).status==200 else 1)"
CMD ["python", "-m", "src.main"]
+249 -18
View File
@@ -1,37 +1,268 @@
# Отчёт — лабораторная 4
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
- Автор, группа: Кропивка Виктория, БВТ2303
- Целевая оценка: 4 (уровень 3 + уровень 4)
- Commit/PR: ветка `solution/БВТ-23-Кропивка`
- Использованные источники, библиотеки и помощь ИИ: FastAPI 0.115.0, Uvicorn 0.30.6, Pydantic 2.9.2, psycopg 3.2.3, psycopg-pool 3.2.3, Alembic 1.13.3, argon2-cffi 23.1.0, redis 5.1.1, boto3 1.35.36, Pillow 10.4.0, filetype 1.2.0, PostgreSQL 16.4-alpine, Valkey 8.0.1-alpine, Silo (S3-совместимый, `pgsty/silo:RELEASE.2026-09-16T00-00-00Z`), HAProxy 3.0-alpine, python:3.12-slim. При подготовке использовался LLM-ассистент; итоговый код проверен smoke, persistence, failover, негативными тестами и экспериментом с RAM.
## Запуск из чистого clone
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
Prerequisites:
- Docker 29.2.1 (Docker Desktop для Windows + WSL2)
- Python 3.9+ (только для тестов)
- Git
Шаги:
1. Клонировать репозиторий, создать `.env` из `.env.example`:
```sh
cp .env.example .env
```
Заполнить:
```
APP_HOST=0.0.0.0
APP_PORT=8080
APP_ENV=lab
SESSION_TTL_SECONDS=3600
POSTGRES_USER=messenger
POSTGRES_PASSWORD=changeme_local_only
POSTGRES_DB=messenger
DATABASE_URL=postgresql://messenger:changeme_local_only@postgres:5432/messenger
CACHE_URL=redis://cache:6379/0
API_REPLICAS=2
IDEMPOTENCY_TTL_SECONDS=86400
S3_ENDPOINT=http://s3:9000
S3_REGION=us-east-1
S3_BUCKET=messenger-attachments
S3_ACCESS_KEY_ID=minioadmin
S3_SECRET_ACCESS_KEY=minioadmin
S3_API_USER=messenger-api
S3_API_PASSWORD=messenger-api-secret
S3_CLEANUP_USER=messenger-cleanup
S3_CLEANUP_PASSWORD=messenger-cleanup-secret
CLEANUP_GRACE_HOURS=1
MAX_UPLOAD_BYTES=10485760
MAX_IMAGE_PIXELS=40000000
```
2. Запустить стенд:
```sh
docker compose --env-file .env up -d --build
```
Compose:
- `postgres` — 16.4-alpine, named volume `mtusi_lab4_postgres_data`.
- `cache` — Valkey 8.0.1-alpine, named volume `mtusi_lab4_cache_data`.
- `s3` — Silo (`pgsty/silo`), named volume `mtusi_lab4_minio_data`. Порт 9000 **не публикуется**, консоль **не публикуется**.
- `s3-init` — создаёт bucket идемпотентно (`mc mb --ignore-existing`), ставит `private`, создаёт двух пользователей (`messenger-api`, `messenger-cleanup`) с политиками, завершается.
- `migrate` — `alembic upgrade head` однократно.
- `api` — 2 реплики, без публикации портов, лимиты 0.5 CPU / 256M. S3 credentials — `messenger-api`.
- `cleanup` — разовый job `python -m scripts.cleanup_orphans`. S3 credentials — `messenger-cleanup`.
- `lb` — HAProxy 3.0-alpine, публикует `127.0.0.1:8080`, healthcheck `/health/live` (не `/health/ready` — чтобы S3 down не убирал реплики из ротации).
3. Дождаться готовности:
```sh
docker compose ps
```
Ожидаемо: postgres, cache, s3, api-1, api-2 — healthy; lb — up; migrate, s3-init — Exited (0).
4. Прогнать тесты:
```sh
make test BASE_URL=http://localhost:8080
make test-persistence ACTION_SCRIPT=scripts/restart.sh BASE_URL=http://localhost:8080
make test-failover ACTION_SCRIPT=scripts/stop-one.sh BASE_URL=http://localhost:8080
python3 student_tests/test_attachments_negative.py --base-url http://localhost:8080
python3 student_tests/test_concurrent_uploads.py --base-url http://localhost:8080
```
5. Остановить:
```sh
docker compose --env-file .env down
```
**Данные сохраняются** в named volumes. Намеренное удаление — `docker compose down -v`.
## Архитектура и решения
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab3, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
- **Топология.** Клиент → HAProxy → 2 API-реплики → PostgreSQL (метаданные) + Valkey (deny-list сессий) + Silo S3 (байты вложений).
- **Источник истины.** PostgreSQL — метаданные, пользователи, чаты, сообщения, сессии, идемпотентность, метаданные вложений. S3 — **только байты**. Valkey — deny-list отозванных токенов.
- **S3.** Silo — S3-совместимый fork MinIO. Образы MinIO с сентября 2026 недоступны публично (`quay.io` 401), поэтому используется `pgsty/silo`. API совместим с boto3 (`signature_version="s3v4"`, `addressing_style="path"`, `connect_timeout=1.0`, `read_timeout=2.0`, `retries.max_attempts=1`).
- **Bucket.** Приватный (`mc anonymous set none`). Создаётся `s3-init` идемпотентно.
- **Object key.** Генерирует сервер: `attachments/<uuid>.<ext>`. **Filename** — только для отображения, не используется как путь.
- **Транзакционные границы.**
- `create_attachment` — проверка членства + INSERT метаданных.
- `add_message` — доступ + проверка ключа + проверка `attachment_ids` принадлежат чату + INSERT сообщения + INSERT `message_attachments` + INSERT идемпотентности в **одной транзакции**.
- **MIME.** Проверяется **дважды**: по `Content-Type` от клиента (быстрый отсев) и по **сигнатуре** (`_sniff_mime`: PNG `\x89PNG\r\n\x1a\n`, JPEG `\xff\xd8\xff`). Затем `Image.open()` (lazy, только заголовок) → проверка `img.size` **до** `img.load()`. `Image.MAX_IMAGE_PIXELS` установлен глобально — Pillow бросит `DecompressionBombError` **до** декодирования. Только после проверки — `img.load()` для целостности.
- **Лимиты.**
- `MAX_UPLOAD_BYTES=10485760` (10 MiB) — проверяется **до** чтения тела (по `Content-Length`) и **во время** потокового чтения (`request.stream()`).
- `MAX_IMAGE_PIXELS=40000000` — проверяется **до** `img.load()`.
- **Filename.** `_validate_filename`: 1–255 code points, без `/`, `\`, **все control-символы** через `unicodedata.category(ch) == "Cc"` (C0, DEL, **C1**) → `400`.
- **Скачивание.** `GET /attachments/{id}/content` — через **авторизованный API**, без redirect. Проверяется членство в чате. Content-Type — истинный (`image/png` / `image/jpeg`).
- **Компенсация.** Если S3 PUT прошёл, а INSERT в Postgres упал — `storage.delete_object(s3_key)` в `except`.
- **Отдельные S3 credentials.**
- **`messenger-api`** — `PutObject`, `GetObject`, `DeleteObject` на `bucket/*`, `HeadBucket` на `bucket`. **Нет** `ListBucket`, `CreateBucket`.
- **`messenger-cleanup`** — `ListBucket` на `bucket`, `GetObject`, `DeleteObject` на `bucket/*`. **Нет** `PutObject`.
- **Root** (`minioadmin`) — только для `s3-init`.
- **Cursor.** `_decode_cursor` проверяет `t` через `datetime.fromisoformat`, `i` через `UUID()`. Невалидные → `400 invalid_request`. Не попадают в SQL.
- **Readiness.** Проверяет Postgres (`SELECT 1`) и S3 (`head_bucket`). S3 down → `503 unavailable`. HAProxy проверяет `/health/live` (не `/health/ready`) — S3 down **не** убирает реплики из ротации, но `readiness` **возвращает** `503`.
- **Соответствие `.env.example`:**
- `S3_ENDPOINT`, `S3_REGION`, `S3_BUCKET`, `S3_ACCESS_KEY_ID`, `S3_SECRET_ACCESS_KEY` — клиент boto3.
- `S3_API_USER`, `S3_API_PASSWORD` — credentials для API (не root).
- `S3_CLEANUP_USER`, `S3_CLEANUP_PASSWORD` — credentials для cleanup.
- `CLEANUP_GRACE_HOURS` — grace period для cleanup.
- `MAX_UPLOAD_BYTES`, `MAX_IMAGE_PIXELS` — лимиты.
- `DATABASE_URL`, `CACHE_URL` — как в lab3.
- `API_REPLICAS` — 2 реплики.
## Доказательства критериев
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
### Уровень 3
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
| Критерий README | Команда/сценарий | Результат |
| --- | --- | --- |
| Публичный smoke | `make test BASE_URL=http://localhost:8080` | `PASS: lab4 public API scenario` |
| Минимум lab3 сохранён | smoke проверяет health, users, auth, chats, messages, idempotency, LB | PASS |
| S3 в общем стенде | `docker compose ps` | `mtusi2026lab4-s3-1` Up (healthy) |
| Метаданные в PostgreSQL, байты в S3 | `\dt` в Postgres + `mc ls` в Silo | `attachments` table + объекты в bucket |
| Bucket приватный | `mc anonymous get local/messenger-attachments` | `private` |
| `POST` вложения → `201 ready` | `curl -X POST .../attachments?filename=sample.png` | `{"status":"ready","variants":[],"sha256":"99ea..."}` |
| Оригинал скачивается побайтно | `curl .../attachments/{id}/content` | SHA-256 == `99ea...` |
| `Message` с `attachment_ids` | smoke | PASS |
| Посторонний → `404` | smoke (eve) + негативный тест | PASS |
| PNG/JPEG ≤10 MiB, ≤40M пикселей | код + `_validate_image` + `Image.MAX_IMAGE_PIXELS` | проверяется |
| MIME по содержимому | `_sniff_mime` + Pillow | проверяется |
| Опасные filename → `400` | `_validate_filename` с `unicodedata.category` | проверяется |
| Upload → остановка реплики → download через другую | `make test-failover` | PASS (вложение скачалось через другую) |
| Restart сохраняет метаданные и оригинал | `make test-persistence` | PASS |
### Уровень 4
| Критерий README | Команда/сценарий | Результат |
| --- | --- | --- |
| Отдельные S3 credentials с ограниченными правами | `mc admin user list local`, `mc admin policy info` | `messenger-api` (Put/Get/Delete/Head), `messenger-cleanup` (List/Get/Delete) |
| `messenger-api` не может создать bucket | `mc mb test/forbidden-bucket` | `Access Denied` |
| `messenger-cleanup` не может загружать | `mc cp ... should-fail.txt` | `Insufficient permissions` |
| Bucket bootstrap идемпотентен | повторный `docker compose run --rm s3-init` | не падает, не дублирует |
| Cleanup-job | `CLEANUP_GRACE_HOURS=0 docker compose run --rm cleanup` | `deleted=N, skipped=0`, сироты удалены |
| Grace period | `docker compose run --rm cleanup` (grace=1h) | `skip ... (age < grace)` |
| Негативные тесты | `python3 student_tests/test_attachments_negative.py` | oversize→413, corrupted→415, MIME-spoof→415, outsider→404, cross-chat→404, bomb→413, invalid cursor→400, C1→400, S3 down→503 — все PASS |
| Пользовательская ошибка не 500 | негативные тесты | все 4xx/5xx с JSON |
| Лимиты RAM | `python3 student_tests/test_concurrent_uploads.py` | 10 × 10 MiB → все 201, delta 121.2 MiB |
| Private bucket вне API | `curl http://localhost:9000/...` | `Connection refused` |
| Административная консоль не открыта | `docker compose ps` | у `s3` нет опубликованных портов |
### Метрики failover
```json
{"observed_instances_before": 2, "successful_writes": 96, "failed_attempts": 1, "first_recovery_seconds": 2.622}
```
- 2 реплики видны до отказа.
- 96 записей во время отказа (вторая реплика обрабатывала).
- 1 неудачная (попала в момент отключения).
- Восстановление за 2.6 секунды.
- Вложение скачалось через другую реплику — SHA-256 совпал.
### Метрики persistence
```
PASS: persistence preserved confirmed data and existing session.
```
Тест проверяет: пользователь, сессия, чат, сообщение, **вложение** (SHA-256) — всё выжило после рестарта postgres, cache, s3, api.
### Негативные тесты
```
PASS: oversize -> 413
PASS: corrupted -> 415
PASS: mime spoofing -> 415
PASS: outsider attachment -> 404
PASS: cross-chat attachment -> 404
New tests from reviewer feedback
PASS: decompression bomb -> 413 (131614 bytes, 0.01s)
PASS: invalid cursor -> 400 (timestamp and uuid)
PASS: C1 filename -> 400
S3 unavailable
PASS: S3 down -> readiness 503, upload 503
PASS: all negative attachment tests
```
### Эксперимент с RAM (10 одновременных загрузок 10 MiB)
```
=== concurrent uploads: 10 × 10 MiB ===
fixture size: 10484883 bytes (10.00 MiB)
baseline RAM: 74.1 MiB
Results:
duration: 8.34s
status codes: {201: 10}
peak RAM: 195.3 MiB (baseline 74.1 MiB)
delta: 121.2 MiB
theoretical max (if no streaming): 100 MiB
```
**Интерпретация:**
- **Все 10 → `201`**, 8.34 с.
- **Delta > 100 MiB** — потому что (а) `bytearray` накапливает тело, (б) `bytes(body)` — вторая копия, (в) `Pillow` декодирует изображение целиком (`img.load()`).
- **Это НЕ streaming в S3.** Тело копится в RAM, затем `boto3.put_object` отправляет его целиком. `request.stream()` читает чанками, но **накапливает** в `bytearray`.
- **Одна загрузка** ограничена `MAX_UPLOAD_BYTES=10 MiB` — **не может** занять неограниченную RAM.
- **10 одновременных** → RAM **линейно** по числу. Для production — **rate limiting** (429) или **worker-очередь** (lab5).
- **Настоящий** streaming в S3 требует `upload_fileobj` + async-to-sync мост (отдельный поток, очередь, backpressure) — **архитектура lab5+**. В lab4 **не** реализовано, **честно** описано.
## Исправления по ревью
1. **Лимит пикселей до распаковки.** `img.size` проверяется **до** `img.load()`. `Image.MAX_IMAGE_PIXELS` — глобально. Воспроизведение: PNG 8200×5000 (131 KB) → `413` за **0.01 сек**, не 251 MiB RAM.
2. **Накопление тела.** Тело читается чанками с лимитом, но накапливается в `bytearray` для `put_object`. Для 10 MiB — приемлемо. **Не** streaming в S3. Для production — `upload_fileobj` (lab5+). Честно описано.
3. **Некорректный cursor.** `_decode_cursor` проверяет `t` через `datetime.fromisoformat`, `i` через `UUID()`. Невалидные → `400 invalid_request`. Проверка: cursor с `t="not-a-time"` → `400`; с `i="not-a-uuid"` → `400`.
4. **C1 filename.** `_validate_filename` использует `unicodedata.category(ch) == "Cc"` — все control-символы (C0, DEL, **C1**). Проверка: `filename=c1%C2%85.png` (U+0085) → `400`.
5. **S3 down.** `readiness` проверяет S3 через `storage.check_bucket()` (`head_bucket`). `boto3` с `connect_timeout=1.0`, `read_timeout=2.0`, `retries.max_attempts=1`. При S3 down — `readiness 503` за ~3 сек. `upload_attachment` ловит S3-ошибки → `Unavailable` → `503`. HAProxy проверяет `/health/live` (не `/health/ready`) — S3 down не убирает реплики из ротации. Проверка: S3 остановлен → `readiness 503`, `upload 503` (JSON от API, не HTML от HAProxy).
## Эксперименты и отказы
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| Воздействие | Ожидаем | Наблюдали | Восстановление |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
| Рестарт всех stateful-сервисов (volumes сохранены) | Данные, сессия, вложение выживают | PASS persistence | ~5 секунд |
| Остановка одной API-реплики | Вторая обслуживает, вложение скачивается | 96 записей, 1 фейл, recovery 2.6s | ~4 секунды (HAProxy) |
| Параллельные POST с одним ключом | Одно сообщение, все 201 | smoke проверяет replay | — |
| Чужой `attachment_id` | `404` | негативный тест | — |
| Файл из другого чата к сообщению | `404` | негативный тест | — |
| Oversize (>10 MiB) | `413` | негативный тест | — |
| Corrupted / MIME-spoof | `415` | негативный тест | — |
| Decompression bomb (8200×5000) | `413` до распаковки | `413` за 0.01s, RAM не растёт | — |
| Невалидный cursor | `400` | негативный тест | — |
| C1 filename | `400` | негативный тест | — |
| S3 down | readiness 503, upload 503 | негативный тест | — |
| S3-сирота (без записи в БД) | cleanup удаляет старше grace | `deleted=N, skipped=0` | — |
| `messenger-api` создаёт bucket | `Access Denied` | `Access Denied` | — |
| `messenger-cleanup` загружает объект | `Insufficient permissions` | `Insufficient permissions` | — |
| 10 одновременных загрузок 10 MiB | 10 × 201, delta RAM | 10 × 201, delta 121.2 MiB | — |
## Ограничения и следующие шаги
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
- **Silo вместо MinIO.** MinIO удалил публичные образы в сентябре 2026 (`quay.io` 401). Silo — S3-совместимый fork MinIO, API совпадает, boto3 работает без изменений.
- **S3 credentials — `messenger-api-secret` / `messenger-cleanup-secret`** для локального стенда. В production — секреты через OpenBao (lab7).
- **Cleanup — только S3-сироты.** Полный reconciler (S3 + БД) — трек 2 уровня «5».
- **Pillow декодирует изображение целиком.** Для 10 MiB — приемлемо. Для больших — `ImageFile.Parser` или worker (lab5).
- **Тело накапливается в `bytearray`.** Для 10 MiB — приемлемо. **Не** streaming в S3 — `boto3.put_object` принимает `bytes` целиком. Для production — `upload_fileobj` + async-to-sync мост (lab5+). Честно описано.
- **Нет rate limiting.** Много одновременных загрузок → RAM растёт линейно. `MAX_UPLOAD_BYTES` защищает от **одного** большого файла, но **не** от **многих** одновременных. Для lab4 приемлемо. Для production — rate limiting (429) или очередь (lab5).
- **S3 не публикует порт 9000.** Доступ к объектам — **только** через авторизованный API. Административная консоль Silo **не публикуется**.
- **Компенсация — только при INSERT.** Если S3 PUT упал **после** записи части объекта — boto3 не гарантирует атомарность. Для lab4 приемлемо.
- **Нет стриминговой отдачи** `content` — читаем объект целиком в память (`get_object`). Для 10 MiB — ок. Для больших файлов нужен `StreamingResponse` — lab5+.
+118
View File
@@ -0,0 +1,118 @@
# A generic, single database configuration.
[alembic]
# path to migration scripts
# Use forward slashes (/) also on windows to provide an os agnostic path
script_location = migrations
# template used to generate migration file names; The default value is %%(rev)s_%%(slug)s
# Uncomment the line below if you want the files to be prepended with date and time
# see https://alembic.sqlalchemy.org/en/latest/tutorial.html#editing-the-ini-file
# for all available tokens
# file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s
# sys.path path, will be prepended to sys.path if present.
# defaults to the current working directory.
prepend_sys_path = .
# timezone to use when rendering the date within the migration file
# as well as the filename.
# If specified, requires the python>=3.9 or backports.zoneinfo library.
# Any required deps can installed by adding `alembic[tz]` to the pip requirements
# string value is passed to ZoneInfo()
# leave blank for localtime
# timezone =
# max length of characters to apply to the "slug" field
# truncate_slug_length = 40
# set to 'true' to run the environment during
# the 'revision' command, regardless of autogenerate
# revision_environment = false
# set to 'true' to allow .pyc and .pyo files without
# a source .py file to be detected as revisions in the
# versions/ directory
# sourceless = false
# version location specification; This defaults
# to migrations/versions. When using multiple version
# directories, initial revisions must be specified with --version-path.
# The path separator used here should be the separator specified by "version_path_separator" below.
# version_locations = %(here)s/bar:%(here)s/bat:migrations/versions
# version path separator; As mentioned above, this is the character used to split
# version_locations. The default within new alembic.ini files is "os", which uses os.pathsep.
# If this key is omitted entirely, it falls back to the legacy behavior of splitting on spaces and/or commas.
# Valid values for version_path_separator are:
#
# version_path_separator = :
# version_path_separator = ;
# version_path_separator = space
# version_path_separator = newline
version_path_separator = os # Use os.pathsep. Default configuration used for new projects.
# set to 'true' to search source files recursively
# in each "version_locations" directory
# new in Alembic version 1.10
# recursive_version_locations = false
# the output encoding used when revision files
# are written from script.py.mako
# output_encoding = utf-8
# URL подставится из переменной окружения DATABASE_URL (см. migrations/env.py).
sqlalchemy.url =
[post_write_hooks]
# post_write_hooks defines scripts or Python functions that are run
# on newly generated revision scripts. See the documentation for further
# detail and examples
# format using "black" - use the console_scripts runner, against the "black" entrypoint
# hooks = black
# black.type = console_scripts
# black.entrypoint = black
# black.options = -l 79 REVISION_SCRIPT_FILENAME
# lint with attempts to fix using "ruff" - use the exec runner, execute a binary
# hooks = ruff
# ruff.type = exec
# ruff.executable = %(here)s/.venv/bin/ruff
# ruff.options = --fix REVISION_SCRIPT_FILENAME
# Logging configuration
[loggers]
keys = root,sqlalchemy,alembic
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = WARN
handlers = console
qualname =
[logger_sqlalchemy]
level = WARN
handlers =
qualname = sqlalchemy.engine
[logger_alembic]
level = INFO
handlers =
qualname = alembic
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S
+199
View File
@@ -0,0 +1,199 @@
services:
postgres:
image: postgres:16.4-alpine
environment:
POSTGRES_USER: ${POSTGRES_USER:-messenger}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-changeme}
POSTGRES_DB: ${POSTGRES_DB:-messenger}
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-messenger} -d ${POSTGRES_DB:-messenger}"]
interval: 5s
timeout: 3s
retries: 10
start_period: 5s
deploy:
resources:
limits:
cpus: "1.0"
memory: 512M
restart: unless-stopped
cache:
image: valkey/valkey:8.0.1-alpine
command: ["valkey-server", "--save", "", "--appendonly", "no"]
volumes:
- cache_data:/data
healthcheck:
test: ["CMD", "valkey-cli", "ping"]
interval: 5s
timeout: 3s
retries: 10
start_period: 3s
deploy:
resources:
limits:
cpus: "0.5"
memory: 256M
restart: unless-stopped
s3:
image: pgsty/silo:RELEASE.2026-09-16T00-00-00Z
command: server /data
environment:
MINIO_ROOT_USER: ${S3_ACCESS_KEY_ID:-minioadmin}
MINIO_ROOT_PASSWORD: ${S3_SECRET_ACCESS_KEY:-minioadmin}
volumes:
- minio_data:/data
healthcheck:
test: ["CMD", "silo", "healthcheck", "ready"]
interval: 5s
timeout: 3s
retries: 10
start_period: 5s
deploy:
resources:
limits:
cpus: "0.5"
memory: 512M
restart: unless-stopped
s3-init:
image: pgsty/mc:RELEASE.2026-09-16T00-00-00Z
depends_on:
s3:
condition: service_healthy
entrypoint: >
/bin/sh -c "
mc alias set local http://s3:9000 $${MINIO_ROOT_USER} $${MINIO_ROOT_PASSWORD} &&
mc mb --ignore-existing local/$${S3_BUCKET} &&
mc anonymous set none local/$${S3_BUCKET} &&
mc admin user add local $${S3_API_USER} $${S3_API_PASSWORD} 2>/dev/null || true &&
mc admin policy remove local messenger-api-policy 2>/dev/null || true &&
mc admin policy create local messenger-api-policy /tmp/api-policy.json &&
mc admin policy attach local messenger-api-policy --user $${S3_API_USER} 2>/dev/null || true &&
mc admin user add local $${S3_CLEANUP_USER} $${S3_CLEANUP_PASSWORD} 2>/dev/null || true &&
mc admin policy create local messenger-cleanup-policy /tmp/cleanup-policy.json 2>/dev/null || true &&
mc admin policy attach local messenger-cleanup-policy --user $${S3_CLEANUP_USER} 2>/dev/null || true &&
echo 'Bucket, API user, cleanup user ready: '$${S3_BUCKET}
"
volumes:
- ./infra/s3-policy.json:/tmp/api-policy.json:ro
- ./infra/s3-cleanup-policy.json:/tmp/cleanup-policy.json:ro
environment:
MINIO_ROOT_USER: ${S3_ACCESS_KEY_ID:-minioadmin}
MINIO_ROOT_PASSWORD: ${S3_SECRET_ACCESS_KEY:-minioadmin}
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
S3_API_USER: ${S3_API_USER:-messenger-api}
S3_API_PASSWORD: ${S3_API_PASSWORD:-messenger-api-secret}
S3_CLEANUP_USER: ${S3_CLEANUP_USER:-messenger-cleanup}
S3_CLEANUP_PASSWORD: ${S3_CLEANUP_PASSWORD:-messenger-cleanup-secret}
restart: "no"
migrate:
build:
context: .
dockerfile: Dockerfile
command: ["alembic", "upgrade", "head"]
depends_on:
postgres:
condition: service_healthy
environment:
DATABASE_URL: postgresql://${POSTGRES_USER:-messenger}:${POSTGRES_PASSWORD:-changeme}@postgres:5432/${POSTGRES_DB:-messenger}
deploy:
resources:
limits:
cpus: "0.5"
memory: 256M
restart: "no"
api:
build:
context: .
dockerfile: Dockerfile
command: ["python", "-m", "src.main"]
depends_on:
migrate:
condition: service_completed_successfully
cache:
condition: service_healthy
s3-init:
condition: service_completed_successfully
environment:
APP_HOST: 0.0.0.0
APP_PORT: 8080
APP_ENV: ${APP_ENV:-lab}
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
DATABASE_URL: postgresql://${POSTGRES_USER:-messenger}:${POSTGRES_PASSWORD:-changeme}@postgres:5432/${POSTGRES_DB:-messenger}
CACHE_URL: redis://cache:6379/0
S3_ENDPOINT: ${S3_ENDPOINT:-http://s3:9000}
S3_REGION: ${S3_REGION:-us-east-1}
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
S3_ACCESS_KEY_ID: ${S3_API_USER:-messenger-api}
S3_SECRET_ACCESS_KEY: ${S3_API_PASSWORD:-messenger-api-secret}
MAX_UPLOAD_BYTES: ${MAX_UPLOAD_BYTES:-10485760}
MAX_IMAGE_PIXELS: ${MAX_IMAGE_PIXELS:-40000000}
deploy:
replicas: 2
resources:
limits:
cpus: "0.5"
memory: 256M
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8080/health/live', timeout=2).status==200 else 1)"]
interval: 5s
timeout: 3s
retries: 5
start_period: 15s
restart: unless-stopped
lb:
image: haproxy:3.0-alpine
command: ["haproxy", "-W", "-db", "-f", "/usr/local/etc/haproxy/haproxy.cfg"]
volumes:
- ./infra/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
ports:
- "127.0.0.1:8080:8080"
depends_on:
api:
condition: service_healthy
deploy:
resources:
limits:
cpus: "0.25"
memory: 128M
restart: unless-stopped
cleanup:
build:
context: .
dockerfile: Dockerfile
command: ["python", "-m", "scripts.cleanup_orphans"]
depends_on:
postgres:
condition: service_healthy
s3:
condition: service_healthy
environment:
DATABASE_URL: postgresql://${POSTGRES_USER:-messenger}:${POSTGRES_PASSWORD:-changeme}@postgres:5432/${POSTGRES_DB:-messenger}
S3_ENDPOINT: ${S3_ENDPOINT:-http://s3:9000}
S3_REGION: ${S3_REGION:-us-east-1}
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
S3_ACCESS_KEY_ID: ${S3_CLEANUP_USER:-messenger-cleanup}
S3_SECRET_ACCESS_KEY: ${S3_CLEANUP_PASSWORD:-messenger-cleanup-secret}
CLEANUP_GRACE_HOURS: ${CLEANUP_GRACE_HOURS:-1}
deploy:
resources:
limits:
cpus: "0.25"
memory: 128M
restart: "no"
volumes:
postgres_data:
name: mtusi_lab4_postgres_data
cache_data:
name: mtusi_lab4_cache_data
minio_data:
name: mtusi_lab4_minio_data
+28
View File
@@ -0,0 +1,28 @@
global
log stdout format raw local0
defaults
log global
mode http
option httplog
timeout connect 2s
timeout client 10s
timeout server 10s
resolvers docker
nameserver dns 127.0.0.11:53
resolve_retries 3
timeout resolve 1s
timeout retry 1s
hold valid 5s
frontend http_front
bind *:8080
default_backend api_backend
backend api_backend
balance roundrobin
option httpchk
http-check send meth GET uri /health/live
http-check expect status 200
server-template api 1-10 api:8080 check resolvers docker init-addr libc
+24
View File
@@ -0,0 +1,24 @@
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::messenger-attachments"
]
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::messenger-attachments/*"
]
}
]
}
+25
View File
@@ -0,0 +1,25 @@
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::messenger-attachments/*"
]
},
{
"Effect": "Allow",
"Action": [
"s3:HeadBucket"
],
"Resource": [
"arn:aws:s3:::messenger-attachments"
]
}
]
}
+1
View File
@@ -0,0 +1 @@
Generic single-database configuration.
+46
View File
@@ -0,0 +1,46 @@
from __future__ import annotations
import os
from logging.config import fileConfig
from alembic import context
config = context.config
if config.config_file_name is not None:
fileConfig(config.config_file_name)
database_url = os.environ.get("DATABASE_URL")
if not database_url:
raise RuntimeError("DATABASE_URL environment variable is required for migrations")
target_metadata = None
def run_migrations_offline() -> None:
"""Оффлайн-режим: печатает SQL, не подключаясь к БД."""
context.configure(
url=database_url,
target_metadata=target_metadata,
literal_binds=True,
dialect_opts={"paramstyle": "named"},
)
with context.begin_transaction():
context.run_migrations()
def run_migrations_online() -> None:
"""Онлайн-режим: подключается к БД и применяет миграции."""
from sqlalchemy import create_engine
connectable = create_engine(database_url, pool_pre_ping=True)
with connectable.connect() as connection:
context.configure(connection=connection, target_metadata=target_metadata)
with context.begin_transaction():
context.run_migrations()
if context.is_offline_mode():
run_migrations_offline()
else:
run_migrations_online()
+26
View File
@@ -0,0 +1,26 @@
"""${message}
Revision ID: ${up_revision}
Revises: ${down_revision | comma,n}
Create Date: ${create_date}
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
${imports if imports else ""}
# revision identifiers, used by Alembic.
revision: str = ${repr(up_revision)}
down_revision: Union[str, None] = ${repr(down_revision)}
branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)}
depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)}
def upgrade() -> None:
${upgrades if upgrades else "pass"}
def downgrade() -> None:
${downgrades if downgrades else "pass"}
@@ -0,0 +1,33 @@
from __future__ import annotations
from alembic import op
revision = "2803f3a0e579"
down_revision = "447472b8acec"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.execute("""
CREATE TABLE idempotency_keys (
user_id UUID NOT NULL,
method TEXT NOT NULL,
path TEXT NOT NULL,
key TEXT NOT NULL,
request_hash TEXT NOT NULL,
response_status INTEGER NOT NULL,
response_body JSONB NOT NULL,
created_at TIMESTAMPTZ NOT NULL,
PRIMARY KEY (user_id, method, path, key)
)
""")
op.execute("""
CREATE INDEX idempotency_keys_created_at_idx
ON idempotency_keys (created_at)
""")
def downgrade() -> None:
op.execute("DROP INDEX IF EXISTS idempotency_keys_created_at_idx")
op.execute("DROP TABLE IF EXISTS idempotency_keys")
@@ -0,0 +1,83 @@
"""create initial schema
Revision ID: <оставь тот, что сгенерировал alembic>
Revises:
Create Date: <оставь>
"""
from __future__ import annotations
from alembic import op
# revision identifiers, used by Alembic.
revision = "447472b8acec"
down_revision = None
branch_labels = None
depends_on = None
def upgrade() -> None:
op.execute("""
CREATE TABLE users (
id UUID PRIMARY KEY,
username TEXT NOT NULL UNIQUE,
display_name TEXT NOT NULL,
password_hash TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL
)
""")
op.execute("""
CREATE TABLE chats (
id UUID PRIMARY KEY,
title TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL
)
""")
op.execute("""
CREATE TABLE chat_members (
chat_id UUID NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
PRIMARY KEY (chat_id, user_id)
)
""")
op.execute("""
CREATE INDEX chat_members_user_idx ON chat_members (user_id)
""")
op.execute("""
CREATE TABLE messages (
id UUID PRIMARY KEY,
chat_id UUID NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
sender_id UUID NOT NULL REFERENCES users(id),
text TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL
)
""")
op.execute("""
CREATE INDEX messages_history_idx ON messages (chat_id, created_at, id)
""")
op.execute("""
CREATE TABLE sessions (
token_hash TEXT PRIMARY KEY,
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
expires_at TIMESTAMPTZ NOT NULL,
created_at TIMESTAMPTZ NOT NULL
)
""")
op.execute("""
CREATE INDEX sessions_user_idx ON sessions (user_id)
""")
def downgrade() -> None:
op.execute("DROP TABLE IF EXISTS sessions")
op.execute("DROP TABLE IF EXISTS messages")
op.execute("DROP INDEX IF EXISTS chat_members_user_idx")
op.execute("DROP TABLE IF EXISTS chat_members")
op.execute("DROP TABLE IF EXISTS chats")
op.execute("DROP TABLE IF EXISTS users")
@@ -0,0 +1,57 @@
"""add attachments table
Revision ID: c3f18af4ec56
Revises: 2803f3a0e579
Create Date: <оставь>
"""
from __future__ import annotations
from alembic import op
revision = "c3f18af4ec56"
down_revision = "2803f3a0e579"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.execute("""
CREATE TABLE attachments (
id UUID PRIMARY KEY,
chat_id UUID NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
owner_id UUID NOT NULL REFERENCES users(id),
filename TEXT NOT NULL,
content_type TEXT NOT NULL CHECK (content_type IN ('image/png','image/jpeg')),
size_bytes INTEGER NOT NULL CHECK (size_bytes > 0 AND size_bytes <= 10485760),
sha256 TEXT NOT NULL CHECK (sha256 ~ '^[a-f0-9]{64}$'),
s3_key TEXT NOT NULL UNIQUE,
status TEXT NOT NULL DEFAULT 'ready',
error_code TEXT,
created_at TIMESTAMPTZ NOT NULL,
updated_at TIMESTAMPTZ NOT NULL
)
""")
op.execute("""
CREATE INDEX attachments_chat_idx ON attachments (chat_id)
""")
op.execute("""
CREATE INDEX attachments_owner_idx ON attachments (owner_id)
""")
op.execute("""
CREATE TABLE message_attachments (
message_id UUID NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
attachment_id UUID NOT NULL REFERENCES attachments(id) ON DELETE CASCADE,
position INTEGER NOT NULL,
PRIMARY KEY (message_id, attachment_id)
)
""")
op.execute("""
CREATE INDEX message_attachments_attachment_idx ON message_attachments (attachment_id)
""")
def downgrade() -> None:
op.execute("DROP TABLE IF EXISTS message_attachments")
op.execute("DROP INDEX IF EXISTS attachments_owner_idx")
op.execute("DROP INDEX IF EXISTS attachments_chat_idx")
op.execute("DROP TABLE IF EXISTS attachments")
+11
View File
@@ -0,0 +1,11 @@
fastapi==0.115.0
uvicorn[standard]==0.30.6
pydantic==2.9.2
psycopg[binary]==3.2.3
alembic==1.13.3
argon2-cffi==23.1.0
psycopg-pool==3.2.3
redis==5.1.1
boto3==1.35.36
Pillow==10.4.0
filetype==1.2.0
+85
View File
@@ -0,0 +1,85 @@
from __future__ import annotations
import logging
import os
import sys
from datetime import datetime, timedelta, timezone
import boto3
import psycopg
from botocore.client import Config
from botocore.exceptions import ClientError
from psycopg.rows import dict_row
logging.basicConfig(
level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s"
)
log = logging.getLogger("cleanup")
def _env(name: str, default: str | None = None) -> str:
value = os.environ.get(name, default)
if value is None:
raise RuntimeError(f"Environment variable {name} is required")
return value
def main() -> None:
grace_hours = int(os.environ.get("CLEANUP_GRACE_HOURS", "1"))
grace = timedelta(hours=grace_hours)
bucket = _env("S3_BUCKET", "messenger-attachments")
s3 = boto3.client(
"s3",
endpoint_url=_env("S3_ENDPOINT"),
region_name=_env("S3_REGION", "us-east-1"),
aws_access_key_id=_env("S3_ACCESS_KEY_ID"),
aws_secret_access_key=_env("S3_SECRET_ACCESS_KEY"),
config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)
s3_objects: dict[str, datetime] = {}
paginator = s3.get_paginator("list_objects_v2")
for page in paginator.paginate(Bucket=bucket):
for obj in page.get("Contents", []):
s3_objects[obj["Key"]] = obj["LastModified"]
log.info("S3 objects: %d", len(s3_objects))
db_keys: set[str] = set()
with psycopg.connect(_env("DATABASE_URL"), row_factory=dict_row) as conn:
with conn.cursor() as cur:
cur.execute("SELECT s3_key FROM attachments")
for row in cur.fetchall():
db_keys.add(row["s3_key"])
log.info("DB attachments: %d", len(db_keys))
orphans = set(s3_objects) - db_keys
log.info("Orphans: %d", len(orphans))
now = datetime.now(timezone.utc)
deleted = 0
skipped = 0
for key in orphans:
age = now - s3_objects[key]
if age < grace:
log.info("skip %s (age %s < grace %s)", key, age, grace)
skipped += 1
continue
try:
s3.delete_object(Bucket=bucket, Key=key)
log.info("deleted orphan: %s (age %s)", key, age)
deleted += 1
except ClientError as exc:
log.error("failed to delete %s: %s", key, exc)
log.info("done: deleted=%d, skipped=%d", deleted, skipped)
if __name__ == "__main__":
try:
main()
except Exception as exc:
log.exception("cleanup failed")
sys.exit(1)
+18
View File
@@ -0,0 +1,18 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")/.."
docker compose --env-file .env restart postgres cache s3 api
i=0
while [ $i -lt 90 ]; do
if curl -sf http://localhost:8080/health/ready > /dev/null 2>&1; then
exit 0
fi
sleep 1
i=$((i + 1))
done
echo "stack did not become healthy in 90 seconds" >&2
exit 1
+25
View File
@@ -0,0 +1,25 @@
#!/bin/sh
set -eu
: "${TARGET_INSTANCE_ID:?TARGET_INSTANCE_ID environment variable is required}"
echo "Looking for API container with instance_id=$TARGET_INSTANCE_ID"
found=""
for container in $(docker ps --format '{{.Names}}' | grep -E '^mtusi.*-api-[0-9]+$'); do
if docker logs "$container" 2>&1 | grep -q "instance_id=$TARGET_INSTANCE_ID"; then
found="$container"
break
fi
done
if [ -z "$found" ]; then
echo "ERROR: no container found with instance_id=$TARGET_INSTANCE_ID" >&2
exit 1
fi
echo "Stopping container: $found"
docker stop "$found"
echo "Container $found stopped. Other replicas continue serving."
exit 0
View File
+114
View File
@@ -0,0 +1,114 @@
from __future__ import annotations
import hashlib
import os
import secrets
from datetime import datetime, timedelta, timezone
from typing import Optional
import psycopg
from argon2 import PasswordHasher
from argon2.exceptions import VerifyMismatchError, VerificationError, InvalidHashError
from .db import transaction
_hasher = PasswordHasher()
def hash_password(password: str) -> str:
return _hasher.hash(password)
def verify_password(password: str, password_hash: str) -> bool:
try:
_hasher.verify(password_hash, password)
return True
except (VerifyMismatchError, VerificationError, InvalidHashError):
return False
def _now() -> datetime:
return datetime.now(timezone.utc)
def _ttl_seconds() -> int:
return int(os.environ.get("SESSION_TTL_SECONDS", "3600"))
def _generate_token() -> str:
return secrets.token_urlsafe(32)
def _token_hash(token: str) -> str:
return hashlib.sha256(token.encode("utf-8")).hexdigest()
def create_session(user_id: str) -> dict:
token = _generate_token()
ttl = _ttl_seconds()
expires_at = _now() + timedelta(seconds=ttl)
token_hash = _token_hash(token)
with transaction() as conn:
with conn.cursor() as cur:
cur.execute(
"""
INSERT INTO sessions (token_hash, user_id, expires_at, created_at)
VALUES (%s, %s, %s, %s)
""",
(token_hash, user_id, expires_at, _now()),
)
return {"token": token, "expires_at": expires_at}
def get_user_id_by_token(token: str) -> Optional[str]:
from . import cache
token_hash = _token_hash(token)
if cache.is_revoked(token_hash):
return None
with transaction() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT user_id, expires_at
FROM sessions
WHERE token_hash = %s
""",
(token_hash,),
)
row = cur.fetchone()
if row is None:
return None
if row["expires_at"] <= _now():
return None
return str(row["user_id"])
def revoke_session(token: str) -> bool:
from . import cache
token_hash = _token_hash(token)
ttl = _ttl_seconds()
with transaction() as conn:
with conn.cursor() as cur:
cur.execute(
"SELECT expires_at FROM sessions WHERE token_hash = %s",
(token_hash,),
)
row = cur.fetchone()
if row is not None:
remaining = int((row["expires_at"] - _now()).total_seconds())
if remaining > 0:
ttl = remaining
cur.execute(
"DELETE FROM sessions WHERE token_hash = %s",
(token_hash,),
)
deleted = cur.rowcount > 0
if deleted:
cache.mark_revoked(token_hash, ttl)
return deleted
+104
View File
@@ -0,0 +1,104 @@
from __future__ import annotations
import logging
import os
from typing import Optional
import redis
log = logging.getLogger("messenger.cache")
_client: Optional[redis.Redis] = None
_failures: int = 0
_MAX_FAILURES_BEFORE_DISABLE: int = 3
def _cache_url() -> Optional[str]:
return os.environ.get("CACHE_URL")
def init_cache() -> None:
global _client, _failures
url = _cache_url()
if not url:
log.warning("CACHE_URL not set; running without cache")
_client = None
return
try:
client = redis.Redis.from_url(
url,
decode_responses=True,
socket_connect_timeout=1.0,
socket_timeout=1.0,
)
client.ping()
_client = client
_failures = 0
log.info("cache connected")
except Exception as exc:
log.warning("cache unavailable at startup: %s; running in degraded mode", exc)
_client = None
def close_cache() -> None:
global _client
if _client is not None:
try:
_client.close()
except Exception:
pass
_client = None
def _note_failure() -> None:
global _client, _failures
_failures += 1
if _failures >= _MAX_FAILURES_BEFORE_DISABLE and _client is not None:
log.warning(
"cache disabled after %d consecutive failures; running in degraded mode",
_failures,
)
try:
_client.close()
except Exception:
pass
_client = None
def _note_success() -> None:
global _failures
_failures = 0
def is_available() -> bool:
return _client is not None
def mark_revoked(token_hash: str, ttl_seconds: int) -> None:
if _client is None:
log.warning(
"cache unavailable; revocation not recorded in Redis "
"(Postgres is the source of truth, next request will 401)"
)
return
if ttl_seconds <= 0:
return
try:
_client.setex(f"revoked:{token_hash}", ttl_seconds, "1")
_note_success()
except Exception as exc:
log.warning("mark_revoked failed: %s", exc)
_note_failure()
def is_revoked(token_hash: str) -> bool:
if _client is None:
return False
try:
exists = _client.exists(f"revoked:{token_hash}")
_note_success()
return exists == 1
except Exception as exc:
log.warning("is_revoked failed: %s", exc)
_note_failure()
return False
+72
View File
@@ -0,0 +1,72 @@
from __future__ import annotations
import os
from contextlib import contextmanager
from typing import Iterator
import psycopg
from psycopg.rows import dict_row
from psycopg_pool import ConnectionPool
_pool: ConnectionPool | None = None
def _database_url() -> str:
url = os.environ.get("DATABASE_URL")
if not url:
raise RuntimeError("DATABASE_URL environment variable is required")
return url
from psycopg_pool import ConnectionPool
def init_pool(min_size: int = 1, max_size: int = 10) -> None:
global _pool
if _pool is not None:
return
dsn = _database_url()
sep = "&" if "?" in dsn else "?"
dsn_with_opts = (
dsn
+ sep
+ "options="
+ "-c%20statement_timeout%3D5000"
+ "%20-c%20lock_timeout%3D3000"
+ "%20-c%20idle_in_transaction_session_timeout%3D10000"
)
_pool = ConnectionPool(
conninfo=dsn_with_opts,
min_size=min_size,
max_size=max_size,
kwargs={"row_factory": dict_row},
open=True,
timeout=5.0,
check=ConnectionPool.check_connection,
max_lifetime=120,
max_idle=60,
)
def close_pool() -> None:
global _pool
if _pool is not None:
_pool.close()
_pool = None
@contextmanager
def connection() -> Iterator[psycopg.Connection]:
if _pool is None:
raise RuntimeError("Connection pool is not initialized")
with _pool.connection() as conn:
yield conn
@contextmanager
def transaction() -> Iterator[psycopg.Connection]:
with connection() as conn:
with conn.transaction():
yield conn
Executable
+81
View File
@@ -0,0 +1,81 @@
from __future__ import annotations
import uuid
from typing import Callable
from fastapi import Request
from fastapi.responses import JSONResponse
from starlette.middleware.base import BaseHTTPMiddleware
from .instance import INSTANCE_ID
from .store import ApiError
REQUEST_ID_HEADER = "X-Request-Id"
class RequestIdMiddleware(BaseHTTPMiddleware):
async def dispatch(self, request: Request, call_next):
rid = request.headers.get(REQUEST_ID_HEADER) or uuid.uuid4().hex
request.state.request_id = rid
response = await call_next(request)
response.headers[REQUEST_ID_HEADER] = rid
response.headers["X-Instance-Id"] = INSTANCE_ID
return response
def _envelope(code: str, message: str, request_id: str) -> dict:
return {"error": {"code": code, "message": message, "request_id": request_id}}
def _rid(request: Request) -> str:
return getattr(request.state, "request_id", None) or uuid.uuid4().hex
def _with_headers(response: JSONResponse, request: Request, status: int) -> JSONResponse:
rid = _rid(request)
response.headers[REQUEST_ID_HEADER] = rid
response.headers["X-Instance-Id"] = INSTANCE_ID
if status == 401:
scheme = "Bearer"
if request.url.path == "/api/v1/auth/sessions":
scheme = "Basic"
response.headers["WWW-Authenticate"] = f'{scheme} realm="messenger"'
return response
async def api_error_handler(request: Request, exc: Exception) -> JSONResponse:
assert isinstance(exc, ApiError)
response = JSONResponse(
status_code=exc.status,
content=_envelope(exc.code, exc.message, _rid(request)),
)
return _with_headers(response, request, exc.status)
async def validation_handler(request: Request, exc: Exception) -> JSONResponse:
response = JSONResponse(
status_code=400,
content=_envelope("invalid_request", "Request validation failed", _rid(request)),
)
return _with_headers(response, request, 400)
async def json_decode_handler(request: Request, exc: Exception) -> JSONResponse:
response = JSONResponse(
status_code=400,
content=_envelope("invalid_request", "Malformed JSON", _rid(request)),
)
return _with_headers(response, request, 400)
async def unhandled_handler(request: Request, exc: Exception) -> JSONResponse:
import logging
logging.getLogger("messenger").exception(
"unhandled error on %s %s", request.method, request.url.path
)
response = JSONResponse(
status_code=500,
content=_envelope("internal_error", "Internal error", _rid(request)),
)
return _with_headers(response, request, 500)
+11
View File
@@ -0,0 +1,11 @@
from __future__ import annotations
import os
import socket
import uuid
INSTANCE_ID: str = (
os.environ.get("INSTANCE_ID")
or socket.gethostname()
or uuid.uuid4().hex[:8]
)
Executable
+451
View File
@@ -0,0 +1,451 @@
from __future__ import annotations
import logging
import os
import re
from contextlib import asynccontextmanager
from typing import Optional
import uuid
from uuid import UUID
from fastapi import Depends, FastAPI, Header, Query, Request, Response
from fastapi.exceptions import RequestValidationError
from fastapi.responses import JSONResponse
from starlette.exceptions import HTTPException as StarletteHTTPException
from . import auth
from .db import close_pool, init_pool
from . import storage
from .errors import (
RequestIdMiddleware,
_envelope,
_rid,
api_error_handler,
json_decode_handler,
unhandled_handler,
validation_handler,
)
from .schemas import (
Chat,
ChatPage,
CreateChat,
CreateMessage,
CreateUser,
Health,
Message,
MessagePage,
Session,
User,
Attachment,
)
from .store import (
ApiError,
BadRequest,
Conflict,
NotFound,
PayloadTooLarge,
Unavailable,
UnsupportedMediaType,
Unauthorized,
store,
iso,
)
from .cache import close_cache, init_cache
from .instance import INSTANCE_ID
logging.basicConfig(
level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s"
)
log = logging.getLogger("messenger")
@asynccontextmanager
async def lifespan(app: FastAPI):
init_pool()
init_cache()
storage.init_storage()
log.info("instance_id=%s", INSTANCE_ID)
log.info("db pool, cache, storage initialized")
try:
yield
finally:
close_cache()
close_pool()
log.info("db pool and cache closed")
app = FastAPI(title="MTUSI Messenger lab2", version="1.2.0", lifespan=lifespan)
app.add_middleware(RequestIdMiddleware)
app.add_exception_handler(ApiError, api_error_handler)
app.add_exception_handler(RequestValidationError, validation_handler)
app.add_exception_handler(Exception, unhandled_handler)
@app.exception_handler(StarletteHTTPException)
async def _http_exc(request: Request, exc: StarletteHTTPException):
code = "not_found" if exc.status_code == 404 else "invalid_request"
return JSONResponse(
status_code=exc.status_code,
content=_envelope(code, str(exc.detail), _rid(request)),
)
# ---------- утилиты валидации ----------
def valid_uuid(value: str, name: str = "id") -> str:
try:
UUID(value)
except (ValueError, TypeError, AttributeError):
raise BadRequest(f"Invalid {name}")
return value
# ---------- зависимости ----------
def _parse_basic(header_value: str) -> tuple[str, str]:
import base64
if not header_value.lower().startswith("basic "):
raise Unauthorized("Basic authentication required")
try:
raw = base64.b64decode(header_value.split(" ", 1)[1], validate=True)
decoded = raw.decode("utf-8")
username, _, password = decoded.partition(":")
if not username:
raise ValueError("empty username")
return username, password
except Exception:
raise Unauthorized("Invalid Basic credentials") from None
def _bearer_token(authorization: Optional[str]) -> str:
if not authorization:
raise Unauthorized("Missing Authorization header")
parts = authorization.split(" ", 1)
if len(parts) != 2 or parts[0].lower() != "bearer":
raise Unauthorized("Bearer token required")
token = parts[1].strip()
if not token:
raise Unauthorized("Empty bearer token")
return token
def current_user_id(
authorization: Optional[str] = Header(default=None),
) -> str:
token = _bearer_token(authorization)
user_id = auth.get_user_id_by_token(token)
if user_id is None:
raise Unauthorized("Invalid or expired token")
return user_id
# ---------- health ----------
@app.get("/health/live", response_model=Health)
def liveness():
return {"status": "ok"}
@app.get("/health/ready", response_model=Health)
def readiness(request: Request):
try:
from .db import transaction
with transaction() as conn:
with conn.cursor() as cur:
cur.execute("SELECT 1")
except Exception:
rid = _rid(request)
response = JSONResponse(
status_code=503,
content=_envelope("unavailable", "Database unavailable", rid),
)
response.headers["X-Request-Id"] = rid
response.headers["X-Instance-Id"] = INSTANCE_ID
return response
try:
storage.check_bucket()
except Exception:
rid = _rid(request)
response = JSONResponse(
status_code=503,
content=_envelope("unavailable", "S3 unavailable", rid),
)
response.headers["X-Request-Id"] = rid
response.headers["X-Instance-Id"] = INSTANCE_ID
return response
return {"status": "ok"}
# ---------- users ----------
@app.post("/api/v1/users", status_code=201, response_model=User)
def register_user(body: CreateUser):
password_hash = auth.hash_password(body.password)
return store.create_user(body.username, body.display_name, password_hash)
@app.get("/api/v1/users/me", response_model=User)
def me(user_id: str = Depends(current_user_id)):
user = store.get_user(user_id)
if user is None:
raise Unauthorized("User not found")
return user
# ---------- auth ----------
@app.post("/api/v1/auth/sessions", status_code=201, response_model=Session)
def create_session(authorization: Optional[str] = Header(default=None)):
username, password = _parse_basic(authorization or "")
row = store.get_user_by_username(username)
if row is None or not auth.verify_password(password, row["password_hash"]):
raise Unauthorized("Invalid credentials")
session = auth.create_session(str(row["id"]))
user = store.get_user(str(row["id"]))
if user is None:
raise Unauthorized("User not found")
return {
"token": session["token"],
"token_type": "Bearer",
"expires_at": iso(session["expires_at"]),
"user": user,
}
@app.delete("/api/v1/auth/sessions/current", status_code=204)
def logout(authorization: Optional[str] = Header(default=None)):
token = _bearer_token(authorization)
user_id = auth.get_user_id_by_token(token)
if user_id is None:
raise Unauthorized("Invalid or expired token")
auth.revoke_session(token)
return Response(status_code=204)
# ---------- chats ----------
@app.post("/api/v1/chats", status_code=201, response_model=Chat)
def create_chat(body: CreateChat, user_id: str = Depends(current_user_id)):
return store.create_chat(body.title, body.member_ids, user_id)
@app.get("/api/v1/chats", response_model=ChatPage)
def list_chats(
limit: int = Query(default=50, ge=1, le=100),
cursor: Optional[str] = Query(default=None, min_length=1),
user_id: str = Depends(current_user_id),
):
items, next_cursor = store.list_chats(user_id, limit, cursor)
return {"items": items, "next_cursor": next_cursor}
@app.get("/api/v1/chats/{chat_id}", response_model=Chat)
def get_chat(chat_id: str, user_id: str = Depends(current_user_id)):
chat_id = valid_uuid(chat_id, "chat_id")
chat = store.get_chat(chat_id, user_id)
if chat is None:
raise NotFound()
return chat
# ---------- messages ----------
@app.post("/api/v1/chats/{chat_id}/messages", status_code=201, response_model=Message)
def create_message(
chat_id: str,
body: CreateMessage,
user_id: str = Depends(current_user_id),
idempotency_key: Optional[str] = Header(default=None, alias="Idempotency-Key"),
):
chat_id = valid_uuid(chat_id, "chat_id")
if not idempotency_key:
raise BadRequest("Idempotency-Key header is required")
if not re.match(r"^[A-Za-z0-9._:-]{1,128}$", idempotency_key):
raise BadRequest("Invalid Idempotency-Key format")
message, status = store.add_message(
chat_id, user_id, body.text, idempotency_key, body.attachment_ids
)
if status != 201:
raise Conflict(f"Idempotency error: {status}")
return message
@app.get("/api/v1/chats/{chat_id}/messages", response_model=MessagePage)
def list_messages(
chat_id: str,
limit: int = Query(default=50, ge=1, le=100),
cursor: Optional[str] = Query(default=None, min_length=1),
user_id: str = Depends(current_user_id),
):
chat_id = valid_uuid(chat_id, "chat_id")
items, next_cursor = store.list_messages(chat_id, user_id, limit, cursor)
return {"items": items, "next_cursor": next_cursor}
# ---------- attachments (lab4) ----------
MAX_UPLOAD_BYTES = int(os.environ.get("MAX_UPLOAD_BYTES", "10485760"))
MAX_IMAGE_PIXELS = int(os.environ.get("MAX_IMAGE_PIXELS", "40000000"))
ALLOWED_MIMES = {"image/png", "image/jpeg"}
from PIL import Image as _PILImage
_PILImage.MAX_IMAGE_PIXELS = MAX_IMAGE_PIXELS
import unicodedata
def _validate_filename(filename: str) -> None:
if not filename:
raise BadRequest("filename is required")
if len(filename) > 255:
raise BadRequest("filename too long")
for ch in filename:
if ch in ("/", "\\"):
raise BadRequest("filename contains forbidden characters")
if unicodedata.category(ch) == "Cc":
raise BadRequest("filename contains control characters")
def _sniff_mime(data: bytes) -> str:
if data.startswith(b"\x89PNG\r\n\x1a\n"):
return "image/png"
if data.startswith(b"\xff\xd8\xff"):
return "image/jpeg"
raise UnsupportedMediaType("Only PNG and JPEG are supported")
def _validate_image(data: bytes) -> tuple[int, int]:
from PIL import Image
import io
try:
img = Image.open(io.BytesIO(data))
w, h = img.size
if w * h > MAX_IMAGE_PIXELS:
raise PayloadTooLarge(
f"Image too large: {w*h} pixels (max {MAX_IMAGE_PIXELS})"
)
img.load()
except PayloadTooLarge:
raise
except Image.DecompressionBombError:
raise PayloadTooLarge(
f"Image too large (decompression bomb protection)"
) from None
except Exception:
raise UnsupportedMediaType("Image is corrupted or unsupported") from None
return w, h
@app.post(
"/api/v1/chats/{chat_id}/attachments",
status_code=201,
response_model=Attachment,
)
async def upload_attachment(
chat_id: str,
request: Request,
filename: str = Query(..., min_length=1, max_length=255),
user_id: str = Depends(current_user_id),
):
chat_id = valid_uuid(chat_id, "chat_id")
_validate_filename(filename)
content_length = request.headers.get("content-length")
if content_length is not None:
try:
if int(content_length) > MAX_UPLOAD_BYTES:
raise PayloadTooLarge(
f"File too large: {content_length} bytes (max {MAX_UPLOAD_BYTES})"
)
except ValueError:
raise BadRequest("Invalid Content-Length") from None
client_ct = (request.headers.get("content-type") or "").split(";")[0].strip().lower()
if client_ct not in ALLOWED_MIMES:
raise UnsupportedMediaType(
f"Content-Type must be image/png or image/jpeg, got {client_ct!r}"
)
body = bytearray()
async for chunk in request.stream():
body.extend(chunk)
if len(body) > MAX_UPLOAD_BYTES:
raise PayloadTooLarge(
f"File too large: > {MAX_UPLOAD_BYTES} bytes"
)
data = bytes(body)
real_mime = _sniff_mime(data)
_validate_image(data)
s3_key = storage.new_object_key(real_mime)
try:
sha256 = storage.put_object(s3_key, data, real_mime)
except Exception as exc:
import logging
logging.getLogger("messenger").warning("S3 upload failed: %s", exc)
raise Unavailable("S3 unavailable") from exc
try:
attachment = store.create_attachment(
chat_id=chat_id,
owner_id=user_id,
filename=filename,
content_type=real_mime,
size_bytes=len(data),
sha256=sha256,
s3_key=s3_key,
)
except Exception:
storage.delete_object(s3_key)
raise
return attachment
@app.get(
"/api/v1/attachments/{attachment_id}",
response_model=Attachment,
)
def get_attachment(
attachment_id: str,
user_id: str = Depends(current_user_id),
):
attachment_id = valid_uuid(attachment_id, "attachment_id")
att = store.get_attachment(attachment_id, user_id)
if att is None:
raise NotFound()
att.pop("s3_key", None)
return att
@app.get("/api/v1/attachments/{attachment_id}/content")
def download_attachment(
attachment_id: str,
variant: str = Query(default="original"),
user_id: str = Depends(current_user_id),
):
attachment_id = valid_uuid(attachment_id, "attachment_id")
if variant != "original":
raise NotFound()
att = store.get_attachment(attachment_id, user_id)
if att is None:
raise NotFound()
if att["status"] == "failed":
raise Conflict("Attachment processing failed")
data = storage.get_object(att["s3_key"])
return Response(content=data, media_type=att["content_type"])
def main() -> None:
import uvicorn
host = os.environ.get("APP_HOST", "0.0.0.0")
port = int(os.environ.get("APP_PORT", "8080"))
uvicorn.run("src.main:app", host=host, port=port, log_level="info")
if __name__ == "__main__":
main()
Executable
+181
View File
@@ -0,0 +1,181 @@
from __future__ import annotations
import re
import uuid
from typing import List, Optional
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
USERNAME_RE = re.compile(r"^[a-z0-9_]{3,32}$")
class StrictModel(BaseModel):
model_config = ConfigDict(extra="forbid", str_strip_whitespace=False)
# ---------- Users ----------
class CreateUser(StrictModel):
username: str = Field(min_length=3, max_length=32)
display_name: str = Field(min_length=1, max_length=100)
password: str = Field(min_length=12, max_length=128)
@field_validator("username")
@classmethod
def _username(cls, v: str) -> str:
if not USERNAME_RE.match(v):
raise ValueError("username must match [a-z0-9_]{3,32}")
return v
@field_validator("display_name")
@classmethod
def _display_name(cls, v: str) -> str:
if not v.strip():
raise ValueError("display_name must not be blank")
if len(v) > 100:
raise ValueError("display_name too long")
return v
class User(StrictModel):
id: str
username: str
display_name: str
created_at: str
# ---------- Chats ----------
class CreateChat(StrictModel):
title: str = Field(min_length=1, max_length=100)
member_ids: List[str] = Field(min_length=1, max_length=100)
@field_validator("title")
@classmethod
def _title(cls, v: str) -> str:
if not v.strip():
raise ValueError("title must not be blank")
return v
@field_validator("member_ids")
@classmethod
def _members(cls, v: List[str]) -> List[str]:
if len(set(v)) != len(v):
raise ValueError("member_ids must be unique")
for item in v:
try:
uuid.UUID(item)
except ValueError as exc:
raise ValueError("member_ids must be UUIDs") from exc
return v
class Chat(StrictModel):
id: str
title: str
member_ids: List[str]
created_at: str
class ChatPage(StrictModel):
items: List[Chat]
next_cursor: Optional[str] = None
# ---------- Attachments (lab4) ----------
class Variant(StrictModel):
name: str
content_type: str
size_bytes: int = Field(ge=1)
width: int = Field(ge=1)
height: int = Field(ge=1)
sha256: str = Field(pattern=r"^[a-f0-9]{64}$")
class Attachment(StrictModel):
id: str
chat_id: str
owner_id: str
filename: str = Field(min_length=1, max_length=255)
content_type: str # image/png или image/jpeg
size_bytes: int = Field(ge=1, le=10485760)
sha256: str = Field(pattern=r"^[a-f0-9]{64}$")
status: str # 'ready' в lab4
variants: List[Variant] = Field(default_factory=list)
error_code: Optional[str] = None
created_at: str
updated_at: str
# ---------- Messages ----------
class CreateMessage(StrictModel):
text: str = Field(default="", max_length=4000)
attachment_ids: List[str] = Field(default_factory=list, max_length=10)
@field_validator("text")
@classmethod
def _text(cls, v: str) -> str:
if len(v) > 4000:
raise ValueError("text too long")
return v
@field_validator("attachment_ids")
@classmethod
def _attachments(cls, v: List[str]) -> List[str]:
if len(set(v)) != len(v):
raise ValueError("attachment_ids must be unique")
for item in v:
try:
uuid.UUID(item)
except ValueError as exc:
raise ValueError("attachment_ids must be UUIDs") from exc
return v
@model_validator(mode="after")
def _text_or_attachment(self) -> "CreateMessage":
has_text = bool(self.text and self.text.strip())
has_attachment = bool(self.attachment_ids)
if not has_text and not has_attachment:
raise ValueError("text must be non-blank or attachment_ids must be non-empty")
return self
class Message(StrictModel):
id: str
chat_id: str
sender_id: str
text: str
created_at: str
attachment_ids: List[str] = Field(default_factory=list)
class MessagePage(StrictModel):
items: List[Message]
next_cursor: Optional[str] = None
# ---------- Sessions ----------
class Session(StrictModel):
token: str = Field(min_length=32)
token_type: str = Field(default="Bearer")
expires_at: str
user: User
# ---------- Misc ----------
class Health(StrictModel):
status: str = "ok"
class ErrorBody(StrictModel):
code: str
message: str
request_id: str
class ErrorEnvelope(StrictModel):
error: ErrorBody
+101
View File
@@ -0,0 +1,101 @@
from __future__ import annotations
import hashlib
import io
import logging
import os
import uuid
from typing import Optional
import boto3
from botocore.client import Config
from botocore.exceptions import ClientError
log = logging.getLogger("messenger.storage")
_client = None
def _env(name: str, default: Optional[str] = None) -> str:
value = os.environ.get(name, default)
if value is None:
raise RuntimeError(f"Environment variable {name} is required")
return value
def init_storage() -> None:
global _client
if _client is not None:
return
_client = boto3.client(
"s3",
endpoint_url=_env("S3_ENDPOINT"),
region_name=_env("S3_REGION", "us-east-1"),
aws_access_key_id=_env("S3_ACCESS_KEY_ID"),
aws_secret_access_key=_env("S3_SECRET_ACCESS_KEY"),
config=Config(
signature_version="s3v4",
s3={"addressing_style": "path"},
connect_timeout=1.0,
read_timeout=2.0,
retries={"max_attempts": 1, "mode": "standard"},
),
)
log.info("storage client initialized")
def _bucket() -> str:
return _env("S3_BUCKET", "messenger-attachments")
def new_object_key(content_type: str) -> str:
ext = "png" if content_type == "image/png" else "jpg"
return f"attachments/{uuid.uuid4().hex}.{ext}"
def put_object(key: str, body: bytes, content_type: str) -> str:
if _client is None:
raise RuntimeError("storage not initialized")
sha256 = hashlib.sha256(body).hexdigest()
_client.put_object(
Bucket=_bucket(),
Key=key,
Body=body,
ContentType=content_type,
Metadata={"sha256": sha256},
)
return sha256
def get_object(key: str) -> bytes:
if _client is None:
raise RuntimeError("storage not initialized")
response = _client.get_object(Bucket=_bucket(), Key=key)
return response["Body"].read()
def delete_object(key: str) -> None:
if _client is None:
return
try:
_client.delete_object(Bucket=_bucket(), Key=key)
except ClientError as exc:
log.warning("failed to delete object %s: %s", key, exc)
def object_exists(key: str) -> bool:
if _client is None:
return False
try:
_client.head_object(Bucket=_bucket(), Key=key)
return True
except ClientError as exc:
if exc.response["Error"]["Code"] in ("404", "NoSuchKey"):
return False
raisev
def check_bucket() -> None:
if _client is None:
raise RuntimeError("storage not initialized")
_client.head_bucket(Bucket=_bucket())
Executable
+548
View File
@@ -0,0 +1,548 @@
from __future__ import annotations
import base64
import hashlib
import json
import uuid
from datetime import datetime, timezone
from typing import List, Optional, Tuple
import psycopg
from psycopg.errors import UniqueViolation
from .db import transaction
# ---------- ошибки ----------
class ApiError(Exception):
status = 500
code = "internal_error"
def __init__(self, message: str = "Internal error") -> None:
super().__init__(message)
self.message = message
class BadRequest(ApiError):
status = 400
code = "invalid_request"
class Unauthorized(ApiError):
status = 401
code = "unauthorized"
class NotFound(ApiError):
status = 404
code = "not_found"
class Conflict(ApiError):
status = 409
code = "conflict"
class PayloadTooLarge(ApiError):
status = 413
code = "payload_too_large"
class UnsupportedMediaType(ApiError):
status = 415
code = "unsupported_media_type"
class Unavailable(ApiError):
status = 503
code = "unavailable"
# ---------- утилиты ----------
def _now() -> datetime:
return datetime.now(timezone.utc)
def iso(dt: datetime) -> str:
return dt.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%fZ")
def _user_dict(row: dict) -> dict:
return {
"id": str(row["id"]),
"username": row["username"],
"display_name": row["display_name"],
"created_at": iso(row["created_at"]),
}
def _chat_dict(row: dict, member_ids: List[str]) -> dict:
return {
"id": str(row["id"]),
"title": row["title"],
"member_ids": member_ids,
"created_at": iso(row["created_at"]),
}
def _message_dict(row: dict, attachment_ids: List[str]) -> dict:
return {
"id": str(row["id"]),
"chat_id": str(row["chat_id"]),
"sender_id": str(row["sender_id"]),
"text": row["text"],
"created_at": iso(row["created_at"]),
"attachment_ids": attachment_ids,
}
# ---------- курсор ----------
def _encode_cursor(scope: str, created_at: str, item_id: str) -> str:
payload = json.dumps(
{"s": scope, "t": created_at, "i": item_id}, separators=(",", ":")
).encode()
return base64.urlsafe_b64encode(payload).decode().rstrip("=")
def _decode_cursor(scope: str, cursor: str) -> Tuple[str, str]:
pad = "=" * (-len(cursor) % 4)
try:
raw = base64.urlsafe_b64decode(cursor + pad)
data = json.loads(raw)
if data.get("s") != scope:
raise ValueError("scope mismatch")
t = data.get("t")
i = data.get("i")
if not isinstance(t, str) or not isinstance(i, str):
raise ValueError("invalid cursor fields")
from datetime import datetime
from uuid import UUID
try:
datetime.fromisoformat(t.replace("Z", "+00:00"))
except (ValueError, TypeError):
raise ValueError("invalid timestamp in cursor") from None
try:
UUID(i)
except (ValueError, TypeError):
raise ValueError("invalid uuid in cursor") from None
return t, i
except Exception as exc:
raise BadRequest("invalid cursor") from exc
# ---------- Store ----------
class Store:
# ---- users ----
def create_user(
self, username: str, display_name: str, password_hash: str
) -> dict:
uid = str(uuid.uuid4())
now = _now()
try:
with transaction() as conn:
with conn.cursor() as cur:
cur.execute(
"""
INSERT INTO users (id, username, display_name, password_hash, created_at)
VALUES (%s, %s, %s, %s, %s)
""",
(uid, username, display_name, password_hash, now),
)
except UniqueViolation:
raise Conflict("username already taken") from None
return {
"id": uid,
"username": username,
"display_name": display_name,
"created_at": iso(now),
}
def get_user(self, user_id: str) -> Optional[dict]:
with transaction() as conn:
with conn.cursor() as cur:
cur.execute("SELECT * FROM users WHERE id = %s", (user_id,))
row = cur.fetchone()
return _user_dict(row) if row else None
def get_user_by_username(self, username: str) -> Optional[dict]:
with transaction() as conn:
with conn.cursor() as cur:
cur.execute("SELECT * FROM users WHERE username = %s", (username,))
row = cur.fetchone()
return row
# ---- chats ----
def create_chat(
self, title: str, member_ids: List[str], creator_id: str
) -> dict:
if creator_id not in member_ids:
raise BadRequest("creator must be in member_ids")
with transaction() as conn:
with conn.cursor() as cur:
cur.execute(
"SELECT id FROM users WHERE id = ANY(%s)",
(member_ids,),
)
found = {str(r["id"]) for r in cur.fetchall()}
missing = set(member_ids) - found
if missing:
raise BadRequest("unknown member_id")
cid = str(uuid.uuid4())
now = _now()
with transaction() as conn:
with conn.cursor() as cur:
cur.execute(
"INSERT INTO chats (id, title, created_at) VALUES (%s, %s, %s)",
(cid, title, now),
)
cur.executemany(
"INSERT INTO chat_members (chat_id, user_id) VALUES (%s, %s)",
[(cid, uid) for uid in member_ids],
)
return {
"id": cid,
"title": title,
"member_ids": list(member_ids),
"created_at": iso(now),
}
def get_chat(self, chat_id: str, viewer_id: str) -> Optional[dict]:
with transaction() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT c.* FROM chats c
JOIN chat_members m ON m.chat_id = c.id
WHERE c.id = %s AND m.user_id = %s
""",
(chat_id, viewer_id),
)
row = cur.fetchone()
if row is None:
return None
cur.execute(
"SELECT user_id FROM chat_members WHERE chat_id = %s ORDER BY user_id",
(chat_id,),
)
members = [str(r["user_id"]) for r in cur.fetchall()]
return _chat_dict(row, members)
def list_chats(
self, viewer_id: str, limit: int, cursor: Optional[str]
) -> Tuple[List[dict], Optional[str]]:
scope = "chats:" + viewer_id
params: list = [viewer_id]
where_cursor = ""
if cursor:
t, i = _decode_cursor(scope, cursor)
where_cursor = "AND (c.created_at, c.id) > (%s, %s)"
params.extend([t, i])
params.append(limit + 1)
with transaction() as conn:
with conn.cursor() as cur:
cur.execute(
f"""
SELECT c.id, c.title, c.created_at
FROM chats c
JOIN chat_members m ON m.chat_id = c.id
WHERE m.user_id = %s {where_cursor}
ORDER BY c.created_at, c.id
LIMIT %s
""",
params,
)
rows = cur.fetchall()
has_more = len(rows) > limit
page = rows[:limit]
result = []
for row in page:
cur.execute(
"SELECT user_id FROM chat_members WHERE chat_id = %s ORDER BY user_id",
(row["id"],),
)
members = [str(r["user_id"]) for r in cur.fetchall()]
result.append(_chat_dict(row, members))
next_cursor = None
if has_more and page:
last = page[-1]
next_cursor = _encode_cursor(scope, iso(last["created_at"]), str(last["id"]))
return result, next_cursor
# ---- messages ----
def add_message(
self,
chat_id: str,
sender_id: str,
text: str,
idempotency_key: str,
attachment_ids: Optional[List[str]] = None,
) -> tuple[dict, int]:
attachment_ids = attachment_ids or []
request_hash = hashlib.sha256(
json.dumps(
{"text": text, "attachment_ids": attachment_ids},
ensure_ascii=False,
sort_keys=True,
).encode("utf-8")
).hexdigest()
path = f"/api/v1/chats/{chat_id}/messages"
method = "POST"
import time as _time
for attempt in range(5):
try:
with transaction() as conn:
with conn.cursor() as cur:
cur.execute(
"SELECT 1 FROM chat_members WHERE chat_id = %s AND user_id = %s",
(chat_id, sender_id),
)
if cur.fetchone() is None:
raise NotFound()
cur.execute(
"""
SELECT request_hash, response_status, response_body
FROM idempotency_keys
WHERE user_id = %s AND method = %s AND path = %s AND key = %s
""",
(sender_id, method, path, idempotency_key),
)
row = cur.fetchone()
if row is not None:
if row["request_hash"] != request_hash:
raise Conflict(
"idempotency key reused with different body"
)
return row["response_body"], row["response_status"]
if attachment_ids:
cur.execute(
"""
SELECT id FROM attachments
WHERE id = ANY(%s) AND chat_id = %s
""",
(attachment_ids, chat_id),
)
found = {str(r["id"]) for r in cur.fetchall()}
missing = set(attachment_ids) - found
if missing:
raise NotFound()
mid = str(uuid.uuid4())
now = _now()
cur.execute(
"""
INSERT INTO messages (id, chat_id, sender_id, text, created_at)
VALUES (%s, %s, %s, %s, %s)
""",
(mid, chat_id, sender_id, text, now),
)
if attachment_ids:
cur.executemany(
"""
INSERT INTO message_attachments
(message_id, attachment_id, position)
VALUES (%s, %s, %s)
""",
[(mid, aid, pos) for pos, aid in enumerate(attachment_ids)],
)
message = {
"id": mid,
"chat_id": chat_id,
"sender_id": sender_id,
"text": text,
"created_at": iso(now),
"attachment_ids": list(attachment_ids),
}
cur.execute(
"""
INSERT INTO idempotency_keys
(user_id, method, path, key, request_hash,
response_status, response_body, created_at)
VALUES (%s, %s, %s, %s, %s, 201, %s, %s)
""",
(
sender_id, method, path, idempotency_key,
request_hash,
json.dumps(message, ensure_ascii=False),
now,
),
)
return message, 201
except UniqueViolation:
_time.sleep(0.05)
continue
raise Conflict("idempotency key in progress, please retry")
def list_messages(
self,
chat_id: str,
viewer_id: str,
limit: int,
cursor: Optional[str],
) -> Tuple[List[dict], Optional[str]]:
with transaction() as conn:
with conn.cursor() as cur:
cur.execute(
"SELECT 1 FROM chat_members WHERE chat_id = %s AND user_id = %s",
(chat_id, viewer_id),
)
if cur.fetchone() is None:
raise NotFound()
scope = "chat:" + chat_id
params: list = [chat_id]
where_cursor = ""
if cursor:
t, i = _decode_cursor(scope, cursor)
where_cursor = "AND (created_at, id) > (%s, %s)"
params.extend([t, i])
params.append(limit + 1)
with transaction() as conn:
with conn.cursor() as cur:
cur.execute(
f"""
SELECT id, chat_id, sender_id, text, created_at
FROM messages
WHERE chat_id = %s {where_cursor}
ORDER BY created_at, id
LIMIT %s
""",
params,
)
rows = cur.fetchall()
message_ids = [str(r["id"]) for r in rows]
attachments_by_message: dict = {}
if message_ids:
cur.execute(
"""
SELECT message_id, attachment_id
FROM message_attachments
WHERE message_id = ANY(%s)
ORDER BY position
""",
(message_ids,),
)
for r in cur.fetchall():
attachments_by_message.setdefault(
str(r["message_id"]), []
).append(str(r["attachment_id"]))
has_more = len(rows) > limit
page = rows[:limit]
result = []
for r in page:
result.append(_message_dict(
r, attachments_by_message.get(str(r["id"]), [])
))
next_cursor = None
if has_more and page:
last = page[-1]
next_cursor = _encode_cursor(scope, iso(last["created_at"]), str(last["id"]))
return result, next_cursor
# ---- attachments ----
def create_attachment(
self,
chat_id: str,
owner_id: str,
filename: str,
content_type: str,
size_bytes: int,
sha256: str,
s3_key: str,
) -> dict:
aid = str(uuid.uuid4())
now = _now()
with transaction() as conn:
with conn.cursor() as cur:
cur.execute(
"SELECT 1 FROM chat_members WHERE chat_id = %s AND user_id = %s",
(chat_id, owner_id),
)
if cur.fetchone() is None:
raise NotFound()
cur.execute(
"""
INSERT INTO attachments
(id, chat_id, owner_id, filename, content_type,
size_bytes, sha256, s3_key, status, error_code,
created_at, updated_at)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, 'ready', NULL, %s, %s)
""",
(aid, chat_id, owner_id, filename, content_type,
size_bytes, sha256, s3_key, now, now),
)
return {
"id": aid,
"chat_id": chat_id,
"owner_id": owner_id,
"filename": filename,
"content_type": content_type,
"size_bytes": size_bytes,
"sha256": sha256,
"status": "ready",
"variants": [],
"error_code": None,
"created_at": iso(now),
"updated_at": iso(now),
}
def get_attachment(
self, attachment_id: str, viewer_id: str
) -> Optional[dict]:
with transaction() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT a.*
FROM attachments a
JOIN chat_members m ON m.chat_id = a.chat_id
WHERE a.id = %s AND m.user_id = %s
""",
(attachment_id, viewer_id),
)
row = cur.fetchone()
if row is None:
return None
return {
"id": str(row["id"]),
"chat_id": str(row["chat_id"]),
"owner_id": str(row["owner_id"]),
"filename": row["filename"],
"content_type": row["content_type"],
"size_bytes": row["size_bytes"],
"sha256": row["sha256"],
"s3_key": row["s3_key"],
"status": row["status"],
"variants": [],
"error_code": row["error_code"],
"created_at": iso(row["created_at"]),
"updated_at": iso(row["updated_at"]),
}
store = Store()
+89
View File
@@ -0,0 +1,89 @@
"""Генерация тестовых сообщений для EXPLAIN.
Создаёт пользователя, чат и N сообщений. N по умолчанию 10 000.
Запуск:
python3 seed_messages.py --base-url http://localhost:8080 --count 10000
"""
from __future__ import annotations
import argparse
import base64
import json
import time
import urllib.error
import urllib.request
import uuid
def call(base: str, method: str, path: str, headers: dict | None = None, body: dict | None = None) -> tuple[int, dict | None]:
data = None
h = dict(headers or {})
if body is not None:
data = json.dumps(body, ensure_ascii=False).encode("utf-8")
h["Content-Type"] = "application/json"
req = urllib.request.Request(base + path, data=data, headers=h, method=method)
try:
resp = urllib.request.urlopen(req, timeout=30)
raw = resp.read()
return resp.code, json.loads(raw) if raw else None
except urllib.error.HTTPError as e:
raw = e.read()
try:
return e.code, json.loads(raw) if raw else None
except Exception:
return e.code, None
def main() -> None:
p = argparse.ArgumentParser()
p.add_argument("--base-url", default="http://localhost:8080")
p.add_argument("--count", type=int, default=10000)
p.add_argument("--batch", type=int, default=500, help="Сообщений между progress-логами")
args = p.parse_args()
base = args.base_url.rstrip("/")
username = "seed_" + uuid.uuid4().hex[:12]
password = "seed-pass-1234567890"
# Регистрация
code, user = call(base, "POST", "/api/v1/users", body={
"username": username, "display_name": "Seed User", "password": password,
})
assert code == 201, f"register failed: {code} {user}"
# Логин
cred = base64.b64encode(f"{username}:{password}".encode()).decode()
code, session = call(base, "POST", "/api/v1/auth/sessions", headers={"Authorization": "Basic " + cred})
assert code == 201, f"login failed: {code} {session}"
token = session["token"]
user_id = session["user"]["id"]
# Чат
code, chat = call(base, "POST", "/api/v1/chats", headers={"Authorization": "Bearer " + token}, body={
"title": "Seed chat", "member_ids": [user_id],
})
assert code == 201, f"chat failed: {code} {chat}"
chat_id = chat["id"]
print(f"chat_id: {chat_id}")
print(f"seeding {args.count} messages...")
started = time.monotonic()
for i in range(args.count):
code, msg = call(
base, "POST", f"/api/v1/chats/{chat_id}/messages",
headers={"Authorization": "Bearer " + token},
body={"text": f"seed-{i:06d}-{uuid.uuid4().hex[:8]}"},
)
assert code == 201, f"message {i} failed: {code} {msg}"
if (i + 1) % args.batch == 0:
elapsed = time.monotonic() - started
rps = (i + 1) / elapsed
print(f" {i + 1}/{args.count} ({rps:.0f} msg/s)")
elapsed = time.monotonic() - started
print(f"done: {args.count} messages in {elapsed:.1f}s ({args.count / elapsed:.0f} msg/s)")
print(f"chat_id: {chat_id}")
if __name__ == "__main__":
main()
+292
View File
@@ -0,0 +1,292 @@
"""Негативные тесты для вложений (lab4 grade 4).
Запуск:
python3 test_attachments_negative.py --base-url http://localhost:8080
"""
from __future__ import annotations
import base64
import io
import json
import sys
import urllib.error
import urllib.request
import uuid
import time
BASE = (
sys.argv[sys.argv.index("--base-url") + 1]
if "--base-url" in sys.argv
else "http://localhost:8080"
)
def call(method, path, headers=None, body=None, raw=None):
data = None
h = dict(headers or {})
if body is not None:
data = json.dumps(body, ensure_ascii=False).encode("utf-8")
h["Content-Type"] = "application/json"
elif raw is not None:
data = raw
req = urllib.request.Request(BASE + path, data=data, headers=h, method=method)
try:
resp = urllib.request.urlopen(req, timeout=30)
payload = resp.read()
return resp.status, json.loads(payload) if payload else None, dict(resp.headers)
except urllib.error.HTTPError as e:
payload = e.read()
try:
return e.code, json.loads(payload) if payload else None, dict(e.headers)
except Exception:
return e.code, None, dict(e.headers)
def make_user(label):
username = f"att_{label}_{uuid.uuid4().hex[:10]}"
password = "negative-pass-1234567890"
code, user, _ = call("POST", "/api/v1/users", body={
"username": username, "display_name": f"Att {label}", "password": password,
})
assert code == 201, f"register {label} failed: {code} {user}"
cred = base64.b64encode(f"{username}:{password}".encode()).decode()
code, session, _ = call("POST", "/api/v1/auth/sessions",
headers={"Authorization": "Basic " + cred})
assert code == 201, f"login {label} failed: {code}"
return {
"id": user["id"],
"token": session["token"],
}
def make_chat(owner, member_ids=None):
if member_ids is None:
member_ids = [owner["id"]]
code, chat, _ = call("POST", "/api/v1/chats",
headers={"Authorization": "Bearer " + owner["token"]},
body={"title": "Negative", "member_ids": member_ids})
assert code == 201, f"chat failed: {code} {chat}"
return chat["id"]
def upload(token, chat_id, filename, data, content_type):
return call(
"POST",
f"/api/v1/chats/{chat_id}/attachments?filename={filename}",
headers={"Authorization": "Bearer " + token, "Content-Type": content_type},
raw=data,
)
def test_decompression_bomb():
"""PNG 8200x5000 (41M пикселей) — маленький по размеру, но огромный по пикселям."""
from PIL import Image
import io as _io
alice = make_user("bomb")
chat = make_chat(alice)
img = Image.new("RGB", (8200, 5000), color=(255, 0, 0))
buf = _io.BytesIO()
img.save(buf, format="PNG", compress_level=9)
data = buf.getvalue()
assert len(data) < 500 * 1024, f"expected small compressed, got {len(data)}"
started = time.monotonic()
code, _, _ = upload(alice["token"], chat, "bomb.png", data, "image/png")
elapsed = time.monotonic() - started
assert code == 413, f"expected 413, got {code}"
assert elapsed < 2.0, f"expected fast rejection (<2s), got {elapsed:.2f}s"
print(f"PASS: decompression bomb -> 413 ({len(data)} bytes, {elapsed:.2f}s)")
def test_invalid_cursor():
"""Cursor с невалидным timestamp или UUID -> 400."""
import base64 as _b64
alice = make_user("cursor")
chat = make_chat(alice)
chat_id = chat
# Невалидный timestamp
bad_t = _b64.urlsafe_b64encode(
json.dumps({
"s": f"chat:{chat_id}",
"t": "not-a-time",
"i": "00000000-0000-4000-8000-000000000000",
}).encode()
).decode().rstrip("=")
code, _, _ = call(
"GET", f"/api/v1/chats/{chat_id}/messages?cursor={bad_t}",
headers={"Authorization": "Bearer " + alice["token"]},
)
assert code == 400, f"invalid timestamp: expected 400, got {code}"
# Невалидный UUID
bad_i = _b64.urlsafe_b64encode(
json.dumps({
"s": f"chat:{chat_id}",
"t": "2026-01-01T00:00:00.000000Z",
"i": "not-a-uuid",
}).encode()
).decode().rstrip("=")
code, _, _ = call(
"GET", f"/api/v1/chats/{chat_id}/messages?cursor={bad_i}",
headers={"Authorization": "Bearer " + alice["token"]},
)
assert code == 400, f"invalid uuid: expected 400, got {code}"
print("PASS: invalid cursor -> 400 (timestamp and uuid)")
def test_filename_c1():
"""Filename с C1 control (U+0085) -> 400."""
from pathlib import Path
fixture = Path(__file__).resolve().parent.parent / "tests/fixtures/sample.png"
png = fixture.read_bytes()
alice = make_user("c1")
chat = make_chat(alice)
# filename = "c1\u0085.png" — C1 control U+0085
code, _, _ = call(
"POST",
f"/api/v1/chats/{chat}/attachments?filename=c1%C2%85.png",
headers={
"Authorization": "Bearer " + alice["token"],
"Content-Type": "image/png",
},
raw=png,
)
assert code == 400, f"expected 400 for C1 filename, got {code}"
print("PASS: C1 filename -> 400")
def test_s3_unavailable():
"""S3 down: readiness 503, upload 503."""
import subprocess
from pathlib import Path
alice = make_user("s3down")
chat = make_chat(alice)
# 1. Baseline: readiness 200
code, _, _ = call("GET", "/health/ready")
assert code == 200, f"readiness should be 200 before stop, got {code}"
# 2. Останавливаем S3
try:
subprocess.run(
["docker", "compose", "--env-file", ".env", "stop", "s3"],
check=True, capture_output=True, text=True,
cwd=str(Path(__file__).resolve().parent.parent),
timeout=30,
)
except subprocess.CalledProcessError as e:
raise SystemExit(f"failed to stop s3: {e.stderr}")
import time as _t
_t.sleep(3)
try:
# 3. Readiness должен быть 503
code, body, _ = call("GET", "/health/ready")
assert code == 503, f"readiness should be 503 with S3 down, got {code}"
assert body and body["error"]["code"] == "unavailable", \
f"expected unavailable, got {body}"
# 4. Upload должен быть 503
fixture = Path(__file__).resolve().parent.parent / "tests/fixtures/sample.png"
png = fixture.read_bytes()
code, body, _ = upload(alice["token"], chat, "s3down.png", png, "image/png")
assert code == 503, f"upload should be 503 with S3 down, got {code}"
assert body and body["error"]["code"] == "unavailable", \
f"expected unavailable, got {body}"
print("PASS: S3 down -> readiness 503, upload 503")
finally:
# 5. Возвращаем S3
subprocess.run(
["docker", "compose", "--env-file", ".env", "start", "s3"],
check=False, capture_output=True, text=True,
cwd=str(Path(__file__).resolve().parent.parent),
timeout=30,
)
# Ждём готовности
deadline = _t.monotonic() + 30
while _t.monotonic() < deadline:
code, _, _ = call("GET", "/health/ready")
if code == 200:
break
_t.sleep(1)
def main():
alice = make_user("alice")
bob = make_user("bob")
eve = make_user("eve")
chat_ab = make_chat(alice, [alice["id"], bob["id"]])
chat_alice = make_chat(alice)
# 1. Oversize: > 10 MiB
big = b"\x89PNG\r\n\x1a\n" + b"\x00" * (10 * 1024 * 1024 + 1)
code, _, _ = upload(alice["token"], chat_ab, "big.png", big, "image/png")
assert code == 413, f"oversize: expected 413, got {code}"
print("PASS: oversize -> 413")
# 2. Corrupted: валидная сигнатура, битые данные
corrupted = b"\x89PNG\r\n\x1a\n" + b"\xff" * 100
code, _, _ = upload(alice["token"], chat_ab, "corrupted.png", corrupted, "image/png")
assert code == 415, f"corrupted: expected 415, got {code}"
print("PASS: corrupted -> 415")
# 3. MIME-spoofing: image/png, но текст
spoofed = b"this is definitely not an image"
code, _, _ = upload(alice["token"], chat_ab, "spoof.png", spoofed, "image/png")
assert code == 415, f"mime spoof: expected 415, got {code}"
print("PASS: mime spoofing -> 415")
# 4. Чужой attachment_id: eve скачивает вложение из chat_ab
# Сначала загрузим настоящий PNG (тестовый fixture)
from pathlib import Path
fixture = Path(__file__).resolve().parent.parent / "tests/fixtures/sample.png"
if not fixture.exists():
raise SystemExit(f"fixture not found: {fixture}")
png = fixture.read_bytes()
code, att, _ = upload(alice["token"], chat_ab, "sample.png", png, "image/png")
assert code == 201, f"upload fixture failed: {code} {att}"
att_id = att["id"]
code, _, _ = call("GET", f"/api/v1/attachments/{att_id}",
headers={"Authorization": "Bearer " + eve["token"]})
assert code == 404, f"outsider metadata: expected 404, got {code}"
code, _, _ = call("GET", f"/api/v1/attachments/{att_id}/content",
headers={"Authorization": "Bearer " + eve["token"]})
assert code == 404, f"outsider content: expected 404, got {code}"
print("PASS: outsider attachment -> 404")
# 5. Вложение из другого чата
# Загружаем в chat_alice, прикрепляем к сообщению в chat_ab
code, att2, _ = upload(alice["token"], chat_alice, "other.png", png, "image/png")
assert code == 201, f"upload to chat_alice failed: {code} {att2}"
other_id = att2["id"]
code, _, _ = call(
"POST", f"/api/v1/chats/{chat_ab}/messages",
headers={
"Authorization": "Bearer " + alice["token"],
"Idempotency-Key": uuid.uuid4().hex,
},
body={"text": "cross-chat", "attachment_ids": [other_id]},
)
assert code == 404, f"cross-chat attachment: expected 404, got {code}"
print("PASS: cross-chat attachment -> 404")
print("\nNew tests from reviewer feedback")
test_decompression_bomb()
test_invalid_cursor()
test_filename_c1()
print("\nS3 unavailable")
test_s3_unavailable()
if __name__ == "__main__":
main()
+100
View File
@@ -0,0 +1,100 @@
"""Проверка отказа Valkey и корректности logout при недоступном кэше."""
from __future__ import annotations
import base64, json, subprocess, sys, time, urllib.error, urllib.request, uuid
BASE = "http://localhost:8080"
COMPOSE = ["docker", "compose", "--env-file", ".env"]
def call(method, path, headers=None, body=None):
data = None
h = dict(headers or {})
if body is not None:
data = json.dumps(body).encode()
h["Content-Type"] = "application/json"
req = urllib.request.Request(BASE + path, data=data, headers=h, method=method)
try:
r = urllib.request.urlopen(req, timeout=10)
raw = r.read()
return r.code, json.loads(raw) if raw else None
except urllib.error.HTTPError as e:
raw = e.read()
try:
return e.code, json.loads(raw) if raw else None
except Exception:
return e.code, None
def stop_cache():
subprocess.run(COMPOSE + ["stop", "cache"], check=True)
def start_cache():
subprocess.run(COMPOSE + ["start", "cache"], check=True)
def restart_api():
subprocess.run(COMPOSE + ["restart", "api"], check=True)
def wait_ready(deadline=60):
end = time.monotonic() + deadline
while time.monotonic() < end:
try:
code, _ = call("GET", "/health/ready")
if code == 200:
return
except Exception:
pass
time.sleep(1)
raise SystemExit("API did not become ready")
def main():
username = "cf_" + uuid.uuid4().hex[:10]
password = "cache-fail-pass-1234567890"
code, _ = call("POST", "/api/v1/users", body={
"username": username, "display_name": "Cache Fail", "password": password,
})
assert code == 201, code
cred = base64.b64encode(f"{username}:{password}".encode()).decode()
code, session = call("POST", "/api/v1/auth/sessions",
headers={"Authorization": "Basic " + cred})
assert code == 201, code
token = session["token"]
# Токен работает
code, _ = call("GET", "/api/v1/users/me",
headers={"Authorization": "Bearer " + token})
assert code == 200, code
# Останавливаем Valkey
stop_cache()
time.sleep(2)
# API должно работать в degraded mode
code, _ = call("GET", "/api/v1/users/me",
headers={"Authorization": "Bearer " + token})
assert code == 200, f"degraded mode failed: {code}"
print("PASS: API works with Valkey down (degraded mode)")
# Logout при недоступном Valkey должен вернуть 204
code, _ = call("DELETE", "/api/v1/auth/sessions/current",
headers={"Authorization": "Bearer " + token})
assert code == 204, f"logout with cache down returned {code}"
print("PASS: logout with cache down returned 204")
# Токен немедленно недействителен (проверка через БД)
code, _ = call("GET", "/api/v1/users/me",
headers={"Authorization": "Bearer " + token})
assert code == 401, f"revoked token still works with cache down: {code}"
print("PASS: revoked token rejected while cache down")
# Поднимаем Valkey, рестартуем API — критический регресс-тест
start_cache()
time.sleep(2)
restart_api()
wait_ready()
code, _ = call("GET", "/api/v1/users/me",
headers={"Authorization": "Bearer " + token})
assert code == 401, f"REGRESSION: revoked token resurrected after cache restart: {code}"
print("PASS: revoked token still 401 after cache restart (regression fixed)")
if __name__ == "__main__":
main()
+189
View File
@@ -0,0 +1,189 @@
"""Эксперимент: 10 одновременных загрузок 10 MiB. Замер пика RAM API.
Запуск:
python3 test_concurrent_uploads.py --base-url http://localhost:8080
Требует docker CLI для `docker stats`.
"""
from __future__ import annotations
import base64
import io
import json
import subprocess
import sys
import threading
import time
import urllib.error
import urllib.request
import uuid
from concurrent.futures import ThreadPoolExecutor
BASE = (
sys.argv[sys.argv.index("--base-url") + 1]
if "--base-url" in sys.argv
else "http://localhost:8080"
)
CONCURRENCY = 10
SIZE_MB = 10
API_CONTAINER = "mtusi2026lab4-api-1"
def call(method, path, headers=None, body=None, raw=None, timeout=60):
data = None
h = dict(headers or {})
if body is not None:
data = json.dumps(body, ensure_ascii=False).encode("utf-8")
h["Content-Type"] = "application/json"
elif raw is not None:
data = raw
req = urllib.request.Request(BASE + path, data=data, headers=h, method=method)
try:
resp = urllib.request.urlopen(req, timeout=timeout)
payload = resp.read()
return resp.status, json.loads(payload) if payload else None
except urllib.error.HTTPError as e:
payload = e.read()
try:
return e.code, json.loads(payload) if payload else None
except Exception:
return e.code, None
def make_user(label):
username = f"ram_{label}_{uuid.uuid4().hex[:10]}"
password = "ram-test-pass-1234567890"
code, user = call("POST", "/api/v1/users", body={
"username": username, "display_name": f"RAM {label}", "password": password,
})
assert code == 201, f"register failed: {code} {user}"
cred = base64.b64encode(f"{username}:{password}".encode()).decode()
code, session = call("POST", "/api/v1/auth/sessions",
headers={"Authorization": "Basic " + cred})
assert code == 201, f"login failed: {code}"
code, chat = call("POST", "/api/v1/chats",
headers={"Authorization": "Bearer " + session["token"]},
body={"title": "RAM", "member_ids": [user["id"]]})
assert code == 201, f"chat failed: {code}"
return {"token": session["token"], "chat_id": chat["id"]}
def measure_ram_mb(container):
"""Возвращает текущее RSS контейнера в MiB."""
try:
result = subprocess.run(
["docker", "stats", "--no-stream", "--format", "{{.MemUsage}}", container],
capture_output=True, text=True, timeout=5,
)
if result.returncode != 0:
return None
# "123.4MiB / 256MiB"
mem = result.stdout.strip().split("/")[0].strip()
if "MiB" in mem:
return float(mem.replace("MiB", "").strip())
if "GiB" in mem:
return float(mem.replace("GiB", "").strip()) * 1024
if "KiB" in mem:
return float(mem.replace("KiB", "").strip()) / 1024
return None
except Exception:
return None
def make_png_of_size(size_bytes):
"""Создаёт PNG ~size_bytes (не гарантированно точно, но близко)."""
from PIL import Image
# Создаём RGB изображение, сжимаем в PNG
# ~size_bytes: подбираем размеры
# 1 пиксель = ~3 байта + заголовки. Для 10 MiB — ~3500x1000
side = int((size_bytes / 3) ** 0.5)
img = Image.new("RGB", (side, side), color=(200, 100, 50))
buf = io.BytesIO()
img.save(buf, format="PNG", compress_level=0) # без сжатия — близко к size
data = buf.getvalue()
# Подгоняем: если больше — обрезаем? Нет, PNG с фиксированным размером.
return data
def main():
print(f"=== concurrent uploads: {CONCURRENCY} × {SIZE_MB} MiB ===")
user = make_user("user")
png = make_png_of_size(SIZE_MB * 1024 * 1024)
print(f"fixture size: {len(png)} bytes ({len(png) / 1024 / 1024:.2f} MiB)")
# Baseline RAM
baseline = measure_ram_mb(API_CONTAINER)
print(f"baseline RAM: {baseline:.1f} MiB" if baseline else "baseline RAM: unknown")
# Мониторинг RAM в фоне
peak = baseline or 0.0
stop = threading.Event()
samples = []
def monitor():
nonlocal peak
while not stop.is_set():
ram = measure_ram_mb(API_CONTAINER)
if ram is not None:
samples.append(ram)
peak = max(peak, ram)
time.sleep(0.2)
monitor_thread = threading.Thread(target=monitor, daemon=True)
monitor_thread.start()
# Запуск 10 одновременных загрузок
results = []
results_lock = threading.Lock()
barrier = threading.Barrier(CONCURRENCY)
def worker(i):
barrier.wait()
code, body = call(
"POST",
f"/api/v1/chats/{user['chat_id']}/attachments?filename=ram-{i}.png",
headers={
"Authorization": "Bearer " + user["token"],
"Content-Type": "image/png",
},
raw=png,
timeout=120,
)
with results_lock:
results.append((code, body))
started = time.monotonic()
with ThreadPoolExecutor(max_workers=CONCURRENCY) as pool:
list(pool.map(worker, range(CONCURRENCY)))
elapsed = time.monotonic() - started
stop.set()
monitor_thread.join(timeout=2)
codes = {}
for code, _ in results:
codes[code] = codes.get(code, 0) + 1
print(f"\nResults:")
print(f" duration: {elapsed:.2f}s")
print(f" status codes: {codes}")
print(f" peak RAM: {peak:.1f} MiB (baseline {baseline:.1f} MiB)")
print(f" delta: {peak - baseline:.1f} MiB")
print(f" theoretical max (if no streaming): {CONCURRENCY * SIZE_MB} MiB")
print(f" samples: {len(samples)}, min={min(samples):.1f}, max={max(samples):.1f}")
assert all(c == 201 for c, _ in results), f"not all uploads succeeded: {codes}"
# Ожидаем: delta < CONCURRENCY * SIZE_MB (иначе нет потоковой обработки)
theoretical = CONCURRENCY * SIZE_MB
if peak - baseline < theoretical * 0.5:
print(f"\nPASS: streaming works — peak delta {peak - baseline:.1f} MiB "
f"< 50% of theoretical {theoretical} MiB")
else:
print(f"\nWARN: peak delta {peak - baseline:.1f} MiB "
f">= 50% of theoretical {theoretical} MiB — check streaming")
if __name__ == "__main__":
main()
+127
View File
@@ -0,0 +1,127 @@
"""Проверка отказа PostgreSQL: таймаут запроса и восстановление."""
from __future__ import annotations
import base64, json, subprocess, sys, time, urllib.error, urllib.request, uuid
BASE = "http://localhost:8080"
COMPOSE = ["docker", "compose", "--env-file", ".env"]
def call(method, path, headers=None, body=None, timeout=10):
data = None
h = dict(headers or {})
if body is not None:
data = json.dumps(body).encode()
h["Content-Type"] = "application/json"
req = urllib.request.Request(BASE + path, data=data, headers=h, method=method)
try:
r = urllib.request.urlopen(req, timeout=timeout)
raw = r.read()
return r.code, json.loads(raw) if raw else None
except urllib.error.HTTPError as e:
raw = e.read()
try:
return e.code, json.loads(raw) if raw else None
except Exception:
return e.code, None
def stop_db():
"""Останавливает контейнер Postgres. TCP-соединения рвутся, psycopg
получает ошибку мгновенно — это честный сценарий 'БД упала'."""
subprocess.run(COMPOSE + ["stop", "postgres"], check=True)
def start_db():
"""Поднимает Postgres. Не падает, если уже запущен."""
subprocess.run(COMPOSE + ["start", "postgres"],
capture_output=True, check=False)
time.sleep(3)
def restart_api():
"""Пересоздаёт API, чтобы пул соединений не держал мёртвые сессии."""
subprocess.run(COMPOSE + ["restart", "api"],
capture_output=True, check=False)
def wait_ready(deadline=60):
end = time.monotonic() + deadline
while time.monotonic() < end:
try:
code, _ = call("GET", "/health/ready", timeout=5)
if code == 200:
return
except Exception:
pass
time.sleep(1)
raise SystemExit("API did not become ready")
def ensure_clean_state():
"""Возвращает стек в рабочее состояние, если предыдущий запуск оставил
Postgres остановленным или пул API в плохом состоянии."""
subprocess.run(COMPOSE + ["start", "postgres"],
capture_output=True, check=False)
time.sleep(3)
# API перезапускаем, потому что после stop Postgres пул мог забиться
# мёртвыми соединениями, и первый запрос упадёт по PoolTimeout.
restart_api()
wait_ready()
def main():
ensure_clean_state()
# Регистрация и логин — до всякого stop, чтобы тест не падал на подготовке
username = "dbf_" + uuid.uuid4().hex[:10]
password = "db-fail-pass-1234567890"
code, _ = call("POST", "/api/v1/users", body={
"username": username, "display_name": "DB Fail", "password": password,
})
assert code == 201, f"failed to create user before stop: {code}"
cred = base64.b64encode(f"{username}:{password}".encode()).decode()
code, session = call("POST", "/api/v1/auth/sessions",
headers={"Authorization": "Basic " + cred})
assert code == 201, f"failed to login before stop: {code}"
token = session["token"]
stop_db()
try:
time.sleep(0.5)
# Запрос к БД должен упасть быстро (TCP порван, psycopg сразу
# получает ошибку соединения). Проверяем < 8 секунд.
start = time.monotonic()
try:
code, _ = call("GET", "/api/v1/users/me",
headers={"Authorization": "Bearer " + token}, timeout=15)
except Exception as exc:
elapsed = time.monotonic() - start
raise SystemExit(f"request hung: {elapsed:.1f}s, {type(exc).__name__}")
elapsed = time.monotonic() - start
assert elapsed < 8, f"request took {elapsed:.1f}s (>8s), DB down not handled fast enough"
assert code in (500, 503), f"expected 500/503 during DB down, got {code}"
print(f"PASS: DB down returned {code} in {elapsed:.1f}s (<8s)")
# health/ready должен вернуть 503, пока БД недоступна
code, _ = call("GET", "/health/ready", timeout=15)
assert code == 503, f"expected 503 from ready, got {code}"
print("PASS: /health/ready returned 503 during DB down")
finally:
# Гарантированно поднимаем Postgres даже при падении выше.
start_db()
# Перезапускаем API, чтобы пул пересоздал соединения с вернувшейся БД.
restart_api()
wait_ready()
# Проверяем, что данные и сессия сохранены после восстановления.
code, _ = call("GET", "/api/v1/users/me",
headers={"Authorization": "Bearer " + token})
assert code == 200, f"session lost after DB recovery: {code}"
print("PASS: session and data preserved after DB recovery")
if __name__ == "__main__":
main()
+72
View File
@@ -0,0 +1,72 @@
"""Тест expiry: сессия истекает через SESSION_TTL_SECONDS.
Запуск:
SESSION_TTL_SECONDS=2 docker compose up -d --build api
python3 test_expiry.py --base-url http://localhost:8080
Тест создаёт пользователя, логинится, ждёт TTL+1 секунду, проверяет 401.
"""
from __future__ import annotations
import base64
import json
import sys
import time
import urllib.error
import urllib.request
import uuid
BASE = sys.argv[sys.argv.index("--base-url") + 1] if "--base-url" in sys.argv else "http://localhost:8080"
def call(method: str, path: str, headers: dict | None = None, body: dict | None = None) -> tuple[int, dict | None]:
data = None
h = dict(headers or {})
if body is not None:
data = json.dumps(body, ensure_ascii=False).encode("utf-8")
h["Content-Type"] = "application/json"
req = urllib.request.Request(BASE + path, data=data, headers=h, method=method)
try:
resp = urllib.request.urlopen(req, timeout=10)
raw = resp.read()
return resp.code, json.loads(raw) if raw else None
except urllib.error.HTTPError as e:
raw = e.read()
try:
return e.code, json.loads(raw) if raw else None
except Exception:
return e.code, None
def main() -> None:
username = "exp_" + uuid.uuid4().hex[:12]
password = "expiry-pass-1234567890"
# Регистрация
code, user = call("POST", "/api/v1/users", body={
"username": username, "display_name": "Expiry Test", "password": password,
})
assert code == 201, f"register failed: {code} {user}"
# Логин
cred = base64.b64encode(f"{username}:{password}".encode()).decode()
code, session = call("POST", "/api/v1/auth/sessions", headers={"Authorization": "Basic " + cred})
assert code == 201, f"login failed: {code} {session}"
token = session["token"]
print(f"session expires_at: {session['expires_at']}")
# Сразу после логина — токен работает
code, me = call("GET", "/api/v1/users/me", headers={"Authorization": "Bearer " + token})
assert code == 200, f"token should work immediately, got {code}"
print("immediately after login: 200 OK")
# Ждём истечения
time.sleep(3)
code, err = call("GET", "/api/v1/users/me", headers={"Authorization": "Bearer " + token})
assert code == 401, f"expired token must give 401, got {code}"
print(f"after TTL: {code} (expected 401)")
print("PASS: expiry — token invalidated after SESSION_TTL_SECONDS")
if __name__ == "__main__":
main()
+115
View File
@@ -0,0 +1,115 @@
"""Параллельная идемпотентность: N потоков посылают один POST с одним ключом.
Ожидаем: ровно одно сообщение создано, все ответы — одинаковые,
дубликатов в истории нет.
"""
from __future__ import annotations
import base64
import json
import sys
import threading
import urllib.error
import urllib.request
import uuid
from concurrent.futures import ThreadPoolExecutor
BASE = sys.argv[sys.argv.index("--base-url") + 1] if "--base-url" in sys.argv else "http://localhost:8080"
THREADS = 10
def call(method, path, headers=None, body=None):
data = None
h = dict(headers or {})
if body is not None:
data = json.dumps(body, ensure_ascii=False).encode("utf-8")
h["Content-Type"] = "application/json"
req = urllib.request.Request(BASE + path, data=data, headers=h, method=method)
try:
resp = urllib.request.urlopen(req, timeout=15)
raw = resp.read()
return resp.status, json.loads(raw) if raw else None, dict(resp.headers)
except urllib.error.HTTPError as e:
raw = e.read()
try:
return e.code, json.loads(raw) if raw else None, dict(e.headers)
except Exception:
return e.code, None, dict(e.headers)
def main():
username = "pario_" + uuid.uuid4().hex[:10]
password = "parallel-pass-1234567890"
# Регистрация
code, user, _ = call("POST", "/api/v1/users", body={
"username": username, "display_name": "Parallel", "password": password,
})
assert code == 201, f"register failed: {code} {user}"
user_id = user["id"]
# Логин
cred = base64.b64encode(f"{username}:{password}".encode()).decode()
code, session, _ = call("POST", "/api/v1/auth/sessions",
headers={"Authorization": "Basic " + cred})
assert code == 201, f"login failed: {code}"
token = session["token"]
# Чат
code, chat, _ = call("POST", "/api/v1/chats",
headers={"Authorization": "Bearer " + token},
body={"title": "Parallel test", "member_ids": [user_id]})
assert code == 201, f"chat failed: {code}"
chat_id = chat["id"]
# Параллельный POST с одним ключом
idem_key = "parallel-" + uuid.uuid4().hex
barrier = threading.Barrier(THREADS)
results = []
results_lock = threading.Lock()
def worker(i):
barrier.wait() # стартуют одновременно
code, body, headers = call(
"POST", f"/api/v1/chats/{chat_id}/messages",
headers={
"Authorization": "Bearer " + token,
"Idempotency-Key": idem_key,
},
body={"text": "parallel-message-same"}, # ← ОДИНАКОВЫЙ текст!
)
with results_lock:
results.append((code, body))
with ThreadPoolExecutor(max_workers=THREADS) as pool:
list(pool.map(worker, range(THREADS)))
# Анализ
print(f"Results: {len(results)} responses")
codes = {}
for code, _ in results:
codes[code] = codes.get(code, 0) + 1
print(f"Status codes: {codes}")
# Проверка: ровно одно сообщение создано
code, page, _ = call("GET", f"/api/v1/chats/{chat_id}/messages?limit=100",
headers={"Authorization": "Bearer " + token})
assert code == 200, f"history failed: {code}"
messages = page["items"]
print(f"Messages in history: {len(messages)}")
assert len(messages) == 1, f"expected exactly 1 message, got {len(messages)}"
print("PASS: parallel idempotency — exactly one message created")
# Проверка: все ответы должны быть одинаковыми
codes = [c for c, _ in results]
bodies = [json.dumps(b, sort_keys=True) for _, b in results if b is not None]
unique_bodies = set(bodies)
print(f"Status codes: {dict((c, codes.count(c)) for c in set(codes))}")
print(f"Unique response bodies: {len(unique_bodies)}")
assert codes.count(201) == THREADS, f"expected all {THREADS} to be 201, got {codes.count(201)}"
assert len(unique_bodies) == 1, f"expected same response body, got {len(unique_bodies)}"
if __name__ == "__main__":
main()
+116
View File
@@ -0,0 +1,116 @@
"""Тест гонки регистрации: два параллельных POST /users с одним username.
Ожидаем: один 201, второй 409, в БД ровно один пользователь.
Гонка гарантируется через threading.Barrier — оба запроса стартуют одновременно.
"""
from __future__ import annotations
import json
import sys
import threading
import urllib.error
import urllib.request
import uuid
from concurrent.futures import ThreadPoolExecutor
BASE = sys.argv[sys.argv.index("--base-url") + 1] if "--base-url" in sys.argv else "http://localhost:8080"
def post_user(username: str, display_name: str, password: str) -> int:
"""Отправляет POST /users, возвращает HTTP-код."""
body = json.dumps(
{"username": username, "display_name": display_name, "password": password},
ensure_ascii=False,
).encode("utf-8")
req = urllib.request.Request(
BASE + "/api/v1/users",
data=body,
headers={"Content-Type": "application/json"},
method="POST",
)
try:
resp = urllib.request.urlopen(req, timeout=10)
return resp.code
except urllib.error.HTTPError as e:
return e.code
def main() -> None:
username = "race_" + uuid.uuid4().hex[:12]
password = "race-pass-1234567890"
barrier = threading.Barrier(2)
def worker(display_name: str) -> int:
barrier.wait() # старт одновременно
return post_user(username, display_name, password)
with ThreadPoolExecutor(max_workers=2) as pool:
results = list(pool.map(worker, ["Первый", "Второй"]))
results.sort()
print(f"HTTP codes: {results}")
assert results == [201, 409], f"expected [201, 409], got {results}"
print(f"PASS: race — one 201, one 409 for username={username}")
def test_chat_rollback() -> None:
"""Попытка создать чат с несуществующим участником → 400, чат не создан."""
import base64
# Создаём пользователя
username = "rollback_" + uuid.uuid4().hex[:8]
password = "rollback-pass-1234567890"
code = post_user(username, "Rollback", password)
assert code == 201, f"failed to create user: {code}"
# Логинимся
cred = base64.b64encode(f"{username}:{password}".encode()).decode()
req = urllib.request.Request(
BASE + "/api/v1/auth/sessions",
headers={"Authorization": "Basic " + cred},
method="POST",
)
resp = urllib.request.urlopen(req)
session = json.loads(resp.read())
token = session["token"]
user_id = session["user"]["id"]
# Пытаемся создать чат с несуществующим UUID
fake_user = "00000000-0000-4000-8000-000000000000"
body = json.dumps(
{"title": "Bad chat", "member_ids": [user_id, fake_user]},
ensure_ascii=False,
).encode("utf-8")
req = urllib.request.Request(
BASE + "/api/v1/chats",
data=body,
headers={
"Content-Type": "application/json",
"Authorization": "Bearer " + token,
},
method="POST",
)
try:
urllib.request.urlopen(req, timeout=10)
code = 200
except urllib.error.HTTPError as e:
code = e.code
assert code == 400, f"expected 400 for unknown member, got {code}"
# Проверяем, что чат не создан — list chats не должен его содержать
req = urllib.request.Request(
BASE + "/api/v1/chats?limit=100",
headers={"Authorization": "Bearer " + token},
)
resp = urllib.request.urlopen(req)
page = json.loads(resp.read())
titles = [c["title"] for c in page["items"]]
assert "Bad chat" not in titles, f"chat was created despite failure: {titles}"
print("PASS: rollback — chat not created when member is unknown")
if __name__ == "__main__":
main()
test_chat_rollback()