Compare commits
10
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f2e074d9f2 | ||
|
|
255c11a822 | ||
|
|
9c15fc3744 | ||
|
|
55bb55d639 | ||
|
|
dc4f5c6c7f | ||
|
|
9347fe35ae | ||
|
|
412b465b85 | ||
|
|
22f7bcddf5 | ||
|
|
bd41f2d93d | ||
|
|
7d4cf242e3 |
@@ -1,7 +1,10 @@
|
||||
# Local/private state
|
||||
/docs/plans/
|
||||
/LAB_NOTES.md
|
||||
.env
|
||||
.env.*
|
||||
!.env.example
|
||||
/compose.local.yaml
|
||||
secrets/
|
||||
certs/private/
|
||||
*.key
|
||||
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
FROM python:3.12.12-slim-bookworm
|
||||
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1
|
||||
WORKDIR /app
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt && useradd --system --uid 10001 --home /app app
|
||||
COPY --chown=app:app src/ src/
|
||||
COPY --chown=app:app migrations/ migrations/
|
||||
COPY --chown=app:app scripts/migrate.py scripts/migrate.py
|
||||
COPY --chown=app:app scripts/reconcile-attachments.py scripts/reconcile-attachments.py
|
||||
USER app
|
||||
EXPOSE 8080
|
||||
CMD ["python", "-m", "src.server"]
|
||||
@@ -1,37 +1,75 @@
|
||||
# Отчёт — лабораторная 4
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
- Автор, группа: Самохвалов Андрей Ремович, БПИ-2302.
|
||||
- Целевая оценка: 5, трек 2 «Согласованность двух хранилищ». Уровни 3 и 4 и повторяемый эксперимент трека 5 проверены.
|
||||
- Ветка: `solution/bpi2302-samohvalov`; commit/PR указываются после публикации.
|
||||
- Источники: `README.md`, `COURSE.md`, `contracts/README.md`, `contracts/openapi.json`, публичные тесты, [MinIO release](https://github.com/minio/minio/releases/tag/RELEASE.2025-10-15T17-29-55Z), [образ](https://hub.docker.com/r/alpine/minio/tags), документация MinIO для IAM. Python/FastAPI, PostgreSQL, MinIO Python SDK; проект подготовлен с помощью ИИ Codex с локальной проверкой ниже.
|
||||
|
||||
## Запуск из чистого clone
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
Требуются Docker Compose и Python 3.9+. Стенд использует только синтетические данные и слушает `127.0.0.1` на `APP_PORT` (по умолчанию 8080). В репозитории нет паролей:
|
||||
|
||||
```sh
|
||||
cp .env.example .env
|
||||
python3 scripts/setup-local-secrets.py
|
||||
docker compose --env-file .env config --quiet
|
||||
docker compose --env-file .env up -d --build --wait --scale api=2
|
||||
make check
|
||||
make test
|
||||
```
|
||||
|
||||
`setup-local-secrets.py` создаёт отсутствующие secret-файлы с правами 0600 и не перезаписывает существующие. Миграции выполняет одноразовый `migrate` под PostgreSQL advisory lock. `s3-volume-init` задаёт владельца нового volume, затем MinIO стартует без root; одноразовый `s3-bootstrap` идемпотентно создаёт закрытый bucket и две ограниченные учётные записи. Повторный `docker compose --env-file .env run --rm s3-bootstrap` безопасен. Для остановки — `docker compose --env-file .env down` без `-v`; для намеренного удаления **только учебных данных** после проверки — `docker compose --env-file .env down -v`.
|
||||
|
||||
Версии: Python 3.12.12, PostgreSQL 17.6, HAProxy 3.0.11, MinIO `RELEASE.2025-10-15T17-29-55Z` (образ закреплён digest), MinIO Client `2025.7.21-debian-12-r3` (digest). Проверенный локальный стенд — Colima/arm64 macOS, 2 CPU, 2 054 840 320 байт памяти. Локальная сеть не разрешала DNS `files.pythonhosted.org`, поэтому сборка для проверки использовала **только локальный ignored** `compose.local.yaml` с временным IP CDN; в чистом clone он не нужен при обычном доступе к PyPI.
|
||||
|
||||
## Архитектура и решения
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab3, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
Клиент → HAProxy → две API-реплики. PostgreSQL хранит пользователей, сессии, чаты, сообщения, ключи идемпотентности и метаданные вложений. MinIO хранит только байты оригиналов в private bucket `messenger-attachments`; локальная файловая система API не хранит оригиналы. Миграция `004_attachments.sql` связывает вложение с чатом; `005_attachment_owner.sql` добавляет владельца и время обновления по OpenAPI. Состав сообщения хранится в `message_attachments` с FK по паре `(id, chat_id)`, чтобы исключить межчатовые ссылки на уровне БД. Статусы таблицы допускают дальнейший переход к асинхронной lab5, но lab4 возвращает только `ready` после обеих устойчивых записей.
|
||||
|
||||
| Сервис | Образ/роль | Health и volume | Секреты и доступ |
|
||||
| --- | --- | --- | --- |
|
||||
| `lb` | HAProxy 3.0.11, единственный host port | проверка backend, без volume | секретов нет |
|
||||
| `api ×2` | Python 3.12.12, 0,5 CPU/256 MiB на реплику | `/health/ready`, без volume | только DB URL и ограниченные S3 credentials |
|
||||
| `postgres` | PostgreSQL 17.6 | `pg_isready`, `postgres_data` | только DB password |
|
||||
| `migrate` | одноразовый Python | завершение 0 | только DB URL |
|
||||
| `s3` | MinIO security release, 0,5 CPU/384 MiB | `/minio/health/ready`, `s3_data` | только root credentials, наружный порт и консоль не опубликованы |
|
||||
| `s3-bootstrap` | одноразовый MinIO Client | завершение 0 | root и создаваемые ограниченные credentials |
|
||||
| `s3-volume-init` | одноразовая настройка владельца volume | завершение 0 | секретов нет |
|
||||
| `reconcile` | одноразовый Python, профиль `maintenance` | завершение 0, без порта и volume | DB URL и отдельные S3 maintenance credentials |
|
||||
|
||||
`infra/minio-api-policy.json` разрешает только операции над объектами одного bucket и чтение его location; создание bucket, изменение IAM и административные операции не разрешены. Отдельная maintenance policy добавляет list для будущей сверки. Bucket не имеет анонимного чтения. Приватность bucket не заменяет проверку членства в чате на API: upload проверяет его до чтения тела, metadata/content — до доступа к S3. Публичный ответ не содержит object key, endpoint и credentials.
|
||||
|
||||
Upload читает raw body в `SpooledTemporaryFile` с порогом RAM 1 MiB и останавливается после 10 MiB; проверяет формат и MIME через Pillow, до декодирования отклоняет свыше 40 млн пикселей, затем полностью проверяет PNG/JPEG. Имя файла применяется только для отображения, object key — случайный UUID в `originals/`. S3 PUT предшествует INSERT/COMMIT в PostgreSQL; `201 ready` возвращается лишь после обоих успехов. При достоверной ошибке до COMMIT объект удаляется. Если подтверждение COMMIT потеряно, удалять небезопасно: объект остаётся кандидатом для сверки после grace period. Если и компенсационное DELETE отказало, key записывается во внутренний лог; reconciler позднее удаляет только старый объект без metadata. Клиенту при ошибке зависимости приходит `503`, а прозрачный retry POST не выполняется. Для S3 заданы connect/read timeouts 2/10 с без внутренних повторов; `/health/ready` проверяет БД и доступ к приватному S3 через разрешённый HEAD отсутствующего probe key.
|
||||
|
||||
POST сообщения принимает до 10 уникальных ID из этого чата, включая пустой текст с вложением. Проверка вложений, запись сообщения, связей и результата идемпотентности выполняются в одной транзакции; порядок ID сохраняется. Повтор ключа с тем же содержимым возвращает прежний Message. `GET` metadata и content сначала проверяют действующую сессию и членство в чате; неизвестный и чужой ID одинаково дают 404. Исходник отдаётся потоково через API, отсутствующий вариант — 404, недоступный S3-объект — 503.
|
||||
|
||||
`docker compose --profile maintenance run --rm reconcile` выполняет **dry-run по умолчанию**. `--apply` удаляет только объекты `originals/<uuid>` без metadata старше 3600 с: перед DELETE повторно проверяются запись БД, размер, ETag, время изменения и возраст. Неизвестные ключи не удаляются. Для metadata без S3-объекта выдаётся ручной инцидент; metadata сохраняется. Восстановление требует `--restore-id` и read-only fixture с точным размером и SHA-256. Файл проверяется до любых удалений при `--apply`; несоответствие останавливает run. Повторный apply не меняет уже восстановленный объект. У API нет `ListBucket`, у отдельной maintenance-учётной записи он есть; обе не имеют прав создавать bucket или менять IAM.
|
||||
|
||||
## Доказательства критериев
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Доказательство |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
| Перенос lab3 | Compose ×2, текстовая часть `make test`, `make test-persistence`, адресный failover обеих реплик | Текстовый API совместим, прежний токен/сообщение после restart; восстановление записи после остановки каждой реплики 3,123 и 3,149 с | локальные тесты U1; полный smoke остановился на ещё не реализованном upload |
|
||||
| Приватный S3 и bootstrap | `docker compose run --rm s3-bootstrap` дважды; `docker compose exec -T api python - < student-tests/test_storage_bootstrap.py` | Два повтора завершились 0; S3 PUT/GET ограниченным API пользователем совпадают по байтам, анонимный GET — 403, создание чужого bucket — AccessDenied | `evidence/storage-bootstrap-2026-10-02.txt` |
|
||||
| Миграция и транзакции | `docker compose exec -T api python - < student-tests/test_database.py`; `... test_auth.py` | Миграции 1–5 применены повторяемо, 2 DB и 3 auth теста PASS | `evidence/storage-bootstrap-2026-10-02.txt` |
|
||||
| Raw PNG/JPEG upload и ограничения | `BASE_URL=http://localhost:8084 python3 -m unittest discover -s student-tests -p 'test_attachment_upload.py' -v` | 4 набора PASS: PNG/JPEG `201 ready`, корректные hash/size/type; 10 MiB+1 и >40 млн px — 413; повреждение/MIME spoof — 415; опасное имя — 400, чужой чат — 404 | `student-tests/test_attachment_upload.py`, синтетический JPEG fixture |
|
||||
| Отказ S3 | `docker compose stop s3`; `docker compose exec -T api python - < student-tests/test_s3_outage.py`; `docker compose start s3` | При реально остановленном S3 readiness=503, upload=503, число metadata строк не изменилось; S3 восстановлен и healthy | `student-tests/test_s3_outage.py` |
|
||||
| Промежуток S3/БД и обрыв потока | `docker compose exec -T api python - < student-tests/test_upload_failure.py` | 4/4 PASS: PUT failure не вызывает SQL; ошибка до INSERT компенсируется DELETE; оборванный поток не начинает PUT; неоднозначный COMMIT оставляет объект для сверки | `evidence/reconcile-u5-2026-10-02.txt` |
|
||||
| Ограничение RAM | `BASE_URL=http://localhost:8084 python3 scripts/measure-upload-memory.py` | 10/10 одновременных upload одного PNG 9 441 653 байт, 10 разных ready ID и верный SHA-256, 6,145 с; sampled peak API-1/2 114,7/115,5 MiB при лимите 256 MiB | `evidence/upload-memory-2026-10-02.csv` |
|
||||
| Сообщения, ACL и скачивание | `python3 -m unittest discover -s student-tests -p test_attachment_access.py -v`; `make test BASE_URL=http://127.0.0.1:8084` | 3 собственных теста OK; публичный сценарий PASS; link/replay и bytes на обеих репликах, чужой объект 404 | `evidence/attachment-access-2026-10-02.txt` |
|
||||
| Сохранность S3 и отказ API | `make test-persistence ... ACTION_SCRIPT=scripts/restart.sh`; `make test-failover ... ACTION_SCRIPT=scripts/stop-one.sh`; `python3 student-tests/test_attachment_failover.py` | Токен, Message, metadata и bytes пережили restart API/DB/S3; публичный failover восстановил запись за 2,826 и 3,020 с; адресный тест остановил каждую загрузившую реплику и скачал те же bytes через другую | `evidence/attachment-access-2026-10-02.txt` |
|
||||
| Сверка orphan/missing | `docker compose --profile maintenance run --rm reconcile`; `student-tests/test_reconcile.py` в maintenance-контейнере | 2/2 PASS: old orphan после PG rollback удалён по истечении grace; fresh и привязанный сохранены; missing дал 503, неверный fixture не изменил данные, верный восстановил bytes/200; повтор идемпотентен | `evidence/reconcile-u5-2026-10-02.txt`, три JSON результата |
|
||||
| Права S3 | `student-tests/test_storage_permissions.py` в API и maintenance-контейнерах | 1/1 PASS для каждой роли; API не может list, обе не могут создавать bucket, анонимный GET 403, консоль не опубликована | `evidence/reconcile-u5-2026-10-02.txt` |
|
||||
| Трек 5: два отказа и стоимость сверки | `sh scripts/fault-injection.sh`; повторные dry-run/apply; 3 замера на каждом размере inventory | Old orphan удалён, missing восстановлен только по проверенному fixture; fresh/linked сохранены; повторы без изменений. Для 333/334 и 533/534 строк/объектов медианы wall 21,143 и 24,219 мс, CPU 7,420 и 9,993 мс | `evidence/reconciliation/analysis-2026-10-02.md`, `run-20261002T111817Z.json` |
|
||||
|
||||
## Эксперименты и отказы
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
Граница PUT/COMMIT проверена fault-инъекциями. Повторяемый сценарий трека 5 создал orphan после успешного S3 PUT и отклонённой PostgreSQL-транзакции, а также missing после COMMIT метаданных и удаления S3-объекта. Dry-run не изменил данные. Apply удалил только старый orphan; свежий и связанный с сообщением объекты остались. Missing без fixture остался ручным инцидентом, неверный fixture был отвергнут до мутаций, проверенный по размеру и SHA-256 fixture восстановил исходные байты. Повторные dry-run/apply не сделали новых изменений. Для теста удаления grace уменьшался до 5 секунд только на синтетическом наборе; обычный запуск использует 3600 секунд. Список объектов MinIO содержит миллисекунды, а HEAD округляет время до секунд: перед удалением сверяются секунды, ETag и размер, затем возраст и ссылка в БД. После раннего канонического dry-run 28 metadata, 29 objects, 1 свежий orphan был защищён и ни один объект не удалён.
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
Для двух размеров inventory сценарий выполнил по три последовательных dry-run в одном maintenance-процессе: 333 metadata/334 objects — wall 19,390/21,143/23,322 мс, CPU 6,447/7,420/8,516 мс; 533/534 — wall 16,585/24,219/44,148 мс, CPU 9,993/9,997/9,500 мс. RSS процесса между сериями изменился с 68 152 до 68 936 KiB, high-water 126 608 KiB был достигнут до этих сканов. Больший inventory потребовал больше CPU в этой серии; wall сильно колебался. Шесть замеров на одном локальном стенде не определяют закон масштабирования; ресурсы PostgreSQL и S3 в них не включены. Полные сырые результаты и способ воспроизведения приведены в `evidence/reconciliation/`.
|
||||
|
||||
Замер RAM проводился на локальном Colima/arm64, 2 CPU и 2 GiB, API ×2 с 0,5 CPU/256 MiB на экземпляр, 10 потоков с барьером старта. CSV содержит baseline и два Docker stats снимка во время 6,145 с серии: peak означает максимум **наблюдённых** снимков, а не точный аппаратный high-water mark. Все 10 ответов 201 имели уникальные ID и совпадающий SHA-256. Создание body по `os.urandom` оставляет воспроизводимыми размер/разрешение/алгоритм, но не конкретные байты и hash между прогонами.
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
Публичный `make test` проходит; отказные сценарии подтверждены отдельными проверками. Missing без проверенного источника остаётся ручным инцидентом, автоматической реконструкции чужих байтов нет. Один MinIO/PostgreSQL/LB и один Docker-host остаются одиночными точками отказа. Локальный HTTP допустим только для изолированного стенда с синтетическими данными; удалённой демонстрации нужен TLS.
|
||||
|
||||
+157
@@ -0,0 +1,157 @@
|
||||
name: mtusi2026lab4
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:17.6-bookworm
|
||||
environment:
|
||||
POSTGRES_DB: messenger
|
||||
POSTGRES_USER: messenger
|
||||
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
|
||||
secrets: [postgres_password]
|
||||
volumes: [postgres_data:/var/lib/postgresql/data]
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U messenger -d messenger"]
|
||||
interval: 2s
|
||||
timeout: 2s
|
||||
retries: 20
|
||||
|
||||
migrate:
|
||||
build: .
|
||||
command: ["python", "scripts/migrate.py"]
|
||||
environment:
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
secrets: [database_url]
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
restart: "no"
|
||||
|
||||
s3-volume-init:
|
||||
image: alpine/minio:RELEASE.2025-10-15T17-29-55Z@sha256:cf23643a6cf9ce159c57643ceb88279e431262282428c9e0bf3a7ef1a97e84b4
|
||||
user: "0:0"
|
||||
entrypoint: ["/bin/sh", "-c"]
|
||||
command: ["chown -R 100:100 /data"]
|
||||
volumes: [s3_data:/data]
|
||||
restart: "no"
|
||||
|
||||
s3:
|
||||
image: alpine/minio:RELEASE.2025-10-15T17-29-55Z@sha256:cf23643a6cf9ce159c57643ceb88279e431262282428c9e0bf3a7ef1a97e84b4
|
||||
command: ["server", "/data", "--console-address", ":9001"]
|
||||
environment:
|
||||
MINIO_ROOT_USER_FILE: /run/secrets/minio_root_user
|
||||
MINIO_ROOT_PASSWORD_FILE: /run/secrets/minio_root_password
|
||||
secrets: [minio_root_user, minio_root_password]
|
||||
volumes: [s3_data:/data]
|
||||
depends_on:
|
||||
s3-volume-init:
|
||||
condition: service_completed_successfully
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "wget -q -O /dev/null http://127.0.0.1:9000/minio/health/ready"]
|
||||
interval: 2s
|
||||
timeout: 2s
|
||||
retries: 30
|
||||
cpus: 0.5
|
||||
mem_limit: 384m
|
||||
|
||||
s3-bootstrap:
|
||||
image: bitnamilegacy/minio-client:2025.7.21-debian-12-r3@sha256:73bd39f7899a0cef12b8dd5df13aa93a3ed1aaa44236542442e9ac76819ac158
|
||||
user: "0:0"
|
||||
entrypoint: ["/bin/sh", "/bootstrap.sh"]
|
||||
command: []
|
||||
volumes:
|
||||
- ./infra/bootstrap-s3.sh:/bootstrap.sh:ro
|
||||
- ./infra/minio-api-policy.json:/policies/minio-api-policy.json:ro
|
||||
- ./infra/minio-maintenance-policy.json:/policies/minio-maintenance-policy.json:ro
|
||||
secrets:
|
||||
- minio_root_user
|
||||
- minio_root_password
|
||||
- s3_access_key_id
|
||||
- s3_secret_access_key
|
||||
- s3_maintenance_key_id
|
||||
- s3_maintenance_secret_key
|
||||
depends_on:
|
||||
s3:
|
||||
condition: service_healthy
|
||||
restart: "no"
|
||||
|
||||
api:
|
||||
build: .
|
||||
scale: 2
|
||||
environment:
|
||||
APP_HOST: 0.0.0.0
|
||||
APP_PORT: 8080
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
|
||||
S3_ENDPOINT: s3:9000
|
||||
S3_BUCKET: messenger-attachments
|
||||
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
|
||||
secrets: [database_url, s3_access_key_id, s3_secret_access_key]
|
||||
depends_on:
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
s3-bootstrap:
|
||||
condition: service_completed_successfully
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=2).close()"]
|
||||
interval: 2s
|
||||
timeout: 3s
|
||||
retries: 20
|
||||
start_period: 5s
|
||||
stop_grace_period: 10s
|
||||
cpus: 0.5
|
||||
mem_limit: 256m
|
||||
|
||||
reconcile:
|
||||
profiles: [maintenance]
|
||||
build: .
|
||||
command: ["python", "scripts/reconcile-attachments.py"]
|
||||
environment:
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
S3_ENDPOINT: s3:9000
|
||||
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_maintenance_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_maintenance_secret_key
|
||||
secrets: [database_url, s3_maintenance_key_id, s3_maintenance_secret_key]
|
||||
volumes: ["./tests/fixtures:/fixtures:ro"]
|
||||
depends_on:
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
s3-bootstrap:
|
||||
condition: service_completed_successfully
|
||||
restart: "no"
|
||||
cpus: 0.5
|
||||
mem_limit: 256m
|
||||
|
||||
lb:
|
||||
image: haproxy:3.0.11-alpine
|
||||
volumes:
|
||||
- ./infra/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
|
||||
ports:
|
||||
- "127.0.0.1:${APP_PORT:-8080}:8080"
|
||||
depends_on:
|
||||
api:
|
||||
condition: service_healthy
|
||||
cpus: 0.5
|
||||
mem_limit: 128m
|
||||
|
||||
secrets:
|
||||
postgres_password:
|
||||
file: ./secrets/postgres_password
|
||||
database_url:
|
||||
file: ./secrets/database_url
|
||||
minio_root_user:
|
||||
file: ./secrets/minio_root_user
|
||||
minio_root_password:
|
||||
file: ./secrets/minio_root_password
|
||||
s3_access_key_id:
|
||||
file: ./secrets/s3_access_key_id
|
||||
s3_secret_access_key:
|
||||
file: ./secrets/s3_secret_access_key
|
||||
s3_maintenance_key_id:
|
||||
file: ./secrets/s3_maintenance_key_id
|
||||
s3_maintenance_secret_key:
|
||||
file: ./secrets/s3_maintenance_secret_key
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
s3_data:
|
||||
@@ -0,0 +1,25 @@
|
||||
Lab4 U4 local verification, 2026-10-02, Colima arm64, APP_PORT=8084.
|
||||
Synthetic data only; no tokens, passwords, object keys or image bodies recorded.
|
||||
|
||||
docker compose -f compose.yaml -f compose.local.yaml --env-file .env up -d --build --wait --scale api=2
|
||||
Result: postgres/s3/lb and both API replicas healthy; migrate/bootstrap one-shots exited 0.
|
||||
|
||||
BASE_URL=http://127.0.0.1:8084 python3 -m unittest discover -s student-tests -p test_attachment_access.py -v
|
||||
Result: 3 tests OK: message link/replay, ACL and invalid links, bytes served by both replicas.
|
||||
|
||||
make test BASE_URL=http://127.0.0.1:8084
|
||||
Result: PASS lab4 public API scenario.
|
||||
|
||||
make test-persistence BASE_URL=http://127.0.0.1:8084 ACTION_SCRIPT=scripts/restart.sh
|
||||
Result: PASS; hook stopped API, LB, PostgreSQL and S3, then restarted without removing volumes.
|
||||
Existing token, confirmed message, attachment metadata and original bytes survived.
|
||||
|
||||
make test-failover BASE_URL=http://127.0.0.1:8084 ACTION_SCRIPT=scripts/stop-one.sh
|
||||
Result, run 1: two instances observed; 112 confirmed writes, 1 failed attempt,
|
||||
first recovery 2.826 seconds; existing token/message/original preserved.
|
||||
Result, run 2 after restore: two instances observed; 113 confirmed writes,
|
||||
1 failed attempt, first recovery 3.020 seconds; existing token/message/original preserved.
|
||||
|
||||
BASE_URL=http://127.0.0.1:8084 python3 student-tests/test_attachment_failover.py
|
||||
Result: stopped each of the two distinct replicas after it accepted an upload;
|
||||
the other replica served byte-identical original; both were restored between runs.
|
||||
@@ -0,0 +1,19 @@
|
||||
Lab 4 U5, local Colima/arm64, 2026-10-02, synthetic data only.
|
||||
Service setup: Compose API x2 + PostgreSQL + private MinIO. Default reconciliation grace: 3600 s.
|
||||
The ignored compose.local.yaml only supplied a temporary package-CDN address for local builds.
|
||||
|
||||
Checks:
|
||||
- `docker compose ... exec -T api python - < student-tests/test_upload_failure.py`: 4/4 PASS, including uncertain COMMIT retaining the object.
|
||||
- `docker compose ... exec -T api python - < student-tests/test_storage_permissions.py`: 1/1 PASS; API identity cannot ListBucket or create a bucket, anonymous GET 403.
|
||||
- `docker compose ... --profile maintenance run --rm -T reconcile python - < student-tests/test_storage_permissions.py`: 1/1 PASS; separate maintenance identity can list but cannot create a bucket, anonymous GET 403, root/API secrets absent.
|
||||
- `docker compose ... --profile maintenance run --rm -T reconcile python - < student-tests/test_reconcile.py`: 2/2 PASS. Old orphan originated from S3 PUT followed by PostgreSQL FK rollback; missing originated from deleting an existing S3 object while ready metadata remained. Dry-run did not mutate. With a test-only 5-second grace, apply deleted the old orphan while preserving a fresh orphan and a message-linked attachment. A wrong fixture was rejected before any deletion; missing content returned HTTP 503. The fixture with matching size/SHA-256 restored the original key, after which API content returned the exact original bytes. Repeat restore was `already_present`.
|
||||
- `docker compose ... ps s3`: healthy S3; only internal `9000-9001/tcp`, no host port published.
|
||||
|
||||
CLI evidence (separate runs, production grace 3600 s):
|
||||
- `reconcile-u5-dry-run.json`: 28 metadata / 28 objects; 1 fresh orphan protected, 1 missing/manual incident, 0 deleted, 5.176 ms.
|
||||
- `reconcile-u5-restore.json`: verified fixture restored that synthetic missing object; 0 deleted, 9.130 ms.
|
||||
- `reconcile-u5-repeat.json`: restore `already_present`, no remaining missing ID, 0 deleted, 7.657 ms.
|
||||
|
||||
MinIO list timestamps include milliseconds while HEAD timestamps have second precision. Before deletion, the reconciler compares their second-level time, ETag and size, then rechecks the age and PostgreSQL reference. The first integration run exposed this representation difference; after the fix, both tests passed.
|
||||
|
||||
Limits: these are single-run local timings, not a throughput estimate. The 5-second grace is used only for synthetic integration data; routine maintenance defaults to one hour. Missing bytes without a verified source remain a manual incident, with metadata retained. A test-created fresh orphan remains in the inventory until it ages beyond grace and a later apply run handles it.
|
||||
@@ -0,0 +1 @@
|
||||
{"deleted": [], "duration_ms": 5.176, "fresh_orphans": ["originals/c2046814-1d5e-429b-b537-168d68d43f40"], "grace_seconds": 3600, "inventory": {"metadata": 28, "objects": 28}, "manual_incidents": ["ee0c99ff-772d-4957-91aa-8a599a67578a"], "missing_attachment_ids": ["ee0c99ff-772d-4957-91aa-8a599a67578a"], "mode": "dry_run", "old_orphans": [], "preserved_after_recheck": [], "restore": null, "unknown_keys_preserved": []}
|
||||
@@ -0,0 +1 @@
|
||||
{"deleted": [], "duration_ms": 7.657, "fresh_orphans": ["originals/c2046814-1d5e-429b-b537-168d68d43f40"], "grace_seconds": 3600, "inventory": {"metadata": 28, "objects": 29}, "manual_incidents": [], "missing_attachment_ids": [], "mode": "apply", "old_orphans": [], "preserved_after_recheck": [], "restore": "already_present", "unknown_keys_preserved": []}
|
||||
@@ -0,0 +1 @@
|
||||
{"deleted": [], "duration_ms": 9.13, "fresh_orphans": ["originals/c2046814-1d5e-429b-b537-168d68d43f40"], "grace_seconds": 3600, "inventory": {"metadata": 28, "objects": 28}, "manual_incidents": [], "missing_attachment_ids": ["ee0c99ff-772d-4957-91aa-8a599a67578a"], "mode": "apply", "old_orphans": [], "preserved_after_recheck": [], "restore": "restored", "unknown_keys_preserved": []}
|
||||
@@ -0,0 +1,24 @@
|
||||
# Reconciliation fault injection and inventory cost
|
||||
|
||||
## Hypothesis
|
||||
|
||||
The reconciler can identify an orphan left by a successful S3 PUT followed by a failed PostgreSQL write, and a missing S3 object referenced by committed metadata. Applying repairs should delete only an orphan older than the grace period, restore missing bytes only from a matching fixture, and leave fresh or linked objects untouched. A second pass should find no work for the repaired cases.
|
||||
|
||||
## Reproduce
|
||||
|
||||
On the local Compose stand with `.env` configured, run `sh scripts/fault-injection.sh`. The local build used `COMPOSE_FILE=compose.yaml:compose.local.yaml` because this host needed a temporary Docker DNS override; the override is ignored by Git. The script builds and runs the maintenance service and writes anonymized JSON under `evidence/reconciliation/`. It uses a synthetic user, chat, message, attachments, and object keys. Repeated runs append more synthetic inventory to persistent volumes, so compare measurements only within one run.
|
||||
|
||||
The PostgreSQL fault is an invalid foreign key insert after a successful S3 PUT. The transaction rolls back, leaving an object with no metadata. The S3 fault deletes an object after its metadata commits. A 5-second grace period lets the test age the first orphan, then create a fresh one. Production maintenance defaults to one hour. A message links another attachment throughout the test.
|
||||
|
||||
## Observed result
|
||||
|
||||
Raw result: [run-20261002T111817Z.json](run-20261002T111817Z.json). The initial dry-run saw 283 metadata rows and 285 S3 objects, including the injected old orphan, fresh orphan, and missing object. The dry-run left them unchanged. A deliberately wrong source was rejected before deletion. Apply without a source deleted the old orphan, retained the missing metadata as a manual incident, and preserved the fresh and linked objects. A second dry-run found the old orphan gone and the missing incident present; a repeated delete made no change to the injected orphan. Apply with the correct fixture restored the exact SHA-256 bytes. Repeated restore returned `already_present`, and the final dry-run found neither injected inconsistency.
|
||||
|
||||
Inventory scans used three sequential dry-run trials per size in the same maintenance process. CPU is user plus system time for that process; RSS is Linux `VmRSS`, and high-water RSS is `ru_maxrss`. Times are milliseconds, RSS is KiB.
|
||||
|
||||
| Inventory | Trial wall times | Trial CPU times | Median wall | Median CPU | RSS before → after across trials | Peak RSS |
|
||||
| --- | --- | --- | ---: | ---: | --- | ---: |
|
||||
| 333 metadata / 334 objects | 19.390, 21.143, 23.322 | 6.447, 7.420, 8.516 | 21.143 | 7.420 | 68152 → 68512 | 126608 |
|
||||
| 533 metadata / 534 objects | 16.585, 24.219, 44.148 | 9.993, 9.997, 9.500 | 24.219 | 9.993 | 68512 → 68936 | 126608 |
|
||||
|
||||
The larger inventory used more CPU in this run; wall time varied widely, including one faster larger-inventory trial. These six samples on one local host do not establish a scaling law. The measurements exclude PostgreSQL and S3 server resources and include pre-existing data in persistent volumes. The process high-water RSS was reached before these scans, so it cannot isolate the scan's peak allocation; the before/after RSS readings show only retained process memory.
|
||||
@@ -0,0 +1 @@
|
||||
{"faults": {"apply_without_source": {"fresh_preserved": true, "linked_preserved": true, "missing_incident_retained": true, "old_deleted": true}, "post_delete_dry_run": {"missing_incident_retained": true, "old_absent": true}, "post_restore_dry_run": {"missing_absent": true, "old_absent": true}, "preview": {"fresh_detected": true, "inventory": {"metadata": 283, "objects": 285}, "linked_preserved": true, "missing_detected": true, "old_detected": true}, "repeat": "already_present", "repeat_delete_no_change": true, "restore": "restored", "wrong_fixture_rejected_without_delete": true}, "limits": "Single local host, 3 sequential dry-run scans per size; CPU/RSS are maintenance-process measurements only.", "measurements": [{"cpu_system_ms": 0.031, "cpu_user_ms": 6.416, "label": "plus_50", "metadata": 333, "objects": 334, "reconciler_ms": 19.349, "rss_after_kib": 68468, "rss_before_kib": 68152, "rss_hwm_kib": 126608, "trial": 1, "wall_ms": 19.39}, {"cpu_system_ms": 0.0, "cpu_user_ms": 7.42, "label": "plus_50", "metadata": 333, "objects": 334, "reconciler_ms": 21.098, "rss_after_kib": 68500, "rss_before_kib": 68468, "rss_hwm_kib": 126608, "trial": 2, "wall_ms": 21.143}, {"cpu_system_ms": 0.02, "cpu_user_ms": 8.496, "label": "plus_50", "metadata": 333, "objects": 334, "reconciler_ms": 23.282, "rss_after_kib": 68512, "rss_before_kib": 68500, "rss_hwm_kib": 126608, "trial": 3, "wall_ms": 23.322}, {"cpu_system_ms": 0.993, "cpu_user_ms": 9.0, "label": "plus_250", "metadata": 533, "objects": 534, "reconciler_ms": 16.523, "rss_after_kib": 68908, "rss_before_kib": 68512, "rss_hwm_kib": 126608, "trial": 1, "wall_ms": 16.585}, {"cpu_system_ms": 0.007, "cpu_user_ms": 9.99, "label": "plus_250", "metadata": 533, "objects": 534, "reconciler_ms": 24.163, "rss_after_kib": 68936, "rss_before_kib": 68908, "rss_hwm_kib": 126608, "trial": 2, "wall_ms": 24.219}, {"cpu_system_ms": 0.0, "cpu_user_ms": 9.5, "label": "plus_250", "metadata": 533, "objects": 534, "reconciler_ms": 44.084, "rss_after_kib": 68936, "rss_before_kib": 68936, "rss_hwm_kib": 126608, "trial": 3, "wall_ms": 44.148}], "summary": {"plus_250": {"median_cpu_ms": 9.993, "median_wall_ms": 24.219, "metadata": 533, "objects": 534}, "plus_50": {"median_cpu_ms": 7.42, "median_wall_ms": 21.143, "metadata": 333, "objects": 334}}}
|
||||
@@ -0,0 +1,12 @@
|
||||
Lab 4 U2 local Colima/arm64; 2 CPU, 2054840320 bytes RAM; synthetic data only.
|
||||
Image: alpine/minio:RELEASE.2025-10-15T17-29-55Z, digest sha256:cf23643a6cf9ce159c57643ceb88279e431262282428c9e0bf3a7ef1a97e84b4.
|
||||
The official minio/minio tag could not be pulled (repository unavailable); pinned alpine/minio security-release build was used.
|
||||
Build used ignored local compose.local.yaml with temporary PyPI CDN DNS override; ordinary compose.yaml remains independent of this host workaround.
|
||||
|
||||
docker compose --env-file .env config --quiet: PASS.
|
||||
docker compose --env-file .env up -d --wait --scale api=2: PASS after adding one-time s3-volume-init for non-root MinIO write permissions.
|
||||
docker compose --env-file .env run --rm s3-bootstrap: PASS, repeated twice after initial bootstrap, no duplicate/failure.
|
||||
docker compose --env-file .env exec -T api python - < student-tests/test_storage_bootstrap.py: 1 PASS; PUT/GET exact bytes, anonymous GET 403, foreign bucket create AccessDenied.
|
||||
docker compose --env-file .env exec -T api python - < student-tests/test_database.py: 2 PASS; migrations 1-4, race/rollback/keyset history.
|
||||
docker compose --env-file .env exec -T api python - < student-tests/test_auth.py: 3 PASS; Basic/Bearer and short TTL.
|
||||
No credentials, session tokens or object keys are included in this evidence.
|
||||
@@ -0,0 +1,7 @@
|
||||
seconds,container,used_mib,limit_mib
|
||||
0,mtusi2026lab4-api-1,48.58,256.0
|
||||
0,mtusi2026lab4-api-2,51.2,256.0
|
||||
2.129,mtusi2026lab4-api-1,114.7,256.0
|
||||
2.129,mtusi2026lab4-api-2,115.5,256.0
|
||||
4.141,mtusi2026lab4-api-1,114.7,256.0
|
||||
4.141,mtusi2026lab4-api-2,115.5,256.0
|
||||
|
@@ -0,0 +1,16 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
|
||||
read_secret() {
|
||||
tr -d '\n' < "/run/secrets/$1"
|
||||
}
|
||||
|
||||
mc alias set local http://s3:9000 "$(read_secret minio_root_user)" "$(read_secret minio_root_password)" >/dev/null
|
||||
mc mb --ignore-existing local/messenger-attachments >/dev/null
|
||||
mc anonymous set none local/messenger-attachments >/dev/null
|
||||
mc admin policy create local messenger-api /policies/minio-api-policy.json >/dev/null
|
||||
mc admin policy create local messenger-maintenance /policies/minio-maintenance-policy.json >/dev/null
|
||||
mc admin user add local "$(read_secret s3_access_key_id)" "$(read_secret s3_secret_access_key)" >/dev/null
|
||||
mc admin policy attach local messenger-api --user "$(read_secret s3_access_key_id)" >/dev/null
|
||||
mc admin user add local "$(read_secret s3_maintenance_key_id)" "$(read_secret s3_maintenance_secret_key)" >/dev/null
|
||||
mc admin policy attach local messenger-maintenance --user "$(read_secret s3_maintenance_key_id)" >/dev/null
|
||||
@@ -0,0 +1,26 @@
|
||||
global
|
||||
log stdout format raw local0
|
||||
|
||||
defaults
|
||||
mode http
|
||||
log global
|
||||
option httplog
|
||||
timeout connect 2s
|
||||
timeout client 15s
|
||||
timeout server 15s
|
||||
|
||||
resolvers docker
|
||||
nameserver docker 127.0.0.11:53
|
||||
resolve_retries 2
|
||||
timeout retry 1s
|
||||
hold valid 1s
|
||||
|
||||
frontend public
|
||||
bind *:8080
|
||||
default_backend api
|
||||
|
||||
backend api
|
||||
balance roundrobin
|
||||
option httpchk GET /health/ready
|
||||
http-check expect status 200
|
||||
server-template api 1-4 api:8080 check inter 1s fall 2 rise 1 resolvers docker init-addr none
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": ["s3:GetBucketLocation"],
|
||||
"Resource": ["arn:aws:s3:::messenger-attachments"]
|
||||
},
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
|
||||
"Resource": ["arn:aws:s3:::messenger-attachments/*"]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": ["s3:GetBucketLocation", "s3:ListBucket"],
|
||||
"Resource": ["arn:aws:s3:::messenger-attachments"]
|
||||
},
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
|
||||
"Resource": ["arn:aws:s3:::messenger-attachments/*"]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
CREATE TABLE users (
|
||||
id uuid PRIMARY KEY,
|
||||
username text NOT NULL UNIQUE CHECK (username ~ '^[a-z0-9_]{3,32}$'),
|
||||
display_name text NOT NULL CHECK (char_length(display_name) BETWEEN 1 AND 100),
|
||||
password_hash text NOT NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
|
||||
CREATE TABLE chats (
|
||||
id uuid PRIMARY KEY,
|
||||
title text NOT NULL CHECK (char_length(title) BETWEEN 1 AND 100),
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
|
||||
CREATE INDEX chats_history_idx ON chats (created_at, id);
|
||||
|
||||
CREATE TABLE memberships (
|
||||
chat_id uuid NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||
user_id uuid NOT NULL REFERENCES users(id),
|
||||
position smallint NOT NULL CHECK (position BETWEEN 0 AND 99),
|
||||
PRIMARY KEY (chat_id, user_id),
|
||||
UNIQUE (chat_id, position)
|
||||
);
|
||||
|
||||
CREATE INDEX memberships_user_idx ON memberships (user_id, chat_id);
|
||||
|
||||
CREATE TABLE messages (
|
||||
id uuid PRIMARY KEY,
|
||||
chat_id uuid NOT NULL,
|
||||
sender_id uuid NOT NULL,
|
||||
text text NOT NULL CHECK (char_length(text) BETWEEN 1 AND 4000),
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
FOREIGN KEY (chat_id, sender_id) REFERENCES memberships(chat_id, user_id)
|
||||
);
|
||||
|
||||
CREATE INDEX messages_history_idx ON messages (chat_id, created_at, id);
|
||||
@@ -0,0 +1,7 @@
|
||||
CREATE TABLE sessions (
|
||||
token_hash bytea PRIMARY KEY,
|
||||
user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
expires_at timestamptz NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX sessions_user_idx ON sessions (user_id);
|
||||
@@ -0,0 +1,11 @@
|
||||
CREATE TABLE message_replays (
|
||||
user_id uuid NOT NULL REFERENCES users(id),
|
||||
chat_id uuid NOT NULL REFERENCES chats(id),
|
||||
request_key text NOT NULL,
|
||||
request_hash bytea NOT NULL,
|
||||
message_id uuid REFERENCES messages(id),
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
PRIMARY KEY (user_id, chat_id, request_key)
|
||||
);
|
||||
|
||||
CREATE INDEX message_replays_expiry_idx ON message_replays (created_at);
|
||||
@@ -0,0 +1,32 @@
|
||||
CREATE TABLE attachments (
|
||||
id uuid PRIMARY KEY,
|
||||
chat_id uuid NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||
filename text NOT NULL,
|
||||
object_key text NOT NULL UNIQUE,
|
||||
content_type text NOT NULL CHECK (content_type IN ('image/png', 'image/jpeg')),
|
||||
size_bytes bigint NOT NULL CHECK (size_bytes BETWEEN 1 AND 10485760),
|
||||
sha256 text NOT NULL CHECK (sha256 ~ '^[0-9a-f]{64}$'),
|
||||
status text NOT NULL DEFAULT 'ready' CHECK (status IN ('queued', 'processing', 'ready', 'failed')),
|
||||
error_code text,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
UNIQUE (id, chat_id)
|
||||
);
|
||||
|
||||
CREATE INDEX attachments_chat_idx ON attachments (chat_id, created_at);
|
||||
|
||||
ALTER TABLE messages DROP CONSTRAINT messages_text_check;
|
||||
ALTER TABLE messages ADD CONSTRAINT messages_text_check CHECK (char_length(text) BETWEEN 0 AND 4000);
|
||||
ALTER TABLE messages ADD CONSTRAINT messages_id_chat_unique UNIQUE (id, chat_id);
|
||||
|
||||
CREATE TABLE message_attachments (
|
||||
message_id uuid NOT NULL,
|
||||
chat_id uuid NOT NULL,
|
||||
attachment_id uuid NOT NULL,
|
||||
position smallint NOT NULL CHECK (position BETWEEN 0 AND 9),
|
||||
PRIMARY KEY (message_id, attachment_id),
|
||||
UNIQUE (message_id, position),
|
||||
FOREIGN KEY (message_id, chat_id) REFERENCES messages(id, chat_id) ON DELETE CASCADE,
|
||||
FOREIGN KEY (attachment_id, chat_id) REFERENCES attachments(id, chat_id)
|
||||
);
|
||||
|
||||
CREATE INDEX message_attachments_attachment_idx ON message_attachments (attachment_id);
|
||||
@@ -0,0 +1,2 @@
|
||||
ALTER TABLE attachments ADD COLUMN owner_id uuid NOT NULL REFERENCES users(id);
|
||||
ALTER TABLE attachments ADD COLUMN updated_at timestamptz NOT NULL DEFAULT clock_timestamp();
|
||||
@@ -0,0 +1,6 @@
|
||||
-- PostgreSQL text cannot contain U+0000; bytea preserves the complete UTF-8 payload.
|
||||
-- API validation enforces the 4000-character limit (up to 16000 UTF-8 bytes).
|
||||
ALTER TABLE messages DROP CONSTRAINT messages_text_check;
|
||||
ALTER TABLE messages ALTER COLUMN text TYPE bytea USING convert_to(text, 'UTF8');
|
||||
ALTER TABLE messages ADD CONSTRAINT messages_text_check
|
||||
CHECK (octet_length(text) <= 16000);
|
||||
@@ -0,0 +1,9 @@
|
||||
fastapi==0.118.3
|
||||
pydantic==2.12.4
|
||||
uvicorn==0.38.0
|
||||
psycopg[binary,pool]==3.2.12
|
||||
psycopg-pool==3.3.3
|
||||
argon2-cffi==25.1.0
|
||||
minio==7.2.20
|
||||
Pillow==11.3.0
|
||||
urllib3==2.8.0
|
||||
Executable
+17
@@ -0,0 +1,17 @@
|
||||
#!/bin/sh
|
||||
# Run on an isolated synthetic-data Compose stand. COMPOSE_FILE may include a local override.
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
mkdir -p evidence/reconciliation
|
||||
stamp=$(date -u +%Y%m%dT%H%M%SZ)
|
||||
temporary="evidence/reconciliation/.run-$stamp.tmp"
|
||||
output="evidence/reconciliation/run-$stamp.json"
|
||||
trap 'rm -f "$temporary"' EXIT HUP INT TERM
|
||||
|
||||
docker compose --env-file .env --profile maintenance build reconcile
|
||||
docker compose --env-file .env --profile maintenance run --rm -T reconcile python - \
|
||||
< student-tests/test_reconcile_research.py > "$temporary"
|
||||
python3 -m json.tool "$temporary" >/dev/null
|
||||
mv "$temporary" "$output"
|
||||
trap - EXIT HUP INT TERM
|
||||
printf 'Reconciliation evidence: %s\n' "$output"
|
||||
@@ -0,0 +1,93 @@
|
||||
"""Ten concurrent bounded PNG uploads plus Docker memory samples; BASE_URL overrides localhost:8080."""
|
||||
|
||||
import csv
|
||||
import hashlib
|
||||
import os
|
||||
import re
|
||||
import struct
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
import zlib
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
from client import Client # noqa: E402
|
||||
|
||||
API_NAMES = ("mtusi2026lab4-api-1", "mtusi2026lab4-api-2")
|
||||
EVIDENCE = ROOT / "evidence/upload-memory-2026-10-02.csv"
|
||||
|
||||
|
||||
def png_chunk(name: bytes, data: bytes) -> bytes:
|
||||
return struct.pack(">I", len(data)) + name + data + struct.pack(">I", zlib.crc32(name + data))
|
||||
|
||||
|
||||
def large_png() -> bytes:
|
||||
width, height = 2048, 1536
|
||||
rows = b"".join(b"\0" + os.urandom(width * 3) for _ in range(height))
|
||||
return b"\x89PNG\r\n\x1a\n" + png_chunk(b"IHDR", struct.pack(">IIBBBBB", width, height, 8, 2, 0, 0, 0)) + png_chunk(b"IDAT", zlib.compress(rows, 1)) + png_chunk(b"IEND", b"")
|
||||
|
||||
|
||||
def mib(text: str) -> float:
|
||||
match = re.fullmatch(r"([0-9.]+)(B|KiB|MiB|GiB)", text.strip())
|
||||
if not match:
|
||||
raise ValueError("unexpected Docker memory unit")
|
||||
return float(match.group(1)) * {"B": 1 / 1048576, "KiB": 1 / 1024, "MiB": 1, "GiB": 1024}[match.group(2)]
|
||||
|
||||
|
||||
def memory_samples(elapsed: float) -> list[tuple[float, str, float, float]]:
|
||||
command = ["rtk", "proxy", "docker", "stats", "--no-stream", "--format", "{{.Name}},{{.MemUsage}}", *API_NAMES]
|
||||
output = subprocess.run(command, check=True, capture_output=True, text=True, timeout=10).stdout
|
||||
rows = []
|
||||
for line in output.splitlines():
|
||||
name, usage = line.split(",", 1)
|
||||
used, limit = usage.split("/", 1)
|
||||
rows.append((round(elapsed, 3), name, round(mib(used), 3), round(mib(limit), 3)))
|
||||
if len(rows) != 2:
|
||||
raise AssertionError("expected two API container memory samples")
|
||||
return rows
|
||||
|
||||
|
||||
def main() -> None:
|
||||
client = Client(timeout=60)
|
||||
actor, _, _, chat = client.bootstrap()
|
||||
data = large_png()
|
||||
if len(data) > 10 * 1024 * 1024:
|
||||
raise AssertionError("synthetic PNG exceeds upload limit")
|
||||
expected_hash = hashlib.sha256(data).hexdigest()
|
||||
barrier = threading.Barrier(10)
|
||||
|
||||
def send(index: int) -> dict:
|
||||
barrier.wait(timeout=10)
|
||||
result = Client(timeout=60).call(
|
||||
"POST", f"/api/v1/chats/{chat['id']}/attachments?filename=load-{index}.png", 201,
|
||||
actor=actor, raw=data, headers={"Content-Type": "image/png"}, schema="Attachment",
|
||||
)["json"]
|
||||
if result["sha256"] != expected_hash or result["size_bytes"] != len(data):
|
||||
raise AssertionError("upload bytes/hash mismatch")
|
||||
return result
|
||||
|
||||
start = time.monotonic()
|
||||
rows = memory_samples(0)
|
||||
with ThreadPoolExecutor(max_workers=10) as workers:
|
||||
futures = [workers.submit(send, index) for index in range(10)]
|
||||
while not all(future.done() for future in futures):
|
||||
rows.extend(memory_samples(time.monotonic() - start))
|
||||
results = [future.result() for future in futures]
|
||||
rows.extend(memory_samples(time.monotonic() - start))
|
||||
if len({result["id"] for result in results}) != 10:
|
||||
raise AssertionError("uploads did not create ten unique attachments")
|
||||
EVIDENCE.parent.mkdir(exist_ok=True)
|
||||
with EVIDENCE.open("w", newline="") as target:
|
||||
writer = csv.writer(target, lineterminator="\n")
|
||||
writer.writerow(("seconds", "container", "used_mib", "limit_mib"))
|
||||
writer.writerows(rows)
|
||||
peaks = {name: max(row[2] for row in rows if row[1] == name) for name in API_NAMES}
|
||||
print(f"PASS: 10/10 uploads; PNG={len(data)} bytes; elapsed={time.monotonic()-start:.3f}s; peaks={peaks}; evidence={EVIDENCE.name}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,30 @@
|
||||
"""Apply numbered SQL migrations once, atomically and under a DB lock."""
|
||||
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
import psycopg
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
from src.db import database_url
|
||||
|
||||
|
||||
def main() -> None:
|
||||
migration_dir = Path(__file__).resolve().parents[1] / "migrations"
|
||||
with psycopg.connect(database_url(), connect_timeout=5, options="-c statement_timeout=30000") as conn:
|
||||
with conn.transaction():
|
||||
conn.execute("SELECT pg_advisory_xact_lock(2026, 2)")
|
||||
conn.execute(
|
||||
"CREATE TABLE IF NOT EXISTS schema_migrations ("
|
||||
"version integer PRIMARY KEY, applied_at timestamptz NOT NULL DEFAULT clock_timestamp())"
|
||||
)
|
||||
for path in sorted(migration_dir.glob("[0-9][0-9][0-9]_*.sql")):
|
||||
version = int(path.name.split("_", 1)[0])
|
||||
if conn.execute("SELECT 1 FROM schema_migrations WHERE version=%s", (version,)).fetchone():
|
||||
continue
|
||||
conn.execute(path.read_text(encoding="utf-8"))
|
||||
conn.execute("INSERT INTO schema_migrations (version) VALUES (%s)", (version,))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,173 @@
|
||||
"""Compare PostgreSQL attachment metadata with private S3 originals.
|
||||
|
||||
Dry-run is the default. Run in the Compose maintenance profile with its separate S3 identity.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from uuid import UUID
|
||||
|
||||
import psycopg
|
||||
from minio import Minio
|
||||
from minio.error import S3Error
|
||||
from psycopg.rows import dict_row
|
||||
from urllib3 import PoolManager, Retry, Timeout
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
from src.db import database_url # noqa: E402
|
||||
|
||||
BUCKET = "messenger-attachments"
|
||||
ORIGINAL = re.compile(r"originals/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\Z")
|
||||
MAX_SOURCE = 10 * 1024 * 1024
|
||||
|
||||
|
||||
def secret(name: str) -> str:
|
||||
return Path(os.environ[f"{name}_FILE"]).read_text(encoding="utf-8").strip()
|
||||
|
||||
|
||||
def storage() -> Minio:
|
||||
return Minio(
|
||||
os.environ.get("S3_ENDPOINT", "s3:9000"),
|
||||
access_key=secret("S3_ACCESS_KEY_ID"), secret_key=secret("S3_SECRET_ACCESS_KEY"),
|
||||
secure=False, region="us-east-1",
|
||||
http_client=PoolManager(timeout=Timeout(connect=2, read=10), retries=Retry(total=0)),
|
||||
)
|
||||
|
||||
|
||||
def age_seconds(last_modified: datetime, now: datetime) -> float:
|
||||
if last_modified is None or last_modified.tzinfo is None:
|
||||
return 0 # Unknown age is never safe to delete.
|
||||
return max(0, (now - last_modified).total_seconds())
|
||||
|
||||
|
||||
def absent(client: Minio, key: str) -> bool:
|
||||
try:
|
||||
client.stat_object(BUCKET, key)
|
||||
return False
|
||||
except S3Error as exc:
|
||||
if exc.code == "NoSuchKey":
|
||||
return True
|
||||
raise
|
||||
|
||||
|
||||
def source_hash(path: Path) -> tuple[int, str]:
|
||||
size, digest = 0, hashlib.sha256()
|
||||
with path.open("rb") as stream:
|
||||
while chunk := stream.read(1024 * 1024):
|
||||
size += len(chunk)
|
||||
if size > MAX_SOURCE:
|
||||
raise ValueError("restore source exceeds 10 MiB")
|
||||
digest.update(chunk)
|
||||
return size, digest.hexdigest()
|
||||
|
||||
|
||||
def delete_orphan(client: Minio, conn: psycopg.Connection, item, grace: int) -> bool:
|
||||
key = item.object_name
|
||||
if conn.execute("SELECT 1 FROM attachments WHERE object_key=%s", (key,)).fetchone():
|
||||
return False
|
||||
try:
|
||||
current = client.stat_object(BUCKET, key)
|
||||
except S3Error as exc:
|
||||
if exc.code == "NoSuchKey":
|
||||
return False
|
||||
raise
|
||||
if (
|
||||
current.etag != item.etag or current.last_modified.replace(microsecond=0) != item.last_modified.replace(microsecond=0)
|
||||
or current.size != item.size or age_seconds(current.last_modified, datetime.now(timezone.utc)) < grace
|
||||
):
|
||||
return False
|
||||
# Upload keys are random and S3 PUT + PostgreSQL INSERT is bounded; grace and recheck protect live uploads.
|
||||
client.remove_object(BUCKET, key)
|
||||
return True
|
||||
|
||||
|
||||
def restore_missing(client: Minio, conn: psycopg.Connection, attachment_id: UUID, source: Path, apply: bool) -> str:
|
||||
size, digest = source_hash(source)
|
||||
row = conn.execute(
|
||||
"SELECT id,object_key,size_bytes,sha256,content_type,status FROM attachments WHERE id=%s FOR UPDATE",
|
||||
(attachment_id,),
|
||||
).fetchone()
|
||||
if row is None or row["status"] != "ready":
|
||||
raise ValueError("ready attachment not found")
|
||||
if not ORIGINAL.fullmatch(row["object_key"]):
|
||||
raise ValueError("invalid original object key")
|
||||
if (size, digest) != (row["size_bytes"], row["sha256"]):
|
||||
raise ValueError("restore source size/SHA-256 mismatch")
|
||||
if not absent(client, row["object_key"]):
|
||||
return "already_present"
|
||||
if not apply:
|
||||
return "would_restore"
|
||||
with source.open("rb") as stream:
|
||||
client.put_object(BUCKET, row["object_key"], stream, size, content_type=row["content_type"])
|
||||
if client.stat_object(BUCKET, row["object_key"]).size != size:
|
||||
raise RuntimeError("restored object size mismatch")
|
||||
return "restored"
|
||||
|
||||
|
||||
def reconcile(client: Minio, conn: psycopg.Connection, apply: bool, grace: int, restore_id: UUID | None, source: Path | None) -> dict:
|
||||
start = time.monotonic()
|
||||
metadata = conn.execute(
|
||||
"SELECT id,object_key,size_bytes,sha256,content_type,status FROM attachments"
|
||||
).fetchall()
|
||||
objects = list(client.list_objects(BUCKET, prefix="originals/", recursive=True))
|
||||
by_key = {row["object_key"]: row for row in metadata}
|
||||
by_object = {item.object_name: item for item in objects}
|
||||
now = datetime.now(timezone.utc)
|
||||
old = [item for item in objects if item.object_name not in by_key and ORIGINAL.fullmatch(item.object_name)
|
||||
and age_seconds(item.last_modified, now) >= grace]
|
||||
fresh = [item.object_name for item in objects if item.object_name not in by_key and ORIGINAL.fullmatch(item.object_name)
|
||||
and age_seconds(item.last_modified, now) < grace]
|
||||
unknown = [item.object_name for item in objects if item.object_name not in by_key and not ORIGINAL.fullmatch(item.object_name)]
|
||||
missing = [row for row in metadata if row["status"] == "ready" and row["object_key"] not in by_object
|
||||
and absent(client, row["object_key"])]
|
||||
|
||||
result = {
|
||||
"mode": "apply" if apply else "dry_run", "grace_seconds": grace,
|
||||
"inventory": {"objects": len(objects), "metadata": len(metadata)},
|
||||
"old_orphans": [item.object_name for item in old], "fresh_orphans": fresh,
|
||||
"unknown_keys_preserved": unknown, "missing_attachment_ids": [str(row["id"]) for row in missing],
|
||||
"manual_incidents": [str(row["id"]) for row in missing],
|
||||
"deleted": [], "preserved_after_recheck": [], "restore": None,
|
||||
}
|
||||
if restore_id is not None and apply:
|
||||
# Reject an invalid source before any orphan deletion in this apply run.
|
||||
restore_missing(client, conn, restore_id, source, False)
|
||||
if apply:
|
||||
for item in old:
|
||||
(result["deleted"] if delete_orphan(client, conn, item, grace) else result["preserved_after_recheck"]).append(item.object_name)
|
||||
if restore_id is not None:
|
||||
result["restore"] = restore_missing(client, conn, restore_id, source, apply)
|
||||
if result["restore"] in {"restored", "already_present"}:
|
||||
result["manual_incidents"] = [item for item in result["manual_incidents"] if item != str(restore_id)]
|
||||
result["duration_ms"] = round((time.monotonic() - start) * 1000, 3)
|
||||
return result
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--apply", action="store_true", help="perform safe deletions and selected verified restore")
|
||||
parser.add_argument("--grace-seconds", type=int, default=3600, help="minimum age before deleting an orphan (default 3600)")
|
||||
parser.add_argument("--restore-id", type=UUID, help="one missing attachment to restore")
|
||||
parser.add_argument("--source", type=Path, help="read-only original fixture with matching size and SHA-256")
|
||||
args = parser.parse_args()
|
||||
if args.grace_seconds < 1 or (args.restore_id is None) != (args.source is None):
|
||||
parser.error("grace must be positive; restore-id and source must be supplied together")
|
||||
try:
|
||||
with psycopg.connect(database_url(), row_factory=dict_row, connect_timeout=5, options="-c statement_timeout=30000") as conn:
|
||||
result = reconcile(storage(), conn, args.apply, args.grace_seconds, args.restore_id, args.source)
|
||||
print(json.dumps(result, ensure_ascii=False, sort_keys=True))
|
||||
return 0
|
||||
except (psycopg.Error, S3Error, OSError, ValueError, RuntimeError) as exc:
|
||||
print(json.dumps({"error": type(exc).__name__, "message": "reconciliation stopped without further mutation"}))
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Executable
+5
@@ -0,0 +1,5 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
docker compose --env-file .env stop api lb postgres s3
|
||||
docker compose --env-file .env up -d --wait --scale api=2
|
||||
@@ -0,0 +1,42 @@
|
||||
"""Create local Compose secrets without printing or overwriting existing values."""
|
||||
|
||||
import os
|
||||
import secrets
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1] / "secrets"
|
||||
|
||||
|
||||
def write(name: str, value: str) -> None:
|
||||
fd = os.open(ROOT / name, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
|
||||
with os.fdopen(fd, "w") as out:
|
||||
out.write(value)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ROOT.mkdir(mode=0o700, exist_ok=True)
|
||||
db_names = {"postgres_password", "database_url"}
|
||||
s3_names = {
|
||||
"minio_root_user", "minio_root_password", "s3_access_key_id", "s3_secret_access_key",
|
||||
"s3_maintenance_key_id", "s3_maintenance_secret_key",
|
||||
}
|
||||
for names in (db_names, s3_names):
|
||||
existing = {name for name in names if (ROOT / name).exists()}
|
||||
if existing and existing != names:
|
||||
raise SystemExit(f"Incomplete secret group: {', '.join(sorted(names - existing))}")
|
||||
if not (ROOT / "postgres_password").exists():
|
||||
password = secrets.token_urlsafe(32)
|
||||
write("postgres_password", password)
|
||||
write("database_url", f"postgresql://messenger:{password}@postgres:5432/messenger")
|
||||
if not (ROOT / "minio_root_user").exists():
|
||||
write("minio_root_user", "lab4admin")
|
||||
write("minio_root_password", secrets.token_urlsafe(32))
|
||||
write("s3_access_key_id", secrets.token_hex(10))
|
||||
write("s3_secret_access_key", secrets.token_urlsafe(32))
|
||||
write("s3_maintenance_key_id", secrets.token_hex(10))
|
||||
write("s3_maintenance_secret_key", secrets.token_urlsafe(32))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+13
@@ -0,0 +1,13 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
: "${TARGET_INSTANCE_ID:?test provides target API instance alias}"
|
||||
for container in $(docker compose --env-file .env ps -q api); do
|
||||
instance=$(docker exec "$container" python -c "import urllib.request; print(urllib.request.urlopen('http://127.0.0.1:8080/health/live', timeout=2).headers['X-Instance-Id'])")
|
||||
if [ "$instance" = "$TARGET_INSTANCE_ID" ]; then
|
||||
docker stop -t 5 "$container" >/dev/null
|
||||
exit 0
|
||||
fi
|
||||
done
|
||||
echo "target API instance not found" >&2
|
||||
exit 1
|
||||
+117
@@ -0,0 +1,117 @@
|
||||
"""Messenger HTTP routes."""
|
||||
|
||||
import re
|
||||
from uuid import UUID
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request, Response
|
||||
|
||||
from .auth import basic_credentials, bearer_token
|
||||
from .attachments import storage_ready, upload
|
||||
from .attachment_read import download, get_view
|
||||
from .models import CreateChat, CreateMessage, CreateUser
|
||||
from .store import store
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/health/live")
|
||||
def live():
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
@router.get("/health/ready")
|
||||
def ready():
|
||||
result = store.ready()
|
||||
storage_ready()
|
||||
return result
|
||||
|
||||
|
||||
@router.post("/api/v1/users", status_code=201)
|
||||
def register_user(body: CreateUser):
|
||||
return store.register(body.username, body.display_name, body.password)
|
||||
|
||||
|
||||
@router.post("/api/v1/auth/sessions", status_code=201)
|
||||
async def create_session(request: Request, response: Response):
|
||||
if await request.body():
|
||||
raise HTTPException(400)
|
||||
username, password = basic_credentials(request.headers.get("authorization"))
|
||||
response.headers["Cache-Control"] = "no-store"
|
||||
return store.create_session(username, password)
|
||||
|
||||
|
||||
def current_user(request: Request) -> dict:
|
||||
return store.authenticated(bearer_token(request.headers.get("authorization")))
|
||||
|
||||
|
||||
def parse_limit(value: str) -> int:
|
||||
try:
|
||||
number = int(value) if value.isdecimal() else 0
|
||||
except ValueError:
|
||||
raise HTTPException(400) from None
|
||||
if not 1 <= number <= 100:
|
||||
raise HTTPException(400)
|
||||
return number
|
||||
|
||||
|
||||
@router.delete("/api/v1/auth/sessions/current", status_code=204)
|
||||
def revoke_session(request: Request):
|
||||
store.revoke_session(bearer_token(request.headers.get("authorization")))
|
||||
return Response(status_code=204)
|
||||
|
||||
|
||||
@router.get("/api/v1/users/me")
|
||||
def user_me(actor: dict = Depends(current_user)):
|
||||
return actor
|
||||
|
||||
|
||||
@router.post("/api/v1/chats", status_code=201)
|
||||
def create_chat(body: CreateChat, actor: dict = Depends(current_user)):
|
||||
return store.create_chat(actor["id"], body.title, body.member_ids)
|
||||
|
||||
|
||||
@router.get("/api/v1/chats")
|
||||
def list_chats(
|
||||
limit: str = Query("50"), cursor: str | None = Query(None, min_length=1), actor: dict = Depends(current_user)
|
||||
):
|
||||
return store.list_chats(actor["id"], parse_limit(limit), cursor)
|
||||
|
||||
|
||||
@router.get("/api/v1/chats/{chat_id}")
|
||||
def get_chat(chat_id: UUID, actor: dict = Depends(current_user)):
|
||||
return store.chat(actor["id"], str(chat_id))
|
||||
|
||||
|
||||
@router.post("/api/v1/chats/{chat_id}/attachments", status_code=201)
|
||||
async def upload_attachment(chat_id: UUID, request: Request, filename: str = Query(...), actor: dict = Depends(current_user)):
|
||||
return await upload(request, str(chat_id), actor["id"], filename)
|
||||
|
||||
|
||||
@router.get("/api/v1/attachments/{attachment_id}")
|
||||
def get_attachment(attachment_id: UUID, actor: dict = Depends(current_user)):
|
||||
return get_view(actor["id"], attachment_id)
|
||||
|
||||
|
||||
@router.get("/api/v1/attachments/{attachment_id}/content")
|
||||
def get_attachment_content(
|
||||
attachment_id: UUID, variant: str = Query("original"), actor: dict = Depends(current_user)
|
||||
):
|
||||
return download(actor["id"], attachment_id, variant)
|
||||
|
||||
|
||||
@router.post("/api/v1/chats/{chat_id}/messages", status_code=201)
|
||||
def create_message(chat_id: UUID, body: CreateMessage, request: Request, actor: dict = Depends(current_user)):
|
||||
key = request.headers.get("Idempotency-Key")
|
||||
if key is None or not re.fullmatch(r"[A-Za-z0-9._:-]{1,128}", key):
|
||||
raise HTTPException(400)
|
||||
return store.create_message(actor["id"], str(chat_id), body.text, key, body.attachment_ids)
|
||||
|
||||
|
||||
@router.get("/api/v1/chats/{chat_id}/messages")
|
||||
def list_messages(
|
||||
chat_id: UUID,
|
||||
limit: str = Query("50"),
|
||||
cursor: str | None = Query(None, min_length=1),
|
||||
actor: dict = Depends(current_user),
|
||||
):
|
||||
return store.list_messages(actor["id"], str(chat_id), parse_limit(limit), cursor)
|
||||
@@ -0,0 +1,54 @@
|
||||
"""Authorize metadata and stream originals from the private bucket."""
|
||||
|
||||
import logging
|
||||
from uuid import UUID
|
||||
|
||||
from fastapi import HTTPException
|
||||
from fastapi.responses import StreamingResponse
|
||||
|
||||
from .attachments import BUCKET, attachment_view, storage
|
||||
from .store import connection
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def metadata(actor_id: str, attachment_id: UUID) -> dict:
|
||||
with connection() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT a.* FROM attachments a "
|
||||
"JOIN memberships m ON m.chat_id=a.chat_id "
|
||||
"WHERE a.id=%s AND m.user_id=%s",
|
||||
(attachment_id, UUID(actor_id)),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
raise HTTPException(404)
|
||||
return row
|
||||
|
||||
|
||||
def download(actor_id: str, attachment_id: UUID, variant: str) -> StreamingResponse:
|
||||
row = metadata(actor_id, attachment_id)
|
||||
if row["status"] != "ready":
|
||||
raise HTTPException(409)
|
||||
if variant != "original":
|
||||
raise HTTPException(404)
|
||||
try:
|
||||
obj = storage().get_object(BUCKET, row["object_key"])
|
||||
except Exception as exc:
|
||||
logger.warning("S3 read failed for attachment %s (%s)", attachment_id, type(exc).__name__)
|
||||
raise HTTPException(503) from None
|
||||
|
||||
def chunks():
|
||||
try:
|
||||
yield from obj.stream(64 * 1024)
|
||||
finally:
|
||||
obj.close()
|
||||
obj.release_conn()
|
||||
|
||||
return StreamingResponse(
|
||||
chunks(), media_type=row["content_type"], headers={"Content-Length": str(row["size_bytes"])}
|
||||
)
|
||||
|
||||
|
||||
def get_view(actor_id: str, attachment_id: UUID) -> dict:
|
||||
return attachment_view(metadata(actor_id, attachment_id))
|
||||
@@ -0,0 +1,151 @@
|
||||
"""Private S3 access shared by attachment routes and maintenance jobs."""
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import logging
|
||||
import os
|
||||
import unicodedata
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
from tempfile import SpooledTemporaryFile
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from fastapi import HTTPException, Request
|
||||
from minio import Minio
|
||||
from minio.error import S3Error
|
||||
from PIL import Image, UnidentifiedImageError
|
||||
from urllib3 import PoolManager, Retry, Timeout
|
||||
|
||||
from .store import connection, store, timestamp
|
||||
|
||||
|
||||
BUCKET = "messenger-attachments"
|
||||
MAX_BYTES = 10 * 1024 * 1024
|
||||
MAX_PIXELS = 40_000_000
|
||||
Image.MAX_IMAGE_PIXELS = MAX_PIXELS
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def secret(name: str) -> str:
|
||||
return Path(os.environ[f"{name}_FILE"]).read_text(encoding="utf-8").strip()
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def storage() -> Minio:
|
||||
return Minio(
|
||||
os.environ.get("S3_ENDPOINT", "s3:9000"),
|
||||
access_key=secret("S3_ACCESS_KEY_ID"),
|
||||
secret_key=secret("S3_SECRET_ACCESS_KEY"),
|
||||
secure=False,
|
||||
region="us-east-1",
|
||||
http_client=PoolManager(timeout=Timeout(connect=2, read=10), retries=Retry(total=0)),
|
||||
)
|
||||
|
||||
|
||||
def storage_ready() -> None:
|
||||
try:
|
||||
storage().stat_object(BUCKET, "health/ready-probe")
|
||||
except S3Error as exc:
|
||||
if exc.code != "NoSuchKey":
|
||||
raise HTTPException(503) from None
|
||||
except Exception:
|
||||
raise HTTPException(503) from None
|
||||
|
||||
|
||||
def attachment_view(row: dict) -> dict:
|
||||
return {
|
||||
"id": str(row["id"]), "chat_id": str(row["chat_id"]), "owner_id": str(row["owner_id"]),
|
||||
"filename": row["filename"], "content_type": row["content_type"],
|
||||
"size_bytes": row["size_bytes"], "sha256": row["sha256"], "status": row["status"],
|
||||
"variants": [], "error_code": row["error_code"],
|
||||
"created_at": timestamp(row["created_at"]), "updated_at": timestamp(row["updated_at"]),
|
||||
}
|
||||
|
||||
|
||||
def validate_filename(filename: str) -> None:
|
||||
if not 1 <= len(filename) <= 255 or filename in {".", ".."}:
|
||||
raise HTTPException(400)
|
||||
if any(char in "/\\" or unicodedata.category(char) == "Cc" for char in filename):
|
||||
raise HTTPException(400)
|
||||
|
||||
|
||||
def validate_image(stream, declared_type: str) -> str:
|
||||
try:
|
||||
with Image.open(stream) as image:
|
||||
actual_type = {"PNG": "image/png", "JPEG": "image/jpeg"}.get(image.format)
|
||||
if actual_type is None or actual_type != declared_type:
|
||||
raise HTTPException(415)
|
||||
if image.width * image.height > MAX_PIXELS:
|
||||
raise HTTPException(413)
|
||||
image.verify()
|
||||
stream.seek(0)
|
||||
with Image.open(stream) as image:
|
||||
image.load()
|
||||
return actual_type
|
||||
except Image.DecompressionBombError:
|
||||
raise HTTPException(413) from None
|
||||
except (UnidentifiedImageError, OSError, ValueError, SyntaxError):
|
||||
raise HTTPException(415) from None
|
||||
|
||||
|
||||
def persist_attachment(chat_id: str, actor_id: str, filename: str, content_type: str, size: int, sha256: str, stream) -> dict:
|
||||
attachment_id = uuid4()
|
||||
object_key = f"originals/{attachment_id}"
|
||||
stream.seek(0)
|
||||
try:
|
||||
storage().put_object(BUCKET, object_key, stream, size, content_type=content_type)
|
||||
except Exception:
|
||||
logger.exception("S3 upload failed for attachment %s", attachment_id)
|
||||
raise HTTPException(503) from None
|
||||
|
||||
commit_uncertain = False
|
||||
try:
|
||||
with connection() as conn:
|
||||
row = conn.execute(
|
||||
"INSERT INTO attachments (id,chat_id,owner_id,filename,object_key,content_type,size_bytes,sha256) "
|
||||
"VALUES (%s,%s,%s,%s,%s,%s,%s,%s) "
|
||||
"RETURNING id,chat_id,owner_id,filename,content_type,size_bytes,sha256,status,error_code,created_at,updated_at",
|
||||
(attachment_id, UUID(chat_id), UUID(actor_id), filename, object_key, content_type, size, sha256),
|
||||
).fetchone()
|
||||
commit_uncertain = True
|
||||
except Exception:
|
||||
if commit_uncertain:
|
||||
# The COMMIT may have succeeded despite a lost acknowledgment. Reconciliation decides later.
|
||||
logger.error("Attachment commit outcome unknown; retain object %s for reconciliation", object_key)
|
||||
else:
|
||||
try:
|
||||
storage().remove_object(BUCKET, object_key)
|
||||
except Exception:
|
||||
logger.exception("Orphan cleanup failed for object %s", object_key)
|
||||
raise
|
||||
return attachment_view(row)
|
||||
|
||||
|
||||
async def upload(request: Request, chat_id: str, actor_id: str, filename: str) -> dict:
|
||||
await asyncio.to_thread(store.chat, actor_id, chat_id)
|
||||
validate_filename(filename)
|
||||
declared_type = request.headers.get("content-type", "").split(";", 1)[0].strip().lower()
|
||||
if declared_type not in {"image/png", "image/jpeg"}:
|
||||
raise HTTPException(415)
|
||||
length = request.headers.get("content-length")
|
||||
if length is not None:
|
||||
try:
|
||||
if int(length) > MAX_BYTES:
|
||||
raise HTTPException(413)
|
||||
except ValueError:
|
||||
raise HTTPException(400) from None
|
||||
|
||||
with SpooledTemporaryFile(max_size=1024 * 1024) as stream:
|
||||
digest = hashlib.sha256()
|
||||
size = 0
|
||||
async for chunk in request.stream():
|
||||
size += len(chunk)
|
||||
if size > MAX_BYTES:
|
||||
raise HTTPException(413)
|
||||
stream.write(chunk)
|
||||
digest.update(chunk)
|
||||
stream.seek(0)
|
||||
validate_image(stream, declared_type)
|
||||
return await asyncio.to_thread(
|
||||
persist_attachment, chat_id, actor_id, filename, declared_type, size, digest.hexdigest(), stream
|
||||
)
|
||||
+43
@@ -0,0 +1,43 @@
|
||||
"""Parse HTTP credentials without exposing them in errors or logs."""
|
||||
|
||||
import base64
|
||||
import binascii
|
||||
import hashlib
|
||||
import re
|
||||
|
||||
from fastapi import HTTPException
|
||||
|
||||
|
||||
BASIC_CHALLENGE = 'Basic realm="messenger", charset="UTF-8"'
|
||||
BEARER_CHALLENGE = 'Bearer realm="messenger"'
|
||||
USERNAME = re.compile(r"[a-z0-9_]{3,32}\Z")
|
||||
|
||||
|
||||
def unauthorized(challenge: str) -> HTTPException:
|
||||
return HTTPException(401, headers={"WWW-Authenticate": challenge})
|
||||
|
||||
|
||||
def basic_credentials(header: str | None) -> tuple[str, str]:
|
||||
if not header or not header.startswith("Basic "):
|
||||
raise unauthorized(BASIC_CHALLENGE)
|
||||
try:
|
||||
raw = base64.b64decode(header[6:], validate=True).decode("utf-8")
|
||||
username, password = raw.split(":", 1)
|
||||
except (binascii.Error, UnicodeError, ValueError):
|
||||
raise unauthorized(BASIC_CHALLENGE) from None
|
||||
if not USERNAME.fullmatch(username) or not 12 <= len(password) <= 128:
|
||||
raise unauthorized(BASIC_CHALLENGE)
|
||||
return username, password
|
||||
|
||||
|
||||
def bearer_token(header: str | None) -> str:
|
||||
if not header or not header.startswith("Bearer "):
|
||||
raise unauthorized(BEARER_CHALLENGE)
|
||||
token = header[7:]
|
||||
if not 32 <= len(token) <= 256 or not token.isascii() or not re.fullmatch(r"[A-Za-z0-9_-]+", token):
|
||||
raise unauthorized(BEARER_CHALLENGE)
|
||||
return token
|
||||
|
||||
|
||||
def token_hash(token: str) -> bytes:
|
||||
return hashlib.sha256(token.encode("ascii")).digest()
|
||||
@@ -0,0 +1,38 @@
|
||||
"""Bounded PostgreSQL pool shared by API storage and health checks."""
|
||||
|
||||
import os
|
||||
import atexit
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
|
||||
import psycopg
|
||||
from psycopg.rows import dict_row
|
||||
from psycopg_pool import ConnectionPool
|
||||
from psycopg_pool import PoolTimeout
|
||||
|
||||
|
||||
def database_url() -> str:
|
||||
path = os.environ.get("DATABASE_URL_FILE")
|
||||
if path:
|
||||
return Path(path).read_text(encoding="utf-8").strip()
|
||||
return os.environ.get("DATABASE_URL", "postgresql:///messenger")
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def pool() -> ConnectionPool:
|
||||
result = ConnectionPool(
|
||||
database_url(), min_size=0, max_size=8, timeout=5,
|
||||
kwargs={"connect_timeout": 5, "options": "-c statement_timeout=5000", "row_factory": dict_row},
|
||||
open=True,
|
||||
)
|
||||
atexit.register(result.close)
|
||||
return result
|
||||
|
||||
|
||||
def ready() -> bool:
|
||||
try:
|
||||
with pool().connection() as conn:
|
||||
conn.execute("SELECT 1 FROM users LIMIT 0")
|
||||
return True
|
||||
except (psycopg.Error, PoolTimeout, OSError):
|
||||
return False
|
||||
+67
@@ -0,0 +1,67 @@
|
||||
"""FastAPI entry point and contract-wide response handling."""
|
||||
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import FastAPI, HTTPException, Request
|
||||
from fastapi.exceptions import RequestValidationError
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
from .api import router
|
||||
|
||||
app = FastAPI()
|
||||
app.include_router(router)
|
||||
INSTANCE_ID = uuid4().hex
|
||||
|
||||
ERRORS = {
|
||||
400: ("invalid_request", "Неверный запрос"),
|
||||
401: ("unauthorized", "Требуется действительный пользователь"),
|
||||
404: ("not_found", "Объект не найден"),
|
||||
409: ("conflict", "Конфликт данных"),
|
||||
413: ("payload_too_large", "Слишком большой запрос"),
|
||||
415: ("unsupported_media_type", "Неподдерживаемый тип данных"),
|
||||
429: ("rate_limited", "Слишком много запросов"),
|
||||
503: ("unavailable", "Сервис недоступен"),
|
||||
}
|
||||
|
||||
|
||||
def error_response(request: Request, status: int, extra_headers: dict | None = None) -> JSONResponse:
|
||||
code, message = ERRORS.get(status, ("internal_error", "Внутренняя ошибка"))
|
||||
request_id = request.state.request_id
|
||||
return JSONResponse(
|
||||
status_code=status,
|
||||
content={"error": {"code": code, "message": message, "request_id": request_id}},
|
||||
headers={"X-Request-Id": request_id, "X-Instance-Id": INSTANCE_ID, **(extra_headers or {})},
|
||||
)
|
||||
|
||||
|
||||
@app.middleware("http")
|
||||
async def request_contract(request: Request, call_next):
|
||||
request.state.request_id = str(uuid4())
|
||||
is_upload = request.url.path.startswith("/api/v1/chats/") and request.url.path.endswith("/attachments")
|
||||
if request.method == "POST" and request.url.path.startswith("/api/v1/") and request.url.path != "/api/v1/auth/sessions" and not is_upload:
|
||||
media_type, _, charset = request.headers.get("content-type", "").lower().partition(";")
|
||||
if media_type.strip() != "application/json" or (charset and charset.strip() != "charset=utf-8"):
|
||||
return error_response(request, 415)
|
||||
try:
|
||||
(await request.body()).decode("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
return error_response(request, 400)
|
||||
response = await call_next(request)
|
||||
response.headers["X-Request-Id"] = request.state.request_id
|
||||
response.headers["X-Instance-Id"] = INSTANCE_ID
|
||||
return response
|
||||
|
||||
|
||||
@app.exception_handler(RequestValidationError)
|
||||
async def validation_error(request: Request, exc: RequestValidationError):
|
||||
return error_response(request, 400)
|
||||
|
||||
|
||||
@app.exception_handler(HTTPException)
|
||||
async def http_error(request: Request, exc: HTTPException):
|
||||
return error_response(request, exc.status_code, exc.headers)
|
||||
|
||||
|
||||
@app.exception_handler(Exception)
|
||||
async def unexpected_error(request: Request, exc: Exception):
|
||||
return error_response(request, 500)
|
||||
@@ -0,0 +1,68 @@
|
||||
"""Request bodies from the lab 1 contract."""
|
||||
|
||||
from uuid import UUID
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
|
||||
|
||||
|
||||
class StrictBody(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", strict=True)
|
||||
|
||||
|
||||
class CreateUser(StrictBody):
|
||||
username: str = Field(pattern=r"^[a-z0-9_]{3,32}$")
|
||||
display_name: str = Field(min_length=1, max_length=100)
|
||||
password: str = Field(min_length=12, max_length=128)
|
||||
|
||||
@field_validator("display_name")
|
||||
@classmethod
|
||||
def nonblank_name(cls, value: str) -> str:
|
||||
if not value.strip():
|
||||
raise ValueError("blank display name")
|
||||
return value
|
||||
|
||||
@field_validator("password")
|
||||
@classmethod
|
||||
def nonblank_password(cls, value: str) -> str:
|
||||
if not value.strip():
|
||||
raise ValueError("blank password")
|
||||
return value
|
||||
|
||||
|
||||
class CreateChat(StrictBody):
|
||||
title: str = Field(min_length=1, max_length=100)
|
||||
member_ids: list[str] = Field(min_length=1, max_length=100)
|
||||
|
||||
@field_validator("title")
|
||||
@classmethod
|
||||
def nonblank_title(cls, value: str) -> str:
|
||||
if not value.strip():
|
||||
raise ValueError("blank title")
|
||||
return value
|
||||
|
||||
@field_validator("member_ids")
|
||||
@classmethod
|
||||
def valid_members(cls, value: list[str]) -> list[str]:
|
||||
members = [str(UUID(member_id)) for member_id in value]
|
||||
if len(set(members)) != len(members):
|
||||
raise ValueError("duplicate member")
|
||||
return members
|
||||
|
||||
|
||||
class CreateMessage(StrictBody):
|
||||
text: str = Field(max_length=4000)
|
||||
attachment_ids: list[str] = Field(default_factory=list, max_length=10)
|
||||
|
||||
@field_validator("attachment_ids")
|
||||
@classmethod
|
||||
def valid_attachments(cls, value: list[str]) -> list[str]:
|
||||
ids = [str(UUID(item)) for item in value]
|
||||
if len(set(ids)) != len(ids):
|
||||
raise ValueError("duplicate attachment")
|
||||
return ids
|
||||
|
||||
@model_validator(mode="after")
|
||||
def text_or_attachment(self):
|
||||
if not self.text.strip() and not self.attachment_ids:
|
||||
raise ValueError("message needs text or attachment")
|
||||
return self
|
||||
@@ -0,0 +1,14 @@
|
||||
"""Run the ASGI worker as the container's main process."""
|
||||
|
||||
import os
|
||||
|
||||
import uvicorn
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
uvicorn.run(
|
||||
"src.main:app",
|
||||
host=os.environ.get("APP_HOST", "0.0.0.0"),
|
||||
port=int(os.environ.get("APP_PORT", "8080")),
|
||||
workers=1,
|
||||
)
|
||||
+297
@@ -0,0 +1,297 @@
|
||||
"""PostgreSQL-backed users, chats and message history."""
|
||||
|
||||
import base64
|
||||
import binascii
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import secrets
|
||||
from contextlib import contextmanager
|
||||
from datetime import datetime, timezone
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
import psycopg
|
||||
from argon2 import PasswordHasher
|
||||
from argon2.exceptions import InvalidHashError, VerificationError
|
||||
from fastapi import HTTPException
|
||||
from psycopg.errors import UniqueViolation
|
||||
from psycopg_pool import PoolTimeout
|
||||
|
||||
from . import db
|
||||
from .auth import BASIC_CHALLENGE, BEARER_CHALLENGE, token_hash, unauthorized
|
||||
|
||||
password_hasher = PasswordHasher()
|
||||
|
||||
|
||||
def timestamp(value: datetime) -> str:
|
||||
return value.astimezone(timezone.utc).isoformat(timespec="microseconds").replace("+00:00", "Z")
|
||||
|
||||
|
||||
def encode_cursor(resource: str, item: dict) -> str:
|
||||
value = [resource, item["created_at"], item["id"]]
|
||||
return base64.urlsafe_b64encode(json.dumps(value, separators=(",", ":")).encode()).decode().rstrip("=")
|
||||
|
||||
|
||||
def decode_cursor(cursor: str | None, resource: str) -> tuple[datetime, UUID] | None:
|
||||
if cursor is None:
|
||||
return None
|
||||
try:
|
||||
raw = base64.b64decode(cursor + "=" * (-len(cursor) % 4), altchars=b"-_", validate=True)
|
||||
value = json.loads(raw)
|
||||
if not isinstance(value, list) or len(value) != 3 or value[0] != resource:
|
||||
raise ValueError
|
||||
date_text, item_id = value[1:]
|
||||
if not isinstance(date_text, str) or not date_text.endswith("Z") or not isinstance(item_id, str):
|
||||
raise ValueError
|
||||
date = datetime.fromisoformat(date_text[:-1] + "+00:00")
|
||||
if date.utcoffset().total_seconds() != 0:
|
||||
raise ValueError
|
||||
return date, UUID(item_id)
|
||||
except (binascii.Error, UnicodeError, ValueError, TypeError, AttributeError):
|
||||
raise HTTPException(400) from None
|
||||
|
||||
|
||||
@contextmanager
|
||||
def connection():
|
||||
try:
|
||||
with db.pool().connection() as conn:
|
||||
yield conn
|
||||
except (psycopg.Error, PoolTimeout, OSError):
|
||||
raise HTTPException(503) from None
|
||||
|
||||
|
||||
def user_view(row: dict) -> dict:
|
||||
return {
|
||||
"id": str(row["id"]), "username": row["username"],
|
||||
"display_name": row["display_name"], "created_at": timestamp(row["created_at"]),
|
||||
}
|
||||
|
||||
|
||||
def chat_view(row: dict) -> dict:
|
||||
return {
|
||||
"id": str(row["id"]), "title": row["title"],
|
||||
"member_ids": [str(member_id) for member_id in row["member_ids"]],
|
||||
"created_at": timestamp(row["created_at"]),
|
||||
}
|
||||
|
||||
|
||||
def message_view(row: dict) -> dict:
|
||||
return {
|
||||
"id": str(row["id"]), "chat_id": str(row["chat_id"]),
|
||||
"sender_id": str(row["sender_id"]), "text": row["text"].decode("utf-8"),
|
||||
"created_at": timestamp(row["created_at"]),
|
||||
"attachment_ids": [str(attachment_id) for attachment_id in row.get("attachment_ids", [])],
|
||||
}
|
||||
|
||||
|
||||
CHAT_COLUMNS = """c.id, c.title, c.created_at,
|
||||
ARRAY(SELECT m.user_id FROM memberships m WHERE m.chat_id=c.id ORDER BY m.position) AS member_ids"""
|
||||
|
||||
MESSAGE_COLUMNS = """m.id, m.chat_id, m.sender_id, m.text, m.created_at,
|
||||
ARRAY(SELECT a.attachment_id FROM message_attachments a
|
||||
WHERE a.message_id=m.id ORDER BY a.position) AS attachment_ids"""
|
||||
|
||||
|
||||
class Store:
|
||||
def ready(self) -> dict:
|
||||
if not db.ready():
|
||||
raise HTTPException(503)
|
||||
return {"status": "ok"}
|
||||
|
||||
def register(self, username: str, display_name: str, password: str) -> dict:
|
||||
password_hash = password_hasher.hash(password)
|
||||
with connection() as conn:
|
||||
try:
|
||||
row = conn.execute(
|
||||
"INSERT INTO users (id, username, display_name, password_hash) VALUES (%s,%s,%s,%s) "
|
||||
"RETURNING id,username,display_name,created_at",
|
||||
(uuid4(), username, display_name, password_hash),
|
||||
).fetchone()
|
||||
except UniqueViolation:
|
||||
raise HTTPException(409) from None
|
||||
return user_view(row)
|
||||
|
||||
def user(self, user_id: str | None) -> dict:
|
||||
if not user_id:
|
||||
raise HTTPException(401)
|
||||
try:
|
||||
user_uuid = UUID(user_id)
|
||||
except (ValueError, TypeError):
|
||||
raise HTTPException(401) from None
|
||||
with connection() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT id,username,display_name,created_at FROM users WHERE id=%s", (user_uuid,)
|
||||
).fetchone()
|
||||
if row is None:
|
||||
raise HTTPException(401)
|
||||
return user_view(row)
|
||||
|
||||
def create_session(self, username: str, password: str) -> dict:
|
||||
with connection() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT id,username,display_name,created_at,password_hash FROM users WHERE username=%s",
|
||||
(username,),
|
||||
).fetchone()
|
||||
try:
|
||||
valid = row is not None and password_hasher.verify(row["password_hash"], password)
|
||||
except (VerificationError, InvalidHashError):
|
||||
valid = False
|
||||
if not valid:
|
||||
raise unauthorized(BASIC_CHALLENGE)
|
||||
token = secrets.token_urlsafe(32)
|
||||
ttl = int(os.environ.get("SESSION_TTL_SECONDS", "3600"))
|
||||
expiry = conn.execute(
|
||||
"INSERT INTO sessions (token_hash,user_id,expires_at) "
|
||||
"VALUES (%s,%s,clock_timestamp()+make_interval(secs => %s)) RETURNING expires_at",
|
||||
(token_hash(token), row["id"], ttl),
|
||||
).fetchone()["expires_at"]
|
||||
return {"token": token, "token_type": "Bearer", "expires_at": timestamp(expiry), "user": user_view(row)}
|
||||
|
||||
def authenticated(self, token: str) -> dict:
|
||||
with connection() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT u.id,u.username,u.display_name,u.created_at FROM sessions s "
|
||||
"JOIN users u ON u.id=s.user_id "
|
||||
"WHERE s.token_hash=%s AND s.expires_at>clock_timestamp()",
|
||||
(token_hash(token),),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
raise unauthorized(BEARER_CHALLENGE)
|
||||
return user_view(row)
|
||||
|
||||
def revoke_session(self, token: str) -> None:
|
||||
with connection() as conn:
|
||||
row = conn.execute(
|
||||
"DELETE FROM sessions WHERE token_hash=%s AND expires_at>clock_timestamp() RETURNING token_hash",
|
||||
(token_hash(token),),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
raise unauthorized(BEARER_CHALLENGE)
|
||||
|
||||
def create_chat(self, actor_id: str, title: str, member_ids: list[str]) -> dict:
|
||||
if actor_id not in member_ids:
|
||||
raise HTTPException(400)
|
||||
members = [UUID(member_id) for member_id in member_ids]
|
||||
with connection() as conn:
|
||||
count = conn.execute("SELECT count(*) AS n FROM users WHERE id=ANY(%s)", (members,)).fetchone()["n"]
|
||||
if count != len(members):
|
||||
raise HTTPException(400)
|
||||
row = conn.execute("INSERT INTO chats (id,title) VALUES (%s,%s) RETURNING id,title,created_at", (uuid4(), title)).fetchone()
|
||||
for position, member_id in enumerate(members):
|
||||
conn.execute(
|
||||
"INSERT INTO memberships (chat_id,user_id,position) VALUES (%s,%s,%s)",
|
||||
(row["id"], member_id, position),
|
||||
)
|
||||
return chat_view({**row, "member_ids": members})
|
||||
|
||||
def chat(self, actor_id: str, chat_id: str) -> dict:
|
||||
with connection() as conn:
|
||||
row = conn.execute(
|
||||
f"SELECT {CHAT_COLUMNS} FROM chats c "
|
||||
"JOIN memberships access ON access.chat_id=c.id "
|
||||
"WHERE access.user_id=%s AND c.id=%s",
|
||||
(UUID(actor_id), UUID(chat_id)),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
raise HTTPException(404)
|
||||
return chat_view(row)
|
||||
|
||||
def list_chats(self, actor_id: str, limit: int, cursor: str | None) -> dict:
|
||||
after = decode_cursor(cursor, f"chats:{actor_id}")
|
||||
query = (f"SELECT {CHAT_COLUMNS} FROM chats c "
|
||||
"JOIN memberships access ON access.chat_id=c.id WHERE access.user_id=%s")
|
||||
params = [UUID(actor_id)]
|
||||
if after:
|
||||
query += " AND (c.created_at,c.id)>(%s,%s)"
|
||||
params.extend(after)
|
||||
query += " ORDER BY c.created_at,c.id LIMIT %s"
|
||||
params.append(limit + 1)
|
||||
with connection() as conn:
|
||||
rows = conn.execute(query, params).fetchall()
|
||||
items = [chat_view(row) for row in rows[:limit]]
|
||||
return {"items": items, "next_cursor": encode_cursor(f"chats:{actor_id}", items[-1]) if len(rows) > limit else None}
|
||||
|
||||
def create_message(self, actor_id: str, chat_id: str, text: str, key: str, attachment_ids: list[str] | None = None) -> dict:
|
||||
ids = [UUID(item) for item in attachment_ids or []]
|
||||
if len(ids) > 10 or len(set(ids)) != len(ids) or (not text.strip() and not ids):
|
||||
raise HTTPException(400)
|
||||
chat_uuid, actor_uuid = UUID(chat_id), UUID(actor_id)
|
||||
request_hash = hashlib.sha256(json.dumps(
|
||||
{"text": text, "attachment_ids": [str(item) for item in ids]}, ensure_ascii=False, sort_keys=True,
|
||||
).encode()).digest()
|
||||
with connection() as conn:
|
||||
member = conn.execute(
|
||||
"SELECT 1 FROM memberships WHERE chat_id=%s AND user_id=%s", (chat_uuid, actor_uuid)
|
||||
).fetchone()
|
||||
if member is None:
|
||||
raise HTTPException(404)
|
||||
conn.execute(
|
||||
"DELETE FROM message_replays WHERE user_id=%s AND chat_id=%s AND request_key=%s "
|
||||
"AND created_at<clock_timestamp()-interval '24 hours'",
|
||||
(actor_uuid, chat_uuid, key),
|
||||
)
|
||||
reserved = conn.execute(
|
||||
"INSERT INTO message_replays (user_id,chat_id,request_key,request_hash) VALUES (%s,%s,%s,%s) "
|
||||
"ON CONFLICT DO NOTHING RETURNING request_key",
|
||||
(actor_uuid, chat_uuid, key, request_hash),
|
||||
).fetchone()
|
||||
if reserved is None:
|
||||
replay = conn.execute(
|
||||
f"SELECT r.request_hash,{MESSAGE_COLUMNS} "
|
||||
"FROM message_replays r JOIN messages m ON m.id=r.message_id "
|
||||
"WHERE r.user_id=%s AND r.chat_id=%s AND r.request_key=%s",
|
||||
(actor_uuid, chat_uuid, key),
|
||||
).fetchone()
|
||||
if replay is None:
|
||||
raise HTTPException(503)
|
||||
if replay["request_hash"] != request_hash:
|
||||
raise HTTPException(409)
|
||||
return message_view(replay)
|
||||
if ids:
|
||||
attachments = conn.execute(
|
||||
"SELECT id,status FROM attachments WHERE chat_id=%s AND id=ANY(%s)",
|
||||
(chat_uuid, ids),
|
||||
).fetchall()
|
||||
if len(attachments) != len(ids):
|
||||
raise HTTPException(404)
|
||||
if any(item["status"] == "failed" for item in attachments):
|
||||
raise HTTPException(409)
|
||||
row = conn.execute(
|
||||
"INSERT INTO messages (id,chat_id,sender_id,text) VALUES (%s,%s,%s,%s) "
|
||||
"RETURNING id,chat_id,sender_id,text,created_at",
|
||||
(uuid4(), chat_uuid, actor_uuid, text.encode("utf-8")),
|
||||
).fetchone()
|
||||
for position, attachment_id in enumerate(ids):
|
||||
conn.execute(
|
||||
"INSERT INTO message_attachments (message_id,chat_id,attachment_id,position) "
|
||||
"VALUES (%s,%s,%s,%s)",
|
||||
(row["id"], chat_uuid, attachment_id, position),
|
||||
)
|
||||
conn.execute(
|
||||
"UPDATE message_replays SET message_id=%s WHERE user_id=%s AND chat_id=%s AND request_key=%s",
|
||||
(row["id"], actor_uuid, chat_uuid, key),
|
||||
)
|
||||
return message_view({**row, "attachment_ids": ids})
|
||||
|
||||
def list_messages(self, actor_id: str, chat_id: str, limit: int, cursor: str | None) -> dict:
|
||||
after = decode_cursor(cursor, f"messages:{chat_id}")
|
||||
chat_uuid, actor_uuid = UUID(chat_id), UUID(actor_id)
|
||||
query = f"SELECT {MESSAGE_COLUMNS} FROM messages m WHERE m.chat_id=%s"
|
||||
params = [chat_uuid]
|
||||
if after:
|
||||
query += " AND (m.created_at,m.id)>(%s,%s)"
|
||||
params.extend(after)
|
||||
query += " ORDER BY m.created_at,m.id LIMIT %s"
|
||||
params.append(limit + 1)
|
||||
with connection() as conn:
|
||||
member = conn.execute(
|
||||
"SELECT 1 FROM memberships WHERE chat_id=%s AND user_id=%s", (chat_uuid, actor_uuid)
|
||||
).fetchone()
|
||||
if member is None:
|
||||
raise HTTPException(404)
|
||||
rows = conn.execute(query, params).fetchall()
|
||||
items = [message_view(row) for row in rows[:limit]]
|
||||
return {"items": items, "next_cursor": encode_cursor(f"messages:{chat_id}", items[-1]) if len(rows) > limit else None}
|
||||
|
||||
|
||||
store = Store()
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 633 B |
@@ -0,0 +1,20 @@
|
||||
"""Invalid pagination limits must remain client errors."""
|
||||
|
||||
import unittest
|
||||
|
||||
from fastapi import HTTPException
|
||||
|
||||
from src.api import parse_limit
|
||||
|
||||
|
||||
class LimitTests(unittest.TestCase):
|
||||
def test_valid_and_extremely_long_limits(self):
|
||||
self.assertEqual(parse_limit("100"), 100)
|
||||
for value in ("0", "101", "-1", "x", "9" * 4301):
|
||||
with self.assertRaises(HTTPException) as failure:
|
||||
parse_limit(value)
|
||||
self.assertEqual(failure.exception.status_code, 400)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,75 @@
|
||||
"""Black-box attachment ACL, message links and idempotency on a running lab4 stack."""
|
||||
|
||||
import sys
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from uuid import uuid4
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tests"))
|
||||
from client import Client
|
||||
|
||||
|
||||
class AttachmentAccessTests(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
cls.client = Client()
|
||||
cls.alice, cls.bob, cls.eve, cls.chat = cls.client.bootstrap()
|
||||
cls.other_chat = cls.client.call(
|
||||
"POST", "/api/v1/chats", 201, actor=cls.alice,
|
||||
body={"title": "Other", "member_ids": [cls.alice["id"], cls.bob["id"]]}, schema="Chat",
|
||||
)["json"]
|
||||
cls.attachment, cls.content = cls.client.attachment(cls.alice, cls.chat)
|
||||
cls.other_attachment, _ = cls.client.attachment(cls.alice, cls.other_chat)
|
||||
|
||||
def test_message_links_and_replay(self):
|
||||
key = uuid4().hex
|
||||
path = f"/api/v1/chats/{self.chat['id']}/messages"
|
||||
body = {"text": "", "attachment_ids": [self.attachment["id"]]}
|
||||
first = self.client.call("POST", path, 201, body=body, actor=self.bob,
|
||||
headers={"Idempotency-Key": key}, schema="Message")["json"]
|
||||
same = self.client.call("POST", path, 201, body={"attachment_ids": body["attachment_ids"], "text": ""},
|
||||
actor=self.bob, headers={"Idempotency-Key": key}, schema="Message")["json"]
|
||||
self.assertEqual(first, same)
|
||||
self.assertEqual(first["attachment_ids"], [self.attachment["id"]])
|
||||
items = self.client.call("GET", path, actor=self.alice, schema="MessagePage")["json"]["items"]
|
||||
self.assertEqual([item["id"] for item in items].count(first["id"]), 1)
|
||||
self.client.call("POST", path, 409, body={"text": "changed", "attachment_ids": body["attachment_ids"]},
|
||||
actor=self.bob, headers={"Idempotency-Key": key})
|
||||
|
||||
def test_access_and_invalid_links(self):
|
||||
attachment_path = f"/api/v1/attachments/{self.attachment['id']}"
|
||||
metadata = self.client.call("GET", attachment_path, actor=self.bob, schema="Attachment")["json"]
|
||||
self.assertEqual(metadata["id"], self.attachment["id"])
|
||||
self.assertNotIn("object_key", metadata)
|
||||
self.assertNotIn("s3", str(metadata).lower())
|
||||
self.assertEqual(self.client.call("GET", attachment_path + "/content", actor=self.bob)["bytes"], self.content)
|
||||
self.client.call("GET", attachment_path + "/content?variant=thumbnail", 404, actor=self.bob)
|
||||
for suffix in ("", "/content"):
|
||||
self.client.call("GET", attachment_path + suffix, 401)
|
||||
for suffix in ("", "/content"):
|
||||
self.client.call("GET", attachment_path + suffix, 404, actor=self.eve)
|
||||
path = f"/api/v1/chats/{self.chat['id']}/messages"
|
||||
for attachment_id in (str(uuid4()), self.other_attachment["id"]):
|
||||
self.client.call("POST", path, 404, actor=self.alice,
|
||||
body={"text": "", "attachment_ids": [attachment_id]},
|
||||
headers={"Idempotency-Key": uuid4().hex})
|
||||
self.client.call("POST", path, 400, actor=self.alice, body={"text": ""},
|
||||
headers={"Idempotency-Key": uuid4().hex})
|
||||
self.client.call("POST", path, 400, actor=self.alice,
|
||||
body={"text": "", "attachment_ids": [self.attachment["id"]] * 2},
|
||||
headers={"Idempotency-Key": uuid4().hex})
|
||||
|
||||
def test_other_replica_reads_same_bytes(self):
|
||||
path = f"/api/v1/attachments/{self.attachment['id']}/content"
|
||||
seen = set()
|
||||
for _ in range(20):
|
||||
result = self.client.call("GET", path, actor=self.bob)
|
||||
self.assertEqual(result["bytes"], self.content)
|
||||
seen.add(result["headers"]["X-Instance-Id"])
|
||||
if len(seen) >= 2:
|
||||
break
|
||||
self.assertGreaterEqual(len(seen), 2)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,58 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Stop the replica that accepted an upload, then read its bytes elsewhere."""
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
from client import Client, Failure # noqa: E402
|
||||
|
||||
|
||||
def main():
|
||||
client = Client(timeout=2)
|
||||
alice, bob, _, chat = client.bootstrap()
|
||||
original = (ROOT / "tests/fixtures/sample.png").read_bytes()
|
||||
observed = {client.call("GET", "/api/v1/users/me", actor=bob)["headers"]["X-Instance-Id"] for _ in range(30)}
|
||||
if len(observed) != 2:
|
||||
raise Failure("two API replicas were not observed before stop")
|
||||
for target in sorted(observed):
|
||||
for _ in range(10):
|
||||
result = client.call(
|
||||
"POST", f"/api/v1/chats/{chat['id']}/attachments?filename=sample.png", 201,
|
||||
actor=alice, raw=original, headers={"Content-Type": "image/png"}, schema="Attachment",
|
||||
)
|
||||
if result["headers"]["X-Instance-Id"] == target:
|
||||
break
|
||||
else:
|
||||
raise Failure("upload did not reach the selected replica")
|
||||
attachment_id = result["json"]["id"]
|
||||
env = {**os.environ, "TARGET_INSTANCE_ID": target}
|
||||
try:
|
||||
subprocess.run([str(ROOT / "scripts/stop-one.sh")], cwd=ROOT, env=env, check=True)
|
||||
deadline = time.monotonic() + 10
|
||||
while True:
|
||||
try:
|
||||
read = client.call("GET", f"/api/v1/attachments/{attachment_id}/content", actor=bob)
|
||||
if read["headers"]["X-Instance-Id"] == target:
|
||||
raise Failure("stopped replica still answered")
|
||||
if read["bytes"] != original:
|
||||
raise Failure("attachment bytes changed after replica stop")
|
||||
break
|
||||
except Failure:
|
||||
if time.monotonic() >= deadline:
|
||||
raise
|
||||
time.sleep(0.2)
|
||||
finally:
|
||||
subprocess.run(["docker", "compose", "--env-file", ".env", "up", "-d", "--wait", "--scale", "api=2"],
|
||||
cwd=ROOT, check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
print(json.dumps({"stopped_upload_replicas": len(observed), "download_on_other_replica": True,
|
||||
"bytes_equal": True}, sort_keys=True))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,63 @@
|
||||
"""Black-box upload and validation checks; BASE_URL defaults to localhost:8080."""
|
||||
|
||||
import hashlib
|
||||
import struct
|
||||
import sys
|
||||
import unittest
|
||||
import zlib
|
||||
from pathlib import Path
|
||||
from urllib.parse import quote
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
from client import Client # noqa: E402
|
||||
|
||||
|
||||
PNG = (ROOT / "tests/fixtures/sample.png").read_bytes()
|
||||
JPEG = (ROOT / "student-tests/fixtures/sample.jpg").read_bytes()
|
||||
|
||||
|
||||
class UploadTests(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
cls.client = Client()
|
||||
cls.alice, _, cls.eve, cls.chat = cls.client.bootstrap()
|
||||
cls.path = f"/api/v1/chats/{cls.chat['id']}/attachments"
|
||||
|
||||
def upload(self, content=PNG, media_type="image/png", filename="sample.png", expected=201, actor=None):
|
||||
return self.client.call(
|
||||
"POST", self.path + "?filename=" + quote(filename, safe=""), expected,
|
||||
actor=actor or self.alice, raw=content, headers={"Content-Type": media_type},
|
||||
schema="Attachment" if expected == 201 else None,
|
||||
)
|
||||
|
||||
def test_png_and_jpeg(self):
|
||||
for content, media_type, filename in ((PNG, "image/png", "sample.png"), (JPEG, "image/jpeg", "sample.jpg")):
|
||||
attachment = self.upload(content, media_type, filename)["json"]
|
||||
self.assertEqual(attachment["status"], "ready")
|
||||
self.assertEqual(attachment["owner_id"], self.alice["id"])
|
||||
self.assertEqual(attachment["chat_id"], self.chat["id"])
|
||||
self.assertEqual(attachment["content_type"], media_type)
|
||||
self.assertEqual(attachment["size_bytes"], len(content))
|
||||
self.assertEqual(attachment["sha256"], hashlib.sha256(content).hexdigest())
|
||||
|
||||
def test_rejects_oversized_and_oversized_pixels(self):
|
||||
self.upload(b"x" * (10 * 1024 * 1024 + 1), expected=413)
|
||||
changed = bytearray(PNG)
|
||||
changed[16:24] = struct.pack(">II", 6325, 6325)
|
||||
changed[29:33] = struct.pack(">I", zlib.crc32(changed[12:29]))
|
||||
self.upload(bytes(changed), expected=413)
|
||||
|
||||
def test_rejects_invalid_image_and_mime_spoofing(self):
|
||||
self.upload(PNG[:-10], expected=415)
|
||||
self.upload(PNG, "image/jpeg", expected=415)
|
||||
self.upload(b"not an image", expected=415)
|
||||
|
||||
def test_rejects_unsafe_names_and_outsider(self):
|
||||
for name in ("../bad.png", "folder\\bad.png", "bad\x00name.png", "bad\nname.png", ""):
|
||||
self.upload(filename=name, expected=400)
|
||||
self.upload(expected=404, actor=self.eve)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,175 @@
|
||||
"""HTTP and database checks for durable Basic/Bearer sessions."""
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from uuid import uuid4
|
||||
|
||||
import psycopg
|
||||
from src.db import database_url
|
||||
|
||||
|
||||
URL = database_url() if os.environ.get("DATABASE_URL") or os.environ.get("DATABASE_URL_FILE") else None
|
||||
|
||||
|
||||
@unittest.skipUnless(URL, "set DATABASE_URL or DATABASE_URL_FILE to a disposable PostgreSQL database")
|
||||
class AuthTests(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
subprocess.run([sys.executable, "scripts/migrate.py"], check=True)
|
||||
with socket.socket() as sock:
|
||||
sock.bind(("127.0.0.1", 0))
|
||||
cls.port = sock.getsockname()[1]
|
||||
cls.start_server()
|
||||
|
||||
@classmethod
|
||||
def start_server(cls, ttl="3600"):
|
||||
cls.instance_id = None
|
||||
env = {**os.environ, "SESSION_TTL_SECONDS": ttl}
|
||||
cls.server = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "src.main:app", "--host", "127.0.0.1", "--port", str(cls.port)],
|
||||
env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
|
||||
)
|
||||
for _ in range(100):
|
||||
if cls.server.poll() is not None:
|
||||
raise AssertionError("API server exited during startup")
|
||||
try:
|
||||
cls.call("GET", "/health/ready", expected=200)
|
||||
return
|
||||
except (AssertionError, urllib.error.URLError):
|
||||
time.sleep(0.05)
|
||||
raise AssertionError("API server did not become ready")
|
||||
|
||||
@classmethod
|
||||
def stop_server(cls):
|
||||
cls.server.terminate()
|
||||
try:
|
||||
cls.server.wait(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
cls.server.kill()
|
||||
cls.server.wait(timeout=5)
|
||||
|
||||
@classmethod
|
||||
def tearDownClass(cls):
|
||||
cls.stop_server()
|
||||
|
||||
@classmethod
|
||||
def call(cls, method, path, *, body=None, authorization=None, user_id=None, expected=200):
|
||||
headers = {"Accept": "application/json"}
|
||||
data = None
|
||||
if body is not None:
|
||||
data = json.dumps(body, ensure_ascii=False).encode()
|
||||
headers["Content-Type"] = "application/json"
|
||||
if authorization is not None:
|
||||
headers["Authorization"] = authorization
|
||||
if user_id is not None:
|
||||
headers["X-User-Id"] = user_id
|
||||
request = urllib.request.Request(
|
||||
f"http://127.0.0.1:{cls.port}{path}", data=data, headers=headers, method=method,
|
||||
)
|
||||
try:
|
||||
response = urllib.request.urlopen(request, timeout=5)
|
||||
except urllib.error.HTTPError as error:
|
||||
response = error
|
||||
with response:
|
||||
status = response.status
|
||||
response_headers = response.headers
|
||||
data = response.read()
|
||||
assert status == expected, (method, path, expected, status)
|
||||
request_id = response_headers.get("X-Request-Id")
|
||||
assert request_id
|
||||
instance_id = response_headers.get("X-Instance-Id")
|
||||
assert instance_id
|
||||
if cls.instance_id is None:
|
||||
cls.instance_id = instance_id
|
||||
else:
|
||||
assert instance_id == cls.instance_id
|
||||
payload = json.loads(data) if data else None
|
||||
if status >= 400:
|
||||
assert payload["error"]["request_id"] == request_id
|
||||
assert "password" not in data.decode().lower()
|
||||
assert "token" not in data.decode().lower()
|
||||
return payload, response_headers
|
||||
|
||||
def user(self, password="Пароль:очень-длинный-123"):
|
||||
username = "u_" + uuid4().hex[:16]
|
||||
body = {"username": username, "display_name": "Алиса", "password": password}
|
||||
user, _ = self.call("POST", "/api/v1/users", body=body, expected=201)
|
||||
self.assertNotIn("password", user)
|
||||
return user, password
|
||||
|
||||
def login(self, username, password, expected=201):
|
||||
encoded = base64.b64encode(f"{username}:{password}".encode()).decode()
|
||||
return self.call("POST", "/api/v1/auth/sessions", authorization="Basic " + encoded, expected=expected)
|
||||
|
||||
def test_basic_bearer_logout_and_storage(self):
|
||||
alice, password = self.user()
|
||||
bob, _ = self.user()
|
||||
session, headers = self.login(alice["username"], password)
|
||||
self.assertEqual(session["token_type"], "Bearer")
|
||||
self.assertEqual(session["user"], alice)
|
||||
self.assertGreaterEqual(len(base64.urlsafe_b64decode(session["token"] + "=" * (-len(session["token"]) % 4))), 32)
|
||||
self.assertEqual(headers.get("Cache-Control"), "no-store")
|
||||
bearer = "Bearer " + session["token"]
|
||||
me, _ = self.call("GET", "/api/v1/users/me", authorization=bearer, user_id=bob["id"])
|
||||
self.assertEqual(me, alice)
|
||||
with psycopg.connect(URL) as conn:
|
||||
row = conn.execute("SELECT token_hash,expires_at FROM sessions WHERE user_id=%s", (alice["id"],)).fetchone()
|
||||
password_hash = conn.execute("SELECT password_hash FROM users WHERE id=%s", (alice["id"],)).fetchone()[0]
|
||||
self.assertEqual(row[0], hashlib.sha256(session["token"].encode()).digest())
|
||||
self.assertNotEqual(password_hash, password)
|
||||
self.assertTrue(password_hash.startswith("$argon2id$"))
|
||||
self.call("DELETE", "/api/v1/auth/sessions/current", authorization=bearer, expected=204)
|
||||
for method, path in (("GET", "/api/v1/users/me"), ("DELETE", "/api/v1/auth/sessions/current")):
|
||||
_, headers = self.call(method, path, authorization=bearer, expected=401)
|
||||
self.assertEqual(headers.get("WWW-Authenticate"), 'Bearer realm="messenger"')
|
||||
|
||||
def test_challenges_and_uniform_credentials(self):
|
||||
user, password = self.user()
|
||||
failures = []
|
||||
for username, candidate in ((user["username"], "wrong-password-123"), ("missing_user_123", password)):
|
||||
payload, headers = self.login(username, candidate, expected=401)
|
||||
failures.append((payload["error"]["code"], payload["error"]["message"]))
|
||||
self.assertEqual(headers.get("WWW-Authenticate"), 'Basic realm="messenger", charset="UTF-8"')
|
||||
self.assertEqual(failures[0], failures[1])
|
||||
encoded = base64.b64encode(f"{user['username']}:{password}".encode()).decode()
|
||||
for authorization in (None, "Basic " + encoded, "Bearer garbage"):
|
||||
_, headers = self.call("GET", "/api/v1/users/me", authorization=authorization, user_id=user["id"], expected=401)
|
||||
self.assertEqual(headers.get("WWW-Authenticate"), 'Bearer realm="messenger"')
|
||||
_, headers = self.call("POST", "/api/v1/auth/sessions", authorization="Bearer garbage", expected=401)
|
||||
self.assertEqual(headers.get("WWW-Authenticate"), 'Basic realm="messenger", charset="UTF-8"')
|
||||
self.call("POST", "/api/v1/auth/sessions", authorization="Basic " + encoded, body={"unexpected": True}, expected=400)
|
||||
|
||||
def test_z_short_ttl_survives_server_restart_without_extension(self):
|
||||
self.stop_server()
|
||||
self.start_server(ttl="2")
|
||||
try:
|
||||
user, password = self.user()
|
||||
session, _ = self.login(user["username"], password)
|
||||
bearer = "Bearer " + session["token"]
|
||||
self.call("GET", "/api/v1/users/me", authorization=bearer)
|
||||
with psycopg.connect(URL) as conn:
|
||||
before = conn.execute("SELECT expires_at FROM sessions WHERE user_id=%s", (user["id"],)).fetchone()[0]
|
||||
self.stop_server()
|
||||
self.start_server(ttl="3600")
|
||||
with psycopg.connect(URL) as conn:
|
||||
after = conn.execute("SELECT expires_at FROM sessions WHERE user_id=%s", (user["id"],)).fetchone()[0]
|
||||
self.assertEqual(before, after)
|
||||
time.sleep(2.1)
|
||||
_, headers = self.call("GET", "/api/v1/users/me", authorization=bearer, expected=401)
|
||||
self.assertEqual(headers.get("WWW-Authenticate"), 'Bearer realm="messenger"')
|
||||
finally:
|
||||
self.stop_server()
|
||||
self.start_server()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,86 @@
|
||||
"""Integration checks against a disposable PostgreSQL database."""
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import unittest
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from uuid import uuid4
|
||||
|
||||
import psycopg
|
||||
|
||||
from src.db import database_url
|
||||
from src.store import Store
|
||||
|
||||
|
||||
URL = database_url() if os.environ.get("DATABASE_URL") or os.environ.get("DATABASE_URL_FILE") else None
|
||||
|
||||
|
||||
@unittest.skipUnless(URL, "set DATABASE_URL or DATABASE_URL_FILE to a disposable PostgreSQL database")
|
||||
class DatabaseTests(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
for _ in range(2):
|
||||
subprocess.run([sys.executable, "scripts/migrate.py"], check=True)
|
||||
cls.store = Store()
|
||||
|
||||
def user(self):
|
||||
name = "u_" + uuid4().hex[:16]
|
||||
return self.store.register(name, "Студент", "Test-password-123")
|
||||
|
||||
def test_migration_and_unique_username_race(self):
|
||||
with psycopg.connect(URL) as conn:
|
||||
versions = conn.execute("SELECT version FROM schema_migrations").fetchall()
|
||||
self.assertEqual(versions, [(1,), (2,), (3,), (4,), (5,), (9,)])
|
||||
name = "u_" + uuid4().hex[:16]
|
||||
|
||||
def register(_):
|
||||
try:
|
||||
return self.store.register(name, "Студент", "Test-password-123")
|
||||
except Exception as exc:
|
||||
return exc
|
||||
|
||||
with ThreadPoolExecutor(max_workers=2) as pool:
|
||||
results = list(pool.map(register, range(2)))
|
||||
self.assertEqual(sum(isinstance(result, dict) for result in results), 1)
|
||||
self.assertEqual([result.status_code for result in results if not isinstance(result, dict)], [409])
|
||||
with psycopg.connect(URL) as conn:
|
||||
self.assertEqual(conn.execute("SELECT count(*) FROM users WHERE username=%s", (name,)).fetchone()[0], 1)
|
||||
password_hash = conn.execute("SELECT password_hash FROM users WHERE username=%s", (name,)).fetchone()[0]
|
||||
self.assertTrue(password_hash.startswith("$argon2id$"))
|
||||
self.assertNotIn("password", str(next(result for result in results if isinstance(result, dict))))
|
||||
subprocess.run([sys.executable, "scripts/migrate.py"], check=True)
|
||||
with psycopg.connect(URL) as conn:
|
||||
self.assertEqual(conn.execute("SELECT count(*) FROM users WHERE username=%s", (name,)).fetchone()[0], 1)
|
||||
|
||||
def test_chat_rollback_and_keyset_history(self):
|
||||
alice, bob, eve = self.user(), self.user(), self.user()
|
||||
with psycopg.connect(URL) as conn:
|
||||
before = conn.execute("SELECT count(*) FROM chats").fetchone()[0]
|
||||
with self.assertRaises(Exception) as failure:
|
||||
self.store.create_chat(alice["id"], "invalid", [alice["id"], str(uuid4())])
|
||||
self.assertEqual(failure.exception.status_code, 400)
|
||||
with psycopg.connect(URL) as conn:
|
||||
self.assertEqual(conn.execute("SELECT count(*) FROM chats").fetchone()[0], before)
|
||||
|
||||
chat = self.store.create_chat(alice["id"], "Valid", [alice["id"], bob["id"]])
|
||||
chat2 = self.store.create_chat(alice["id"], "Second", [alice["id"]])
|
||||
chat_page = self.store.list_chats(alice["id"], 1, None)
|
||||
self.assertEqual(chat_page["items"], [chat])
|
||||
self.assertEqual(self.store.list_chats(alice["id"], 1, chat_page["next_cursor"]), {"items": [chat2], "next_cursor": None})
|
||||
first = self.store.create_message(alice["id"], chat["id"], "первое", uuid4().hex)
|
||||
second = self.store.create_message(bob["id"], chat["id"], "второе", uuid4().hex)
|
||||
page = self.store.list_messages(alice["id"], chat["id"], 1, None)
|
||||
self.assertEqual(page["items"], [first])
|
||||
self.assertEqual(self.store.list_messages(bob["id"], chat["id"], 1, page["next_cursor"]), {"items": [second], "next_cursor": None})
|
||||
self.assertEqual(Store().chat(bob["id"], chat["id"]), chat)
|
||||
with self.assertRaises(Exception) as wrong_cursor:
|
||||
self.store.list_messages(alice["id"], chat2["id"], 1, page["next_cursor"])
|
||||
self.assertEqual(wrong_cursor.exception.status_code, 400)
|
||||
with self.assertRaises(Exception) as hidden:
|
||||
self.store.chat(eve["id"], chat["id"])
|
||||
self.assertEqual(hidden.exception.status_code, 404)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,53 @@
|
||||
"""Concurrent PostgreSQL-backed message retries create one row."""
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import unittest
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from uuid import uuid4
|
||||
|
||||
import psycopg
|
||||
|
||||
from src.store import Store
|
||||
|
||||
|
||||
URL = os.environ.get("DATABASE_URL")
|
||||
|
||||
|
||||
@unittest.skipUnless(URL, "set DATABASE_URL to a disposable PostgreSQL database")
|
||||
class IdempotencyTests(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
subprocess.run([sys.executable, "scripts/migrate.py"], check=True)
|
||||
cls.store = Store()
|
||||
|
||||
def test_parallel_replay_conflict_and_scope(self):
|
||||
alice = self.store.register("u_" + uuid4().hex[:16], "Алиса", "Test-password-123")
|
||||
bob = self.store.register("u_" + uuid4().hex[:16], "Боб", "Test-password-123")
|
||||
chat = self.store.create_chat(alice["id"], "Test", [alice["id"], bob["id"]])
|
||||
other_chat = self.store.create_chat(alice["id"], "Other", [alice["id"]])
|
||||
key = uuid4().hex
|
||||
|
||||
def send(_):
|
||||
return Store().create_message(alice["id"], chat["id"], "same body", key, [] if _ else None)
|
||||
|
||||
with ThreadPoolExecutor(max_workers=2) as pool:
|
||||
first, replay = list(pool.map(send, range(2)))
|
||||
self.assertEqual(first, replay)
|
||||
with psycopg.connect(URL) as conn:
|
||||
count = conn.execute(
|
||||
"SELECT count(*) FROM messages WHERE chat_id=%s AND sender_id=%s AND text=%s",
|
||||
(chat["id"], alice["id"], b"same body"),
|
||||
).fetchone()[0]
|
||||
self.assertEqual(count, 1)
|
||||
|
||||
with self.assertRaises(Exception) as conflict:
|
||||
self.store.create_message(alice["id"], chat["id"], "different body", key)
|
||||
self.assertEqual(conflict.exception.status_code, 409)
|
||||
self.assertNotEqual(self.store.create_message(bob["id"], chat["id"], "same body", key)["id"], first["id"])
|
||||
self.assertNotEqual(self.store.create_message(alice["id"], other_chat["id"], "same body", key)["id"], first["id"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,54 @@
|
||||
"""Every API response identifies the live replica process."""
|
||||
|
||||
import json
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
|
||||
class InstanceHeaderTests(unittest.TestCase):
|
||||
def test_process_alias_on_success_and_error(self):
|
||||
with socket.socket() as sock:
|
||||
sock.bind(("127.0.0.1", 0))
|
||||
port = sock.getsockname()[1]
|
||||
server = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "src.main:app", "--host", "127.0.0.1", "--port", str(port)],
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
|
||||
)
|
||||
try:
|
||||
for _ in range(100):
|
||||
try:
|
||||
with urllib.request.urlopen(f"http://127.0.0.1:{port}/health/live", timeout=1) as response:
|
||||
live_id = response.headers.get("X-Instance-Id")
|
||||
self.assertEqual(json.load(response), {"status": "ok"})
|
||||
break
|
||||
except urllib.error.URLError:
|
||||
if server.poll() is not None:
|
||||
self.fail("API exited before readiness")
|
||||
time.sleep(0.05)
|
||||
else:
|
||||
self.fail("API did not start")
|
||||
self.assertTrue(live_id)
|
||||
for path in ("/health/live", "/no-such-route"):
|
||||
try:
|
||||
response = urllib.request.urlopen(f"http://127.0.0.1:{port}{path}", timeout=1)
|
||||
except urllib.error.HTTPError as error:
|
||||
response = error
|
||||
with response:
|
||||
self.assertEqual(response.headers.get("X-Instance-Id"), live_id)
|
||||
self.assertTrue(response.headers.get("X-Request-Id"))
|
||||
finally:
|
||||
server.terminate()
|
||||
try:
|
||||
server.wait(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
server.kill()
|
||||
server.wait(timeout=5)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,27 @@
|
||||
"""Text-only messages retain lab3 behavior under the lab4 wire schema."""
|
||||
|
||||
import unittest
|
||||
from datetime import datetime, timezone
|
||||
from uuid import uuid4
|
||||
|
||||
from src.models import CreateMessage
|
||||
from src.store import message_view
|
||||
|
||||
|
||||
class Lab3CompatibilityTests(unittest.TestCase):
|
||||
def test_missing_and_explicit_empty_attachments_are_equivalent(self):
|
||||
missing = CreateMessage.model_validate({"text": "hello"})
|
||||
explicit = CreateMessage.model_validate({"text": "hello", "attachment_ids": []})
|
||||
self.assertEqual(missing, explicit)
|
||||
self.assertEqual(missing.attachment_ids, [])
|
||||
|
||||
def test_text_message_response_has_empty_attachment_ids(self):
|
||||
row = {
|
||||
"id": uuid4(), "chat_id": uuid4(), "sender_id": uuid4(),
|
||||
"text": "hello", "created_at": datetime.now(timezone.utc),
|
||||
}
|
||||
self.assertEqual(message_view(row)["attachment_ids"], [])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,156 @@
|
||||
"""Run with DATABASE_URL pointing to a disposable PostgreSQL database.
|
||||
|
||||
Checks real HTTP writes, persisted history, retries, and migration of old rows.
|
||||
"""
|
||||
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from uuid import uuid4
|
||||
|
||||
import psycopg
|
||||
from psycopg.conninfo import make_conninfo
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
LAB = json.loads((ROOT / "lab.json").read_text())["number"]
|
||||
URL = os.environ.get("DATABASE_URL")
|
||||
|
||||
|
||||
@unittest.skipUnless(URL, "set DATABASE_URL to a disposable database")
|
||||
class MessageUnicodeTests(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
subprocess.run([sys.executable, "scripts/migrate.py"], cwd=ROOT, check=True)
|
||||
with socket.socket() as sock:
|
||||
sock.bind(("127.0.0.1", 0))
|
||||
cls.port = sock.getsockname()[1]
|
||||
cls.start_server()
|
||||
cls.addClassCleanup(cls.stop_server)
|
||||
|
||||
@classmethod
|
||||
def start_server(cls):
|
||||
cls.server = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "src.main:app", "--host", "127.0.0.1", "--port", str(cls.port)],
|
||||
cwd=ROOT, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
|
||||
env={**os.environ, "VALKEY_URL": "redis://127.0.0.1:1/0", "OTEL_SDK_DISABLED": "true"},
|
||||
)
|
||||
for _ in range(100):
|
||||
if cls.server.poll() is not None:
|
||||
raise AssertionError("API exited during startup")
|
||||
try:
|
||||
cls.call("GET", "/health/live")
|
||||
return
|
||||
except (AssertionError, urllib.error.URLError):
|
||||
time.sleep(0.05)
|
||||
cls.stop_server()
|
||||
raise AssertionError("API did not become ready")
|
||||
|
||||
@classmethod
|
||||
def stop_server(cls):
|
||||
cls.server.terminate()
|
||||
try:
|
||||
cls.server.wait(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
cls.server.kill()
|
||||
cls.server.wait(timeout=5)
|
||||
|
||||
@classmethod
|
||||
def call(cls, method, path, body=None, headers=None, expected=200):
|
||||
request = urllib.request.Request(
|
||||
f"http://127.0.0.1:{cls.port}{path}", method=method,
|
||||
data=json.dumps(body).encode() if body is not None else None,
|
||||
headers={"Content-Type": "application/json", **(headers or {})},
|
||||
)
|
||||
try:
|
||||
response = urllib.request.urlopen(request, timeout=10)
|
||||
except urllib.error.HTTPError as exc:
|
||||
response = exc
|
||||
with response:
|
||||
payload = response.read()
|
||||
assert response.status == expected, (method, path, response.status, expected, payload)
|
||||
return json.loads(payload) if payload else None
|
||||
|
||||
def test_http_round_trip_after_restart_and_replay(self):
|
||||
username, password = "u_" + uuid4().hex[:16], "Test-password-123"
|
||||
user = self.call("POST", "/api/v1/users", {
|
||||
"username": username, "display_name": "Unicode", "password": password,
|
||||
}, expected=201)
|
||||
session = self.call("POST", "/api/v1/auth/sessions", headers={
|
||||
"Authorization": "Basic " + base64.b64encode(f"{username}:{password}".encode()).decode(),
|
||||
}, expected=201)
|
||||
auth = {"Authorization": "Bearer " + session["token"]}
|
||||
chat = self.call("POST", "/api/v1/chats", {
|
||||
"title": "Unicode", "member_ids": [user["id"]],
|
||||
}, auth, 201)
|
||||
path = f"/api/v1/chats/{chat['id']}/messages"
|
||||
texts = ["\0", "\0Привет\0🙂\0", r"literal \u0000 \0 \\", "🙂" * 3999 + "\0", "\0" * 4000]
|
||||
messages, keys = [], []
|
||||
for text in texts:
|
||||
key = uuid4().hex
|
||||
message = self.call("POST", path, {"text": text}, {**auth, "Idempotency-Key": key}, 201)
|
||||
self.assertEqual(message["text"], text)
|
||||
messages.append(message)
|
||||
keys.append(key)
|
||||
for text in ("", "\0" * 4001, "🙂" * 4001):
|
||||
self.call("POST", path, {"text": text}, {**auth, "Idempotency-Key": uuid4().hex}, 400)
|
||||
self.stop_server()
|
||||
self.start_server()
|
||||
found, cursor = [], None
|
||||
while True:
|
||||
page = self.call("GET", path + "?limit=1" + (
|
||||
"&cursor=" + urllib.parse.quote(cursor, safe="") if cursor else ""
|
||||
), headers=auth)
|
||||
found.extend(page["items"])
|
||||
cursor = page["next_cursor"]
|
||||
if cursor is None:
|
||||
break
|
||||
self.assertEqual(found, messages)
|
||||
if LAB >= 3:
|
||||
for text, message, key in zip(texts, messages, keys):
|
||||
replay = self.call("POST", path, {"text": text}, {**auth, "Idempotency-Key": key}, 201)
|
||||
self.assertEqual(replay, message)
|
||||
self.call("POST", path, {"text": "different"}, {**auth, "Idempotency-Key": keys[0]}, 409)
|
||||
self.assertEqual(len(self.call("GET", path, headers=auth)["items"]), len(messages))
|
||||
|
||||
def test_upgrade_preserves_old_text_and_is_repeatable(self):
|
||||
schema = "unicode_" + uuid4().hex
|
||||
with psycopg.connect(URL, autocommit=True) as conn:
|
||||
conn.execute(f'CREATE SCHEMA "{schema}"')
|
||||
isolated = make_conninfo(URL, options=f"-c search_path={schema}")
|
||||
try:
|
||||
old_text = "Старое 🙂 \\ literal \\u0000"
|
||||
with psycopg.connect(isolated) as conn:
|
||||
conn.execute("CREATE TABLE schema_migrations (version integer PRIMARY KEY)")
|
||||
for migration in sorted((ROOT / "migrations").glob("*.sql")):
|
||||
version = int(migration.name.split("_", 1)[0])
|
||||
if version >= 9:
|
||||
continue
|
||||
conn.execute(migration.read_text())
|
||||
conn.execute("INSERT INTO schema_migrations VALUES (%s)", (version,))
|
||||
user, chat, message = uuid4(), uuid4(), uuid4()
|
||||
conn.execute("INSERT INTO users (id,username,display_name,password_hash) VALUES (%s,'old','Old','hash')", (user,))
|
||||
conn.execute("INSERT INTO chats (id,title) VALUES (%s,'Old')", (chat,))
|
||||
conn.execute("INSERT INTO memberships (chat_id,user_id,position) VALUES (%s,%s,0)", (chat, user))
|
||||
conn.execute("INSERT INTO messages (id,chat_id,sender_id,text) VALUES (%s,%s,%s,%s)", (message, chat, user, old_text))
|
||||
for _ in range(2):
|
||||
subprocess.run([sys.executable, "scripts/migrate.py"], cwd=ROOT, check=True)
|
||||
with psycopg.connect(isolated) as conn:
|
||||
conn.execute((ROOT / "migrations/009_message_text_utf8.sql").read_text())
|
||||
saved, = conn.execute("SELECT text FROM messages WHERE id=%s", (message,)).fetchone()
|
||||
self.assertEqual(saved.decode("utf-8"), old_text)
|
||||
finally:
|
||||
with psycopg.connect(URL, autocommit=True) as conn:
|
||||
conn.execute(f'DROP SCHEMA "{schema}" CASCADE')
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,142 @@
|
||||
"""Run inside the maintenance image: python - < student-tests/test_reconcile.py."""
|
||||
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import io
|
||||
import time
|
||||
import unittest
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import Mock
|
||||
from urllib.error import HTTPError
|
||||
from urllib.request import Request, urlopen
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
import psycopg
|
||||
from psycopg.rows import dict_row
|
||||
|
||||
from src.db import database_url
|
||||
from src.store import store
|
||||
|
||||
spec = importlib.util.spec_from_file_location("reconciler", "scripts/reconcile-attachments.py")
|
||||
reconciler = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(reconciler)
|
||||
|
||||
|
||||
class ReconcileTests(unittest.TestCase):
|
||||
def test_delete_rechecks_database_and_object_version(self):
|
||||
key = f"originals/{uuid4()}"
|
||||
item = SimpleNamespace(object_name=key, etag="old", size=10,
|
||||
last_modified=datetime.now(timezone.utc) - timedelta(hours=2))
|
||||
conn, client = Mock(), Mock()
|
||||
conn.execute.return_value.fetchone.return_value = {"id": uuid4()}
|
||||
self.assertFalse(reconciler.delete_orphan(client, conn, item, 3600))
|
||||
client.remove_object.assert_not_called()
|
||||
|
||||
conn.execute.return_value.fetchone.return_value = None
|
||||
client.stat_object.return_value = SimpleNamespace(etag="replaced", size=10, last_modified=item.last_modified)
|
||||
self.assertFalse(reconciler.delete_orphan(client, conn, item, 3600))
|
||||
client.remove_object.assert_not_called()
|
||||
|
||||
def test_orphans_missing_repair_and_live_reference(self):
|
||||
client = reconciler.storage()
|
||||
fixture = Path("/fixtures/sample.png")
|
||||
data = fixture.read_bytes()
|
||||
digest = hashlib.sha256(data).hexdigest()
|
||||
username = "reconcile_" + uuid4().hex[:16]
|
||||
owner = store.register(username, "Reconciler test", "Test-password-123")
|
||||
session = store.create_session(username, "Test-password-123")
|
||||
chat = store.create_chat(owner["id"], "Reconciler test", [owner["id"]])
|
||||
active_id, missing_id = uuid4(), uuid4()
|
||||
active_key, missing_key = f"originals/{active_id}", f"originals/{missing_id}"
|
||||
old_key, fresh_key = f"originals/{uuid4()}", f"originals/{uuid4()}"
|
||||
|
||||
def put(key):
|
||||
client.put_object(reconciler.BUCKET, key, io.BytesIO(data), len(data), content_type="image/png")
|
||||
|
||||
put(active_key)
|
||||
put(missing_key)
|
||||
put(old_key)
|
||||
with self.assertRaises(psycopg.errors.ForeignKeyViolation):
|
||||
with psycopg.connect(database_url()) as failed:
|
||||
failed.execute(
|
||||
"INSERT INTO attachments (id,chat_id,owner_id,filename,object_key,content_type,size_bytes,sha256) "
|
||||
"VALUES (%s,%s,%s,%s,%s,%s,%s,%s)",
|
||||
(UUID(old_key.split("/")[1]), uuid4(), UUID(owner["id"]), "orphan.png", old_key,
|
||||
"image/png", len(data), digest),
|
||||
)
|
||||
with psycopg.connect(database_url(), row_factory=dict_row) as conn:
|
||||
for item_id, key in ((active_id, active_key), (missing_id, missing_key)):
|
||||
conn.execute(
|
||||
"INSERT INTO attachments (id,chat_id,owner_id,filename,object_key,content_type,size_bytes,sha256) "
|
||||
"VALUES (%s,%s,%s,%s,%s,%s,%s,%s)",
|
||||
(item_id, UUID(chat["id"]), UUID(owner["id"]), "sample.png", key, "image/png", len(data), digest),
|
||||
)
|
||||
message_id = uuid4()
|
||||
conn.execute(
|
||||
"INSERT INTO messages (id,chat_id,sender_id,text) VALUES (%s,%s,%s,%s)",
|
||||
(message_id, UUID(chat["id"]), UUID(owner["id"]), b""),
|
||||
)
|
||||
conn.execute(
|
||||
"INSERT INTO message_attachments (message_id,chat_id,attachment_id,position) VALUES (%s,%s,%s,0)",
|
||||
(message_id, UUID(chat["id"]), active_id),
|
||||
)
|
||||
|
||||
client.remove_object(reconciler.BUCKET, missing_key)
|
||||
|
||||
time.sleep(6)
|
||||
put(fresh_key)
|
||||
with psycopg.connect(database_url(), row_factory=dict_row) as conn:
|
||||
preview = reconciler.reconcile(client, conn, False, 5, None, None)
|
||||
self.assertIn(old_key, preview["old_orphans"])
|
||||
self.assertIn(fresh_key, preview["fresh_orphans"])
|
||||
self.assertIn(str(missing_id), preview["missing_attachment_ids"])
|
||||
self.assertNotIn(active_key, preview["old_orphans"])
|
||||
self.assertFalse(reconciler.absent(client, old_key))
|
||||
|
||||
wrong = Path("/tmp/reconcile-wrong-source")
|
||||
wrong.write_bytes(b"not the original")
|
||||
with self.assertRaisesRegex(ValueError, "size/SHA-256 mismatch"):
|
||||
reconciler.reconcile(client, conn, True, 5, missing_id, wrong)
|
||||
self.assertFalse(reconciler.absent(client, old_key))
|
||||
|
||||
applied = reconciler.reconcile(client, conn, True, 5, None, None)
|
||||
self.assertIn(old_key, applied["deleted"])
|
||||
self.assertTrue(reconciler.absent(client, old_key))
|
||||
self.assertFalse(reconciler.absent(client, fresh_key))
|
||||
self.assertFalse(reconciler.absent(client, active_key))
|
||||
self.assertTrue(reconciler.absent(client, missing_key))
|
||||
request = Request(
|
||||
f"http://api:8080/api/v1/attachments/{missing_id}/content",
|
||||
headers={"Authorization": "Bearer " + session["token"]},
|
||||
)
|
||||
with self.assertRaises(HTTPError) as unavailable:
|
||||
urlopen(request, timeout=5)
|
||||
self.assertEqual(unavailable.exception.code, 503)
|
||||
|
||||
with self.assertRaisesRegex(ValueError, "size/SHA-256 mismatch"):
|
||||
reconciler.restore_missing(client, conn, missing_id, wrong, True)
|
||||
self.assertTrue(reconciler.absent(client, missing_key))
|
||||
self.assertEqual(reconciler.restore_missing(client, conn, missing_id, fixture, True), "restored")
|
||||
self.assertEqual(reconciler.restore_missing(client, conn, missing_id, fixture, True), "already_present")
|
||||
final = reconciler.reconcile(client, conn, False, 5, None, None)
|
||||
self.assertNotIn(old_key, final["old_orphans"])
|
||||
self.assertNotIn(str(missing_id), final["missing_attachment_ids"])
|
||||
self.assertIsNotNone(conn.execute(
|
||||
"SELECT 1 FROM message_attachments WHERE attachment_id=%s", (active_id,)
|
||||
).fetchone())
|
||||
|
||||
response = client.get_object(reconciler.BUCKET, missing_key)
|
||||
try:
|
||||
self.assertEqual(hashlib.sha256(response.read()).hexdigest(), digest)
|
||||
finally:
|
||||
response.close()
|
||||
response.release_conn()
|
||||
with urlopen(request, timeout=5) as response:
|
||||
self.assertEqual(response.read(), data)
|
||||
print("PASS: old orphan dry-run/apply, fresh and linked objects retained, missing restored from verified fixture")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,200 @@
|
||||
"""Repeatable lab 4 consistency experiment; run inside the maintenance Compose service.
|
||||
|
||||
Writes one JSON result to stdout. It creates synthetic objects and metadata, but never
|
||||
deletes the measured baseline or the fresh/linked objects after the experiment.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
import resource
|
||||
import statistics
|
||||
import time
|
||||
from pathlib import Path
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
import psycopg
|
||||
from PIL import Image
|
||||
from psycopg.rows import dict_row
|
||||
|
||||
from src.db import database_url
|
||||
from src.store import store
|
||||
|
||||
spec = importlib.util.spec_from_file_location("reconciler", "scripts/reconcile-attachments.py")
|
||||
reconciler = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(reconciler)
|
||||
|
||||
|
||||
def put(client, key: str, data: bytes) -> None:
|
||||
client.put_object(reconciler.BUCKET, key, io.BytesIO(data), len(data), content_type="image/png")
|
||||
|
||||
|
||||
def rss_kib() -> int:
|
||||
for line in Path("/proc/self/status").read_text().splitlines():
|
||||
if line.startswith("VmRSS:"):
|
||||
return int(line.split()[1])
|
||||
raise RuntimeError("VmRSS unavailable")
|
||||
|
||||
|
||||
def scan(client, conn, label: str, trial: int) -> dict:
|
||||
before = resource.getrusage(resource.RUSAGE_SELF)
|
||||
rss_before = rss_kib()
|
||||
start = time.perf_counter()
|
||||
result = reconciler.reconcile(client, conn, False, 3600, None, None)
|
||||
wall_ms = (time.perf_counter() - start) * 1000
|
||||
after = resource.getrusage(resource.RUSAGE_SELF)
|
||||
return {
|
||||
"label": label, "trial": trial, "objects": result["inventory"]["objects"],
|
||||
"metadata": result["inventory"]["metadata"],
|
||||
"reconciler_ms": result["duration_ms"], "wall_ms": round(wall_ms, 3),
|
||||
"cpu_user_ms": round((after.ru_utime - before.ru_utime) * 1000, 3),
|
||||
"cpu_system_ms": round((after.ru_stime - before.ru_stime) * 1000, 3),
|
||||
"rss_before_kib": rss_before, "rss_after_kib": rss_kib(), "rss_hwm_kib": after.ru_maxrss,
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
client = reconciler.storage()
|
||||
fixture = Path("/fixtures/sample.png")
|
||||
original = fixture.read_bytes()
|
||||
digest = hashlib.sha256(original).hexdigest()
|
||||
username = "research_" + uuid4().hex[:16]
|
||||
owner = store.register(username, "Reconciliation research", "Test-password-123")
|
||||
chat = store.create_chat(owner["id"], "Reconciliation research", [owner["id"]])
|
||||
owner_id, chat_id = UUID(owner["id"]), UUID(chat["id"])
|
||||
active_id, missing_id, old_id, fresh_id = (uuid4() for _ in range(4))
|
||||
active_key, missing_key, old_key, fresh_key = (f"originals/{item}" for item in (active_id, missing_id, old_id, fresh_id))
|
||||
put(client, active_key, original)
|
||||
put(client, missing_key, original)
|
||||
put(client, old_key, original)
|
||||
try:
|
||||
with psycopg.connect(database_url()) as failed:
|
||||
failed.execute(
|
||||
"INSERT INTO attachments (id,chat_id,owner_id,filename,object_key,content_type,size_bytes,sha256) "
|
||||
"VALUES (%s,%s,%s,%s,%s,%s,%s,%s)",
|
||||
(old_id, uuid4(), owner_id, "orphan.png", old_key, "image/png", len(original), digest),
|
||||
)
|
||||
except psycopg.errors.ForeignKeyViolation:
|
||||
pass
|
||||
else:
|
||||
raise AssertionError("PostgreSQL fault injection did not roll back")
|
||||
|
||||
with psycopg.connect(database_url(), row_factory=dict_row) as conn:
|
||||
for attachment_id, key in ((active_id, active_key), (missing_id, missing_key)):
|
||||
conn.execute(
|
||||
"INSERT INTO attachments (id,chat_id,owner_id,filename,object_key,content_type,size_bytes,sha256) "
|
||||
"VALUES (%s,%s,%s,%s,%s,%s,%s,%s)",
|
||||
(attachment_id, chat_id, owner_id, "sample.png", key, "image/png", len(original), digest),
|
||||
)
|
||||
message_id = uuid4()
|
||||
conn.execute("INSERT INTO messages (id,chat_id,sender_id,text) VALUES (%s,%s,%s,%s)",
|
||||
(message_id, chat_id, owner_id, b""))
|
||||
conn.execute("INSERT INTO message_attachments (message_id,chat_id,attachment_id,position) VALUES (%s,%s,%s,0)",
|
||||
(message_id, chat_id, active_id))
|
||||
|
||||
client.remove_object(reconciler.BUCKET, missing_key)
|
||||
time.sleep(6) # An explicit test-only 5-second grace; routine maintenance uses one hour.
|
||||
put(client, fresh_key, original)
|
||||
|
||||
with psycopg.connect(database_url(), row_factory=dict_row) as conn:
|
||||
preview = reconciler.reconcile(client, conn, False, 5, None, None)
|
||||
assert old_key in preview["old_orphans"] and fresh_key in preview["fresh_orphans"]
|
||||
assert str(missing_id) in preview["manual_incidents"] and active_key not in preview["old_orphans"]
|
||||
assert not reconciler.absent(client, old_key) # Dry-run did not delete.
|
||||
wrong = Path("/tmp/research-wrong-source")
|
||||
wrong.write_bytes(b"wrong fixture")
|
||||
try:
|
||||
reconciler.reconcile(client, conn, True, 5, missing_id, wrong)
|
||||
except ValueError:
|
||||
pass
|
||||
else:
|
||||
raise AssertionError("wrong fixture accepted")
|
||||
assert not reconciler.absent(client, old_key)
|
||||
|
||||
apply_without_source = reconciler.reconcile(client, conn, True, 5, None, None)
|
||||
assert old_key in apply_without_source["deleted"]
|
||||
assert str(missing_id) in apply_without_source["manual_incidents"]
|
||||
assert reconciler.absent(client, old_key) and reconciler.absent(client, missing_key)
|
||||
assert not reconciler.absent(client, fresh_key) and not reconciler.absent(client, active_key)
|
||||
assert conn.execute("SELECT 1 FROM message_attachments WHERE attachment_id=%s", (active_id,)).fetchone()
|
||||
|
||||
post_delete_preview = reconciler.reconcile(client, conn, False, 5, None, None)
|
||||
assert old_key not in post_delete_preview["old_orphans"]
|
||||
assert str(missing_id) in post_delete_preview["manual_incidents"]
|
||||
assert fresh_key in post_delete_preview["fresh_orphans"]
|
||||
repeat_delete = reconciler.reconcile(client, conn, True, 5, None, None)
|
||||
assert old_key not in repeat_delete["deleted"]
|
||||
|
||||
restored = reconciler.reconcile(client, conn, True, 3600, missing_id, fixture)
|
||||
assert restored["restore"] == "restored" and not reconciler.absent(client, missing_key)
|
||||
repeat = reconciler.reconcile(client, conn, True, 3600, missing_id, fixture)
|
||||
assert repeat["restore"] == "already_present" and not repeat["deleted"]
|
||||
assert str(missing_id) not in repeat["manual_incidents"]
|
||||
assert not reconciler.absent(client, fresh_key) and not reconciler.absent(client, active_key)
|
||||
post_restore_preview = reconciler.reconcile(client, conn, False, 5, None, None)
|
||||
assert old_key not in post_restore_preview["old_orphans"]
|
||||
assert str(missing_id) not in post_restore_preview["manual_incidents"]
|
||||
conn.commit()
|
||||
|
||||
tiny = io.BytesIO()
|
||||
Image.new("RGB", (4, 4), (5, 30, 80)).save(tiny, format="PNG")
|
||||
data = tiny.getvalue()
|
||||
tiny_hash = hashlib.sha256(data).hexdigest()
|
||||
|
||||
def add_batch(count: int) -> None:
|
||||
rows = []
|
||||
for _ in range(count):
|
||||
attachment_id = uuid4()
|
||||
key = f"originals/{attachment_id}"
|
||||
put(client, key, data)
|
||||
rows.append((attachment_id, chat_id, owner_id, "inventory.png", key, "image/png", len(data), tiny_hash))
|
||||
with conn.cursor() as cursor:
|
||||
cursor.executemany(
|
||||
"INSERT INTO attachments (id,chat_id,owner_id,filename,object_key,content_type,size_bytes,sha256) "
|
||||
"VALUES (%s,%s,%s,%s,%s,%s,%s,%s)", rows,
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
measurements = []
|
||||
for label, additional in (("plus_50", 50), ("plus_250", 200)):
|
||||
add_batch(additional)
|
||||
measurements.extend(scan(client, conn, label, trial) for trial in range(1, 4))
|
||||
assert measurements[3]["objects"] >= measurements[0]["objects"] + 200
|
||||
assert measurements[3]["metadata"] >= measurements[0]["metadata"] + 200
|
||||
|
||||
response = client.get_object(reconciler.BUCKET, missing_key)
|
||||
try:
|
||||
assert hashlib.sha256(response.read()).hexdigest() == digest
|
||||
finally:
|
||||
response.close()
|
||||
response.release_conn()
|
||||
|
||||
summary = {label: {
|
||||
"median_wall_ms": round(statistics.median(item["wall_ms"] for item in measurements if item["label"] == label), 3),
|
||||
"median_cpu_ms": round(statistics.median(item["cpu_user_ms"] + item["cpu_system_ms"] for item in measurements if item["label"] == label), 3),
|
||||
"objects": next(item["objects"] for item in measurements if item["label"] == label),
|
||||
"metadata": next(item["metadata"] for item in measurements if item["label"] == label),
|
||||
} for label in ("plus_50", "plus_250")}
|
||||
print(json.dumps({
|
||||
"faults": {
|
||||
"preview": {"old_detected": old_key in preview["old_orphans"], "fresh_detected": fresh_key in preview["fresh_orphans"],
|
||||
"missing_detected": str(missing_id) in preview["missing_attachment_ids"],
|
||||
"linked_preserved": active_key not in preview["old_orphans"], "inventory": preview["inventory"]},
|
||||
"wrong_fixture_rejected_without_delete": True,
|
||||
"apply_without_source": {"old_deleted": old_key in apply_without_source["deleted"],
|
||||
"missing_incident_retained": str(missing_id) in apply_without_source["manual_incidents"],
|
||||
"fresh_preserved": True, "linked_preserved": True},
|
||||
"post_delete_dry_run": {"old_absent": old_key not in post_delete_preview["old_orphans"],
|
||||
"missing_incident_retained": str(missing_id) in post_delete_preview["manual_incidents"]},
|
||||
"repeat_delete_no_change": old_key not in repeat_delete["deleted"],
|
||||
"restore": restored["restore"], "repeat": repeat["restore"],
|
||||
"post_restore_dry_run": {"old_absent": old_key not in post_restore_preview["old_orphans"],
|
||||
"missing_absent": str(missing_id) not in post_restore_preview["manual_incidents"]},
|
||||
}, "measurements": measurements, "summary": summary,
|
||||
"limits": "Single local host, 3 sequential dry-run scans per size; CPU/RSS are maintenance-process measurements only."
|
||||
}, sort_keys=True))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,46 @@
|
||||
"""Run inside an API container while S3 is stopped; no secrets are printed."""
|
||||
|
||||
import io
|
||||
import unittest
|
||||
from urllib.error import HTTPError
|
||||
from urllib.request import Request, urlopen
|
||||
from uuid import uuid4
|
||||
|
||||
from PIL import Image
|
||||
|
||||
from src.db import pool
|
||||
from src.store import store
|
||||
|
||||
|
||||
def status(path: str, data: bytes | None = None, headers: dict | None = None) -> int:
|
||||
request = Request("http://127.0.0.1:8080" + path, data=data, headers=headers or {}, method="POST" if data is not None else "GET")
|
||||
try:
|
||||
with urlopen(request, timeout=15) as response:
|
||||
return response.status
|
||||
except HTTPError as error:
|
||||
return error.code
|
||||
|
||||
|
||||
class S3OutageTests(unittest.TestCase):
|
||||
def test_unavailable_without_false_ready_or_metadata(self):
|
||||
username = "s3_outage_" + uuid4().hex[:16]
|
||||
user = store.register(username, "S3 outage", "Test-password-123")
|
||||
session = store.create_session(username, "Test-password-123")
|
||||
chat = store.create_chat(user["id"], "S3 outage", [user["id"]])
|
||||
with pool().connection() as conn:
|
||||
before = conn.execute("SELECT count(*) AS n FROM attachments WHERE chat_id=%s", (chat["id"],)).fetchone()["n"]
|
||||
data = io.BytesIO()
|
||||
Image.new("RGB", (4, 3), (20, 60, 120)).save(data, format="PNG")
|
||||
self.assertEqual(status("/health/ready"), 503)
|
||||
self.assertEqual(
|
||||
status(f"/api/v1/chats/{chat['id']}/attachments?filename=outage.png", data.getvalue(),
|
||||
{"Content-Type": "image/png", "Authorization": "Bearer " + session["token"]}),
|
||||
503,
|
||||
)
|
||||
with pool().connection() as conn:
|
||||
after = conn.execute("SELECT count(*) AS n FROM attachments WHERE chat_id=%s", (chat["id"],)).fetchone()["n"]
|
||||
self.assertEqual(after, before)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,38 @@
|
||||
"""Run inside an API container after S3 bootstrap."""
|
||||
|
||||
import io
|
||||
import unittest
|
||||
from uuid import uuid4
|
||||
from urllib.error import HTTPError
|
||||
from urllib.request import urlopen
|
||||
|
||||
from minio.error import S3Error
|
||||
|
||||
from src.attachments import BUCKET, storage
|
||||
|
||||
|
||||
class StorageBootstrapTests(unittest.TestCase):
|
||||
def test_private_bucket_and_scoped_api_credentials(self):
|
||||
client = storage()
|
||||
key = "test-bootstrap/" + uuid4().hex
|
||||
data = b"private attachment fixture"
|
||||
client.put_object(BUCKET, key, io.BytesIO(data), len(data), content_type="application/octet-stream")
|
||||
try:
|
||||
response = client.get_object(BUCKET, key)
|
||||
try:
|
||||
self.assertEqual(response.read(), data)
|
||||
finally:
|
||||
response.close()
|
||||
response.release_conn()
|
||||
with self.assertRaises(HTTPError) as denied:
|
||||
urlopen(f"http://s3:9000/{BUCKET}/{key}", timeout=3)
|
||||
self.assertEqual(denied.exception.code, 403)
|
||||
with self.assertRaises(S3Error) as forbidden:
|
||||
client.make_bucket("forbidden-" + uuid4().hex)
|
||||
self.assertEqual(forbidden.exception.code, "AccessDenied")
|
||||
finally:
|
||||
client.remove_object(BUCKET, key)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,45 @@
|
||||
"""Run in both API and maintenance containers; tests their distinct S3 identities."""
|
||||
|
||||
import io
|
||||
import os
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from urllib.error import HTTPError
|
||||
from urllib.request import urlopen
|
||||
from uuid import uuid4
|
||||
|
||||
from minio.error import S3Error
|
||||
|
||||
from src.attachments import BUCKET, storage
|
||||
|
||||
|
||||
class StoragePermissionsTests(unittest.TestCase):
|
||||
def test_scoped_identity_and_private_bucket(self):
|
||||
maintenance = os.environ["S3_ACCESS_KEY_ID_FILE"].endswith("/s3_maintenance_key_id")
|
||||
self.assertFalse(Path("/run/secrets/minio_root_user").exists())
|
||||
if maintenance:
|
||||
self.assertFalse(Path("/run/secrets/s3_access_key_id").exists())
|
||||
else:
|
||||
self.assertFalse(Path("/run/secrets/s3_maintenance_key_id").exists())
|
||||
client = storage()
|
||||
key = "permissions/" + uuid4().hex
|
||||
client.put_object(BUCKET, key, io.BytesIO(b"private"), 7)
|
||||
try:
|
||||
if maintenance:
|
||||
list(client.list_objects(BUCKET, prefix="originals/", recursive=True))
|
||||
else:
|
||||
with self.assertRaises(S3Error) as denied:
|
||||
list(client.list_objects(BUCKET, prefix="originals/", recursive=True))
|
||||
self.assertEqual(denied.exception.code, "AccessDenied")
|
||||
with self.assertRaises(S3Error) as denied:
|
||||
client.make_bucket("forbidden-" + uuid4().hex)
|
||||
self.assertEqual(denied.exception.code, "AccessDenied")
|
||||
with self.assertRaises(HTTPError) as denied:
|
||||
urlopen(f"http://s3:9000/{BUCKET}/{key}", timeout=3)
|
||||
self.assertEqual(denied.exception.code, 403)
|
||||
finally:
|
||||
client.remove_object(BUCKET, key)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,68 @@
|
||||
"""Run inside an API image: python - < student-tests/test_upload_failure.py."""
|
||||
|
||||
import asyncio
|
||||
import io
|
||||
import unittest
|
||||
from contextlib import contextmanager
|
||||
from unittest.mock import Mock, patch
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import HTTPException
|
||||
|
||||
from src import attachments
|
||||
|
||||
|
||||
class UploadFailureTests(unittest.TestCase):
|
||||
def test_post_put_database_failure_compensates(self):
|
||||
client = Mock()
|
||||
with patch.object(attachments, "storage", return_value=client), patch.object(
|
||||
attachments, "connection", side_effect=HTTPException(503)
|
||||
):
|
||||
with self.assertRaises(HTTPException) as failure:
|
||||
attachments.persist_attachment(str(uuid4()), str(uuid4()), "test.png", "image/png", 3, "0" * 64, io.BytesIO(b"png"))
|
||||
self.assertEqual(failure.exception.status_code, 503)
|
||||
key = client.put_object.call_args.args[1]
|
||||
client.remove_object.assert_called_once_with(attachments.BUCKET, key)
|
||||
|
||||
def test_s3_failure_does_not_touch_database(self):
|
||||
client = Mock()
|
||||
client.put_object.side_effect = OSError("simulated outage")
|
||||
with patch.object(attachments, "storage", return_value=client), patch.object(attachments, "connection") as db:
|
||||
with self.assertRaises(HTTPException) as failure:
|
||||
attachments.persist_attachment(str(uuid4()), str(uuid4()), "test.png", "image/png", 3, "0" * 64, io.BytesIO(b"png"))
|
||||
self.assertEqual(failure.exception.status_code, 503)
|
||||
db.assert_not_called()
|
||||
|
||||
def test_ambiguous_commit_keeps_object_for_reconciliation(self):
|
||||
@contextmanager
|
||||
def ambiguous_commit():
|
||||
connection = Mock()
|
||||
connection.execute.return_value.fetchone.return_value = {}
|
||||
yield connection
|
||||
raise HTTPException(503)
|
||||
|
||||
client = Mock()
|
||||
with patch.object(attachments, "storage", return_value=client), patch.object(
|
||||
attachments, "connection", side_effect=ambiguous_commit
|
||||
):
|
||||
with self.assertRaises(HTTPException) as failure:
|
||||
attachments.persist_attachment(str(uuid4()), str(uuid4()), "test.png", "image/png", 3, "0" * 64, io.BytesIO(b"png"))
|
||||
self.assertEqual(failure.exception.status_code, 503)
|
||||
client.remove_object.assert_not_called()
|
||||
|
||||
def test_interrupted_request_never_starts_s3_put(self):
|
||||
class Interrupted:
|
||||
headers = {"content-type": "image/png"}
|
||||
|
||||
async def stream(self):
|
||||
yield b"partial"
|
||||
raise OSError("client disconnected")
|
||||
|
||||
with patch.object(attachments.store, "chat"), patch.object(attachments, "storage") as client:
|
||||
with self.assertRaises(OSError):
|
||||
asyncio.run(attachments.upload(Interrupted(), str(uuid4()), str(uuid4()), "test.png"))
|
||||
client.assert_not_called()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user