Author SHA1 Message Date
Exodia a5b0ac9b33 fix 2026-11-13 14:09:36 +03:00
Exodia 125a5e6409 ЛР 4 — БПИ2303 — Щербаков — на 3 2026-11-13 14:02:00 +03:00
39 changed files with 2142 additions and 48 deletions
+6 -13
View File
@@ -1,20 +1,13 @@
# Copy to .env for your LOCAL synthetic test environment; never commit .env.
# These are logical interface suggestions; map actual framework names in REPORT.md.
APP_HOST=0.0.0.0
# python scripts/bootstrap.py creates .env and random local secrets.
# Only LB is published; choose another free loopback port when needed.
APP_PORT=8080
APP_ENV=lab
SESSION_TTL_SECONDS=3600
# Choose credentials locally. File paths contain no secret values.
DATABASE_URL_FILE=/run/secrets/database_url
# Optional on 3, required use on 4 in lab2; Redis OR Valkey:
SESSION_STORE=postgres
CACHE_URL_FILE=/run/secrets/cache_url
API_REPLICAS=2
IDEMPOTENCY_TTL_SECONDS=86400
S3_ENDPOINT=http://s3:9000
S3_ENDPOINT=http://s3:8333
S3_REGION=us-east-1
S3_BUCKET=messenger-attachments
S3_ACCESS_KEY_ID_FILE=/run/secrets/s3_access_key_id
S3_SECRET_ACCESS_KEY_FILE=/run/secrets/s3_secret_access_key
MAX_UPLOAD_BYTES=10485760
MAX_IMAGE_PIXELS=40000000
# Internal API: 0.0.0.0:8080, exactly two services api1/api2.
# Credentials are mounted via Compose secrets, never placed in this file.
# Idempotency lifetime is fixed at 86400 seconds in migration 002.
+5
View File
@@ -0,0 +1,5 @@
*.sh text eol=lf
*.cmd text eol=crlf
*.py text eol=lf
*.sql text eol=lf
*.yaml text eol=lf
+23 -14
View File
@@ -1,18 +1,27 @@
## Лабораторная 4
Автор / группа: TODO
Целевая оценка (3/4/5): TODO
Трек 5, если заявлен: TODO
Автор / группа: Щербаков Марк Игоревич / БПИ2303
Целевая оценка (3/4/5): 3
Трек 5, если заявлен: не заявлен
Что реализовано и что перенесено: TODO
Команды запуска из чистого clone: TODO
Результаты проверок и ссылка на REPORT.md: TODO
Ограничения / известные ошибки: TODO
Источники и использование ИИ: TODO
Из lab3 перенесены PostgreSQL, сессии, две API за HAProxy и идемпотентность.
Добавлены приватный SeaweedFS S3, вложения PNG/JPEG, проверка bytes/pixels/MIME,
связь с сообщениями и авторизованный download без изменения оригинала.
- [ ] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
- [ ] Контракт и публичные тесты не ослаблены.
- [ ] make check и make test выполнены; приложен реальный вывод.
- [ ] Сценарии отказов/безопасности своего уровня воспроизведены.
- [ ] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
- [ ] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
Запуск: `python scripts/bootstrap.py`, затем
`docker compose --env-file .env up -d --build --wait`.
Результаты: smoke и 23 собственных теста, persistence, failover обеих API,
остановка загрузившей API, private S3 GET=403 и отказ S3 прошли.
Инструкция: [REPORT.md](../REPORT.md); протоколы: [evidence](../evidence/README.md).
Ограничения: один host/LB/PostgreSQL/S3, нет reconciler orphan-объектов,
API использует общую административную S3 identity, уровни 4/5 не заявляются.
Источники приведены в REPORT.md. Использован OpenAI Codex.
- [x] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
- [x] Контракт и публичные тесты не ослаблены.
- [x] Эквиваленты make check и make test выполнены Python-командами; приложен реальный вывод.
- [x] Сценарии отказов/безопасности своего уровня воспроизведены.
- [x] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
- [x] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
+24
View File
@@ -0,0 +1,24 @@
name: Application
on: [push, pull_request]
permissions:
contents: read
jobs:
application:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- run: python scripts/bootstrap.py
- run: chmod +x scripts/restart.sh scripts/stop-one.sh
- run: docker compose --env-file .env up -d --build --wait
- run: make check test PYTHON=python
- run: python scripts/verify.py --output evidence/grade3-checks.txt
- if: always()
uses: actions/upload-artifact@v4
with:
name: grade3-evidence
path: evidence/grade3-checks.txt
- if: always()
run: docker compose down
+14
View File
@@ -0,0 +1,14 @@
FROM python:3.12.12-slim-bookworm
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 APP_HOST=0.0.0.0 APP_PORT=8080
WORKDIR /app
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt \
&& groupadd --gid 10001 messenger \
&& useradd --uid 10001 --gid messenger --no-create-home --shell /usr/sbin/nologin messenger
COPY --chown=10001:10001 src/ ./src/
COPY --chown=10001:10001 migrations/ ./migrations/
USER 10001:10001
EXPOSE 8080
STOPSIGNAL SIGTERM
HEALTHCHECK --interval=5s --timeout=5s --start-period=10s --retries=5 CMD ["python", "-m", "src.healthcheck"]
CMD ["python", "-m", "src.server"]
+7
View File
@@ -28,3 +28,10 @@ test-persistence:
test-failover:
@test -n "$(ACTION_SCRIPT)" || (echo "Set ACTION_SCRIPT to your implemented executable stop-one hook"; exit 2)
$(PYTHON) tests/resilience.py --mode failover --action-script "$(ACTION_SCRIPT)" --base-url "$(BASE_URL)" $(TLS_ARGS)
.PHONY: test-student verify
test-student:
BASE_URL="$(BASE_URL)" $(PYTHON) -m unittest discover -s student-tests -v
verify:
$(PYTHON) scripts/verify.py --base-url "$(BASE_URL)" --output evidence/grade3-checks.txt
+261 -21
View File
@@ -1,37 +1,277 @@
# Отчёт — лабораторная 4
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
- Автор: Щербаков Марк Игоревич, БПИ2303.
- Целевая оценка: **«3»**. Уровни 4/5 и исследовательские треки не заявляются.
- Ветка: solution/BPI2303-Scherbakov.
- Название коммита решения: «ЛР 4 — БПИ2303 — Щербаков — на 3».
- История коммитов: git log --oneline. Текст сдачи: [SUBMISSION.md](SUBMISSION.md).
- Помощь ИИ: OpenAI Codex — изучение lab1–3, реализация, проверки и отчёт.
- Основа: собственные предыдущие работы; чужие студенческие решения не использовались.
## Запуск из чистого clone
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
Нужны Docker с Linux-контейнерами, Compose v2 и Python 3.12 для проверок.
Все зависимости поднимаются из текущего каталога. Соседние lab1–3 для запуска не нужны.
## Архитектура и решения
~~~sh
python scripts/bootstrap.py
docker compose --env-file .env config --quiet
docker compose --env-file .env up -d --build --wait
python scripts/check_contract.py
python tests/smoke.py --base-url http://localhost:8080
~~~
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab3, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
Bootstrap создаёт .env и случайные учебные credentials в secrets/, не выводит
и не перезаписывает существующие файлы. Они исключены из Git и Docker build context.
Файлы postgres_password и database_url задают PostgreSQL; s3_access_key_id,
s3_secret_access_key и s3_config.json задают согласованную S3 identity.
В конфигурации нет anonymous identity. Секреты из lab3 не копируются.
На проверенном Windows-хосте в .env установлен **APP_PORT=18084**.
Адрес стенда: **http://127.0.0.1:18084**. Для повторения результатов установить
этот порт перед запуском. В .env.example по умолчанию задан 8080.
При наличии HTTP proxy добавить localhost и 127.0.0.1 в NO_PROXY.
Скрипт полной проверки сам задаёт NO_PROXY.
Миграции выполняет одноразовый сервис migrate после готовности PostgreSQL.
Сервис s3-init ждёт S3, создаёт bucket, если его нет, и устанавливает private ACL.
API запускаются после успешного завершения обоих jobs; Exited (0) у них — норма.
Повторный bootstrap bucket не удаляет объекты. Изменение credentials требует
согласованной перенастройки хранилища и API.
Полная проверка с реальными отказами:
~~~sh
python scripts/verify.py --base-url http://127.0.0.1:18084 --output evidence/grade3-checks.txt
~~~
Скрипт выполняет публичный smoke, собственные тесты, повтор миграций, restart
пяти сервисов, остановку загрузившей файл API (по очереди обеих), анонимный S3 GET,
отказ S3 и публичные failover-сценарии обеих API. Он создаёт синтетические данные
и временно останавливает сервисы только проекта mtusi2026lab4; volumes сохраняются.
Требуется доступ к Docker.
Отдельные проверки в PowerShell:
~~~powershell
$env:BASE_URL = 'http://127.0.0.1:18084'
python -m unittest discover -s student-tests -v
python tests/resilience.py --mode persistence --action-script scripts/restart.cmd --base-url $env:BASE_URL
python tests/resilience.py --mode failover --action-script scripts/stop-one.cmd --base-url $env:BASE_URL
docker compose --env-file .env up -d --wait api1 api2
~~~
Linux/macOS: выполнить chmod +x scripts/restart.sh scripts/stop-one.sh
и использовать .sh вместо .cmd. С GNU Make:
~~~sh
make check test test-student PYTHON=python BASE_URL=http://127.0.0.1:18084
make test-persistence PYTHON=python BASE_URL=http://127.0.0.1:18084 ACTION_SCRIPT=scripts/restart.sh
make test-failover PYTHON=python BASE_URL=http://127.0.0.1:18084 ACTION_SCRIPT=scripts/stop-one.sh
docker compose --env-file .env up -d --wait api1 api2
~~~
На Windows проверки выполнены прямыми Python-командами, эквивалентными Make targets.
Штатная остановка: docker compose --env-file .env down — volumes сохраняются.
**Намеренный сброс всех учебных данных:** docker compose --env-file .env down -v.
Команда удаления volumes не используется в hooks и проверках.
## Архитектура и версии
~~~mermaid
flowchart LR
C[Клиенты] --> LB[HAProxy / localhost:APP_PORT]
LB --> A1[api1]
LB --> A2[api2]
A1 --> PG[(PostgreSQL: metadata / sessions / message keys)]
A2 --> PG
A1 --> S3[(SeaweedFS: private originals)]
A2 --> S3
PG --> PV[postgres_data]
S3 --> SV[s3_data]
~~~
| Роль / service | Версия | Readiness / bootstrap | Хранение / сеть |
| --- | --- | --- | --- |
| api1, api2 | Python 3.12.12, собственный Dockerfile | /health/ready: PostgreSQL + schema 003 + S3 HeadBucket | Read-only root, без data volume и опубликованных портов |
| postgres | postgres:16.10-bookworm | pg_isready; отдельный migrate | postgres_data, только внутренняя сеть |
| s3 | chrislusf/seaweedfs:3.85 | s3-init ждёт подписанный ListBuckets; API проверяет HeadBucket | s3_data на /data; S3 :8333, порты наружу не публикуются |
| lb | haproxy:3.0.11-alpine | HTTP checks API каждую секунду, fall=2/rise=1 | Только 127.0.0.1:APP_PORT → :8080 |
API: 1 CPU / 512 MiB на реплику; PostgreSQL: 1 CPU / 256 MiB;
S3: 1 CPU / 512 MiB; LB: 0.5 CPU / 128 MiB.
Миграции и bucket job: 0.5 CPU / 128 MiB.
Docker secrets монтируются в /run/secrets; значения не входят в образ и HTTP-ответы.
Runtime-зависимости закреплены в requirements.txt: psycopg 3.2.6,
psycopg-pool 3.2.6, typing_extensions 4.16.0, boto3 1.40.61, Pillow 12.0.0.
Реально разрешённые транзитивные версии сохранены в evidence/final-checks.txt.
Полного lockfile с hashes нет. Образы MinIO в официальных реестрах оказались
недоступны при подготовке; выбран разрешённый заданием SeaweedFS.
SeaweedFS server -filer -s3 запускает master, volume server, filer и S3 gateway.
Рабочий каталог образа — /data; он целиком находится в persistent volume,
включая метаданные filer и данные volumes. Размер volume ограничен 100 MiB,
максимум 8 volumes — учебная настройка ёмкости без репликации на другой host.
## Что перенесено и изменено
Из lab1 унаследованы HTTP transport, строгая валидация, ошибки, keyset-пагинация
и завершение по SIGTERM. Lab2 перевела состояние в PostgreSQL, добавила scrypt
и общие Bearer-сессии. Из lab3 перенесены эти исходники, неизменённые миграции
001/002, собственные тесты, две API за HAProxy и message_keys.
Предыдущие каталоги не изменялись. README задания, COURSE, contracts и публичные
tests текущей lab4 сохранены; соседние skeleton поверх них не копировались.
Новая миграция 003 создаёт Attachment с FK на чат и владельца-участника,
размером, MIME, SHA-256, случайным object_key и UTC timestamps.
Бинарных столбцов в PostgreSQL нет. Поле messages.attachment_ids —
JSONB-массив до десяти уникальных UUID; все ссылки проверяются в транзакции
отправки, участники неизменяемы. История возвращает attachment_ids,
в том числе [] для старых текстовых сообщений.
Текст с отсутствующим attachment_ids и с [] семантически равен.
Идемпотентность сравнивает text и упорядоченный список нормализованных UUID.
Блокировка строки чата в PostgreSQL сериализует конкурирующие отправки всех API;
запись сообщения и результата ключа коммитятся атомарно. Ключ действует 24 часа.
Сессии, HMAC cursor key и результат повторной отправки переживают restart.
Синхронная загрузка:
1. Bearer-сессия и членство в чате проверяются до чтения тела.
2. filename — 1–255 Unicode code points. Slash, backslash, NUL, управляющие
символы, surrogate и пустое/пробельное имя отклоняются. Это только отображаемое имя.
3. Требуется Content-Length, Transfer-Encoding не принимается. До чтения
проверяется максимум 10 MiB; читается ровно объявленное число байтов.
4. Pillow открывает только PNG/JPEG, сравнивает настоящий MIME с Content-Type,
проверяет размер из заголовка до load(), затем выполняет verify() и полное
декодирование для отбрасывания повреждённых данных.
Предел — 40 млн пикселей включительно.
5. S3 PUT по случайному ключу originals/<uuid> → INSERT метаданных → COMMIT →
201 ready. Поля variants=[], error_code=null. Оригинал не перекодируется.
На API нет локального файла оригинала. Один одновременный decode на реплику
ограничен semaphore; остальные загрузки получают 503 через 1 секунду ожидания.
Тело максимум 10 MiB хранится в RAM; это ограниченная буферизация, а не потоковая
обработка уровня 4. Сокет клиента: 10 s; S3 connect/read: 1/3 s, без retry SDK;
LB client/server: 30/20 s. Ограничения bytes/pixels можно только уменьшить env.
Опыт десяти параллельных uploads не проводился; общая защита от DoS не заявляется.
Download: Bearer → membership query → S3 GET → ограниченное чтение →
проверка size/SHA-256 → bytes с истинным Content-Type, X-Request-Id и X-Instance-Id.
Редиректа и публичного S3 URL нет. Посторонний получает одинаковый 404
для metadata и content. Неизвестный вариант даёт 404.
Публичные metadata не раскрывают object_key.
/health/live не обращается к хранилищам. /health/ready требует PostgreSQL,
миграцию 003 и S3, поскольку оригинал — обязательный синхронный путь.
При отказе S3 LB выводит обе реплики из балансировки.
Текстовые операции и metadata сами по себе могли бы работать с PostgreSQL,
но политика общей readiness намеренно строже.
Настройки .env.example используются Compose: APP_PORT — внешний порт LB,
APP_ENV/SESSION_TTL_SECONDS — среда/сессии, S3_ENDPOINT/REGION/BUCKET —
S3 client и bootstrap, MAX_UPLOAD_BYTES/MAX_IMAGE_PIXELS — ограничения API.
Внутренние APP_HOST=0.0.0.0, APP_PORT=8080 и пути к secret-файлам заданы
непосредственно в compose.yaml. API_REPLICAS не требуется: явно объявлены api1/api2.
## Доказательства критериев
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
Полный реальный вывод отказов:
[evidence/grade3-checks.txt](evidence/grade3-checks.txt).
Финальная сборка, smoke, 23 собственных теста и версии зависимостей:
[evidence/final-checks.txt](evidence/final-checks.txt).
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
Первый полный прогон содержал 21 тест; затем добавлены два граничных сценария,
все 23 теста выполнены на финальной сборке.
Контракт и публичные tests сверены с Git.
## Эксперименты и отказы
Полная факультативная OpenAPI-валидация не выполнялась: на host отсутствует
openapi-spec-validator. Для неё выполнить:
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
~~~sh
python -m pip install -r requirements-dev.txt
python scripts/validate_openapi.py
~~~
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
Обязательный check_contract прошёл.
## Ограничения и следующие шаги
Дата испытаний: 13 ноября 2026. Windows-хост, Docker Desktop 4.77.0,
Engine 29.5.3; Docker VM: 8 CPU, 2 967 707 648 байт RAM.
Это ресурсы VM; модель физического CPU и пределы производительности не исследовались.
Версии Docker, instance aliases, контейнеры, StartedAt, mounts и результаты
записаны в протокол автоматически.
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
| Критерий уровня 3 | Команда / сценарий | Наблюдение |
| --- | --- | --- |
| Предыдущий минимум и публичный smoke | scripts/check_contract.py; tests/smoke.py; student-tests | Контрактный сценарий и собственные проверки API/сессий/пагинации/идемпотентности прошли |
| S3 в общем стенде, метаданные отдельно | compose config; state() в verify.py; миграция 003 | PostgreSQL и SeaweedFS с отдельными volumes; API без data volume |
| 201 ready, оригинал побайтно | smoke; test_original_and_message_shared_with_member; test_jpeg_original_and_truncation | PNG/JPEG, размер и SHA-256 совпадают; attachment-only сообщение читает другой участник |
| Приватный bucket и ACL чата | verify_upload_failover; test_acl_before_image_validation | Прямой unsigned GET существующего S3-объекта — 403; посторонний в API — 404 |
| Byte/pixel/MIME/filename limits | test_byte_limit_before_reading_body; test_image_validation; test_filename | 413 до передачи oversized тела; 415 для подмены/порчи; 400 для опасного имени |
| Граничные допустимые размеры | test_maximum_byte_size_accepted; test_exactly_40_million_pixels_accepted | Приняты 10 MiB включительно и валидный PNG 8000×5000 |
| Остановка загрузившей API | verify_upload_failover, обе API по очереди | Останавливается alias из ответа upload; download другой репликой побайтно совпадает |
| Restart всех stores и API | Публичный persistence с restart.cmd/.sh + inspect до/после | StartedAt меняется у s3/postgres/api1/api2/lb, volumes сохраняются; прежний token и файл работают |
| Минимум failover lab3 | Публичный failover обеих API; inspect exited | Подтверждённые сообщения и файл сохранены, retry без дублей, запись восстанавливается за бюджет теста |
| Недоступность S3 | verify_s3_failure | Прямо в API: live=200, ready=503, metadata=200, upload/download=503; число metadata не увеличилось |
Проверки завершились успешно: публичные persistence и failover обеих API,
анонимный S3 GET=403, контролируемый отказ S3, 23 собственных теста.
В полном прогоне: остановка api1 — 97 успешных записей, 0 неудачных POST,
first_recovery=3.844 s; остановка api2 — 96 записей, 0 неудачных POST,
first_recovery=4.750 s. Параллельные чтения: 148/145 успешных, по одной переходной
ошибке. first_recovery включает запуск Docker hook; это не длительность полного
простоя. Показатели относятся к этому локальному эксперименту.
## Отказы, согласованность и ограничения
Повторы migration jobs сверяются по checksum под advisory lock;
повторный запуск не меняет количества строк.
При restart публичная проверка сохраняет прежнюю сессию, сообщения и файл;
дополнительные проверки сохраняют cursor и результат идемпотентного POST.
Отдельно inspect подтверждает реальный перезапуск процессов.
| Воздействие | Ожидаемый результат | Проверяемое наблюдение |
| --- | --- | --- |
| Остановить загрузившую API | Файл доступен другой реплике | Два UPLOAD_FAILOVER с разными stopped/download aliases и одинаковым SHA-256 |
| Остановить одну API во время POST | Вторая продолжает запись; retry без дублей | Публичный failover: successes, failures, first_recovery_seconds в протоколе |
| Перезапустить весь стенд | Сохранить volumes, токены и данные | BEFORE_RESTART/AFTER_RESTART, persistence PASS |
| Остановить S3 | Контролируемые 503, без ложного 201 | S3 outage PASS, неизменное число attachments; после start оригинал совпадает |
| GET чужого файла | Скрыть существование объекта | 404 для metadata/content, 401 без сессии |
У PostgreSQL и S3 нет общей транзакции.
Если PUT успешен, а INSERT/COMMIT не подтверждён, клиент не получает 201;
возможен orphan object. При неопределённом результате COMMIT объект не удаляется,
чтобы не разрушить уже зафиксированную запись.
Автоматические reconciler/компенсация с grace period не реализованы
и относятся к уровню 4/5. При отказе S3 во время GET/PUT API возвращает 503;
ошибка записи не фиксируется как успешный результат идемпотентности.
Учебное упрощение: одна S3 identity с административными правами используется
API и bootstrap. Секреты не публикуются, anonymous identity отсутствует;
отдельные ограниченные credentials API относятся к уровню 4.
Сеть Compose считается доверенной; внутренние HTTP-интерфейсы SeaweedFS доступны
соседним контейнерам, наружу не опубликованы.
Это не защита от компрометации контейнера внутри сети.
Один Docker-host, LB, PostgreSQL и SeaweedFS остаются точками единого отказа.
Нет репликации S3/БД, backup, TLS, workers, presigned upload, thumbnail,
защиты от потери volumes/host и гарантии доступности при отказе stores.
Проверены остановка отдельной API и сохранность при полном restart;
они не доказывают промышленную HA. Нагрузочное исследование, RPS и p95/p99
для уровня 3 не заявляются. Файлы и пользователи в проверках синтетические.
## Источники
- [Обязательная семантика lab4](contracts/README.md) и [OpenAPI](contracts/openapi.json).
- [SeaweedFS S3 API](https://github.com/seaweedfs/seaweedfs/wiki/Amazon-S3-API):
server с S3 gateway, конфигурация identities.
- [Pillow Image](https://pillow.readthedocs.io/en/stable/reference/Image.html):
open/verify/load, размеры до декодирования и защита от decompression bomb.
- [Boto3 PUT](https://docs.aws.amazon.com/boto3/latest/reference/services/s3/client/put_object.html)
и [GET](https://docs.aws.amazon.com/boto3/latest/reference/services/s3/client/get_object.html):
сохранение bytes и чтение оригинала.
- [HAProxy 3.0](https://docs.haproxy.org/3.0/configuration.html).
- [psycopg transactions](https://www.psycopg.org/psycopg3/docs/basic/transactions.html).
+17
View File
@@ -0,0 +1,17 @@
# Лабораторная 4 — БПИ2303 — Щербаков — на 3
Перенесена собственная реализация lab3: PostgreSQL, сессии, HAProxy,
две API-реплики и атомарная идемпотентность. Добавлены приватное S3-хранилище
SeaweedFS, миграция вложений, проверка PNG/JPEG и авторизованное скачивание.
Запуск: `python scripts/bootstrap.py`, затем
`docker compose --env-file .env up -d --build --wait`.
Локальный проверенный порт в `.env`: `APP_PORT=18084`.
Полная проверка:
`python scripts/verify.py --base-url http://127.0.0.1:18084 --output evidence/grade3-checks.txt`.
Инструкция и ограничения: [REPORT.md](REPORT.md).
Реальный протокол: [evidence/grade3-checks.txt](evidence/grade3-checks.txt).
Обязательные контракты и публичные tests сохранены. Использован OpenAI Codex.
Уровни 4/5 не заявляются. Commit/push/PR выполняются отдельно.
+144
View File
@@ -0,0 +1,144 @@
name: mtusi2026lab4
x-api: &api
build: .
image: mtusi2026lab4-api:local
environment: &api-env
APP_HOST: 0.0.0.0
APP_PORT: 8080
APP_ENV: ${APP_ENV:-lab}
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
DATABASE_URL_FILE: /run/secrets/database_url
S3_ENDPOINT: ${S3_ENDPOINT:-http://s3:8333}
S3_REGION: ${S3_REGION:-us-east-1}
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
MAX_UPLOAD_BYTES: ${MAX_UPLOAD_BYTES:-10485760}
MAX_IMAGE_PIXELS: ${MAX_IMAGE_PIXELS:-40000000}
secrets: [database_url, s3_access_key_id, s3_secret_access_key]
depends_on:
migrate:
condition: service_completed_successfully
s3-init:
condition: service_completed_successfully
restart: unless-stopped
read_only: true
cap_drop: [ALL]
security_opt: [no-new-privileges:true]
stop_grace_period: 7s
cpus: 1.0
mem_limit: 512m
services:
postgres:
image: postgres:16.10-bookworm
environment:
POSTGRES_USER: messenger
POSTGRES_DB: messenger
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
secrets: [postgres_password]
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test: [CMD-SHELL, "pg_isready -U messenger -d messenger"]
interval: 2s
timeout: 3s
retries: 30
restart: unless-stopped
cpus: 1.0
mem_limit: 256m
migrate:
build: .
image: mtusi2026lab4-api:local
command: [python, -m, src.database]
environment:
DATABASE_URL_FILE: /run/secrets/database_url
secrets: [database_url]
depends_on:
postgres:
condition: service_healthy
restart: "no"
healthcheck:
disable: true
read_only: true
cap_drop: [ALL]
security_opt: [no-new-privileges:true]
cpus: 0.5
mem_limit: 128m
s3:
image: chrislusf/seaweedfs:3.85
command: [server, -dir=/data, -ip=s3, -master.volumeSizeLimitMB=100, -volume.max=8, -filer, -s3, -s3.config=/run/secrets/s3_config]
secrets: [s3_config]
volumes:
- s3_data:/data
restart: unless-stopped
cpus: 1.0
mem_limit: 512m
s3-init:
build: .
image: mtusi2026lab4-api:local
command: [python, -m, src.bucket_bootstrap]
environment:
S3_ENDPOINT: ${S3_ENDPOINT:-http://s3:8333}
S3_REGION: ${S3_REGION:-us-east-1}
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
secrets: [s3_access_key_id, s3_secret_access_key]
depends_on: [s3]
restart: "no"
healthcheck:
disable: true
read_only: true
cap_drop: [ALL]
security_opt: [no-new-privileges:true]
cpus: 0.5
mem_limit: 128m
api1:
<<: *api
environment:
<<: *api-env
INSTANCE_ID: api1
api2:
<<: *api
environment:
<<: *api-env
INSTANCE_ID: api2
lb:
image: haproxy:3.0.11-alpine
volumes:
- ./infra/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
ports:
- "127.0.0.1:${APP_PORT:-8080}:8080"
depends_on:
api1:
condition: service_healthy
api2:
condition: service_healthy
restart: unless-stopped
stop_grace_period: 7s
cpus: 0.5
mem_limit: 128m
volumes:
postgres_data:
s3_data:
secrets:
postgres_password:
file: ./secrets/postgres_password
database_url:
file: ./secrets/database_url
s3_access_key_id:
file: ./secrets/s3_access_key_id
s3_secret_access_key:
file: ./secrets/s3_secret_access_key
s3_config:
file: ./secrets/s3_config.json
+15
View File
@@ -3,3 +3,18 @@
Храните небольшие обезличенные результаты: команды и stdout проверок, JSON/CSV агрегатов нагрузки, EXPLAIN, dashboard JSON, текст разбора отказа. Укажите дату, commit, аппаратные лимиты и профиль нагрузки.
Не добавляйте .env, session tokens, cookies, приватные ключи, credentials, реальные сообщения, raw dumps и signed URLs. Для скриншотов скрывайте секреты. Большие логи/бинарные traces сохраняйте отдельно по правилам преподавателя; одного скриншота без методики недостаточно.
## Выполненные проверки решения на 3
- [grade3-checks.txt](grade3-checks.txt) — smoke, 21 первоначальный собственный тест,
миграции, inspect, persistence, остановка загрузивших API, private S3,
отказ/восстановление S3 и failover обеих API.
- [final-checks.txt](final-checks.txt) — финальная сборка, smoke, все 23 собственных
теста (добавлены точные границы 10 MiB и 40 млн пикселей), runtime dependencies,
состояние сервисов и неизменность публичного контракта/tests относительно Git.
Команда воспроизведения полного набора:
~~~sh
python scripts/verify.py --base-url http://127.0.0.1:18084 --output evidence/grade3-checks.txt
~~~
+58
View File
@@ -0,0 +1,58 @@
UTC 2026-11-13T10:54:55.166348+00:00
COMMAND C:\Python312\python.exe scripts/check_contract.py
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
COMMAND C:\Python312\python.exe tests/smoke.py --base-url http://127.0.0.1:18084
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
COMMAND C:\Python312\python.exe -m unittest discover -s student-tests -v
test_access_before_payload_validation (test_api.API.test_access_before_payload_validation) ... ok
test_bad_credentials_and_schemes (test_api.API.test_bad_credentials_and_schemes) ... ok
test_chat_atomic_invalid_members (test_api.API.test_chat_atomic_invalid_members) ... ok
test_concurrent_messages_and_keyset (test_api.API.test_concurrent_messages_and_keyset) ... ok
test_cursor_scope_and_tampering (test_api.API.test_cursor_scope_and_tampering) ... ok
test_json_and_sender_validation (test_api.API.test_json_and_sender_validation) ... ok
test_logout_only_revokes_current_session (test_api.API.test_logout_only_revokes_current_session) ... ok
test_registration_validation (test_api.API.test_registration_validation) ... ok
test_unicode_password_with_colon_and_max_length (test_api.API.test_unicode_password_with_colon_and_max_length) ... ok
test_username_race (test_api.API.test_username_race) ... ok
test_acl_before_image_validation (test_attachments.Attachments.test_acl_before_image_validation) ... ok
test_attachment_scope_and_idempotency (test_attachments.Attachments.test_attachment_scope_and_idempotency) ... ok
test_byte_limit_before_reading_body (test_attachments.Attachments.test_byte_limit_before_reading_body) ... ok
test_exactly_40_million_pixels_accepted (test_attachments.Attachments.test_exactly_40_million_pixels_accepted) ... ok
test_filename (test_attachments.Attachments.test_filename) ... ok
test_image_validation (test_attachments.Attachments.test_image_validation) ... ok
test_jpeg_original_and_truncation (test_attachments.Attachments.test_jpeg_original_and_truncation) ... ok
test_maximum_byte_size_accepted (test_attachments.Attachments.test_maximum_byte_size_accepted) ... ok
test_original_and_message_shared_with_member (test_attachments.Attachments.test_original_and_message_shared_with_member) ... ok
test_conflicting_bodies_race (test_idempotency.Idempotency.test_conflicting_bodies_race) ... ok
test_invalid_keys_and_no_cached_error (test_idempotency.Idempotency.test_invalid_keys_and_no_cached_error) ... ok
test_key_scope_and_access (test_idempotency.Idempotency.test_key_scope_and_access) ... ok
test_same_key_concurrent_across_replicas (test_idempotency.Idempotency.test_same_key_concurrent_across_replicas) ... PASS: concurrent_retries=12 observed_replicas=2 committed_messages=1
ok
----------------------------------------------------------------------
Ran 23 tests in 32.470s
OK
COMMAND docker compose --env-file .env exec -T api1 python -m pip freeze
WARNING: The directory '/home/messenger/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
boto3==1.40.61
botocore==1.40.76
jmespath==1.1.0
pillow==12.0.0
psycopg==3.2.6
psycopg-binary==3.2.6
psycopg-pool==3.2.6
python-dateutil==2.9.0.post0
s3transfer==0.14.0
six==1.17.0
typing_extensions==4.16.0
urllib3==2.8.0
COMMAND docker compose --env-file .env ps
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
mtusi2026lab4-api1-1 mtusi2026lab4-api:local "python -m src.server" api1 About a minute ago Up About a minute (healthy) 8080/tcp
mtusi2026lab4-api2-1 mtusi2026lab4-api:local "python -m src.server" api2 About a minute ago Up About a minute (healthy) 8080/tcp
mtusi2026lab4-lb-1 haproxy:3.0.11-alpine "docker-entrypoint.s…" lb 7 minutes ago Up 5 minutes 127.0.0.1:18084->8080/tcp
mtusi2026lab4-postgres-1 postgres:16.10-bookworm "docker-entrypoint.s…" postgres 7 minutes ago Up 5 minutes (healthy) 5432/tcp
mtusi2026lab4-s3-1 chrislusf/seaweedfs:3.85 "/entrypoint.sh serv…" s3 7 minutes ago Up 4 minutes 7333/tcp, 8080/tcp, 8333/tcp, 8888/tcp, 9333/tcp, 18080/tcp, 18888/tcp, 19333/tcp
COMMAND git -c safe.directory=C:/Users/marks/Desktop/vp/mtusi2026lab4 diff --exit-code -- README.md COURSE.md contracts tests
PASS: final source, smoke, all 23 student tests, runtime dependencies and unchanged public contract/tests
+65
View File
@@ -0,0 +1,65 @@
UTC 2026-11-13T10:49:10.611207+00:00
COMMAND python scripts/verify.py --base-url http://127.0.0.1:18084
Client=29.5.3 Server=29.5.3
Docker CPUs=8 memory_bytes=2967707648 OS=Docker Desktop
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
test_access_before_payload_validation (test_api.API.test_access_before_payload_validation) ... ok
test_bad_credentials_and_schemes (test_api.API.test_bad_credentials_and_schemes) ... ok
test_chat_atomic_invalid_members (test_api.API.test_chat_atomic_invalid_members) ... ok
test_concurrent_messages_and_keyset (test_api.API.test_concurrent_messages_and_keyset) ... ok
test_cursor_scope_and_tampering (test_api.API.test_cursor_scope_and_tampering) ... ok
test_json_and_sender_validation (test_api.API.test_json_and_sender_validation) ... ok
test_logout_only_revokes_current_session (test_api.API.test_logout_only_revokes_current_session) ... ok
test_registration_validation (test_api.API.test_registration_validation) ... ok
test_unicode_password_with_colon_and_max_length (test_api.API.test_unicode_password_with_colon_and_max_length) ... ok
test_username_race (test_api.API.test_username_race) ... ok
test_acl_before_image_validation (test_attachments.Attachments.test_acl_before_image_validation) ... ok
test_attachment_scope_and_idempotency (test_attachments.Attachments.test_attachment_scope_and_idempotency) ... ok
test_byte_limit_before_reading_body (test_attachments.Attachments.test_byte_limit_before_reading_body) ... ok
test_filename (test_attachments.Attachments.test_filename) ... ok
test_image_validation (test_attachments.Attachments.test_image_validation) ... ok
test_jpeg_original_and_truncation (test_attachments.Attachments.test_jpeg_original_and_truncation) ... ok
test_original_and_message_shared_with_member (test_attachments.Attachments.test_original_and_message_shared_with_member) ... ok
test_conflicting_bodies_race (test_idempotency.Idempotency.test_conflicting_bodies_race) ... ok
test_invalid_keys_and_no_cached_error (test_idempotency.Idempotency.test_invalid_keys_and_no_cached_error) ... ok
test_key_scope_and_access (test_idempotency.Idempotency.test_key_scope_and_access) ... ok
test_same_key_concurrent_across_replicas (test_idempotency.Idempotency.test_same_key_concurrent_across_replicas) ... PASS: concurrent_retries=12 observed_replicas=2 committed_messages=1
ok
----------------------------------------------------------------------
Ran 21 tests in 29.448s
OK
OBSERVED_INSTANCES ["api1-a65e7971cde3", "api2-ee7a40d9d441"]
Container mtusi2026lab4-migrate-run-3fcdaacd95b0 Creating
Container mtusi2026lab4-migrate-run-3fcdaacd95b0 Created
PASS: migrations applied
Container mtusi2026lab4-migrate-run-5309164ddf9e Creating
Container mtusi2026lab4-migrate-run-5309164ddf9e Created
PASS: migrations applied
PASS: repeated migrations preserve all table counts 71|26|49|57|68|57|3|5
BEFORE_RESTART {"api1": {"id": "82d506213800", "status": "running", "started_at": "2026-11-13T10:48:35.085906013Z", "volumes": [], "ports": {}, "memory_limit": 536870912, "nano_cpus": 1000000000}, "api2": {"id": "65158cbc104d", "status": "running", "started_at": "2026-11-13T10:48:35.090231302Z", "volumes": [], "ports": {}, "memory_limit": 536870912, "nano_cpus": 1000000000}, "postgres": {"id": "337a801d2d8d", "status": "running", "started_at": "2026-11-13T10:48:08.472177118Z", "volumes": ["mtusi2026lab4_postgres_data"], "ports": {}, "memory_limit": 268435456, "nano_cpus": 1000000000}, "s3": {"id": "dabae5d269ac", "status": "running", "started_at": "2026-11-13T10:48:08.462152899Z", "volumes": ["mtusi2026lab4_s3_data"], "ports": {}, "memory_limit": 536870912, "nano_cpus": 1000000000}, "lb": {"id": "3288ada2b3b9", "status": "running", "started_at": "2026-11-13T10:48:41.912294259Z", "volumes": [], "ports": {"8080/tcp": [{"HostIp": "127.0.0.1", "HostPort": "18084"}]}, "memory_limit": 134217728, "nano_cpus": 500000000}}
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
AFTER_RESTART {"api1": {"id": "82d506213800", "status": "running", "started_at": "2026-11-13T10:50:04.482195882Z", "volumes": [], "ports": {}, "memory_limit": 536870912, "nano_cpus": 1000000000}, "api2": {"id": "65158cbc104d", "status": "running", "started_at": "2026-11-13T10:50:04.40460967Z", "volumes": [], "ports": {}, "memory_limit": 536870912, "nano_cpus": 1000000000}, "postgres": {"id": "337a801d2d8d", "status": "running", "started_at": "2026-11-13T10:50:04.716428682Z", "volumes": ["mtusi2026lab4_postgres_data"], "ports": {}, "memory_limit": 268435456, "nano_cpus": 1000000000}, "s3": {"id": "dabae5d269ac", "status": "running", "started_at": "2026-11-13T10:50:04.907037089Z", "volumes": ["mtusi2026lab4_s3_data"], "ports": {}, "memory_limit": 536870912, "nano_cpus": 1000000000}, "lb": {"id": "3288ada2b3b9", "status": "running", "started_at": "2026-11-13T10:50:04.291720252Z", "volumes": [], "ports": {"8080/tcp": [{"HostIp": "127.0.0.1", "HostPort": "18084"}]}, "memory_limit": 134217728, "nano_cpus": 500000000}}
PASS: five containers restarted; volume, sessions, cursor and idempotency result preserved
UPLOAD_FAILOVER {"stopped": "api1-bf729723a3ad", "download_instance": "api2-c0d733cf173b", "sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75", "message_id": "132a8a9f-aab2-4e08-ba11-6cc489acd306"}
UPLOAD_FAILOVER {"stopped": "api2-c0d733cf173b", "download_instance": "api1-a18496561062", "sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75", "message_id": "293f8515-01cc-4360-b87d-d923f6e8d928"}
PASS: anonymous S3 GET of existing object denied (403)
PASS: S3 down: live=200 ready=503 metadata=200 download=503 upload=503
PASS: S3 recovered; original bytes preserved; failed upload created no metadata
FAILOVER_RUN 1
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 97, "failed_attempts": 0, "first_recovery_seconds": 3.844}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
AFTER_FAILOVER {"api1": {"id": "82d506213800", "status": "exited", "started_at": "2026-11-13T10:50:44.539901416Z", "volumes": [], "ports": {}, "memory_limit": 536870912, "nano_cpus": 1000000000}, "api2": {"id": "65158cbc104d", "status": "running", "started_at": "2026-11-13T10:51:02.755073452Z", "volumes": [], "ports": {}, "memory_limit": 536870912, "nano_cpus": 1000000000}, "postgres": {"id": "337a801d2d8d", "status": "running", "started_at": "2026-11-13T10:50:04.716428682Z", "volumes": ["mtusi2026lab4_postgres_data"], "ports": {}, "memory_limit": 268435456, "nano_cpus": 1000000000}, "s3": {"id": "dabae5d269ac", "status": "running", "started_at": "2026-11-13T10:51:31.425009302Z", "volumes": ["mtusi2026lab4_s3_data"], "ports": {}, "memory_limit": 536870912, "nano_cpus": 1000000000}, "lb": {"id": "3288ada2b3b9", "status": "running", "started_at": "2026-11-13T10:50:04.291720252Z", "volumes": [], "ports": {"8080/tcp": [{"HostIp": "127.0.0.1", "HostPort": "18084"}]}, "memory_limit": 134217728, "nano_cpus": 500000000}}
CONCURRENT_READS {"ok": 148, "errors": 1}
FAILOVER_RUN 2
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 96, "failed_attempts": 0, "first_recovery_seconds": 4.75}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
AFTER_FAILOVER {"api1": {"id": "82d506213800", "status": "running", "started_at": "2026-11-13T10:52:31.531295764Z", "volumes": [], "ports": {}, "memory_limit": 536870912, "nano_cpus": 1000000000}, "api2": {"id": "65158cbc104d", "status": "exited", "started_at": "2026-11-13T10:51:02.755073452Z", "volumes": [], "ports": {}, "memory_limit": 536870912, "nano_cpus": 1000000000}, "postgres": {"id": "337a801d2d8d", "status": "running", "started_at": "2026-11-13T10:50:04.716428682Z", "volumes": ["mtusi2026lab4_postgres_data"], "ports": {}, "memory_limit": 268435456, "nano_cpus": 1000000000}, "s3": {"id": "dabae5d269ac", "status": "running", "started_at": "2026-11-13T10:51:31.425009302Z", "volumes": ["mtusi2026lab4_s3_data"], "ports": {}, "memory_limit": 536870912, "nano_cpus": 1000000000}, "lb": {"id": "3288ada2b3b9", "status": "running", "started_at": "2026-11-13T10:50:04.291720252Z", "volumes": [], "ports": {"8080/tcp": [{"HostIp": "127.0.0.1", "HostPort": "18084"}]}, "memory_limit": 134217728, "nano_cpus": 500000000}}
CONCURRENT_READS {"ok": 145, "errors": 1}
PASS: each API stopped separately; reads/writes and original session survive; both replicas restored; S3 bytes checked
PASS: grade-3 verification complete
+6
View File
@@ -1,5 +1,11 @@
# Инфраструктура студента
Реализованный стенд: [compose.yaml](../compose.yaml). Запуск, версии, порты,
секреты и результаты проверок описаны в [REPORT.md](../REPORT.md).
HAProxy использует [haproxy.cfg](haproxy.cfg). SeaweedFS запускает master,
volume server, filer и S3 gateway одним процессом; рабочий каталог `/data`
целиком находится в `s3_data`. Внешних зависимостей от lab1–3 нет.
Здесь разместите конфигурации выбранных сервисов; основную топологию — в корневом compose.yaml/stack.yaml. `compose.example.yaml` — только worksheet ролей и намеренно не запускается. `contracts/infrastructure.json` задаёт минимальные роли, не готовые образы/пароли.
В REPORT.md сопоставьте роль → service name → версия/digest → health/readiness → volume → сеть/порт → способ передачи credentials. Приведите bootstrap (schema, bucket, broker topology, а с lab6 provisioning) и безопасное повторение этих команд.
+42
View File
@@ -0,0 +1,42 @@
global
log stdout format raw local0
maxconn 512
hard-stop-after 5s
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 500ms
timeout client 30s
timeout server 20s
timeout check 1s
retries 1
retry-on conn-failure
option redispatch
resolvers docker
nameserver dns 127.0.0.11:53
resolve_retries 3
timeout resolve 1s
timeout retry 1s
hold valid 1s
hold obsolete 1s
hold nx 1s
hold timeout 1s
hold refused 1s
hold other 1s
frontend public
bind :8080
default_backend replicas
backend replicas
balance roundrobin
option httpchk
http-check send meth GET uri /health/ready ver HTTP/1.1 hdr Host localhost
http-check expect status 200
default-server check inter 1s fall 2 rise 1 resolvers docker resolve-prefer ipv4 init-addr libc,none
server api1 api1:8080
server api2 api2:8080
+36
View File
@@ -0,0 +1,36 @@
CREATE TABLE users (
id uuid PRIMARY KEY,
username text NOT NULL UNIQUE CHECK (username ~ '^[a-z0-9_]{3,32}$'),
display_name text NOT NULL CHECK (char_length(display_name) BETWEEN 1 AND 100),
password_hash text NOT NULL,
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE TABLE chats (
id uuid PRIMARY KEY,
title text NOT NULL CHECK (char_length(title) BETWEEN 1 AND 100),
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE TABLE memberships (
chat_id uuid NOT NULL REFERENCES chats(id),
user_id uuid NOT NULL REFERENCES users(id),
ordinal integer NOT NULL CHECK (ordinal >= 0),
PRIMARY KEY (chat_id, user_id),
UNIQUE (chat_id, ordinal)
);
CREATE INDEX memberships_user ON memberships (user_id, chat_id);
CREATE TABLE messages (
id uuid PRIMARY KEY,
chat_id uuid NOT NULL REFERENCES chats(id),
sender_id uuid NOT NULL,
text text NOT NULL CHECK (char_length(text) BETWEEN 1 AND 4000),
created_at timestamptz NOT NULL,
FOREIGN KEY (chat_id, sender_id) REFERENCES memberships(chat_id, user_id)
);
CREATE INDEX messages_history ON messages (chat_id, created_at, id);
CREATE TABLE sessions (
token_hash text PRIMARY KEY,
user_id uuid NOT NULL REFERENCES users(id),
expires_at timestamptz NOT NULL
);
CREATE INDEX sessions_expiry ON sessions (expires_at);
CREATE TABLE settings (key text PRIMARY KEY, value text NOT NULL);
+11
View File
@@ -0,0 +1,11 @@
-- This table serves only POST /api/v1/chats/{chat_id}/messages.
-- Therefore method and canonical path are implicit in (user_id, chat_id, key).
CREATE TABLE message_keys (
user_id uuid NOT NULL REFERENCES users(id),
chat_id uuid NOT NULL REFERENCES chats(id),
key text NOT NULL CHECK (key ~ '^[A-Za-z0-9._:-]{1,128}$'),
message_id uuid NOT NULL REFERENCES messages(id),
expires_at timestamptz NOT NULL DEFAULT (clock_timestamp() + interval '24 hours'),
PRIMARY KEY (user_id, chat_id, key)
);
CREATE INDEX message_keys_expiry ON message_keys(expires_at);
+19
View File
@@ -0,0 +1,19 @@
CREATE TABLE attachments (
id uuid PRIMARY KEY,
chat_id uuid NOT NULL REFERENCES chats(id),
owner_id uuid NOT NULL,
filename text NOT NULL CHECK (char_length(filename) BETWEEN 1 AND 255),
content_type text NOT NULL CHECK (content_type IN ('image/png','image/jpeg')),
size_bytes integer NOT NULL CHECK (size_bytes BETWEEN 1 AND 10485760),
sha256 text NOT NULL CHECK (sha256 ~ '^[a-f0-9]{64}$'),
object_key text NOT NULL UNIQUE,
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
updated_at timestamptz NOT NULL DEFAULT clock_timestamp(),
FOREIGN KEY (chat_id,owner_id) REFERENCES memberships(chat_id,user_id)
);
CREATE INDEX attachments_chat ON attachments(chat_id);
ALTER TABLE messages DROP CONSTRAINT messages_text_check;
ALTER TABLE messages ADD COLUMN attachment_ids jsonb NOT NULL DEFAULT '[]';
ALTER TABLE messages ADD CHECK (char_length(text) <= 4000);
ALTER TABLE messages ADD CHECK (jsonb_typeof(attachment_ids)='array' AND jsonb_array_length(attachment_ids)<=10);
ALTER TABLE messages ADD CHECK (char_length(text)>0 OR jsonb_array_length(attachment_ids)>0);
+5
View File
@@ -0,0 +1,5 @@
psycopg[binary]==3.2.6
psycopg-pool==3.2.6
typing_extensions==4.16.0
boto3==1.40.61
Pillow==12.0.0
+34
View File
@@ -0,0 +1,34 @@
"""Generate local synthetic credentials without printing or overwriting them."""
from pathlib import Path
import secrets
import json
root = Path(__file__).resolve().parents[1]
directory = root / "secrets"
directory.mkdir(exist_ok=True)
password_file = directory / "postgres_password"
if not password_file.exists():
password_file.write_text(secrets.token_hex(32), encoding="ascii")
url = directory / "database_url"
if not url.exists():
password = password_file.read_text(encoding="ascii").strip()
url.write_text(f"postgresql://messenger:{password}@postgres:5432/messenger", encoding="ascii")
for name, value in (("s3_access_key_id", "lab4" + secrets.token_hex(10)),
("s3_secret_access_key", secrets.token_hex(32))):
path = directory / name
if not path.exists():
path.write_text(value, encoding="ascii")
config = directory / "s3_config.json"
if not config.exists():
config.write_text(json.dumps({"identities": [{
"name": "lab4-local",
"credentials": [{"accessKey": (directory / "s3_access_key_id").read_text().strip(),
"secretKey": (directory / "s3_secret_access_key").read_text().strip()}],
"actions": ["Admin", "Read", "List", "Tagging", "Write"]
}]}), encoding="utf-8")
env = root / ".env"
if not env.exists():
env.write_bytes((root / ".env.example").read_bytes())
print("Local .env and secrets ready; existing files preserved.")
+4
View File
@@ -0,0 +1,4 @@
@echo off
cd /d "%~dp0.."
docker compose --env-file .env restart s3 postgres api1 api2 lb
exit /b %errorlevel%
+4
View File
@@ -0,0 +1,4 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")/.."
docker compose --env-file .env restart s3 postgres api1 api2 lb
+4
View File
@@ -0,0 +1,4 @@
@echo off
cd /d "%~dp0.."
python scripts/stop_one.py
exit /b %errorlevel%
+4
View File
@@ -0,0 +1,4 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")/.."
exec "${PYTHON:-python3}" scripts/stop_one.py
+19
View File
@@ -0,0 +1,19 @@
"""Stop exactly the process identified by TARGET_INSTANCE_ID; never remove volumes."""
import os
from pathlib import Path
import re
import subprocess
ROOT = Path(__file__).resolve().parents[1]
target = os.environ.get("TARGET_INSTANCE_ID", "")
if not re.fullmatch(r"api[12]-[0-9a-f]{12}", target):
raise SystemExit("TARGET_INSTANCE_ID must be the exact X-Instance-Id from a running API")
service = target.split("-", 1)[0]
compose = ["docker", "compose", "--env-file", ".env"]
# Compare to the real API process, rather than trusting a supplied service name.
code = "from urllib.request import urlopen; print(urlopen('http://127.0.0.1:8080/health/live',timeout=2).headers['X-Instance-Id'])"
actual = subprocess.check_output(compose + ["exec", "-T", service, "python", "-c", code], cwd=ROOT, text=True).strip()
if actual != target:
raise SystemExit("Target process changed; refusing to stop another instance")
subprocess.run(compose + ["stop", "--timeout", "6", service], cwd=ROOT, check=True)
print("Stopped " + service + " instance=" + target)
+237
View File
@@ -0,0 +1,237 @@
"""Reproduce grade-3 evidence against the lab4 Compose stack; no credentials in output."""
import argparse
from datetime import datetime, timezone
import json
import os
from pathlib import Path
import subprocess
import sys
import threading
import time
from urllib.parse import quote
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / "tests"))
from client import Client, Failure
COMPOSE = ["docker", "compose", "--env-file", ".env"]
def command(args, capture=False):
result = subprocess.run(args, cwd=ROOT, text=True, encoding="utf-8", errors="replace",
stdout=subprocess.PIPE if capture else sys.stdout,
stderr=subprocess.STDOUT, timeout=180)
if result.returncode:
raise RuntimeError("Command failed: " + args[0])
return result.stdout
def sql(query):
return command(COMPOSE + ["exec", "-T", "postgres", "psql", "-U", "messenger", "-d", "messenger", "-Atc", query], True).strip()
def state():
result = {}
for service in ("api1", "api2", "postgres", "s3", "lb"):
cid = command(COMPOSE + ["ps", "-a", "-q", service], True).strip()
data = json.loads(command(["docker", "inspect", cid], True))[0]
result[service] = {"id": cid[:12], "status": data["State"]["Status"],
"started_at": data["State"]["StartedAt"],
"volumes": [m["Name"] for m in data["Mounts"] if m["Type"] == "volume"],
"ports": data["HostConfig"]["PortBindings"],
"memory_limit": data["HostConfig"]["Memory"],
"nano_cpus": data["HostConfig"]["NanoCpus"]}
return result
def wait_replicas(c, actor):
deadline, seen = time.monotonic() + 60, set()
while time.monotonic() < deadline:
try:
seen.add(c.call("GET", "/api/v1/users/me", actor=actor, schema="User")["headers"]["X-Instance-Id"])
if len(seen) >= 2:
return seen
except Failure:
pass
time.sleep(0.1)
raise AssertionError("Two replicas did not become ready")
def verify_upload_failover(c, a, b, chat):
import hashlib
content = (ROOT / "tests/fixtures/sample.png").read_bytes()
for service in ("api1", "api2"):
wait_replicas(c, a)
for _ in range(30):
response = c.call("POST", "/api/v1/chats/" + chat["id"] + "/attachments?filename=failover.png",
201, actor=a, raw=content, headers={"Content-Type": "image/png"}, schema="Attachment")
target = response["headers"]["X-Instance-Id"]
if target.startswith(service + "-"):
break
else:
raise AssertionError("Upload did not reach expected replica")
att = response["json"]
message = c.message(a, chat, "", attachments=[att["id"]])["json"]
env = dict(os.environ, TARGET_INSTANCE_ID=target)
subprocess.run([sys.executable, "scripts/stop_one.py"], cwd=ROOT, env=env, check=True, timeout=20, stdout=sys.stdout, stderr=subprocess.STDOUT)
try:
assert state()[service]["status"] == "exited"
result = c.call("GET", "/api/v1/attachments/" + att["id"] + "/content", actor=b)
assert result["bytes"] == content
assert result["headers"]["X-Instance-Id"] != target
assert c.call("GET", "/api/v1/attachments/" + att["id"], actor=b, schema="Attachment")["json"] == att
print("UPLOAD_FAILOVER " + json.dumps({"stopped": target, "download_instance": result["headers"]["X-Instance-Id"],
"sha256": hashlib.sha256(result["bytes"]).hexdigest(),
"message_id": message["id"]}), flush=True)
finally:
command(COMPOSE + ["up", "-d", "--wait", "api1", "api2"], True)
# Inspect mounts: original bytes are never persisted in an API volume.
assert all(not state()[service]["volumes"] for service in ("api1", "api2"))
key = sql("SELECT object_key FROM attachments WHERE id='" + att["id"] + "'")
probe = """import urllib.request, urllib.error, sys, os
try:
urllib.request.urlopen(os.environ['S3_ENDPOINT'] + '/' + os.environ['S3_BUCKET'] + '/' + sys.argv[1],timeout=3)
raise AssertionError('Anonymous S3 GET allowed')
except urllib.error.HTTPError as exc:
assert exc.code == 403, exc.code
print('PASS: anonymous S3 GET of existing object denied (403)')
"""
command(COMPOSE + ["exec", "-T", "api1", "python", "-c", probe, key])
def verify_s3_failure(c, a, b, chat):
att, content = c.attachment(a, chat)
before = sql("SELECT count(*) FROM attachments")
# Execute inside the API network namespace: LB correctly removes unready APIs.
# Program travels through stdin; bearer never appears in command arguments/output.
program = """import urllib.request, urllib.error, json, base64
actor = ACTOR
attachment = ATTACHMENT
chat_id = CHAT_ID
content = base64.b64decode(CONTENT)
def request(path, expected, data=None, mime=None):
headers = {'Authorization': 'Bearer ' + actor['token']}
if mime: headers['Content-Type'] = mime
req = urllib.request.Request('http://127.0.0.1:8080' + path, data=data, headers=headers)
try: response = urllib.request.urlopen(req, timeout=8)
except urllib.error.HTTPError as exc: response = exc
with response:
assert response.code == expected, (path, response.code)
result = response.read()
return result
request('/health/live', 200)
request('/health/ready', 503)
request('/api/v1/attachments/' + attachment['id'], 200)
request('/api/v1/attachments/' + attachment['id'] + '/content', 503)
request('/api/v1/chats/' + chat_id + '/attachments?filename=outage.png', 503, content, 'image/png')
print('PASS: S3 down: live=200 ready=503 metadata=200 download=503 upload=503')
"""
import base64
program = program.replace("ACTOR", repr({"token": a["token"]})).replace("ATTACHMENT", repr(att)).replace("CHAT_ID", repr(chat["id"])).replace("CONTENT", repr(base64.b64encode(content).decode()))
command(COMPOSE + ["stop", "--timeout", "10", "s3"], True)
try:
result = subprocess.run(COMPOSE + ["exec", "-T", "api1", "python", "-"], cwd=ROOT,
input=program, text=True, capture_output=True, timeout=30)
assert result.returncode == 0, "S3 outage check failed (details suppressed to protect session)"
print(result.stdout.strip(), flush=True)
assert sql("SELECT count(*) FROM attachments") == before
finally:
command(COMPOSE + ["start", "s3"], True)
wait_replicas(c, a)
assert c.call("GET", "/api/v1/attachments/" + att["id"] + "/content", actor=b)["bytes"] == content
print("PASS: S3 recovered; original bytes preserved; failed upload created no metadata", flush=True)
def main():
p = argparse.ArgumentParser(description=__doc__)
p.add_argument("--base-url", default="http://127.0.0.1:8080")
p.add_argument("--output")
args = p.parse_args()
if args.output:
sys.stdout = open(ROOT / args.output, "w", encoding="utf-8", buffering=1)
sys.stderr = sys.stdout
os.environ.update(BASE_URL=args.base_url, NO_PROXY="localhost,127.0.0.1", no_proxy="localhost,127.0.0.1")
print("UTC " + datetime.now(timezone.utc).isoformat(), flush=True)
print("COMMAND python scripts/verify.py --base-url " + args.base_url, flush=True)
command(["docker", "version", "--format", "Client={{.Client.Version}} Server={{.Server.Version}}"])
command(["docker", "info", "--format", "Docker CPUs={{.NCPU}} memory_bytes={{.MemTotal}} OS={{.OperatingSystem}}"])
command(COMPOSE + ["config", "--quiet"])
command([sys.executable, "scripts/check_contract.py"])
command([sys.executable, "tests/smoke.py", "--base-url", args.base_url])
command([sys.executable, "-m", "unittest", "discover", "-s", "student-tests", "-v"])
c = Client(args.base_url, timeout=5)
a, b, _, chat = c.bootstrap()
seen = wait_replicas(c, a)
print("OBSERVED_INSTANCES " + json.dumps(sorted(seen)), flush=True)
first = c.message(a, chat, "survives restart", key="restart-key")["json"]
second = c.message(b, chat, "cursor after restart")["json"]
path = "/api/v1/chats/" + chat["id"] + "/messages"
page = c.call("GET", path + "?limit=1", actor=a, schema="MessagePage")["json"]
counts = "SELECT (SELECT count(*) FROM users),(SELECT count(*) FROM chats),(SELECT count(*) FROM memberships),(SELECT count(*) FROM messages),(SELECT count(*) FROM sessions),(SELECT count(*) FROM message_keys),(SELECT count(*) FROM schema_migrations),(SELECT count(*) FROM attachments)"
before_counts = sql(counts)
for _ in range(2):
command(COMPOSE + ["run", "--rm", "--no-deps", "migrate"])
assert sql(counts) == before_counts
print("PASS: repeated migrations preserve all table counts " + before_counts, flush=True)
before = state()
print("BEFORE_RESTART " + json.dumps(before), flush=True)
hook = "scripts/restart.cmd" if os.name == "nt" else "scripts/restart.sh"
command([sys.executable, "tests/resilience.py", "--mode", "persistence", "--action-script", hook, "--base-url", args.base_url])
after = state()
print("AFTER_RESTART " + json.dumps(after), flush=True)
for service in before:
assert before[service]["started_at"] != after[service]["started_at"]
assert before[service]["volumes"] == after[service]["volumes"]
assert c.message(a, chat, "survives restart", key="restart-key")["json"] == first
rest = c.call("GET", path + "?limit=1&cursor=" + quote(page["next_cursor"]), actor=a, schema="MessagePage")["json"]
assert page["items"] == [first] and rest["items"] == [second]
print("PASS: five containers restarted; volume, sessions, cursor and idempotency result preserved", flush=True)
verify_upload_failover(c, a, b, chat)
verify_s3_failure(c, a, b, chat)
stopped = set()
hook = "scripts/stop-one.cmd" if os.name == "nt" else "scripts/stop-one.sh"
# Run the unchanged public test until it has selected each real replica.
# Read concurrently with its POST stream to also measure read availability.
for attempt in range(6):
if stopped == {"api1", "api2"}:
break
command(COMPOSE + ["up", "-d", "--wait", "api1", "api2"], True)
wait_replicas(c, a)
done, reads = threading.Event(), {"ok": 0, "errors": 0}
def read_during_failure():
reader = Client(args.base_url, timeout=2)
while not done.is_set():
try:
result = reader.call("GET", path, actor=b, schema="MessagePage")["json"]
assert first in result["items"]
reads["ok"] += 1
except Failure:
reads["errors"] += 1
done.wait(0.1)
worker = threading.Thread(target=read_during_failure)
worker.start()
try:
print("FAILOVER_RUN " + str(attempt + 1), flush=True)
command([sys.executable, "tests/resilience.py", "--mode", "failover", "--action-script", hook, "--base-url", args.base_url])
current = state()
down = {s for s in ("api1", "api2") if current[s]["status"] == "exited"}
assert len(down) == 1
stopped.update(down)
print("AFTER_FAILOVER " + json.dumps(current), flush=True)
assert reads["ok"] >= 5
print("CONCURRENT_READS " + json.dumps(reads), flush=True)
assert c.message(a, chat, "survives restart", key="restart-key")["json"] == first
finally:
done.set()
worker.join(timeout=5)
command(COMPOSE + ["up", "-d", "--wait", "api1", "api2"], True)
assert stopped == {"api1", "api2"}, "Both replicas must be tested"
wait_replicas(c, a)
assert sql("SELECT count(*) FROM chats c WHERE NOT EXISTS (SELECT 1 FROM memberships m WHERE m.chat_id=c.id)") == "0"
print("PASS: each API stopped separately; reads/writes and original session survive; both replicas restored; S3 bytes checked", flush=True)
print("PASS: grade-3 verification complete", flush=True)
if __name__ == "__main__":
main()
+1
View File
@@ -0,0 +1 @@
"""Lab 1: a single-process, in-memory messenger."""
+110
View File
@@ -0,0 +1,110 @@
"""Synchronous bounded binary upload and membership-authorized download."""
import hashlib
import io
import os
import threading
import unicodedata
from uuid import uuid4
from PIL import Image, UnidentifiedImageError
from .common import APIError, invalid, record, string_value, uuid_value
from .objects import Objects
MAX_BYTES = min(10485760, int(os.environ.get("MAX_UPLOAD_BYTES", "10485760")))
MAX_PIXELS = min(40000000, int(os.environ.get("MAX_IMAGE_PIXELS", "40000000")))
# Pillow's hard error triggers at twice this limit, before pixel allocation.
# The exact contract limit is also checked below before load().
Image.MAX_IMAGE_PIXELS = MAX_PIXELS
def filename(query):
if set(query) != {"filename"} or len(query["filename"]) != 1:
raise invalid()
value = string_value(query["filename"][0], 255)
if any(c in "/\\" or unicodedata.category(c) == "Cc" for c in value):
raise invalid()
return value
def image_type(data, declared):
if declared not in ("image/png", "image/jpeg"):
raise APIError(415, "unsupported_media_type", "Expected PNG or JPEG")
try:
with Image.open(io.BytesIO(data), formats=("PNG", "JPEG")) as image:
if image.width * image.height > MAX_PIXELS:
raise APIError(413, "payload_too_large", "Image exceeds pixel limit")
actual = Image.MIME[image.format]
if actual != declared:
raise APIError(415, "unsupported_media_type", "Image MIME mismatch")
image.verify()
# verify() alone does not decode JPEG scan data. Reject truncation too.
with Image.open(io.BytesIO(data), formats=("PNG", "JPEG")) as image:
image.load()
return actual
except Image.DecompressionBombError:
raise APIError(413, "payload_too_large", "Image exceeds pixel limit") from None
except (UnidentifiedImageError, OSError, ValueError, SyntaxError):
raise APIError(415, "unsupported_media_type", "Invalid PNG or JPEG") from None
class Attachments:
def init_attachments(self):
self.objects = Objects()
# One decode per replica bounds aggregate pixel memory at this grade.
self.upload_slot = threading.BoundedSemaphore(1)
def upload(self, chat_id, actor, query, declared, read_bytes):
self.check_chat(chat_id, actor)
name = filename(query)
if not self.upload_slot.acquire(timeout=1):
raise APIError(503, "unavailable", "Upload capacity exhausted")
try:
data = read_bytes(MAX_BYTES)
content_type = image_type(data, declared)
attachment_id = uuid4()
key = "originals/" + uuid4().hex
digest = hashlib.sha256(data).hexdigest()
self.objects.put(key, data, content_type)
# No 201 until both PUT and PostgreSQL COMMIT have completed.
# No DELETE on ambiguous COMMIT: it could remove an acknowledged original.
with self.pool.connection() as conn:
row = conn.execute(
"INSERT INTO attachments(id,chat_id,owner_id,filename,content_type,size_bytes,sha256,object_key) "
"VALUES (%s,%s,%s,%s,%s,%s,%s,%s) RETURNING *",
(attachment_id, uuid_value(chat_id), actor[0], name, content_type, len(data), digest, key),
).fetchone()
return 201, self.attachment_json(row)
finally:
self.upload_slot.release()
def attachment(self, conn, value, actor):
row = conn.execute(
"SELECT a.* FROM attachments a WHERE a.id=%s AND EXISTS "
"(SELECT 1 FROM memberships m WHERE m.chat_id=a.chat_id AND m.user_id=%s)",
(uuid_value(value), actor[0]),
).fetchone()
if row is None:
raise APIError(404, "not_found", "Object not found")
return row
@staticmethod
def attachment_json(row):
return dict(record({k: v for k, v in row.items() if k != "object_key"}),
status="ready", variants=[], error_code=None)
def get_attachment(self, value, actor, query, content=False):
with self.pool.connection() as conn:
row = self.attachment(conn, value, actor)
if not content:
if query:
raise invalid()
return 200, self.attachment_json(row)
if set(query) - {"variant"} or any(len(v) != 1 for v in query.values()):
raise invalid()
if query.get("variant", ["original"])[0] != "original":
raise APIError(404, "not_found", "Variant not found")
data = self.objects.get(row["object_key"], row["size_bytes"])
if hashlib.sha256(data).hexdigest() != row["sha256"]:
raise APIError(503, "unavailable", "Original unavailable")
return 200, (data, row["content_type"])
+32
View File
@@ -0,0 +1,32 @@
"""Idempotently create a private bucket. Uses locally generated lab credentials."""
import os
import time
from botocore.exceptions import BotoCoreError, ClientError
from .objects import client
def main():
s3 = client()
bucket = os.environ.get("S3_BUCKET", "messenger-attachments")
for attempt in range(60):
try:
s3.list_buckets()
break
except (BotoCoreError, ClientError):
if attempt == 59:
raise RuntimeError("S3 did not become available") from None
time.sleep(1)
try:
s3.head_bucket(Bucket=bucket)
except ClientError as exc:
if exc.response["Error"]["Code"] not in ("404", "NoSuchBucket", "NotFound"):
raise RuntimeError("Cannot inspect bucket") from None
s3.create_bucket(Bucket=bucket)
# S3 config has authenticated identities only; anonymous access is absent.
s3.put_bucket_acl(Bucket=bucket, ACL="private")
print("PASS: private bucket ready; authenticated access only")
if __name__ == "__main__":
main()
+50
View File
@@ -0,0 +1,50 @@
from datetime import datetime
from uuid import UUID
import re
class APIError(Exception):
def __init__(self, status, code, message):
self.status, self.code, self.message = status, code, message
def invalid():
return APIError(400, "invalid_request", "Invalid request")
def uuid_value(value):
if not isinstance(value, str) or not re.fullmatch(
r"[0-9a-fA-F]{8}(?:-[0-9a-fA-F]{4}){3}-[0-9a-fA-F]{12}", value
):
raise invalid()
return str(UUID(value))
def string_value(value, maximum):
if (not isinstance(value, str) or not 1 <= len(value) <= maximum
or not value.strip() or "\x00" in value or any(0xD800 <= ord(c) <= 0xDFFF for c in value)):
raise invalid()
return value
def fields(body, expected):
if not isinstance(body, dict) or set(body) != set(expected):
raise invalid()
def unique_object(pairs):
result = {}
for key, value in pairs:
if key in result:
raise invalid()
result[key] = value
return result
def reject_constant(value):
raise invalid()
def record(row):
return {k: (str(v) if isinstance(v, UUID) else
v.isoformat(timespec="microseconds").replace("+00:00", "Z")
if isinstance(v, datetime) else v) for k, v in row.items()}
+44
View File
@@ -0,0 +1,44 @@
"""PostgreSQL configuration and transactional, versioned migrations."""
import hashlib
import os
from pathlib import Path
import secrets
import psycopg
from psycopg.rows import dict_row
from psycopg_pool import ConnectionPool
def database_url():
filename = os.environ.get("DATABASE_URL_FILE")
return Path(filename).read_text().strip() if filename else os.environ["DATABASE_URL"]
def migrate():
with psycopg.connect(database_url(), connect_timeout=3) as conn:
conn.execute("SELECT pg_advisory_xact_lock(2026002)")
conn.execute("CREATE TABLE IF NOT EXISTS schema_migrations "
"(version text PRIMARY KEY, checksum text NOT NULL)")
for path in sorted((Path(__file__).resolve().parents[1] / "migrations").glob("*.sql")):
sql = path.read_text(encoding="utf-8")
checksum = hashlib.sha256(sql.encode()).hexdigest()
old = conn.execute("SELECT checksum FROM schema_migrations WHERE version=%s", (path.name,)).fetchone()
if old:
if old[0] != checksum:
raise RuntimeError("Applied migration checksum changed")
continue
conn.execute(sql)
conn.execute("INSERT INTO schema_migrations VALUES (%s,%s)", (path.name, checksum))
conn.execute("INSERT INTO settings VALUES ('cursor_key',%s) ON CONFLICT DO NOTHING", (secrets.token_hex(32),))
def pool():
return ConnectionPool(database_url(), min_size=1, max_size=8, timeout=2,
max_waiting=32, open=True, check=ConnectionPool.check_connection,
kwargs={"connect_timeout": 3, "row_factory": dict_row,
"options": "-c timezone=UTC -c statement_timeout=3000 -c lock_timeout=2000"})
if __name__ == "__main__":
migrate()
print("PASS: migrations applied")
+18
View File
@@ -0,0 +1,18 @@
"""Container readiness probe; exits nonzero on failure."""
import json
import os
from urllib.request import ProxyHandler, build_opener
def main():
host = os.environ.get("APP_HOST", "0.0.0.0")
if host == "0.0.0.0":
host = "127.0.0.1"
port = int(os.environ.get("APP_PORT", "8080"))
with build_opener(ProxyHandler({})).open(f"http://{host}:{port}/health/ready", timeout=2) as response:
if response.status != 200 or json.load(response) != {"status": "ok"}:
raise SystemExit(1)
if __name__ == "__main__":
main()
+56
View File
@@ -0,0 +1,56 @@
"""Private S3 originals; no local files and no credentials in public metadata."""
import os
from pathlib import Path
import boto3
from botocore.config import Config
from botocore.exceptions import BotoCoreError, ClientError
from .common import APIError
def secret(name):
return Path(os.environ[name + "_FILE"]).read_text(encoding="utf-8").strip()
def client():
return boto3.client(
"s3", endpoint_url=os.environ.get("S3_ENDPOINT", "http://s3:8333"),
region_name=os.environ.get("S3_REGION", "us-east-1"),
aws_access_key_id=secret("S3_ACCESS_KEY_ID"),
aws_secret_access_key=secret("S3_SECRET_ACCESS_KEY"),
config=Config(connect_timeout=1, read_timeout=3,
retries={"total_max_attempts": 1},
s3={"addressing_style": "path"}),
)
class Objects:
def __init__(self):
self.client = client()
self.bucket = os.environ.get("S3_BUCKET", "messenger-attachments")
def call(self, method, **kwargs):
try:
return getattr(self.client, method)(Bucket=self.bucket, **kwargs)
except (BotoCoreError, ClientError):
raise APIError(503, "unavailable", "Object storage unavailable") from None
def ready(self):
self.call("head_bucket")
def put(self, key, data, content_type):
self.call("put_object", Key=key, Body=data, ContentType=content_type)
def get(self, key, expected_size):
response = self.call("get_object", Key=key)
try:
# Bounded even if stored object has been replaced out of band.
data = response["Body"].read(expected_size + 1)
if len(data) != expected_size:
raise APIError(503, "unavailable", "Original unavailable")
return data
except BotoCoreError:
raise APIError(503, "unavailable", "Object storage unavailable") from None
finally:
response["Body"].close()
+38
View File
@@ -0,0 +1,38 @@
"""Signed, resource-bound keyset cursors, inherited from lab1."""
import base64
import binascii
import hmac
import json
import re
from .common import invalid
class Pagination:
def cursor(self, scope, record):
payload = json.dumps([scope, record["created_at"], record["id"]], separators=(",", ":")).encode()
signature = hmac.digest(self.cursor_key, payload, "sha256")
return base64.urlsafe_b64encode(signature + payload).decode().rstrip("=")
def decode_page(self, query, scope):
if set(query) - {"limit", "cursor"} or any(len(v) != 1 for v in query.values()):
raise invalid()
limit = query.get("limit", ["50"])[0]
if not re.fullmatch(r"[0-9]{1,3}", limit) or not 1 <= int(limit) <= 100:
raise invalid()
limit = int(limit)
after = None
if "cursor" in query:
token = query["cursor"][0]
if not re.fullmatch(r"[A-Za-z0-9_-]{1,1024}", token):
raise invalid()
try:
raw = base64.b64decode(token + "=" * (-len(token) % 4), altchars=b"-_", validate=True)
signature, payload = raw[:32], raw[32:]
if not hmac.compare_digest(signature, hmac.digest(self.cursor_key, payload, "sha256")):
raise invalid()
resource, created, record_id = json.loads(payload)
if resource != scope:
raise invalid()
after = (created, record_id)
except (ValueError, UnicodeError, binascii.Error):
raise invalid() from None
return limit, after
+209
View File
@@ -0,0 +1,209 @@
"""Stateless lab4 HTTP replica. Migrations run as a separate Compose job."""
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import logging
import os
import re
import signal
import threading
import time
from urllib.parse import parse_qs, urlsplit
from uuid import uuid4
import psycopg
from .common import APIError, invalid, unique_object, reject_constant
from .store import Store
LOG = logging.getLogger("messenger")
MAX_BODY = 64 * 1024
SHUTDOWN_SECONDS = 5
class Server(ThreadingHTTPServer):
daemon_threads = True
request_queue_size = 128
def __init__(self, address):
self.instance_id = os.environ.get("INSTANCE_ID", "api") + "-" + uuid4().hex[:12]
self.store = Store()
self.stopping = threading.Event()
self.workers = set()
self.workers_lock = threading.Lock()
super().__init__(address, Handler)
self.timeout = 0.2
def process_request_thread(self, request, client_address):
worker = threading.current_thread()
with self.workers_lock:
self.workers.add(worker)
try:
super().process_request_thread(request, client_address)
finally:
with self.workers_lock:
self.workers.discard(worker)
def handle_error(self, request, client_address):
LOG.error("connection_handler_failed")
class Handler(BaseHTTPRequestHandler):
server_version = "Messenger"
sys_version = ""
def setup(self):
super().setup()
self.connection.settimeout(10)
def log_message(self, format, *args):
# Base handler includes raw paths and possibly user-supplied text.
pass
def send_error(self, code, message=None, explain=None):
self.request_id = getattr(self, "request_id", str(uuid4()))
self.reply(400, {"error": {"code": "invalid_request", "message": "Invalid HTTP request", "request_id": self.request_id}})
def read_body(self):
if self.headers.get_content_type() != "application/json":
raise APIError(415, "unsupported_media_type", "Expected application/json")
lengths = self.headers.get_all("Content-Length", [])
if self.headers.get("Transfer-Encoding") or len(lengths) != 1 or not re.fullmatch(r"[0-9]{1,10}", lengths[0]):
raise invalid()
size = int(lengths[0])
if size > MAX_BODY:
raise APIError(413, "payload_too_large", "Request exceeds 65536 bytes")
try:
raw = self.rfile.read(size)
if len(raw) != size:
raise invalid()
return json.loads(raw.decode("utf-8"), object_pairs_hook=unique_object, parse_constant=reject_constant)
except (ValueError, UnicodeError, RecursionError, TimeoutError):
raise invalid() from None
def read_binary(self, maximum):
lengths = self.headers.get_all("Content-Length", [])
if self.headers.get("Transfer-Encoding") or len(lengths) != 1 or not re.fullmatch(r"[0-9]{1,20}", lengths[0]):
raise invalid()
size = int(lengths[0])
if size > maximum:
raise APIError(413, "payload_too_large", "Upload exceeds byte limit")
if size == 0:
raise APIError(415, "unsupported_media_type", "Empty image")
try:
data = self.rfile.read(size)
if len(data) != size:
raise invalid()
return data
except (TimeoutError, OSError):
raise invalid() from None
def reply(self, status, body):
binary = isinstance(body, tuple)
encoded = body[0] if binary else b"" if status == 204 else json.dumps(body, ensure_ascii=False, separators=(",", ":")).encode("utf-8")
self.send_response(status)
self.send_header("Content-Type", body[1] if binary else "application/json; charset=utf-8")
self.send_header("X-Content-Type-Options", "nosniff")
self.send_header("Cache-Control", "no-store")
self.send_header("Content-Length", str(len(encoded)))
self.send_header("X-Request-Id", self.request_id)
self.send_header("X-Instance-Id", self.server.instance_id)
if status == 401:
challenge = 'Basic realm="messenger", charset="UTF-8"' if urlsplit(self.path).path == "/api/v1/auth/sessions" else 'Bearer realm="messenger"'
self.send_header("WWW-Authenticate", challenge)
self.send_header("Connection", "close")
self.end_headers()
self.close_connection = True
self.wfile.write(encoded)
def handle_api(self):
started = time.monotonic()
self.request_id = str(uuid4())
status = 500
try:
server = self.server
assert isinstance(server, Server)
if server.stopping.is_set():
raise APIError(503, "unavailable", "Server is stopping")
parsed = urlsplit(self.path)
path = parsed.path
if path in ("/health/live", "/health/ready") and self.command == "GET":
if path == "/health/ready":
server.store.ready()
status, body = 200, {"status": "ok"}
else:
store = server.store
public = (path, self.command) in (("/api/v1/users", "POST"), ("/api/v1/auth/sessions", "POST"))
actor = None if public else store.authenticate(self.headers.get("Authorization"))
query = parse_qs(parsed.query, keep_blank_values=True, max_num_fields=10, errors="strict")
upload = re.fullmatch(r"/api/v1/chats/([^/]+)/attachments", path)
attachment = re.fullmatch(r"/api/v1/attachments/([^/]+)(/content)?", path)
cached_body = None
if upload and self.command == "POST":
status, body = store.upload(upload[1], actor, query, self.headers.get_content_type(), self.read_binary)
elif attachment and self.command == "GET":
status, body = store.get_attachment(attachment[1], actor, query, bool(attachment[2]))
elif path == "/api/v1/auth/sessions" and self.command == "POST":
if self.headers.get("Transfer-Encoding") or self.headers.get("Content-Length", "0") != "0":
raise invalid()
status, body = store.login(self.headers.get("Authorization"))
else:
if self.command == "POST":
match = re.fullmatch(r"/api/v1/chats/([^/]+)/messages", path)
if match:
store.check_chat(match[1], actor)
cached_body = self.read_body()
keys = self.headers.get_all("Idempotency-Key", [])
key = keys[0] if len(keys) == 1 else None
status, body = store.dispatch(self.command, path, query, actor, lambda: cached_body, key)
except APIError as exc:
status = exc.status
body = {"error": {"code": exc.code, "message": exc.message, "request_id": self.request_id}}
except (psycopg.OperationalError, psycopg.InterfaceError):
status = 503
body = {"error": {"code": "unavailable", "message": "Storage unavailable", "request_id": self.request_id}}
except ValueError:
status = 400
body = {"error": {"code": "invalid_request", "message": "Invalid request", "request_id": self.request_id}}
except Exception as exc:
LOG.error("request_failed request_id=%s exception=%s", self.request_id, type(exc).__name__)
body = {"error": {"code": "internal_error", "message": "Internal error", "request_id": self.request_id}}
try:
self.reply(status, body)
except OSError:
LOG.info("client_disconnected request_id=%s", self.request_id)
finally:
LOG.info("request request_id=%s method=%s status=%d duration_ms=%.3f", self.request_id, self.command, status, (time.monotonic() - started) * 1000)
do_GET = do_POST = do_PUT = do_PATCH = do_DELETE = do_OPTIONS = handle_api
def main():
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
server = Server((os.environ.get("APP_HOST", "0.0.0.0"), int(os.environ.get("APP_PORT", "8080"))))
def stop(signum, frame):
server.stopping.set()
signal.signal(signal.SIGTERM, stop)
signal.signal(signal.SIGINT, stop)
LOG.info("started host=%s port=%d env=%s", *server.server_address, os.environ.get("APP_ENV", "lab"))
try:
while not server.stopping.is_set():
server.handle_request()
finally:
server.stopping.set()
server.server_close()
LOG.info("shutdown_started listener_closed=true budget_seconds=%d", SHUTDOWN_SECONDS)
deadline = time.monotonic() + SHUTDOWN_SECONDS
with server.workers_lock:
workers = list(server.workers)
for worker in workers:
worker.join(max(0, deadline - time.monotonic()))
server.store.pool.close()
LOG.info("shutdown_complete unfinished_requests=%d", sum(w.is_alive() for w in workers))
if __name__ == "__main__":
main()
+210
View File
@@ -0,0 +1,210 @@
"""Shared PostgreSQL state and atomic idempotent message creation."""
import base64
import hashlib
import hmac
import os
import re
import secrets
from uuid import uuid4
import psycopg
from .common import APIError, fields, invalid, string_value, uuid_value
from .database import pool
from .pagination import Pagination
from .attachments import Attachments
from .common import record
def password_hash(password, salt=None):
salt = salt or secrets.token_hex(16)
derived = hashlib.scrypt(password.encode("utf-8"), salt=bytes.fromhex(salt),
n=16384, r=8, p=1, dklen=32)
return "scrypt$16384$8$1$" + salt + "$" + derived.hex()
def unauthorized():
return APIError(401, "unauthorized", "Valid credentials required")
class Store(Attachments, Pagination):
def __init__(self):
self.pool = pool()
self.init_attachments()
self.ttl = int(os.environ.get("SESSION_TTL_SECONDS", "3600"))
if self.ttl <= 0:
raise ValueError("SESSION_TTL_SECONDS must be positive")
with self.pool.connection() as conn:
self.cursor_key = bytes.fromhex(conn.execute("SELECT value FROM settings WHERE key='cursor_key'").fetchone()["value"])
self.dummy_hash = password_hash("dummy-password-not-a-user")
def ready(self):
with self.pool.connection() as conn:
if not conn.execute("SELECT 1 FROM schema_migrations WHERE version='003_attachments.sql'").fetchone():
raise APIError(503, "unavailable", "Schema not ready")
conn.execute("SELECT 1 FROM sessions LIMIT 1")
conn.execute("SELECT 1 FROM message_keys LIMIT 1")
self.objects.ready()
def authenticate(self, header):
if not isinstance(header, str) or not re.fullmatch(r"(?i:Bearer) [A-Za-z0-9_-]{43}", header):
raise unauthorized()
digest = hashlib.sha256(header.split(" ", 1)[1].encode()).hexdigest()
with self.pool.connection() as conn:
row = conn.execute("SELECT user_id FROM sessions WHERE token_hash=%s AND expires_at>clock_timestamp()", (digest,)).fetchone()
if not row:
raise unauthorized()
return (str(row["user_id"]), digest)
def login(self, header):
try:
scheme, encoded = header.split(" ", 1)
if scheme.lower() != "basic":
raise ValueError()
username, password = base64.b64decode(encoded, validate=True).decode("utf-8").split(":", 1)
if not re.fullmatch(r"[a-z0-9_]{3,32}", username) or not 12 <= len(password) <= 128:
raise ValueError()
except (AttributeError, ValueError, UnicodeError):
raise unauthorized() from None
with self.pool.connection() as conn:
row = conn.execute("SELECT * FROM users WHERE username=%s", (username,)).fetchone()
expected = row["password_hash"] if row else self.dummy_hash
actual = password_hash(password, expected.split("$")[4])
if not hmac.compare_digest(actual, expected) or row is None:
raise unauthorized()
del row["password_hash"]
token = secrets.token_urlsafe(32)
digest = hashlib.sha256(token.encode()).hexdigest()
with self.pool.connection() as conn:
expiry = conn.execute("INSERT INTO sessions VALUES (%s,%s,clock_timestamp()+%s*interval '1 second') RETURNING expires_at", (digest, row["id"], self.ttl)).fetchone()
return 201, {"token": token, "token_type": "Bearer", "expires_at": record(expiry)["expires_at"], "user": record(row)}
def chat(self, conn, value, actor, lock=False):
chat_id = uuid_value(value)
row = conn.execute("SELECT c.* FROM chats c WHERE c.id=%s AND EXISTS "
"(SELECT 1 FROM memberships m WHERE m.chat_id=c.id AND m.user_id=%s)" +
(" FOR UPDATE OF c" if lock else ""), (chat_id, actor[0])).fetchone()
if not row:
raise APIError(404, "not_found", "Object not found")
row = record(row)
row["member_ids"] = [str(m["user_id"]) for m in conn.execute("SELECT user_id FROM memberships WHERE chat_id=%s ORDER BY ordinal", (chat_id,))]
return row
def check_chat(self, value, actor):
with self.pool.connection() as conn:
self.chat(conn, value, actor)
def page(self, conn, query, scope, table, actor=None, chat_id=None):
limit, after = self.decode_page(query, scope)
# Table and predicate are selected internally, never supplied by a client.
if table == "messages":
sql = "SELECT id,chat_id,sender_id,text,created_at,attachment_ids FROM messages WHERE chat_id=%s"
params = [chat_id]
else:
sql = "SELECT c.* FROM chats c WHERE EXISTS (SELECT 1 FROM memberships m WHERE m.chat_id=c.id AND m.user_id=%s)"
params = [actor[0]]
if after:
sql += " AND (created_at,id)>(%s::timestamptz,%s::uuid)"
params.extend(after)
sql += " ORDER BY created_at,id LIMIT %s"
params.append(limit + 1)
rows = [record(r) for r in conn.execute(sql, params)]
items = rows[:limit]
if table == "chats":
items = [self.chat(conn, r["id"], actor) for r in items]
return {"items": items, "next_cursor": self.cursor(scope, items[-1]) if len(rows) > limit else None}
def dispatch(self, method, path, query, actor, read_body, idempotency_key=None):
if path == "/api/v1/users" and method == "POST":
body = read_body()
fields(body, ("username", "display_name", "password"))
username = body["username"]
if not isinstance(username, str) or not re.fullmatch(r"[a-z0-9_]{3,32}", username):
raise invalid()
display = string_value(body["display_name"], 100)
password = string_value(body["password"], 128)
if len(password) < 12:
raise invalid()
hashed = password_hash(password)
try:
with self.pool.connection() as conn:
row = conn.execute("INSERT INTO users (id,username,display_name,password_hash) VALUES (%s,%s,%s,%s) RETURNING id,username,display_name,created_at", (uuid4(), username, display, hashed)).fetchone()
return 201, record(row)
except psycopg.errors.UniqueViolation:
raise APIError(409, "conflict", "Username already exists") from None
with self.pool.connection() as conn:
if path == "/api/v1/auth/sessions/current" and method == "DELETE":
deleted = conn.execute("DELETE FROM sessions WHERE token_hash=%s RETURNING user_id", (actor[1],)).fetchone()
if not deleted:
raise unauthorized()
return 204, None
if path == "/api/v1/users/me" and method == "GET":
return 200, record(conn.execute("SELECT id,username,display_name,created_at FROM users WHERE id=%s", (actor[0],)).fetchone())
if path == "/api/v1/chats":
if method == "GET":
return 200, self.page(conn, query, "chats:" + actor[0], "chats", actor=actor)
if method == "POST":
body = read_body()
fields(body, ("title", "member_ids"))
title = string_value(body["title"], 100)
members = body["member_ids"]
if not isinstance(members, list) or not 1 <= len(members) <= 100:
raise invalid()
members = [uuid_value(m) for m in members]
if len(set(members)) != len(members) or actor[0] not in members:
raise invalid()
row = conn.execute("INSERT INTO chats(id,title) VALUES (%s,%s) RETURNING *", (uuid4(), title)).fetchone()
existing = conn.execute("SELECT id FROM users WHERE id=ANY(%s::uuid[])", (members,)).fetchall()
if len(existing) != len(members):
raise invalid() # Rolls back the chat INSERT as well.
with conn.cursor() as cur:
cur.executemany("INSERT INTO memberships VALUES (%s,%s,%s)", [(row["id"], m, i) for i, m in enumerate(members)])
return 201, dict(record(row), member_ids=members)
match = re.fullmatch(r"/api/v1/chats/([^/]+)(/messages)?", path)
if match:
chat = self.chat(conn, match[1], actor, lock=method == "POST")
if match[2] is None and method == "GET":
return 200, chat
if match[2] == "/messages":
if method == "GET":
return 200, self.page(conn, query, "messages:" + chat["id"], "messages", chat_id=chat["id"])
if method == "POST":
if not isinstance(idempotency_key, str) or not re.fullmatch(r"[A-Za-z0-9._:-]{1,128}", idempotency_key):
raise invalid()
body = read_body()
fields(body, ("text", "attachment_ids") if isinstance(body, dict) and "attachment_ids" in body else ("text",))
ids = body.get("attachment_ids", [])
if not isinstance(ids, list) or len(ids) > 10:
raise invalid()
ids = [uuid_value(value) for value in ids]
if len(set(ids)) != len(ids):
raise invalid()
text = body["text"]
if not isinstance(text, str) or len(text) > 4000 or "\x00" in text or any(0xD800 <= ord(c) <= 0xDFFF for c in text):
raise invalid()
if not text.strip() and not ids:
raise invalid()
for value in ids:
attachment = self.attachment(conn, value, actor)
if str(attachment["chat_id"]) != chat["id"]:
raise APIError(404, "not_found", "Object not found")
# The chat row is already locked across ALL API replicas.
# READ COMMITTED sees the preceding writer's committed key.
conn.execute("DELETE FROM message_keys WHERE user_id=%s AND chat_id=%s AND key=%s AND expires_at<=clock_timestamp()",
(actor[0], chat["id"], idempotency_key))
previous = conn.execute("SELECT m.* FROM message_keys k JOIN messages m ON m.id=k.message_id "
"WHERE k.user_id=%s AND k.chat_id=%s AND k.key=%s",
(actor[0], chat["id"], idempotency_key)).fetchone()
if previous:
if previous["text"] != text or previous["attachment_ids"] != ids:
raise APIError(409, "conflict", "Key already used for another message")
return 201, record(previous)
row = conn.execute("INSERT INTO messages(id,chat_id,sender_id,text,created_at,attachment_ids) VALUES (%s,%s,%s,%s,GREATEST(clock_timestamp(), "
"COALESCE((SELECT created_at+interval '1 microsecond' FROM messages "
"WHERE chat_id=%s ORDER BY created_at DESC,id DESC LIMIT 1),clock_timestamp())),%s::jsonb) RETURNING *",
(uuid4(), chat["id"], actor[0], text, chat["id"], psycopg.types.json.Jsonb(ids))).fetchone()
conn.execute("INSERT INTO message_keys(user_id,chat_id,key,message_id) VALUES (%s,%s,%s,%s)",
(actor[0], chat["id"], idempotency_key, row["id"]))
return 201, record(row)
raise APIError(404, "not_found", "Object not found")
Binary file not shown.

After

Width:  |  Height:  |  Size: 637 B

+107
View File
@@ -0,0 +1,107 @@
"""Own HTTP checks, separate from the unchanged public tests."""
import base64
from concurrent.futures import ThreadPoolExecutor
import os
from pathlib import Path
import sys
import unittest
from urllib.parse import quote
from uuid import uuid4
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tests"))
from client import Client
class API(unittest.TestCase):
def setUp(self):
self.c = Client(os.environ.get("BASE_URL", "http://127.0.0.1:8080"))
self.a, self.b, self.e, self.chat = self.c.bootstrap()
self.path = "/api/v1/chats/" + self.chat["id"]
def test_registration_validation(self):
body = dict(self.a["registration"], username="u_" + uuid4().hex[:16])
for password in ("short", "x" * 129, 123, None):
self.c.call("POST", "/api/v1/users", 400, body=dict(body, password=password))
self.c.call("POST", "/api/v1/users", 400, body=dict(body, extra=True))
def test_unicode_password_with_colon_and_max_length(self):
username = "u_" + uuid4().hex[:16]
password = "Я:🔐" + "б" * 125
body = {"username": username, "display_name": "Имя 👋", "password": password}
user = self.c.call("POST", "/api/v1/users", 201, body=body, schema="User")["json"]
credential = base64.b64encode((username + ":" + password).encode()).decode()
result = self.c.call("POST", "/api/v1/auth/sessions", 201, headers={"Authorization": "Basic " + credential}, schema="Session")["json"]
self.assertEqual(user, result["user"])
self.assertGreaterEqual(len(base64.urlsafe_b64decode(result["token"] + "=")), 32)
def test_bad_credentials_and_schemes(self):
for credential in ("Basic ***", "Bearer token", "", "Basic " + base64.b64encode(b"missing:wrong-password").decode()):
self.c.call("POST", "/api/v1/auth/sessions", 401, headers={"Authorization": credential})
self.c.call("GET", "/api/v1/users/me", 401, headers={"X-User-Id": self.a["id"]})
def test_logout_only_revokes_current_session(self):
body = self.a["registration"]
credential = base64.b64encode((body["username"] + ":" + body["password"]).encode()).decode()
other = self.c.call("POST", "/api/v1/auth/sessions", 201, headers={"Authorization": "Basic " + credential}, schema="Session")["json"]
self.c.call("DELETE", "/api/v1/auth/sessions/current", 204, actor=self.a)
self.c.call("GET", "/api/v1/users/me", 401, actor=self.a)
self.c.call("GET", "/api/v1/users/me", actor={"token": other["token"]}, schema="User")
def test_chat_atomic_invalid_members(self):
before = self.c.call("GET", "/api/v1/chats?limit=100", actor=self.a, schema="ChatPage")["json"]
for members in ([self.a["id"], str(uuid4())], [self.a["id"], self.a["id"]], [self.b["id"]]):
self.c.call("POST", "/api/v1/chats", 400, actor=self.a, body={"title": "rollback", "member_ids": members})
after = self.c.call("GET", "/api/v1/chats?limit=100", actor=self.a, schema="ChatPage")["json"]
self.assertEqual(before, after)
def test_username_race(self):
body = dict(self.a["registration"], username="race_" + uuid4().hex[:16])
def register(_):
return Client(self.c.base).call("POST", "/api/v1/users", [201, 409], body=body)["status"]
with ThreadPoolExecutor(max_workers=6) as pool:
statuses = list(pool.map(register, range(6)))
self.assertEqual(statuses.count(201), 1)
self.assertEqual(statuses.count(409), 5)
def test_concurrent_messages_and_keyset(self):
def send(i):
return Client(self.c.base).message(self.a, self.chat, "Поток " + str(i))["json"]
with ThreadPoolExecutor(max_workers=8) as pool:
sent = list(pool.map(send, range(40)))
items, cursor = [], None
while True:
page = self.c.call("GET", self.path + "/messages?limit=7" + ("&cursor=" + quote(cursor) if cursor else ""), actor=self.b, schema="MessagePage")["json"]
items.extend(page["items"])
cursor = page["next_cursor"]
if cursor is None:
break
self.assertEqual(len(items), 40)
self.assertEqual({x["id"] for x in items}, {x["id"] for x in sent})
self.assertEqual(items, sorted(sent, key=lambda x: (x["created_at"], x["id"])))
def test_cursor_scope_and_tampering(self):
for i in range(3):
self.c.message(self.a, self.chat, str(i))
page = self.c.call("GET", self.path + "/messages?limit=1", actor=self.a, schema="MessagePage")["json"]
cursor = page["next_cursor"]
self.c.call("GET", "/api/v1/chats?cursor=" + quote(cursor), 400, actor=self.a)
other = self.c.call("POST", "/api/v1/chats", 201, actor=self.a, body={"title": "Другой", "member_ids": [self.a["id"]]}, schema="Chat")["json"]
self.c.call("GET", "/api/v1/chats/" + other["id"] + "/messages?cursor=" + quote(cursor), 400, actor=self.a)
self.c.call("GET", self.path + "/messages?cursor=" + quote("X" + cursor[1:]), 400, actor=self.a)
def test_json_and_sender_validation(self):
for raw in (b'{"text":"a","text":"b"}', b'{"text":NaN}', b'{', b'{"text":"\\u0000"}'):
self.c.call("POST", self.path + "/messages", 400, actor=self.a, raw=raw, headers={"Content-Type": "application/json", "Idempotency-Key": uuid4().hex})
self.c.call("POST", self.path + "/messages", 400, actor=self.a, body={"text": "hello", "sender_id": self.b["id"]})
self.c.call("POST", self.path + "/messages", 400, actor=self.a, body={"text": " "})
page = self.c.call("GET", self.path + "/messages", actor=self.a, schema="MessagePage")["json"]
self.assertEqual(page["items"], [])
def test_access_before_payload_validation(self):
self.c.call("POST", self.path + "/messages", 404, actor=self.e, body={})
self.c.call("POST", self.path + "/messages", 401, body={})
self.c.call("GET", self.path, 404, actor=self.e)
if __name__ == "__main__":
unittest.main()
+135
View File
@@ -0,0 +1,135 @@
"""Lab4 binary API checks without additional dependencies on the test host."""
import hashlib
import http.client
import os
from pathlib import Path
import struct
import sys
import unittest
from urllib.parse import quote, urlsplit
from uuid import uuid4
import zlib
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tests"))
from client import Client
def png(width, height, pixels=None):
def chunk(name, data):
return struct.pack('!I', len(data)) + name + data + struct.pack('!I', zlib.crc32(name + data))
header = struct.pack('!IIBBBBB', width, height, 8, 2, 0, 0, 0)
return (b'\x89PNG\r\n\x1a\n' + chunk(b'IHDR', header)
+ chunk(b'IDAT', zlib.compress(pixels or b'\0\xff\0\0')) + chunk(b'IEND', b''))
class Attachments(unittest.TestCase):
def setUp(self):
self.c = Client(os.environ.get('BASE_URL', 'http://127.0.0.1:8080'))
self.a, self.b, self.e, self.chat = self.c.bootstrap()
self.path = '/api/v1/chats/' + self.chat['id']
self.data = (Path(__file__).resolve().parents[1] / 'tests/fixtures/sample.png').read_bytes()
def upload(self, data=None, mime='image/png', name='фото.png', expected=201, actor=None):
return self.c.call('POST', self.path + '/attachments?filename=' + quote(name, safe=''),
expected, actor=actor or self.a, raw=self.data if data is None else data,
headers={'Content-Type': mime}, schema='Attachment' if expected == 201 else None)
def test_original_and_message_shared_with_member(self):
att = self.upload()['json']
self.assertEqual(att['sha256'], hashlib.sha256(self.data).hexdigest())
self.assertEqual(att['size_bytes'], len(self.data))
self.assertEqual(att['variants'], [])
base = '/api/v1/attachments/' + att['id']
self.assertEqual(self.c.call('GET', base, actor=self.b, schema='Attachment')['json'], att)
result = self.c.call('GET', base + '/content?variant=original', actor=self.b)
self.assertEqual(result['bytes'], self.data)
self.assertEqual(result['headers'].get_content_type(), 'image/png')
self.c.call('GET', base + '/content?variant=thumbnail', 404, actor=self.b)
msg = self.c.message(self.b, self.chat, '', attachments=[att['id']])['json']
self.assertEqual(self.c.call('GET', self.path + '/messages', actor=self.a,
schema='MessagePage')['json']['items'], [msg])
def test_jpeg_original_and_truncation(self):
data = (Path(__file__).parent / 'fixtures/sample.jpg').read_bytes()
att = self.upload(data=data, mime='image/jpeg', name='photo.jpg')['json']
result = self.c.call('GET', '/api/v1/attachments/' + att['id'] + '/content', actor=self.b)
self.assertEqual(result['bytes'], data)
self.assertEqual(result['headers'].get_content_type(), 'image/jpeg')
self.upload(data=data[:-30], mime='image/jpeg', expected=415)
def test_acl_before_image_validation(self):
self.upload(data=b'broken', expected=404, actor=self.e)
att = self.upload()['json']
for suffix in ('', '/content'):
self.c.call('GET', '/api/v1/attachments/' + att['id'] + suffix, 404, actor=self.e)
self.c.call('GET', '/api/v1/attachments/' + att['id'] + suffix, 401)
def test_filename(self):
for name in ('../photo.png', 'a\\b.png', 'a\x00.png', 'a\n.png', 'a\x7f.png', '', 'x' * 256):
self.upload(name=name, expected=400)
def test_image_validation(self):
self.upload(mime='image/jpeg', expected=415)
self.upload(mime='application/octet-stream', expected=415)
for data in (b'', b'not an image', self.data[:30], self.data[:-20]):
self.upload(data=data, expected=415)
# Header announces 40,010,000 px, with valid CRC; reject before decoding.
self.upload(data=png(10000, 4001), expected=413)
def test_maximum_byte_size_accepted(self):
padding = b'x' * (10485760 - len(self.data) - 12)
chunk = struct.pack('!I', len(padding)) + b'vpAD' + padding + struct.pack('!I', zlib.crc32(b'vpAD' + padding))
data = self.data[:33] + chunk + self.data[33:]
self.assertEqual(len(data), 10485760)
att = self.upload(data=data)['json']
self.assertEqual(att['size_bytes'], 10485760)
result = self.c.call('GET', '/api/v1/attachments/' + att['id'] + '/content', actor=self.b)
self.assertEqual(result['bytes'], data)
def test_exactly_40_million_pixels_accepted(self):
def chunk(name, data):
return struct.pack('!I', len(data)) + name + data + struct.pack('!I', zlib.crc32(name + data))
# Valid 1-bit grayscale PNG, 8000*5000 pixels. Small compressed fixture.
data = (b'\x89PNG\r\n\x1a\n' + chunk(b'IHDR', struct.pack('!IIBBBBB', 8000, 5000, 1, 0, 0, 0, 0))
+ chunk(b'IDAT', zlib.compress(b'\0' * (1001 * 5000))) + chunk(b'IEND', b''))
att = self.upload(data=data)['json']
self.assertEqual(att['sha256'], hashlib.sha256(data).hexdigest())
def test_byte_limit_before_reading_body(self):
url = urlsplit(self.c.base)
conn = http.client.HTTPConnection(url.hostname, url.port, timeout=5)
try:
conn.putrequest('POST', self.path + '/attachments?filename=too-big.png')
conn.putheader('Authorization', 'Bearer ' + self.a['token'])
conn.putheader('Content-Type', 'image/png')
conn.putheader('Content-Length', str(10485761))
conn.endheaders() # No body: server must reject using the header alone.
response = conn.getresponse()
self.assertEqual(response.status, 413)
self.assertTrue(response.getheader('X-Request-Id'))
response.read()
finally:
conn.close()
def test_attachment_scope_and_idempotency(self):
att = self.upload()['json']
other = self.c.call('POST', '/api/v1/chats', 201, actor=self.a,
body={'title': 'other', 'member_ids': [self.a['id']]}, schema='Chat')['json']
for ids in ([att['id']], [str(uuid4())]):
self.c.call('POST', '/api/v1/chats/' + other['id'] + '/messages', 404, actor=self.a,
body={'text': '', 'attachment_ids': ids}, headers={'Idempotency-Key': uuid4().hex})
for ids in ([att['id'], att['id']], 'wrong', [None], [att['id']] * 11):
self.c.call('POST', self.path + '/messages', 400, actor=self.a,
body={'text': '', 'attachment_ids': ids}, headers={'Idempotency-Key': uuid4().hex})
key = uuid4().hex
msg = self.c.message(self.a, self.chat, '', key=key, attachments=[att['id']])['json']
self.assertEqual(self.c.message(self.a, self.chat, '', key=key, attachments=[att['id']])['json'], msg)
self.c.call('POST', self.path + '/messages', 409, actor=self.a,
body={'text': 'changed', 'attachment_ids': []}, headers={'Idempotency-Key': key})
key = uuid4().hex
msg = self.c.message(self.a, self.chat, 'text', key=key)['json']
self.assertEqual(self.c.message(self.a, self.chat, 'text', key=key, attachments=[])['json'], msg)
if __name__ == '__main__':
unittest.main()
+68
View File
@@ -0,0 +1,68 @@
"""Real HTTP races through the LB, with one shared session and key."""
from concurrent.futures import ThreadPoolExecutor
import os
from pathlib import Path
import sys
import threading
import unittest
from uuid import uuid4
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tests"))
from client import Client
class Idempotency(unittest.TestCase):
def setUp(self):
self.c = Client(os.environ.get("BASE_URL", "http://127.0.0.1:8080"))
self.a, self.b, self.e, self.chat = self.c.bootstrap()
self.path = "/api/v1/chats/" + self.chat["id"] + "/messages"
def test_same_key_concurrent_across_replicas(self):
key, barrier = uuid4().hex, threading.Barrier(12)
def send(_):
c = Client(self.c.base)
barrier.wait(timeout=10)
return c.message(self.a, self.chat, "Один результат 👋", key=key)
with ThreadPoolExecutor(max_workers=12) as pool:
results = list(pool.map(send, range(12)))
self.assertGreaterEqual(len({r["headers"]["X-Instance-Id"] for r in results}), 2)
self.assertTrue(all(r["json"] == results[0]["json"] for r in results))
history = self.c.call("GET", self.path, actor=self.b, schema="MessagePage")["json"]
self.assertEqual(history["items"], [results[0]["json"]])
print("PASS: concurrent_retries=12 observed_replicas=2 committed_messages=1", flush=True)
def test_conflicting_bodies_race(self):
key, barrier = uuid4().hex, threading.Barrier(2)
def send(text):
c = Client(self.c.base)
barrier.wait(timeout=10)
return c.call("POST", self.path, [201, 409], actor=self.a,
body={"text": text}, headers={"Idempotency-Key": key})
with ThreadPoolExecutor(max_workers=2) as pool:
results = list(pool.map(send, ["first", "second"]))
self.assertEqual(sorted(r["status"] for r in results), [201, 409])
history = self.c.call("GET", self.path, actor=self.a, schema="MessagePage")["json"]
self.assertEqual(len(history["items"]), 1)
def test_key_scope_and_access(self):
key = uuid4().hex
one = self.c.message(self.a, self.chat, "first", key=key)["json"]
two = self.c.message(self.b, self.chat, "second", key=key)["json"]
other = self.c.call("POST", "/api/v1/chats", 201, actor=self.a,
body={"title": "other", "member_ids": [self.a["id"]]}, schema="Chat")["json"]
three = self.c.message(self.a, other, "third", key=key)["json"]
self.assertEqual(len({m["id"] for m in (one, two, three)}), 3)
self.c.call("POST", self.path, 404, actor=self.e, body={"text": "first"}, headers={"Idempotency-Key": key})
self.c.call("DELETE", "/api/v1/auth/sessions/current", 204, actor=self.a)
self.c.call("POST", self.path, 401, actor=self.a, body={"text": "first"}, headers={"Idempotency-Key": key})
def test_invalid_keys_and_no_cached_error(self):
for key in ("", "a" * 129, "has space", "bad/key", "bad,key"):
self.c.call("POST", self.path, 400, actor=self.a, body={"text": "hello"}, headers={"Idempotency-Key": key})
key = uuid4().hex
self.c.call("POST", self.path, 400, actor=self.a, body={"text": " "}, headers={"Idempotency-Key": key})
self.c.message(self.a, self.chat, "valid after rejected body", key=key)
if __name__ == "__main__":
unittest.main()