ЛР 4 — БПИ2302 — Шурман — на 4 #120

Open
aliska wants to merge 3 commits from aliska/mtusi2026lab4-fork:solution/bpi2302-shurman into main
42 changed files with 1789 additions and 23 deletions
+11
View File
@@ -0,0 +1,11 @@
FROM python:3.12.7-slim-bookworm
WORKDIR /srv
COPY app/requirements.txt app/requirements.txt
RUN pip install --no-cache-dir -r app/requirements.txt && useradd --uid 10001 --create-home app
COPY app app
COPY migrations migrations
USER 10001:10001
ENV PYTHONUNBUFFERED=1 PYTHONDONTWRITEBYTECODE=1
EXPOSE 8080
HEALTHCHECK --interval=5s --timeout=4s --start-period=20s CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8080/health/ready', timeout=3)" || exit 1
CMD ["python", "app/server.py"]
+161 -23
View File
@@ -1,37 +1,175 @@
# Отчёт — лабораторная 4
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
Автор: Шурман А.Р., БПИ2302. Целевая оценка: **4**. Исследовательские треки на 5 не выполнялись.
## Запуск из чистого clone
## Статус проверки
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
Полный прогон выполнен **12 ноября 2026 года** на macOS arm64 / Docker Desktop.
`make check`, полная валидация OpenAPI, 5 unit-тестов и весь `scripts/verify.py`
прошли. Реальные результаты находятся в `evidence/`: smoke, негативные сценарии,
приватность bucket, безопасная очистка, обрыв upload, отказ S3, повторный bootstrap,
persistence, failover и нагрузка. Контракт и публичные тесты не изменены.
Первоначальный `evidence/offline.txt` сохранён как историческая проверка до запуска Docker.
## Архитектура и решения
## Запуск
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab3, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
Нужны Docker с Compose, Python 3.9+ на хосте (для memory/unit — Pillow).
API работает на Python 3.12.7. Новый стенд использует порт **8084**, отдельные volumes
и Compose project `mtusi-lab4`, поэтому не занимает порт предыдущих работ.
## Доказательства критериев
```sh
cd mtusi2026lab4-fork
python3 -m venv .venv
.venv/bin/pip install -r requirements-dev.txt Pillow==11.1.0
python3 scripts/bootstrap.py
docker compose config --quiet
docker compose up -d --build --wait
BASE_URL=http://localhost:8084 .venv/bin/python scripts/verify.py
make validate PYTHON=.venv/bin/python
```
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
Команды выполняются из корня этой лабораторной. Если репозиторий склонирован отдельно,
первую команду замените на переход в его каталог. `bootstrap.py` генерирует локальные
секреты, не выводит и не перезаписывает их. Повторный запуск сохраняет данные.
Для локальных тестов при отсутствии Pillow: отдельное venv и `pip install Pillow==11.1.0`.
Полная проверка OpenAPI: `pip install -r requirements-dev.txt`, затем `make validate`.
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
API: http://localhost:8084; Swagger: http://localhost:8084/docs.
Остановка без потери данных: `docker compose down` (без `-v`).
Возобновление: `docker compose up -d --wait`.
## Эксперименты и отказы
Готовые образы `minio/minio` и `minio/mc` оказались недоступны при проверке
12 ноября 2026 года (Docker Hub: pull access denied; Quay: 401).
Поэтому `infra/s3/Dockerfile` собирает официальные исходники закреплённых релизов
[MinIO RELEASE.2025-04-22T22-12-26Z](https://github.com/minio/minio/tree/RELEASE.2025-04-22T22-12-26Z)
и [mc RELEASE.2025-04-16T18-13-26Z](https://github.com/minio/mc/tree/RELEASE.2025-04-16T18-13-26Z)
в Go 1.24.2. Лицензии включены в образы. Первый запуск требует интернет и время
на скачивание Go-зависимостей и компиляцию; следующие сборки используют Docker cache.
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
## Архитектура
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
HAProxy → две FastAPI-реплики → общий PostgreSQL и приватный MinIO.
Valkey кэширует публичные профили; сессии проверяются в PostgreSQL при каждом запросе.
Исходная основа — собственная лабораторная 2: модели пользователей/чатов,
аутентификация, SQL-хранилище и мигратор. Возможности лабораторной 3 добавлены здесь:
балансировка, `X-Instance-Id`, общая таблица идемпотентности.
Контракт и публичные тесты skeleton не изменены.
## Ограничения и следующие шаги
Каждая реплика возвращает `api1` или `api2`. HAProxy проверяет readiness каждую секунду,
исключает реплику после двух неудач. Он не повторяет POST автоматически. При потере ответа
клиент повторяет сообщение с тем же `Idempotency-Key`. Advisory lock в PostgreSQL
сериализует один ключ пользователя/чата, вставка сообщения и результата атомарны.
Повтор в течение 24 часов возвращает исходный объект; другой текст/список вложений — 409.
Отсутствующий список вложений и `[]` равны. Просроченная запись ключа заменяется при повторном использовании.
История читается из primary без кэша. Keyset cursor не обеспечивает snapshot между страницами.
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
Метаданные Attachment находятся в PostgreSQL, байты — только в S3. Временный файл API
существует исключительно во время приёма и проверки запроса и закрывается по выходу.
Имя пользователя не используется как S3 key. Ключ — `originals/<UUID>`.
Скачивание идёт через API после проверки членства, без redirect и публичных URL.
## Валидация и ресурсы
Загрузка принимает сырые PNG/JPEG, максимум 10 MiB. Content-Length проверяется заранее,
но фактические байты также считаются во время потока. Общий срок приёма — 30 секунд.
Pillow проверяет формат, MIME, структуру и декодирование. Размер не выше 40 млн пикселей
проверяется до загрузки пикселей. Анимированные PNG в этой реализации отклоняются.
Оригинал сохраняется без преобразования; SHA-256 и размер вычисляются при чтении.
Пустые/опасные filename, MIME-подмена и чужие вложения имеют ошибки контракта.
Одновременно каждая реплика обрабатывает максимум две загрузки. Ожидание слота до 0.5 s,
затем явный 503. Две реплики ограничены по 768 MiB и 1 CPU. На диск пишется максимум
10 MiB на активную загрузку, декодирование занимает ограниченную память до размера
изображения. S3: connect timeout 2 s, read timeout 10 s, автоматических повторов нет.
Download читает S3 чанками по 64 KiB. Оригинал не загружается целиком в RAM при скачивании.
## Согласованность S3 и БД
1. API принимает и проверяет временный файл.
2. Отдельная транзакция сохраняет `pending` и назначенный object key до S3 PUT.
3. Новая транзакция блокирует строку, выполняет PUT и переводит её в `ready`.
4. Ответ 201 отправляется только после COMMIT. `pending` не показывается публично.
При обрыве передачи клиента до PUT остаётся только автоматически закрываемый временный
файл. При сбое PUT, падении процесса или неуспешном COMMIT остаётся учтённый `pending`.
Фоновый cleaner раз в минуту выбирает pending старше **1 часа**, блокируя строку
`FOR UPDATE SKIP LOCKED`, удаляет соответствующий S3 key и затем запись БД. Активная
транзакция пропускается. При ошибке S3 строка сохраняется для следующего прохода.
При сбое COMMIT очистка повторит безопасный DELETE отсутствующего объекта.
`ready` никогда не выбирается. Часовой grace period значительно больше таймаутов загрузки.
Это не распределённая транзакция: некоторое время возможны pending-метаданные и лишние байты.
Объекты, вручную записанные администратором без intent, вне модели очистки.
## Приватность и запуск хранилища
MinIO, PostgreSQL, Valkey и API не публикуют порты на хост. Только HAProxy доступен
на loopback. Консоль MinIO слушает 127.0.0.1 внутри его контейнера.
Идемпотентный `s3-init` создаёт bucket, запрещает anonymous и назначает отдельному API user
политику: ListBucket одного bucket и Get/Put/Delete исключительно в `originals/*`.
Root credentials доступны только MinIO и init, API их не получает.
Readiness зависит от PostgreSQL и MinIO. Valkey имеет fallback в PostgreSQL.
Секреты находятся в исключённом из Git каталоге `secrets/`.
| Роль / service | Версия | Готовность | Постоянные данные | Доступ / credentials |
|---|---|---|---|---|
| LB / `lb` | HAProxy 3.0.8-alpine | HTTP `/health/ready` обеих API | нет | только `127.0.0.1:8084` |
| API / `api1`, `api2` | Python 3.12.7; `app/requirements.txt` | PostgreSQL + MinIO | нет | внутренняя сеть; DB/S3 через secrets |
| БД / `postgres` | PostgreSQL 16.6-bookworm | `pg_isready` | `postgres-data` | внутренняя сеть; password secret |
| Кэш / `valkey` | Valkey 8.0.2-alpine | некритичен, fallback в БД | `valkey-data` (AOF) | внутренняя сеть |
| S3 / `s3` | MinIO RELEASE.2025-04-22T22-12-26Z | `/minio/health/live` | `s3-data` | внутренняя сеть; root secrets |
| Bootstrap / `s3-init` | mc RELEASE.2025-04-16T18-13-26Z | успешное завершение | bucket/policy в S3 | root + API secrets |
| Миграции / `migrate` | образ API | успешное завершение | `schema_migrations` в БД | secrets, advisory lock, checksum миграций |
| Очистка / `cleaner` | образ API | повтор каждые 60 s | pending в БД | ограниченные API S3 credentials |
Все сервисы находятся в отдельной Compose-сети `mtusi-lab4_default`.
Повторный `bootstrap.py` сохраняет секреты; повторные `migrate` и `s3-init`
не удаляют данные и выполняются при `docker compose up`.
## Проверки и доказательства
| Критерий | Проверка | Результат 12.11.2026 |
|---|---|---|
| Структура и синтаксис | `make check`, AST всех новых Python-файлов | выполнено |
| Конфигурация | `docker compose config --quiet` | выполнено |
| Реальные PNG/JPEG, MIME, повреждение, пиксели | `python3 student-tests/unit.py` | 5 тестов пройдено |
| Обязательный API и checksum | `make test BASE_URL=http://localhost:8084` | PASS; вывод в `evidence/` |
| Ошибки файлов и ACL, конкуренция ключа | `BASE_URL=http://localhost:8084 python3 student-tests/verify.py` | PASS; вывод в `evidence/` |
| Приватность bucket и безопасная очистка | `student-tests/storage_checks.py` внутри API через общий runner | PASS; вывод в `evidence/` |
| Обрыв загрузки и отказ S3 | `student-tests/outage.py` | PASS; вывод в `evidence/` |
| Повторный bootstrap | повтор `docker compose run --rm s3-init` | PASS; вывод в `evidence/` |
| Перезапуск всех сервисов | persistence + `scripts/restart.sh` | PASS; вывод в `evidence/` |
| Остановка обслужившей реплики | failover + `scripts/stop-one.sh` | PASS; 89 записей, 3 неудачные попытки, восстановление 2.8 s |
| Остановка именно загрузившей реплики | `student-tests/attachment_failover.py` | PASS; upload api1 → download api2; bytes совпадают |
| 10 одновременных загрузок и RSS | `student-tests/memory.py` | 8 × 201, 2 × 503; пик RSS 132796 / 133912 KiB |
`.venv/bin/python scripts/verify.py` выполняет проверки и сохраняет реальные результаты в evidence,
останавливаясь при первом провале. Тест очистки создаёт только синтетические строки и объекты,
моделирует rollback финализации после PUT, проверяет grace period и сохранность ready.
Он удаляет только свои синтетические данные. До запуска убедитесь, что это учебный стенд.
Memory-тест создаёт PNG примерно 9 MiB и одновременно запускает 10 запросов.
Он пишет коды ответов, длительность, выборки VmRSS процессов API и измеренный максимум.
Отказы 503 из-за ограничения слотов учитываются явно; они не записываются как успешные загрузки.
Последовательная выборка `/proc/1/status` может пропустить короткий пик, поэтому результат —
максимум наблюдений, а не гарантированный абсолютный пик. Фактический эксперимент: 10 одновременных запросов по 9 201 866 байт,
8 ответов 201 и 2 ответа 503; время ответа 0.427–0.680 s. Наблюдаемый пик RSS:
api1 — 132 796 KiB (129.7 MiB), api2 — 133 912 KiB (130.8 MiB), при лимите
768 MiB на реплику. Полные выборки сохранены в `evidence/memory.json`.
Это проверка ограниченности ресурсов, а не обещание принять любую конкуренцию без 503.
## Ограничения
Один host, HAProxy, PostgreSQL и MinIO остаются точками отказа. Это отказоустойчивость
к остановке одной API-реплики, не полноценная HA-система. При отказе S3 readiness
становится 503; после восстановления новые обращения повторяются клиентом.
Обрыв S3 посреди уже начатого download закрывает поток, поэтому клиент проверяет длину/checksum.
Нет UI, обработки изображений в очереди, presigned URL и исследовательских треков.
Проверка реальных контейнеров выполнена; при переносе на другую машину её нужно повторить.
## Источники и использование ИИ
Основа — собственные предыдущие лабораторные и skeleton преподавателя;
API и критерии взяты из `contracts/` и `README.md`. MinIO/mc собираются из официальных
исходников по ссылкам выше. При проверке и исправлении работы использован Codex:
анализ кода, исправление сборки S3, дополнительные тесты, запуск стенда и оформление
фактических результатов. Результаты получены запуском, а не сгенерированы.
+113
View File
@@ -0,0 +1,113 @@
"""Binary upload, private download, and durable upload intents."""
import asyncio
import hashlib
import tempfile
import uuid
import unicodedata
from urllib.parse import quote
from fastapi import Depends, Request
from fastapi.responses import StreamingResponse
from starlette.concurrency import run_in_threadpool
import storage
# Two decoded images per replica at most; no unbounded waiting queue.
slots = asyncio.Semaphore(2)
def valid_filename(name):
return (isinstance(name, str) and 1 <= len(name) <= 255 and bool(name.strip())
and not any(c in '/\\' or unicodedata.category(c) == 'Cc' for c in name))
def install(app, pool, current_user, visible_chat, query, wire, ApiError):
def public(row):
row = wire(row)
row.pop('object_key', None)
return dict(row, variants=[], error_code=None)
def visible_attachment(aid, actor):
try:
aid = str(uuid.UUID(aid))
except ValueError:
raise ApiError(400, 'invalid_request', 'Invalid attachment id')
rows = query("""SELECT a.* FROM attachments a JOIN members m ON m.chat_id=a.chat_id
WHERE a.id=%s AND m.user_id=%s AND a.status='ready'""", (aid, actor['id']))
if not rows:
raise ApiError(404, 'not_found', 'Attachment not found')
return rows[0]
def save(file, chat_id, actor, name, mime, size, digest):
try:
storage.validate_image(file, mime)
except storage.TooManyPixels:
raise ApiError(413, 'payload_too_large', 'Too many image pixels')
except storage.InvalidImage:
raise ApiError(415, 'unsupported_media_type', 'Invalid PNG/JPEG or MIME mismatch')
aid = str(uuid.uuid4())
key = 'originals/' + aid
# Commit intent before PUT. Crash/uncertain PUT or COMMIT is recoverable.
query("""INSERT INTO attachments(id,chat_id,owner_id,filename,object_key,content_type,size_bytes,sha256,status)
VALUES (%s,%s,%s,%s,%s,%s,%s,%s,'pending') RETURNING id""",
(aid,chat_id,actor['id'],name,key,mime,size,digest))
with pool.connection() as conn:
# Cleaner skips this row while PUT and commit are in flight.
conn.execute('SELECT id FROM attachments WHERE id=%s FOR UPDATE', (aid,)).fetchone()
storage.client.put_object(Bucket=storage.BUCKET, Key=key, Body=file,
ContentLength=size, ContentType=mime)
row = conn.execute("UPDATE attachments SET status='ready', updated_at=clock_timestamp() WHERE id=%s RETURNING *", (aid,)).fetchone()
return public(row)
@app.post('/api/v1/chats/{chat_id}/attachments', status_code=201)
async def upload(chat_id: str, request: Request, actor=Depends(current_user)):
await run_in_threadpool(visible_chat, chat_id, actor['id'])
name = request.query_params.get('filename')
if set(request.query_params) != {'filename'} or not valid_filename(name):
raise ApiError(400, 'invalid_request', 'Invalid filename')
mime = request.headers.get('content-type', '').split(';')[0].strip().lower()
if mime not in ('image/png', 'image/jpeg'):
raise ApiError(415, 'unsupported_media_type', 'PNG or JPEG required')
length = request.headers.get('content-length')
if length is not None:
if not length.isdigit():
raise ApiError(400, 'invalid_request', 'Invalid Content-Length')
if int(length) > storage.MAX_BYTES:
raise ApiError(413, 'payload_too_large', 'File too large')
try:
await asyncio.wait_for(slots.acquire(), timeout=0.5)
except asyncio.TimeoutError:
raise ApiError(503, 'unavailable', 'Upload capacity reached')
try:
with tempfile.TemporaryFile() as file:
size = 0
digest = hashlib.sha256()
try:
async with asyncio.timeout(30):
async for chunk in request.stream():
size += len(chunk)
if size > storage.MAX_BYTES:
raise ApiError(413, 'payload_too_large', 'File too large')
digest.update(chunk)
await run_in_threadpool(file.write, chunk)
except TimeoutError:
raise ApiError(400, 'invalid_request', 'Upload timed out')
return await run_in_threadpool(save, file, chat_id, actor, name, mime, size, digest.hexdigest())
finally:
slots.release()
@app.get('/api/v1/attachments/{aid}')
def metadata(aid: str, actor=Depends(current_user)):
return public(visible_attachment(aid, actor))
@app.get('/api/v1/attachments/{aid}/content')
def content(aid: str, variant: str = 'original', actor=Depends(current_user)):
row = visible_attachment(aid, actor)
if variant != 'original':
raise ApiError(404, 'not_found', 'Variant not found')
obj = storage.client.get_object(Bucket=storage.BUCKET, Key=row['object_key'])
def chunks():
try:
while data := obj['Body'].read(64 * 1024):
yield data
finally:
obj['Body'].close()
return StreamingResponse(chunks(), media_type=row['content_type'], headers={
'Content-Length': str(row['size_bytes']), 'X-Content-Type-Options': 'nosniff',
'Content-Disposition': "attachment; filename*=UTF-8''" + quote(row['filename'], safe='')})
+34
View File
@@ -0,0 +1,34 @@
"""Remove abandoned upload intents after an hour; never remove ready objects."""
import os
import time
import logging
import psycopg
from psycopg.rows import dict_row
import storage
def sweep():
count = 0
# Small transactions, no unbounded listing of the bucket into memory.
for _ in range(100):
with psycopg.connect(storage.secret('DATABASE_URL'), row_factory=dict_row,
connect_timeout=3, options='-c statement_timeout=15000') as conn:
row = conn.execute("""SELECT id,object_key FROM attachments
WHERE status='pending' AND created_at < clock_timestamp()-interval '1 hour'
ORDER BY created_at FOR UPDATE SKIP LOCKED LIMIT 1""").fetchone()
if not row:
return count
storage.client.delete_object(Bucket=storage.BUCKET, Key=row['object_key'])
conn.execute("DELETE FROM attachments WHERE id=%s AND status='pending'", (row['id'],))
count += 1
return count
if __name__ == '__main__':
while True:
try:
print('Removed abandoned uploads:', sweep(), flush=True)
except Exception:
logging.error('Cleanup failed; will retry next cycle')
if os.environ.get('CLEANUP_ONCE') == '1':
break
time.sleep(60)
+38
View File
@@ -0,0 +1,38 @@
"""Image checks without network access."""
import warnings
from PIL import Image, UnidentifiedImageError
MAX_BYTES = 10_485_760
MAX_PIXELS = 40_000_000
Image.MAX_IMAGE_PIXELS = MAX_PIXELS
class InvalidImage(ValueError):
pass
class TooManyPixels(ValueError):
pass
def validate_image(file, mime):
try:
with warnings.catch_warnings():
warnings.simplefilter('error', Image.DecompressionBombWarning)
file.seek(0)
with Image.open(file) as img:
if img.width * img.height > MAX_PIXELS:
raise TooManyPixels()
actual = {'PNG': 'image/png', 'JPEG': 'image/jpeg'}.get(img.format)
if actual != mime or actual is None:
raise InvalidImage()
if getattr(img, 'n_frames', 1) != 1:
raise InvalidImage()
img.verify()
file.seek(0)
with Image.open(file) as img:
img.load() # Detect truncated JPEG/pixel data, not only its header.
except (Image.DecompressionBombError, Image.DecompressionBombWarning) as exc:
raise TooManyPixels() from exc
except (UnidentifiedImageError, OSError, SyntaxError, ValueError) as exc:
if isinstance(exc, TooManyPixels):
raise
raise InvalidImage() from exc
finally:
file.seek(0)
+21
View File
@@ -0,0 +1,21 @@
"""Atomic, serialized, versioned migrations; changed applied files are rejected."""
import hashlib
import os
from pathlib import Path
import psycopg
with psycopg.connect(Path(os.environ["DATABASE_URL_FILE"]).read_text().strip()) as conn:
conn.execute("SELECT pg_advisory_xact_lock(20260002)")
conn.execute("CREATE TABLE IF NOT EXISTS schema_migrations(version text PRIMARY KEY, checksum text NOT NULL)")
for path in sorted(Path("migrations").glob("*.sql")):
sql = path.read_text()
digest = hashlib.sha256(sql.encode()).hexdigest()
row = conn.execute("SELECT checksum FROM schema_migrations WHERE version=%s", (path.name,)).fetchone()
if row:
if row[0] != digest:
raise RuntimeError("Applied migration changed: " + path.name)
continue
conn.execute(sql)
conn.execute("INSERT INTO schema_migrations VALUES (%s,%s)", (path.name,digest))
print("Applied", path.name)
print("Migrations OK")
+28
View File
@@ -0,0 +1,28 @@
annotated-types==0.8.0
anyio==4.15.1
argon2-cffi==23.1.0
argon2-cffi-bindings==26.1.0
cffi==2.1.1
click==8.5.0
fastapi==0.115.6
h11==0.16.0
idna==3.20
psycopg==3.2.6
psycopg-binary==3.2.6
psycopg-pool==3.3.3
pycparser==3.12
pydantic==2.14.1
pydantic_core==2.50.1
redis==5.2.1
starlette==0.41.3
typing-inspection==0.4.4
typing_extensions==4.16.0
uvicorn==0.34.0
boto3==1.35.99
botocore==1.35.99
s3transfer==0.10.4
jmespath==1.0.1
python-dateutil==2.9.0.post0
six==1.17.0
urllib3==2.3.0
Pillow==11.1.0
+502
View File
@@ -0,0 +1,502 @@
#!/usr/bin/env python3
"""Lab 4 PostgreSQL messenger, adapted from the author's lab 1."""
from __future__ import annotations
import base64
import binascii
import json
import logging
import os
import re
import hashlib
import secrets
from pathlib import Path
from contextlib import asynccontextmanager
import psycopg
from psycopg.rows import dict_row
from psycopg_pool import ConnectionPool, PoolTimeout, TooManyRequests
import redis
from argon2 import PasswordHasher
from argon2.exceptions import VerificationError
import uuid
from datetime import datetime, timezone
from typing import Any, Optional, Tuple
import uvicorn
from fastapi import Depends, FastAPI, Header, Query, Request
from fastapi.exceptions import RequestValidationError
from fastapi.responses import JSONResponse, Response
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
from starlette.exceptions import HTTPException as StarletteHTTPException
USERNAME_RE = re.compile(r"^[a-z0-9_]{3,32}$")
LOG = logging.getLogger("messenger")
INSTANCE_ID = os.environ.get("INSTANCE_ID", str(uuid.uuid4()))
def env_int(name: str, default: int, minimum: int = 0) -> int:
try:
value = int(os.environ.get(name, str(default)))
except ValueError as exc:
raise SystemExit("%s must be an integer" % name) from exc
if value < minimum:
raise SystemExit("%s must be at least %s" % (name, minimum))
return value
class ApiError(Exception):
def __init__(self, status: int, code: str, message: str) -> None:
super().__init__(message)
self.status = status
self.code = code
self.message = message
class StrictModel(BaseModel):
model_config = ConfigDict(extra="forbid")
class Health(StrictModel):
status: str = Field(pattern="^ok$")
class User(StrictModel):
id: uuid.UUID
username: str = Field(pattern=r"^[a-z0-9_]{3,32}$")
display_name: str = Field(min_length=1, max_length=100)
created_at: datetime
class CreateUser(StrictModel):
password: str = Field(min_length=12, max_length=128)
@field_validator("password")
@classmethod
def non_blank_password(cls, value: str) -> str:
if not value.strip():
raise ValueError("password must not be blank")
return value
username: str = Field(pattern=r"^[a-z0-9_]{3,32}$")
display_name: str = Field(min_length=1, max_length=100)
@field_validator("display_name")
@classmethod
def non_blank_display_name(cls, value: str) -> str:
if not value.strip():
raise ValueError("display_name must not be blank")
return value
class Chat(StrictModel):
id: uuid.UUID
title: str = Field(min_length=1, max_length=100)
member_ids: list[uuid.UUID] = Field(min_length=1, max_length=100)
created_at: datetime
class CreateChat(StrictModel):
title: str = Field(min_length=1, max_length=100)
member_ids: list[uuid.UUID] = Field(min_length=1, max_length=100)
@field_validator("title")
@classmethod
def non_blank_title(cls, value: str) -> str:
if not value.strip():
raise ValueError("title must not be blank")
return value
@field_validator("member_ids")
@classmethod
def unique_members(cls, value: list[uuid.UUID]) -> list[uuid.UUID]:
if len(set(value)) != len(value):
raise ValueError("member_ids must be unique")
return value
class Message(StrictModel):
id: uuid.UUID
chat_id: uuid.UUID
sender_id: uuid.UUID
text: str = Field(max_length=4000)
attachment_ids: list[uuid.UUID] = Field(default_factory=list, max_length=10)
created_at: datetime
class CreateMessage(StrictModel):
text: str = Field(max_length=4000)
attachment_ids: list[uuid.UUID] = Field(default_factory=list, max_length=10)
@model_validator(mode="after")
def valid_message(self):
if not self.text.strip() and not self.attachment_ids:
raise ValueError("message must not be empty")
if len(set(self.attachment_ids)) != len(self.attachment_ids):
raise ValueError("duplicate attachment")
return self
class ChatPage(StrictModel):
items: list[Chat]
next_cursor: Optional[str]
class MessagePage(StrictModel):
items: list[Message]
next_cursor: Optional[str]
class ErrorDetails(StrictModel):
code: str
message: str
request_id: str
class ErrorResponse(StrictModel):
error: ErrorDetails
def valid_uuid(value: Any) -> bool:
if not isinstance(value, str):
return False
try:
return str(uuid.UUID(value)) == value.lower()
except (ValueError, AttributeError):
return False
def make_cursor(scope: str, item: dict[str, Any]) -> str:
raw = json.dumps(
{"v": 1, "scope": scope, "created_at": item["created_at"], "id": item["id"]},
separators=(",", ":"),
).encode("utf-8")
return base64.urlsafe_b64encode(raw).rstrip(b"=").decode("ascii")
def read_cursor(value: str, scope: str) -> Tuple[str, str]:
try:
raw = base64.urlsafe_b64decode(value + "=" * (-len(value) % 4))
data = json.loads(raw)
if set(data) != {"v", "scope", "created_at", "id"}:
raise ValueError
if data["v"] != 1 or data["scope"] != scope or not valid_uuid(data["id"]):
raise ValueError
datetime.fromisoformat(data["created_at"].replace("Z", "+00:00"))
return data["created_at"], data["id"]
except (ValueError, TypeError, AttributeError, UnicodeError, json.JSONDecodeError, binascii.Error):
raise ApiError(400, "invalid_request", "Invalid cursor") from None
def secret(name):
return Path(os.environ[name + "_FILE"]).read_text().strip()
POOL = ConnectionPool(secret("DATABASE_URL"), min_size=1, max_size=8, timeout=3,
max_waiting=32, open=False, check=ConnectionPool.check_connection,
kwargs={"row_factory": dict_row, "connect_timeout": 3,
"options": "-c statement_timeout=3000 -c lock_timeout=2000"})
CACHE = redis.Redis.from_url(secret("CACHE_URL"), socket_connect_timeout=0.2,
socket_timeout=0.2, decode_responses=True)
HASHER = PasswordHasher(time_cost=2, memory_cost=19456, parallelism=1)
DUMMY_HASH = HASHER.hash(secrets.token_urlsafe(32))
TTL = env_int("SESSION_TTL_SECONDS", 3600, 1)
@asynccontextmanager
async def lifespan(app):
POOL.open()
try:
yield
finally:
POOL.close()
CACHE.close()
app = FastAPI(title="MTUSI Messenger — Lab 4", version="4.0.0", lifespan=lifespan)
def wire(row):
return {k: (v.isoformat(timespec="microseconds").replace("+00:00", "Z")
if isinstance(v, datetime) else str(v) if isinstance(v, uuid.UUID) else v)
for k, v in row.items()}
def query(sql, params=()):
with POOL.connection() as conn:
return [wire(row) for row in conn.execute(sql, params).fetchall()]
def profile(user_id):
key = "user:" + user_id
try:
value = CACHE.get(key)
if value:
return json.loads(value)
except (redis.RedisError, ValueError):
pass
rows = query("SELECT id, username, display_name, created_at FROM users WHERE id=%s", (user_id,))
if not rows:
raise ApiError(401, "unauthorized", "Invalid session")
user = rows[0]
try:
CACHE.setex(key, 60, json.dumps(user))
except redis.RedisError:
pass
return user
def token_hash(token):
return hashlib.sha256(token.encode()).hexdigest()
def bearer(authorization):
if not authorization or not authorization.startswith("Bearer "):
raise ApiError(401, "unauthorized", "Bearer session required")
token = authorization[7:]
if not re.fullmatch(r"[A-Za-z0-9_-]{43}", token):
raise ApiError(401, "unauthorized", "Invalid session")
return token_hash(token)
def current_user(authorization: Optional[str] = Header(default=None)):
digest = bearer(authorization)
rows = query("SELECT user_id FROM sessions WHERE token_hash=%s AND expires_at > clock_timestamp()", (digest,))
if not rows:
raise ApiError(401, "unauthorized", "Invalid session")
return profile(rows[0]["user_id"])
def visible_chat(chat_id, actor_id):
if not valid_uuid(chat_id):
raise ApiError(400, "invalid_request", "Invalid chat id")
rows = query("SELECT c.* FROM chats c JOIN members m ON m.chat_id=c.id WHERE c.id=%s AND m.user_id=%s", (chat_id, actor_id))
if not rows:
raise ApiError(404, "not_found", "Chat not found")
return chat_members(rows)[0]
def chat_members(rows):
if rows:
members = query("SELECT chat_id, user_id FROM members WHERE chat_id=ANY(%s::uuid[]) ORDER BY user_id", ([r["id"] for r in rows],))
grouped = {r["id"]: [] for r in rows}
for m in members:
grouped[m["chat_id"]].append(m["user_id"])
for row in rows:
row["member_ids"] = grouped[row["id"]]
return rows
def page_args(request, scope, limit, cursor):
if set(request.query_params).difference({"limit", "cursor"}):
raise ApiError(400, "invalid_request", "Invalid query parameters")
return limit, read_cursor(cursor, scope) if cursor is not None else None
def page(rows, limit, scope):
return {"items": rows[:limit], "next_cursor": make_cursor(scope, rows[limit-1]) if len(rows)>limit else None}
def request_id(request: Request) -> str:
return getattr(request.state, "request_id", str(uuid.uuid4()))
def error_response(request: Request, status: int, code: str, message: str) -> JSONResponse:
rid = request_id(request)
response = JSONResponse(
status_code=status,
content={"error": {"code": code, "message": message, "request_id": rid}},
)
response.headers["X-Request-Id"] = rid
response.headers["X-Instance-Id"] = INSTANCE_ID
if status == 401:
response.headers["WWW-Authenticate"] = ('Basic realm="messenger", charset="UTF-8"'
if request.url.path == "/api/v1/auth/sessions" else 'Bearer realm="messenger"')
return response
@app.middleware("http")
async def request_envelope(request: Request, call_next: Any) -> JSONResponse:
request.state.request_id = str(uuid.uuid4())
try:
response = await call_next(request)
except Exception:
LOG.error("request_id=%s unexpected error", request_id(request))
response = error_response(request, 500, "internal_error", "Internal server error")
response.headers["X-Request-Id"] = request_id(request)
response.headers["X-Instance-Id"] = INSTANCE_ID
return response
@app.exception_handler(ApiError)
async def api_error_handler(request: Request, error: ApiError) -> JSONResponse:
return error_response(request, error.status, error.code, error.message)
@app.exception_handler(RequestValidationError)
async def validation_error_handler(request: Request, _error: RequestValidationError) -> JSONResponse:
return error_response(request, 400, "invalid_request", "Request validation failed")
@app.exception_handler(StarletteHTTPException)
async def http_error_handler(request: Request, error: StarletteHTTPException) -> JSONResponse:
if error.status_code == 404:
return error_response(request, 404, "not_found", "Resource not found")
return error_response(request, error.status_code, "invalid_request", "Request failed")
async def require_json(request: Request) -> None:
content_type = request.headers.get("content-type", "").split(";", 1)[0].strip().lower()
if content_type != "application/json":
raise ApiError(415, "unsupported_media_type", "Content-Type must be application/json")
@app.exception_handler(psycopg.Error)
@app.exception_handler(PoolTimeout)
@app.exception_handler(TooManyRequests)
async def database_error(request, error):
return error_response(request, 503, "unavailable", "Database temporarily unavailable")
@app.get("/health/live", response_model=Health)
def health_live():
return {"status": "ok"}
@app.get("/health/ready", response_model=Health)
def health_ready():
query("SELECT 1 FROM schema_migrations LIMIT 1")
storage.client.head_bucket(Bucket=storage.BUCKET)
return {"status": "ok"}
@app.post("/api/v1/users", response_model=User, status_code=201, dependencies=[Depends(require_json)])
def create_user(payload: CreateUser):
password_hash = HASHER.hash(payload.password)
try:
return query("INSERT INTO users (id,username,display_name,password_hash) VALUES (%s,%s,%s,%s) RETURNING id,username,display_name,created_at",
(uuid.uuid4(), payload.username, payload.display_name, password_hash))[0]
except psycopg.errors.UniqueViolation:
raise ApiError(409, "conflict", "Username already exists") from None
@app.post("/api/v1/auth/sessions", status_code=201)
async def login(request: Request, authorization: Optional[str] = Header(default=None)):
# CPU/SQL work runs in FastAPI's worker pool, not in the event loop.
from starlette.concurrency import run_in_threadpool
if await request.body():
raise ApiError(400, "invalid_request", "Session request must not have a body")
return await run_in_threadpool(login_sync, authorization)
def login_sync(authorization):
try:
if not authorization or not authorization.startswith("Basic "):
raise ValueError
username, password = base64.b64decode(authorization[6:], validate=True).decode("utf-8").split(":", 1)
if not USERNAME_RE.fullmatch(username) or not 12 <= len(password) <= 128:
raise ValueError
except (ValueError, UnicodeError, binascii.Error):
raise ApiError(401, "unauthorized", "Invalid credentials") from None
users = query("SELECT * FROM users WHERE username=%s", (username,))
try:
HASHER.verify(users[0]["password_hash"] if users else DUMMY_HASH, password)
except VerificationError:
raise ApiError(401, "unauthorized", "Invalid credentials") from None
if not users:
raise ApiError(401, "unauthorized", "Invalid credentials")
token = secrets.token_urlsafe(32)
expiry = query("INSERT INTO sessions(token_hash,user_id,expires_at) VALUES (%s,%s,clock_timestamp() + %s * interval '1 second') RETURNING expires_at",
(token_hash(token), users[0]["id"], TTL))[0]["expires_at"]
user = users[0]
del user["password_hash"]
return {"token": token, "token_type": "Bearer", "expires_at": expiry, "user": user}
@app.delete("/api/v1/auth/sessions/current", status_code=204)
def logout(actor=Depends(current_user), authorization: Optional[str] = Header(default=None)):
query("DELETE FROM sessions WHERE token_hash=%s RETURNING user_id", (bearer(authorization),))
return Response(status_code=204)
@app.get("/api/v1/users/me", response_model=User)
def get_me(actor=Depends(current_user)):
return actor
@app.post("/api/v1/chats", response_model=Chat, status_code=201, dependencies=[Depends(require_json)])
def create_chat(payload: CreateChat, actor=Depends(current_user)):
members = [str(m) for m in payload.member_ids]
if actor["id"] not in members:
raise ApiError(400, "invalid_request", "The current user must be a member")
try:
with POOL.connection() as conn:
chat = wire(conn.execute("INSERT INTO chats(id,title) VALUES (%s,%s) RETURNING *", (uuid.uuid4(),payload.title)).fetchone())
with conn.cursor() as cur:
cur.executemany("INSERT INTO members(chat_id,user_id) VALUES (%s,%s)", [(chat["id"],m) for m in members])
chat["member_ids"] = members
return chat
except psycopg.errors.ForeignKeyViolation:
raise ApiError(400, "invalid_request", "All chat members must exist") from None
@app.get("/api/v1/chats", response_model=ChatPage)
def list_chats(request: Request, limit: int = Query(50,ge=1,le=100), cursor: Optional[str] = Query(None,min_length=1), actor=Depends(current_user)):
scope = "chats:" + actor["id"]
limit, after = page_args(request,scope,limit,cursor)
sql = "SELECT c.* FROM chats c JOIN members m ON c.id=m.chat_id WHERE m.user_id=%s"
params = [actor["id"]]
if after:
sql += " AND (c.created_at,c.id) > (%s,%s)"
params.extend(after)
rows = query(sql + " ORDER BY c.created_at,c.id LIMIT %s", params + [limit+1])
return page(chat_members(rows),limit,scope)
@app.get("/api/v1/chats/{chat_id}", response_model=Chat)
def get_chat(chat_id: str, actor=Depends(current_user)):
return visible_chat(chat_id,actor["id"])
@app.post("/api/v1/chats/{chat_id}/messages", response_model=Message, status_code=201, dependencies=[Depends(require_json)])
def create_message(chat_id: str, payload: CreateMessage, actor=Depends(current_user),
idempotency_key: Optional[str] = Header(default=None)):
visible_chat(chat_id, actor["id"])
if not idempotency_key or not re.fullmatch(r"[A-Za-z0-9._:-]{1,128}", idempotency_key):
raise ApiError(400, "invalid_request", "Invalid Idempotency-Key")
ids = [str(i) for i in payload.attachment_ids]
body = json.dumps({"text": payload.text, "attachment_ids": ids}, sort_keys=True)
scope = actor["id"] + ":" + chat_id + ":" + idempotency_key
lock = int.from_bytes(hashlib.sha256(scope.encode()).digest()[:8], "big", signed=True)
with POOL.connection() as conn:
conn.execute("SELECT pg_advisory_xact_lock(%s)", (lock,))
previous = conn.execute("SELECT * FROM idempotency WHERE user_id=%s AND chat_id=%s AND key=%s AND expires_at>clock_timestamp()",
(actor["id"], chat_id, idempotency_key)).fetchone()
if previous:
if previous["body"] != body:
raise ApiError(409, "conflict", "Key already used for a different message")
return previous["response"]
if ids:
found = conn.execute("SELECT id FROM attachments WHERE id=ANY(%s::uuid[]) AND chat_id=%s AND status='ready'", (ids, chat_id)).fetchall()
if len(found) != len(ids):
raise ApiError(404, "not_found", "Attachment not found")
msg = wire(conn.execute("INSERT INTO messages(id,chat_id,sender_id,text,attachment_ids) VALUES (%s,%s,%s,%s,%s::uuid[]) RETURNING *",
(uuid.uuid4(),chat_id,actor["id"],payload.text,ids)).fetchone())
msg["attachment_ids"] = ids
from psycopg.types.json import Jsonb
conn.execute("""INSERT INTO idempotency(user_id,chat_id,key,body,response) VALUES (%s,%s,%s,%s,%s)
ON CONFLICT(user_id,chat_id,key) DO UPDATE SET body=EXCLUDED.body,response=EXCLUDED.response,expires_at=clock_timestamp()+interval '24 hours'""",
(actor["id"],chat_id,idempotency_key,body,Jsonb(msg)))
return msg
@app.get("/api/v1/chats/{chat_id}/messages", response_model=MessagePage)
def list_messages(request: Request, chat_id: str, limit: int = Query(50,ge=1,le=100), cursor: Optional[str] = Query(None,min_length=1), actor=Depends(current_user)):
visible_chat(chat_id,actor["id"])
scope = "messages:" + chat_id
limit,after = page_args(request,scope,limit,cursor)
sql = "SELECT * FROM messages WHERE chat_id=%s"
params = [chat_id]
if after:
sql += " AND (created_at,id) > (%s,%s)"
params.extend(after)
return page(query(sql+" ORDER BY created_at,id LIMIT %s",params+[limit+1]),limit,scope)
import storage
from attachments import install
install(app, POOL, current_user, visible_chat, query, wire, ApiError)
@app.exception_handler(storage.StorageError)
@app.exception_handler(storage.ClientError)
async def storage_error(request, error):
return error_response(request, 503, "unavailable", "Storage temporarily unavailable")
if __name__ == "__main__":
uvicorn.run(app, host=os.environ.get("APP_HOST","0.0.0.0"), port=env_int("APP_PORT",8080,1))
+24
View File
@@ -0,0 +1,24 @@
"""Bounded S3 client and image validation shared by the API and cleaner."""
import os
import warnings
from pathlib import Path
import boto3
from botocore.config import Config
from botocore.exceptions import BotoCoreError as StorageError, ClientError
from PIL import Image, UnidentifiedImageError
MAX_BYTES = 10_485_760
MAX_PIXELS = 40_000_000
BUCKET = os.environ.get('S3_BUCKET', 'messenger-attachments')
Image.MAX_IMAGE_PIXELS = MAX_PIXELS
def secret(name):
return Path(os.environ[name + '_FILE']).read_text().strip()
client = boto3.client('s3', endpoint_url=os.environ.get('S3_ENDPOINT', 'http://s3:9000'),
region_name='us-east-1', aws_access_key_id=secret('S3_ACCESS_KEY_ID'),
aws_secret_access_key=secret('S3_SECRET_ACCESS_KEY'),
config=Config(connect_timeout=2, read_timeout=10, retries={'total_max_attempts': 1},
s3={'addressing_style': 'path'}))
from images import validate_image, InvalidImage, TooManyPixels
+107
View File
@@ -0,0 +1,107 @@
name: mtusi-lab4
x-app: &app
build: .
image: mtusi-messenger:lab4
mem_limit: 768m
cpus: 1.0
environment: &env
APP_HOST: 0.0.0.0
APP_PORT: 8080
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
DATABASE_URL_FILE: /run/secrets/database_url
CACHE_URL_FILE: /run/secrets/cache_url
S3_ENDPOINT: http://s3:9000
S3_BUCKET: messenger-attachments
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
secrets: [database_url, cache_url, s3_access_key_id, s3_secret_access_key]
stop_grace_period: 10s
services:
postgres:
image: postgres:16.6-bookworm
environment:
POSTGRES_USER: messenger
POSTGRES_DB: messenger
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
secrets: [postgres_password]
volumes: [postgres-data:/var/lib/postgresql/data]
healthcheck:
test: [CMD-SHELL, 'pg_isready -U messenger -d messenger']
interval: 2s
timeout: 3s
retries: 30
valkey:
image: valkey/valkey:8.0.2-alpine
command: [valkey-server, --appendonly, 'yes', --maxmemory, 64mb, --maxmemory-policy, allkeys-lru]
volumes: [valkey-data:/data]
s3:
build:
context: ./infra/s3
target: server
image: mtusi-minio:RELEASE.2025-04-22T22-12-26Z
command: [server, /data, --console-address, '127.0.0.1:9001']
environment:
MINIO_ROOT_USER_FILE: /run/secrets/s3_root_user
MINIO_ROOT_PASSWORD_FILE: /run/secrets/s3_root_password
secrets: [s3_root_user, s3_root_password]
volumes: [s3-data:/data]
healthcheck:
test: [CMD, curl, -f, http://localhost:9000/minio/health/live]
interval: 3s
timeout: 3s
retries: 30
s3-init:
build:
context: ./infra/s3
target: client
image: mtusi-mc:RELEASE.2025-04-16T18-13-26Z
entrypoint: [/bin/sh, /config/init.sh]
volumes: ['./infra/s3:/config:ro']
secrets: [s3_root_user, s3_root_password, s3_access_key_id, s3_secret_access_key]
depends_on:
s3: {condition: service_healthy}
migrate:
<<: *app
command: [python, app/migrate.py]
healthcheck: {disable: true}
depends_on:
postgres: {condition: service_healthy}
api1:
<<: *app
environment:
<<: *env
INSTANCE_ID: api1
depends_on: &deps
migrate: {condition: service_completed_successfully}
s3-init: {condition: service_completed_successfully}
valkey: {condition: service_started}
api2:
<<: *app
environment:
<<: *env
INSTANCE_ID: api2
depends_on: *deps
cleaner:
<<: *app
command: [python, app/cleanup.py]
healthcheck: {disable: true}
depends_on: *deps
lb:
image: haproxy:3.0.8-alpine
ports: ['127.0.0.1:${HOST_PORT:-8084}:8080']
volumes: ['./infra/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro']
depends_on:
api1: {condition: service_healthy}
api2: {condition: service_healthy}
volumes:
postgres-data:
valkey-data:
s3-data:
secrets:
postgres_password: {file: ./secrets/postgres_password}
database_url: {file: ./secrets/database_url}
cache_url: {file: ./secrets/cache_url}
s3_root_user: {file: ./secrets/s3_root_user}
s3_root_password: {file: ./secrets/s3_root_password}
s3_access_key_id: {file: ./secrets/s3_access_key_id}
s3_secret_access_key: {file: ./secrets/s3_secret_access_key}
+39
View File
@@ -0,0 +1,39 @@
Container mtusi-lab4-api1-1 Stopping
Container mtusi-lab4-api1-1 Stopped
Container mtusi-lab4-valkey-1 Running
Container mtusi-lab4-s3-1 Running
Container mtusi-lab4-api2-1 Running
Container mtusi-lab4-postgres-1 Running
Container mtusi-lab4-postgres-1 Waiting
Container mtusi-lab4-s3-1 Waiting
Container mtusi-lab4-s3-1 Healthy
Container mtusi-lab4-postgres-1 Healthy
Container mtusi-lab4-s3-init-1 Starting
Container mtusi-lab4-migrate-1 Starting
Container mtusi-lab4-s3-init-1 Started
Container mtusi-lab4-migrate-1 Started
Container mtusi-lab4-migrate-1 Waiting
Container mtusi-lab4-s3-init-1 Waiting
Container mtusi-lab4-migrate-1 Waiting
Container mtusi-lab4-s3-init-1 Waiting
Container mtusi-lab4-s3-init-1 Exited
Container mtusi-lab4-migrate-1 Exited
Container mtusi-lab4-migrate-1 Exited
Container mtusi-lab4-s3-init-1 Exited
Container mtusi-lab4-api1-1 Starting
Container mtusi-lab4-api1-1 Started
Container mtusi-lab4-migrate-1 Waiting
Container mtusi-lab4-api1-1 Waiting
Container mtusi-lab4-s3-init-1 Waiting
Container mtusi-lab4-valkey-1 Waiting
Container mtusi-lab4-s3-1 Waiting
Container mtusi-lab4-api2-1 Waiting
Container mtusi-lab4-postgres-1 Waiting
Container mtusi-lab4-postgres-1 Healthy
Container mtusi-lab4-api2-1 Healthy
Container mtusi-lab4-s3-init-1 Exited
Container mtusi-lab4-migrate-1 Exited
Container mtusi-lab4-valkey-1 Healthy
Container mtusi-lab4-s3-1 Healthy
Container mtusi-lab4-api1-1 Healthy
PASS: upload=api1, download=api2, sha256=99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
+6
View File
@@ -0,0 +1,6 @@
Container mtusi-lab4-s3-1 Running
Container mtusi-lab4-s3-1 Waiting
Container mtusi-lab4-s3-1 Healthy
Container mtusi-lab4-s3-init-run-6bff1f84d1f3 Creating
Container mtusi-lab4-s3-init-run-6bff1f84d1f3 Created
Private bucket and restricted API user ready
+2
View File
@@ -0,0 +1,2 @@
python3 scripts/check_contract.py
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
+7
View File
@@ -0,0 +1,7 @@
{"Service": "api1", "ID": "dfb599607e63", "Image": "mtusi-messenger:lab4", "State": "running", "Health": "healthy", "Status": "Up 9 seconds (healthy)", "Ports": "8080/tcp"}
{"Service": "api2", "ID": "0f6254d1d00b", "Image": "mtusi-messenger:lab4", "State": "running", "Health": "healthy", "Status": "Up 9 seconds (healthy)", "Ports": "8080/tcp"}
{"Service": "cleaner", "ID": "91c5a5e1aecb", "Image": "mtusi-messenger:lab4", "State": "running", "Health": "", "Status": "Up Less than a second", "Ports": "8080/tcp"}
{"Service": "lb", "ID": "2b763a6b13e7", "Image": "haproxy:3.0.8-alpine", "State": "running", "Health": "", "Status": "Up 10 seconds", "Ports": "127.0.0.1:8084->8080/tcp"}
{"Service": "postgres", "ID": "1f7074602484", "Image": "postgres:16.6-bookworm", "State": "running", "Health": "healthy", "Status": "Up 10 seconds (healthy)", "Ports": "5432/tcp"}
{"Service": "s3", "ID": "52d093feabad", "Image": "mtusi-minio:RELEASE.2025-04-22T22-12-26Z", "State": "running", "Health": "healthy", "Status": "Up 10 seconds (healthy)", "Ports": ""}
{"Service": "valkey", "ID": "54a602e6f545", "Image": "valkey/valkey:8.0.2-alpine", "State": "running", "Health": "", "Status": "Up 10 seconds", "Ports": "6379/tcp"}
+7
View File
@@ -0,0 +1,7 @@
{"Service": "api1", "ID": "dfb599607e63", "Image": "mtusi-messenger:lab4", "State": "running", "Health": "healthy", "Status": "Up 42 seconds (healthy)", "Ports": "8080/tcp"}
{"Service": "api2", "ID": "0f6254d1d00b", "Image": "mtusi-messenger:lab4", "State": "running", "Health": "healthy", "Status": "Up 42 seconds (healthy)", "Ports": "8080/tcp"}
{"Service": "cleaner", "ID": "91c5a5e1aecb", "Image": "mtusi-messenger:lab4", "State": "running", "Health": "", "Status": "Up 8 seconds", "Ports": "8080/tcp"}
{"Service": "lb", "ID": "2b763a6b13e7", "Image": "haproxy:3.0.8-alpine", "State": "running", "Health": "", "Status": "Up 36 seconds", "Ports": "127.0.0.1:8084->8080/tcp"}
{"Service": "postgres", "ID": "1f7074602484", "Image": "postgres:16.6-bookworm", "State": "running", "Health": "healthy", "Status": "Up 46 seconds (healthy)", "Ports": "5432/tcp"}
{"Service": "s3", "ID": "52d093feabad", "Image": "mtusi-minio:RELEASE.2025-04-22T22-12-26Z", "State": "running", "Health": "healthy", "Status": "Up 4 seconds (healthy)", "Ports": ""}
{"Service": "valkey", "ID": "54a602e6f545", "Image": "valkey/valkey:8.0.2-alpine", "State": "running", "Health": "", "Status": "Up 46 seconds", "Ports": "6379/tcp"}
+4
View File
@@ -0,0 +1,4 @@
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://localhost:8084"
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 89, "failed_attempts": 3, "first_recovery_seconds": 2.8}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
+68
View File
@@ -0,0 +1,68 @@
{
"concurrency": 10,
"file_bytes": 9201866,
"results": [
{
"status": 503,
"seconds": 0.597
},
{
"status": 201,
"seconds": 0.489
},
{
"status": 201,
"seconds": 0.68
},
{
"status": 201,
"seconds": 0.624
},
{
"status": 201,
"seconds": 0.427
},
{
"status": 201,
"seconds": 0.673
},
{
"status": 503,
"seconds": 0.57
},
{
"status": 201,
"seconds": 0.493
},
{
"status": 201,
"seconds": 0.455
},
{
"status": 201,
"seconds": 0.645
}
],
"samples": [
{
"seconds": 0.0,
"api1_rss_kib": 107580,
"api2_rss_kib": 109104
},
{
"seconds": 0.309,
"api1_rss_kib": 128832,
"api2_rss_kib": 133908
},
{
"seconds": 0.639,
"api1_rss_kib": 132796,
"api2_rss_kib": 133912
}
],
"peak_rss_kib": {
"api1": 132796,
"api2": 133912
},
"sampling_note": "Sequential /proc sampling; very brief peaks may be missed. 503 is explicit admission control."
}
+51
View File
@@ -0,0 +1,51 @@
{
"concurrency": 10,
"file_bytes": 9201866,
"results": [
{
"status": 503,
"seconds": 0.597
},
{
"status": 201,
"seconds": 0.489
},
{
"status": 201,
"seconds": 0.68
},
{
"status": 201,
"seconds": 0.624
},
{
"status": 201,
"seconds": 0.427
},
{
"status": 201,
"seconds": 0.673
},
{
"status": 503,
"seconds": 0.57
},
{
"status": 201,
"seconds": 0.493
},
{
"status": 201,
"seconds": 0.455
},
{
"status": 201,
"seconds": 0.645
}
],
"peak_rss_kib": {
"api1": 132796,
"api2": 133912
},
"sampling_note": "Sequential /proc sampling; very brief peaks may be missed. 503 is explicit admission control."
}
+22
View File
@@ -0,0 +1,22 @@
$ make check
python3 scripts/check_contract.py
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
exit=0
$ docker compose config --quiet
exit=0
$ python3 student-tests/unit.py
test_mime_substitution (__main__.Images) ... ok
test_not_image (__main__.Images) ... ok
test_pixel_limit_before_decode (__main__.Images) ... ok
test_png_and_jpeg (__main__.Images) ... ok
test_truncated_pixels (__main__.Images) ... ok
----------------------------------------------------------------------
Ran 5 tests in 0.090s
OK
exit=0
PASS: AST parsed for 15 Python files.
+9
View File
@@ -0,0 +1,9 @@
PASS: interrupted upload did not create an attachment
Container mtusi-lab4-s3-1 Stopping
Container mtusi-lab4-s3-1 Stopped
PASS: unavailable S3 returns 503 from API
Container mtusi-lab4-s3-1 Starting
Container mtusi-lab4-s3-1 Started
Container mtusi-lab4-s3-1 Waiting
Container mtusi-lab4-s3-1 Healthy
PASS: S3 recovery preserves the original bytes
+3
View File
@@ -0,0 +1,3 @@
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:8084"
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
+2
View File
@@ -0,0 +1,2 @@
python3 tests/smoke.py --base-url "http://localhost:8084"
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
+1
View File
@@ -0,0 +1 @@
PASS: private bucket, failed finalization recovery, active lock protection, grace period, ready protection, repeated cleanup
+1
View File
@@ -0,0 +1 @@
PASS: JPEG round-trip, invalid files, ACL, cross-chat references, concurrent idempotency
+10
View File
@@ -0,0 +1,10 @@
test_mime_substitution (__main__.Images) ... ok
test_not_image (__main__.Images) ... ok
test_pixel_limit_before_decode (__main__.Images) ... ok
test_png_and_jpeg (__main__.Images) ... ok
test_truncated_pixels (__main__.Images) ... ok
----------------------------------------------------------------------
Ran 5 tests in 0.714s
OK
+4
View File
@@ -0,0 +1,4 @@
.venv/bin/python scripts/validate_openapi.py
/Users/sleepy13/Desktop/projects/vp/mtusi2026lab4/.venv/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
warnings.warn(
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
+21
View File
@@ -0,0 +1,21 @@
global
log stdout format raw local0
maxconn 128
defaults
mode http
log global
timeout connect 2s
timeout client 40s
timeout server 40s
timeout http-request 10s
timeout queue 3s
frontend http
bind :8080
default_backend api
backend api
balance roundrobin
option httpchk GET /health/ready
http-check expect status 200
retries 0
server api1 api1:8080 check inter 1s fall 2 rise 1
server api2 api2:8080 check inter 1s fall 2 rise 1
+23
View File
@@ -0,0 +1,23 @@
# Build the pinned upstream releases: public MinIO registry images are unavailable.
FROM golang:1.24.2-bookworm AS minio-build
WORKDIR /src
RUN curl -fsSL https://codeload.github.com/minio/minio/tar.gz/refs/tags/RELEASE.2025-04-22T22-12-26Z | tar xz --strip-components=1
RUN CGO_ENABLED=0 go build -trimpath -o /out/minio .
FROM golang:1.24.2-bookworm AS mc-build
WORKDIR /src
RUN curl -fsSL https://codeload.github.com/minio/mc/tar.gz/refs/tags/RELEASE.2025-04-16T18-13-26Z | tar xz --strip-components=1
RUN CGO_ENABLED=0 go build -trimpath -o /out/mc .
FROM debian:bookworm-20250407-slim AS runtime
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates curl && rm -rf /var/lib/apt/lists/*
FROM runtime AS server
COPY --from=minio-build /out/minio /usr/local/bin/minio
COPY --from=minio-build /src/LICENSE /usr/share/licenses/minio/LICENSE
ENTRYPOINT ["minio"]
FROM runtime AS client
COPY --from=mc-build /out/mc /usr/local/bin/mc
COPY --from=mc-build /src/LICENSE /usr/share/licenses/mc/LICENSE
ENTRYPOINT ["mc"]
+9
View File
@@ -0,0 +1,9 @@
#!/bin/sh
set -eu
mc alias set local http://s3:9000 "$(cat /run/secrets/s3_root_user)" "$(cat /run/secrets/s3_root_password)" >/dev/null
mc mb --ignore-existing local/messenger-attachments >/dev/null
mc anonymous set none local/messenger-attachments >/dev/null
mc admin user add local "$(cat /run/secrets/s3_access_key_id)" "$(cat /run/secrets/s3_secret_access_key)" >/dev/null
mc admin policy create local attachments /config/policy.json >/dev/null
mc admin policy attach local attachments --user "$(cat /run/secrets/s3_access_key_id)" >/dev/null
echo 'Private bucket and restricted API user ready'
+7
View File
@@ -0,0 +1,7 @@
{
"Version": "2012-10-17",
"Statement": [
{"Effect":"Allow","Action":["s3:ListBucket"],"Resource":["arn:aws:s3:::messenger-attachments"]},
{"Effect":"Allow","Action":["s3:GetObject","s3:PutObject","s3:DeleteObject"],"Resource":["arn:aws:s3:::messenger-attachments/originals/*"]}
]
}
+13
View File
@@ -0,0 +1,13 @@
CREATE TABLE users (
id uuid PRIMARY KEY, username varchar(32) NOT NULL UNIQUE CHECK (username ~ '^[a-z0-9_]{3,32}$'),
display_name varchar(100) NOT NULL CHECK (length(trim(display_name))>0),
password_hash text NOT NULL, created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE TABLE chats (id uuid PRIMARY KEY, title varchar(100) NOT NULL CHECK(length(trim(title))>0), created_at timestamptz NOT NULL DEFAULT clock_timestamp());
CREATE TABLE members (chat_id uuid REFERENCES chats(id), user_id uuid REFERENCES users(id), PRIMARY KEY(chat_id,user_id));
CREATE INDEX members_user_chat ON members(user_id,chat_id);
CREATE TABLE messages (id uuid PRIMARY KEY, chat_id uuid NOT NULL REFERENCES chats(id), sender_id uuid NOT NULL REFERENCES users(id), text varchar(4000) NOT NULL CHECK(length(trim(text))>0), created_at timestamptz NOT NULL DEFAULT clock_timestamp(), FOREIGN KEY(chat_id,sender_id) REFERENCES members(chat_id,user_id));
CREATE INDEX messages_history ON messages(chat_id,created_at,id);
CREATE INDEX chats_order ON chats(created_at,id);
CREATE TABLE sessions (token_hash char(64) PRIMARY KEY, user_id uuid NOT NULL REFERENCES users(id), expires_at timestamptz NOT NULL);
CREATE INDEX sessions_expiry ON sessions(expires_at);
+18
View File
@@ -0,0 +1,18 @@
ALTER TABLE messages DROP CONSTRAINT messages_text_check;
ALTER TABLE messages ADD COLUMN attachment_ids uuid[] NOT NULL DEFAULT '{}';
ALTER TABLE messages ADD CHECK (length(trim(text))>0 OR cardinality(attachment_ids)>0);
CREATE TABLE idempotency (
user_id uuid REFERENCES users(id), chat_id uuid REFERENCES chats(id), key varchar(128),
body text NOT NULL, response jsonb NOT NULL,
expires_at timestamptz NOT NULL DEFAULT clock_timestamp()+interval '24 hours',
PRIMARY KEY(user_id,chat_id,key)
);
CREATE TABLE attachments (
id uuid PRIMARY KEY, chat_id uuid NOT NULL REFERENCES chats(id), owner_id uuid NOT NULL REFERENCES users(id),
filename varchar(255) NOT NULL, object_key text NOT NULL UNIQUE,
content_type text NOT NULL, size_bytes bigint NOT NULL, sha256 char(64) NOT NULL,
status text NOT NULL CHECK(status IN ('pending','ready')),
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
updated_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE INDEX attachments_pending ON attachments(created_at) WHERE status='pending';
+32
View File
@@ -0,0 +1,32 @@
"""Generate local-only synthetic credentials once; never overwrite them."""
import os
import secrets
from pathlib import Path
root = Path(__file__).resolve().parents[1]
os.umask(0o077)
(root / "secrets").mkdir(exist_ok=True)
p = root / "secrets/postgres_password"
if not p.exists():
p.write_text(secrets.token_hex(32))
values = {"database_url": "postgresql://messenger:" + p.read_text().strip() + "@postgres:5432/messenger", "cache_url": "redis://valkey:6379/0"}
for name, value in values.items():
dest = root / "secrets" / name
if not dest.exists():
dest.write_text(value)
# Docker Desktop bind-mounted secrets need read permission for container UID 10001.
for name in ["postgres_password", "database_url", "cache_url"]:
(root / "secrets" / name).chmod(0o644)
env = root / ".env"
if not env.exists():
env.write_text("SESSION_TTL_SECONDS=3600\nHOST_PORT=8084\n")
print("Local configuration ready (secrets not printed)")
for name, value in {
's3_root_user': 'admin-' + secrets.token_hex(8),
's3_root_password': secrets.token_hex(32),
's3_access_key_id': 'api-' + secrets.token_hex(8),
's3_secret_access_key': secrets.token_hex(32),
}.items():
dest = root / 'secrets' / name
if not dest.exists():
dest.write_text(value)
dest.chmod(0o644)
+4
View File
@@ -0,0 +1,4 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")/.."
docker compose restart postgres valkey s3 api1 api2 cleaner lb
+7
View File
@@ -0,0 +1,7 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")/.."
case "${TARGET_INSTANCE_ID:-}" in
api1|api2) docker compose stop "$TARGET_INSTANCE_ID" ;;
*) echo 'Unknown target instance' >&2; exit 1 ;;
esac
+48
View File
@@ -0,0 +1,48 @@
"""Run all checks and save actual outputs. Stop on the first failure."""
import os
import json
from pathlib import Path
import subprocess
import sys
root=Path(__file__).resolve().parents[1]
os.chdir(root)
os.environ.setdefault('BASE_URL','http://localhost:8084')
(root/'evidence').mkdir(exist_ok=True)
def run(name, args, stdin=None):
result=subprocess.run(args,input=stdin,text=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
output = result.stdout
if name.startswith('containers-') and result.returncode == 0:
fields = ('Service', 'ID', 'Image', 'State', 'Health', 'Status', 'Ports')
output = '\n'.join(json.dumps({key: row.get(key) for key in fields})
for row in map(json.loads, output.splitlines())) + '\n'
output = '\n'.join(line.rstrip() for line in output.splitlines()) + '\n'
(root/'evidence'/name).write_text(output)
print(output,end='')
if result.returncode:
raise SystemExit(result.returncode)
run('check.txt',['make','check'])
run('unit.txt',[sys.executable,'student-tests/unit.py'])
run('smoke.txt',['make','test','BASE_URL='+os.environ['BASE_URL']])
run('student-tests.txt',[sys.executable,'student-tests/verify.py'])
subprocess.run(['docker','compose','stop','cleaner'],check=True)
try:
run('storage-checks.txt',['docker','compose','exec','-T','api1','python','-'],
(root/'student-tests/storage_checks.py').read_text())
finally:
subprocess.run(['docker','compose','start','cleaner'],check=True)
run('outage.txt',[sys.executable,'student-tests/outage.py'])
run('bootstrap-repeat.txt',['docker','compose','run','--rm','s3-init'])
run('containers-before.txt',['docker','compose','ps','--format','json'])
run('persistence.txt',['make','test-persistence','ACTION_SCRIPT=scripts/restart.sh','BASE_URL='+os.environ['BASE_URL']])
run('containers-after.txt',['docker','compose','ps','--format','json'])
try:
run('failover.txt',['make','test-failover','ACTION_SCRIPT=scripts/stop-one.sh','BASE_URL='+os.environ['BASE_URL']])
finally:
subprocess.run(['docker','compose','start','api1','api2'],check=True)
# Compose waits for the API's actual readiness after restoring the replica.
subprocess.run(['docker','compose','up','-d','--wait','api1','api2'],check=True)
run('attachment-failover.txt',[sys.executable,'student-tests/attachment_failover.py'])
run('memory.txt',[sys.executable,'student-tests/memory.py'])
print('All requested checks passed; actual results are in evidence/.')
+34
View File
@@ -0,0 +1,34 @@
"""Stop the exact replica that accepted the upload, then compare original bytes."""
import hashlib
from pathlib import Path
import subprocess
import sys
import time
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'tests'))
from client import Client, Failure, check
c = Client(timeout=10)
a, b, _, chat = c.bootstrap()
png = (Path(__file__).resolve().parents[1] / 'tests/fixtures/sample.png').read_bytes()
result = c.call('POST', '/api/v1/chats/' + chat['id'] + '/attachments?filename=failover.png',
201, actor=a, raw=png, headers={'Content-Type': 'image/png'}, schema='Attachment')
target = result['headers']['X-Instance-Id']
check(target in ('api1', 'api2'), 'unknown upload replica')
try:
subprocess.run(['docker', 'compose', 'stop', target], check=True)
deadline = time.monotonic() + 20
while True:
try:
downloaded = c.call('GET', '/api/v1/attachments/' + result['json']['id'] + '/content', actor=b)
break
except Failure:
if time.monotonic() >= deadline:
raise
time.sleep(0.5)
check(downloaded['headers']['X-Instance-Id'] != target, 'uploader still serves download')
check(downloaded['bytes'] == png, 'original changed after uploader stopped')
print('PASS: upload=' + target + ', download=' + downloaded['headers']['X-Instance-Id']
+ ', sha256=' + hashlib.sha256(png).hexdigest())
finally:
subprocess.run(['docker', 'compose', 'up', '-d', '--wait', 'api1', 'api2'], check=True)
+57
View File
@@ -0,0 +1,57 @@
"""Start ten simultaneous uploads; record actual container RSS via /proc."""
from concurrent.futures import ThreadPoolExecutor
import json
import os
from pathlib import Path
import subprocess
import sys
import threading
import time
sys.path.insert(0,str(Path(__file__).resolve().parents[1]/'tests'))
from client import Client, check
from PIL import Image
import io
c=Client(timeout=45)
a,b,e,chat=c.bootstrap()
file=io.BytesIO()
# Incompressible ~9 MiB RGB PNG rather than ten tiny fixtures.
Image.frombytes('RGB',(1750,1750),os.urandom(1750*1750*3)).save(file,'PNG')
data=file.getvalue()
check(len(data)<=10485760,'fixture too large')
barrier=threading.Barrier(10)
finished=threading.Event()
samples=[]
def rss():
while not finished.is_set():
row={'seconds':round(time.monotonic()-start,3)}
for name in ['api1','api2']:
result=subprocess.run(['docker','compose','exec','-T',name,'python','-c',
"from pathlib import Path; print(next(x.split()[1] for x in Path('/proc/1/status').read_text().splitlines() if x.startswith('VmRSS:')))"] ,capture_output=True,text=True,check=True)
row[name+'_rss_kib']=int(result.stdout.strip())
samples.append(row)
finished.wait(0.1)
def upload(i):
barrier.wait()
response=Client(timeout=45).call('POST','/api/v1/chats/'+chat['id']+'/attachments?filename=load.png',
[201,503],actor=a,raw=data,headers={'Content-Type':'image/png'})
return {'status':response['status'],'seconds':round(response['elapsed'],3)}
start=time.monotonic()
with ThreadPoolExecutor(max_workers=1) as monitor:
future=monitor.submit(rss)
try:
with ThreadPoolExecutor(max_workers=10) as pool:
results=list(pool.map(upload,range(10)))
finally:
finished.set()
future.result()
check(samples,'no memory measurements')
report={'concurrency':10,'file_bytes':len(data),'results':results,'samples':samples,
'peak_rss_kib':{n:max(s[n+'_rss_kib'] for s in samples) for n in ['api1','api2']},
'sampling_note':'Sequential /proc sampling; very brief peaks may be missed. 503 is explicit admission control.'}
Path('evidence/memory.json').write_text(json.dumps(report,indent=2))
check(any(r['status']==201 for r in results),'no upload succeeded')
print(json.dumps({k:v for k,v in report.items() if k!='samples'},indent=2))
+66
View File
@@ -0,0 +1,66 @@
"""Exercise interrupted HTTP upload and unavailable S3 on this Compose project."""
import http.client
import json
import os
from pathlib import Path
import subprocess
import sys
import time
from urllib.parse import urlsplit
sys.path.insert(0,str(Path(__file__).resolve().parents[1]/'tests'))
from client import Client, check
c=Client(timeout=45)
a,b,e,chat=c.bootstrap()
att,png=c.attachment(a,chat)
base=urlsplit(c.base)
check(base.hostname in ('localhost','127.0.0.1'),'test is limited to the local lab')
# Close while the body is incomplete. No intent should be created before validation.
connection=http.client.HTTPConnection(base.hostname,base.port,timeout=5)
connection.putrequest('POST','/api/v1/chats/'+chat['id']+'/attachments?filename=interrupted.png')
connection.putheader('Authorization','Bearer '+a['token'])
connection.putheader('Content-Type','image/png')
connection.putheader('Content-Length',str(len(png)))
connection.endheaders()
connection.send(png[:16])
connection.close()
# Query only the synthetic chat through stdin; credentials/tokens are never printed.
probe="""
import sys
sys.path.insert(0,'/srv/app')
import psycopg,storage
with psycopg.connect(storage.secret('DATABASE_URL')) as db:
n=db.execute("SELECT count(*) FROM attachments WHERE chat_id=%s", (CHAT,)).fetchone()[0]
assert n==1, 'interrupted upload created metadata'
print('PASS: interrupted upload did not create an attachment')
""".replace('CHAT',repr(chat['id']))
subprocess.run(['docker','compose','exec','-T','api1','python','-'],input=probe,text=True,check=True)
# Direct request within api1 ensures the API's structured 503 is tested,
# independently of HAProxy taking all unready instances out of rotation.
probe="""
import urllib.request,urllib.error
req=urllib.request.Request('http://localhost:8080/api/v1/attachments/ATT/content',headers={'Authorization':'Bearer TOKEN'})
try:
urllib.request.urlopen(req,timeout=20)
raise AssertionError('unavailable S3 returned success')
except urllib.error.HTTPError as error:
assert error.code==503, error.code
print('PASS: unavailable S3 returns 503 from API')
""".replace('ATT',att['id']).replace('TOKEN',a['token'])
try:
subprocess.run(['docker','compose','stop','s3'],check=True)
subprocess.run(['docker','compose','exec','-T','api1','python','-'],input=probe,text=True,check=True)
finally:
subprocess.run(['docker','compose','up','-d','--wait','s3'],check=True)
for attempt in range(30):
try:
result=c.call('GET','/api/v1/attachments/'+att['id']+'/content',actor=b)
check(result['bytes']==png,'recovered file differs')
break
except AssertionError:
if attempt==29:
raise
time.sleep(1)
print('PASS: S3 recovery preserves the original bytes')
+66
View File
@@ -0,0 +1,66 @@
"""Run inside the API container: private bucket and cleanup failure recovery."""
import io
import json
from pathlib import Path
import sys
import uuid
import urllib.request
import urllib.error
sys.path.insert(0, '/srv/app')
import psycopg
import storage
from cleanup import sweep
prefix = uuid.uuid4().hex
ids = [str(uuid.uuid4()) for _ in range(3)]
keys = ['originals/'+i for i in ids]
user,chat = str(uuid.uuid4()),str(uuid.uuid4())
dsn=storage.secret('DATABASE_URL')
# Model a process stopping after S3 PUT but before final DB COMMIT.
# Also create a fresh intent and an old ready file; neither may be removed.
try:
with psycopg.connect(dsn) as db:
db.execute('INSERT INTO users(id,username,display_name,password_hash) VALUES (%s,%s,%s,%s)',
(user,'cleanup_'+prefix[:12],'Synthetic cleanup','not-a-login-hash'))
db.execute('INSERT INTO chats(id,title) VALUES (%s,%s)', (chat,'Synthetic cleanup'))
for i,(aid,key) in enumerate(zip(ids,keys)):
db.execute("""INSERT INTO attachments(id,chat_id,owner_id,filename,object_key,content_type,size_bytes,sha256,status,created_at)
VALUES (%s,%s,%s,'test.png',%s,'image/png',3,%s,%s,clock_timestamp()-%s*interval '1 hour')""",
(aid,chat,user,key,'0'*64,'ready' if i==2 else 'pending',2 if i!=1 else 0))
for key in keys:
storage.client.put_object(Bucket=storage.BUCKET,Key=key,Body=b'abc')
try:
urllib.request.urlopen('http://s3:9000/'+storage.BUCKET+'/'+keys[2],timeout=5)
raise AssertionError('Anonymous object access allowed')
except urllib.error.HTTPError as error:
assert error.code==403, error.code
# Roll back a simulated failed finalization; durable pending intent remains.
with psycopg.connect(dsn) as db:
db.execute("UPDATE attachments SET status='ready' WHERE id=%s", (ids[0],))
db.rollback()
# Even an old intent must survive while a live finalization holds its lock.
with psycopg.connect(dsn) as active:
active.execute('SELECT id FROM attachments WHERE id=%s FOR UPDATE', (ids[0],))
sweep()
storage.client.head_object(Bucket=storage.BUCKET, Key=keys[0])
assert active.execute('SELECT status FROM attachments WHERE id=%s', (ids[0],)).fetchone()[0] == 'pending'
sweep()
sweep() # Idempotent after completion.
with psycopg.connect(dsn) as db:
remaining={str(r[0]) for r in db.execute('SELECT id FROM attachments WHERE id=ANY(%s::uuid[])',(ids,))}
assert remaining==set(ids[1:]), remaining
try:
storage.client.head_object(Bucket=storage.BUCKET,Key=keys[0])
raise AssertionError('Abandoned object still exists')
except storage.ClientError as error:
assert error.response['ResponseMetadata']['HTTPStatusCode']==404
for key in keys[1:]:
storage.client.head_object(Bucket=storage.BUCKET,Key=key)
print('PASS: private bucket, failed finalization recovery, active lock protection, grace period, ready protection, repeated cleanup')
finally:
for key in keys:
storage.client.delete_object(Bucket=storage.BUCKET,Key=key)
with psycopg.connect(dsn) as db:
db.execute('DELETE FROM attachments WHERE id=ANY(%s::uuid[])',(ids,))
db.execute('DELETE FROM chats WHERE id=%s',(chat,))
db.execute('DELETE FROM users WHERE id=%s',(user,))
+50
View File
@@ -0,0 +1,50 @@
"""Offline checks of image content, size limits, and MIME detection."""
import io
from pathlib import Path
import struct
import sys
import unittest
import zlib
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'app'))
from PIL import Image
from images import validate_image, InvalidImage, TooManyPixels
class Images(unittest.TestCase):
def picture(self, fmt):
file = io.BytesIO()
Image.new('RGB', (320, 200), (20, 30, 40)).save(file, fmt)
file.seek(0)
return file
def test_png_and_jpeg(self):
for fmt, mime in [('PNG', 'image/png'), ('JPEG', 'image/jpeg')]:
file = self.picture(fmt)
before = file.getvalue()
validate_image(file, mime)
self.assertEqual(file.tell(), 0)
self.assertEqual(file.read(), before)
def test_mime_substitution(self):
with self.assertRaises(InvalidImage):
validate_image(self.picture('PNG'), 'image/jpeg')
def test_not_image(self):
for content in [b'', b'hello', b'<script>alert(1)</script>']:
with self.assertRaises(InvalidImage):
validate_image(io.BytesIO(content), 'image/png')
def test_truncated_pixels(self):
file = self.picture('JPEG')
with self.assertRaises(InvalidImage):
validate_image(io.BytesIO(file.getvalue()[:-20]), 'image/jpeg')
def test_pixel_limit_before_decode(self):
def chunk(kind, data):
return struct.pack('>I', len(data))+kind+data+struct.pack('>I', zlib.crc32(kind+data)&0xffffffff)
huge = b'\x89PNG\r\n\x1a\n'+chunk(b'IHDR',struct.pack('>IIBBBBB', 8000,8000,8,2,0,0,0))
huge += chunk(b'IDAT', zlib.compress(b'\x00'))+chunk(b'IEND', b'')
with self.assertRaises(TooManyPixels):
validate_image(io.BytesIO(huge), 'image/png')
if __name__ == '__main__':
unittest.main(verbosity=2)
+59
View File
@@ -0,0 +1,59 @@
"""Black-box grade-4 checks; creates only synthetic data on the local lab."""
from concurrent.futures import ThreadPoolExecutor
import hashlib
import io
import json
from pathlib import Path
import sys
import threading
import uuid
from PIL import Image
from urllib.parse import quote
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'tests'))
from client import Client, check
c = Client(timeout=45)
a,b,e,chat = c.bootstrap()
base = '/api/v1/chats/' + chat['id']
att, png = c.attachment(a, chat)
# Exercise the second supported format through both stores, not only Pillow.
jpeg_file = io.BytesIO()
Image.new('RGB', (320, 200), (20, 30, 40)).save(jpeg_file, 'JPEG')
jpeg = jpeg_file.getvalue()
jpeg_att = c.call('POST', base+'/attachments?filename=photo.jpg', 201, actor=a,
raw=jpeg, headers={'Content-Type': 'image/jpeg'}, schema='Attachment')['json']
check(jpeg_att['sha256'] == hashlib.sha256(jpeg).hexdigest(), 'JPEG checksum differs')
check(jpeg_att['size_bytes'] == len(jpeg), 'JPEG length differs')
jpeg_download = c.call('GET', '/api/v1/attachments/'+jpeg_att['id']+'/content', actor=b)
check(jpeg_download['bytes'] == jpeg, 'JPEG original differs')
check(jpeg_download['headers'].get_content_type() == 'image/jpeg', 'JPEG MIME differs')
for name in ['../x.png', 'a\\b.png', '\x00.png', '\n.png']:
c.call('POST', base+'/attachments?filename='+quote(name, safe=''), 400, actor=a,
raw=png, headers={'Content-Type':'image/png'})
for data,mime,status in [(b'x'*(10485760+1),'image/png',413),
(b'not an image','image/png',415), (png,'image/jpeg',415),
(png[:20],'image/png',415)]:
c.call('POST', base+'/attachments?filename=x.png', status, actor=a,
raw=data, headers={'Content-Type':mime})
other = c.call('POST','/api/v1/chats',201,actor=a,
body={'title':'Other chat','member_ids':[a['id']]},schema='Chat')['json']
for target,ids,actor in [(other,[att['id']],a), (chat,[str(uuid.uuid4())],a), (chat,[att['id']],e)]:
c.call('POST','/api/v1/chats/'+target['id']+'/messages',404,actor=actor,
body={'text':'','attachment_ids':ids},headers={'Idempotency-Key':uuid.uuid4().hex})
for suffix in ['', '/content']:
c.call('GET','/api/v1/attachments/'+att['id']+suffix,404,actor=e)
# A member other than the uploader can attach and download the same file.
c.message(b, chat, '', attachments=[att['id']])
key = uuid.uuid4().hex
barrier = threading.Barrier(10)
def replay(_):
barrier.wait()
return Client(timeout=45).message(a,chat,'one logical message',key=key)['json']
with ThreadPoolExecutor(max_workers=10) as pool:
messages = list(pool.map(replay,range(10)))
check(all(m == messages[0] for m in messages), 'concurrent replay duplicated messages')
c.call('POST',base+'/messages',409,actor=a,body={'text':'different'},headers={'Idempotency-Key':key})
# Default [] and explicit [] are equivalent.
c.call('POST',base+'/messages',201,actor=a,body={'text':'one logical message','attachment_ids':[]},
headers={'Idempotency-Key':key},schema='Message')
print('PASS: JPEG round-trip, invalid files, ACL, cross-chat references, concurrent idempotency')