Лабораторная 4: S3-хранилище и вложения

This commit is contained in:
Алиса Шурман
2026-11-12 22:27:32 +03:00
parent ec540a1a72
commit 7fbdef3a6a
25 changed files with 1447 additions and 23 deletions
+11
View File
@@ -0,0 +1,11 @@
FROM python:3.12.7-slim-bookworm
WORKDIR /srv
COPY app/requirements.txt app/requirements.txt
RUN pip install --no-cache-dir -r app/requirements.txt && useradd --uid 10001 --create-home app
COPY app app
COPY migrations migrations
USER 10001:10001
ENV PYTHONUNBUFFERED=1 PYTHONDONTWRITEBYTECODE=1
EXPOSE 8080
HEALTHCHECK --interval=5s --timeout=4s --start-period=20s CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8080/health/ready', timeout=3)" || exit 1
CMD ["python", "app/server.py"]
+123 -23
View File
@@ -1,37 +1,137 @@
# Отчёт — лабораторная 4
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
Автор: Шурман А.Р., БПИ2302. Целевая оценка: **4**. Исследовательские треки на 5 не выполнялись.
## Запуск из чистого clone
## Статус проверки
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
Реализация подготовлена, но **полный Docker-прогон ещё не выполнен**. В среде подготовки
нет доступа к Docker socket. Проверены структура задания, Python-синтаксис,
Compose-конфигурация и пять локальных тестов валидации изображений.
Фактические локальные результаты находятся в `evidence/offline.txt`.
Smoke, сохранность, отказ реплики, доступ к приватному bucket, очистка после сбоя
и эксперимент с 10 загрузками необходимо выполнить приведённой ниже командой.
До этого выполнение всех критериев оценки 4 не подтверждено.
## Архитектура и решения
## Запуск
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab3, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
Нужны Docker с Compose, Python 3.9+ на хосте (для memory/unit — Pillow).
API работает на Python 3.12.7. Новый стенд использует порт **8084**, отдельные volumes
и Compose project `mtusi-lab4`, поэтому не занимает порт предыдущих работ.
## Доказательства критериев
```sh
cd labs/mtusi2026lab4
python3 scripts/bootstrap.py
docker compose config --quiet
docker compose up -d --build --wait
BASE_URL=http://localhost:8084 python3 scripts/verify.py
```
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
Команды выполняются из корня этой лабораторной. Если репозиторий склонирован отдельно,
первую команду замените на переход в его каталог. `bootstrap.py` генерирует локальные
секреты, не выводит и не перезаписывает их. Повторный запуск сохраняет данные.
Для локальных тестов при отсутствии Pillow: отдельное venv и `pip install Pillow==11.1.0`.
Полная проверка OpenAPI: `pip install -r requirements-dev.txt`, затем `make validate`.
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
API: http://localhost:8084; Swagger: http://localhost:8084/docs.
Остановка без потери данных: `docker compose down` (без `-v`).
Возобновление: `docker compose up -d --wait`.
## Эксперименты и отказы
## Архитектура
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
HAProxy → две FastAPI-реплики → общий PostgreSQL и приватный MinIO.
Valkey кэширует публичные профили; сессии проверяются в PostgreSQL при каждом запросе.
Исходная основа — собственная лабораторная 2: модели пользователей/чатов,
аутентификация, SQL-хранилище и мигратор. Возможности лабораторной 3 добавлены здесь:
балансировка, `X-Instance-Id`, общая таблица идемпотентности.
Контракт и публичные тесты skeleton не изменены.
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
Каждая реплика возвращает `api1` или `api2`. HAProxy проверяет readiness каждую секунду,
исключает реплику после двух неудач. Он не повторяет POST автоматически. При потере ответа
клиент повторяет сообщение с тем же `Idempotency-Key`. Advisory lock в PostgreSQL
сериализует один ключ пользователя/чата, вставка сообщения и результата атомарны.
Повтор в течение 24 часов возвращает исходный объект; другой текст/список вложений — 409.
Отсутствующий список вложений и `[]` равны. Просроченная запись ключа заменяется при повторном использовании.
История читается из primary без кэша. Keyset cursor не обеспечивает snapshot между страницами.
## Ограничения и следующие шаги
Метаданные Attachment находятся в PostgreSQL, байты — только в S3. Временный файл API
существует исключительно во время приёма и проверки запроса и закрывается по выходу.
Имя пользователя не используется как S3 key. Ключ — `originals/<UUID>`.
Скачивание идёт через API после проверки членства, без redirect и публичных URL.
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
## Валидация и ресурсы
Загрузка принимает сырые PNG/JPEG, максимум 10 MiB. Content-Length проверяется заранее,
но фактические байты также считаются во время потока. Общий срок приёма — 30 секунд.
Pillow проверяет формат, MIME, структуру и декодирование. Размер не выше 40 млн пикселей
проверяется до загрузки пикселей. Анимированные PNG в этой реализации отклоняются.
Оригинал сохраняется без преобразования; SHA-256 и размер вычисляются при чтении.
Пустые/опасные filename, MIME-подмена и чужие вложения имеют ошибки контракта.
Одновременно каждая реплика обрабатывает максимум две загрузки. Ожидание слота до 0.5 s,
затем явный 503. Две реплики ограничены по 768 MiB и 1 CPU. На диск пишется максимум
10 MiB на активную загрузку, декодирование занимает ограниченную память до размера
изображения. S3: connect timeout 2 s, read timeout 10 s, автоматических повторов нет.
Download читает S3 чанками по 64 KiB. Оригинал не загружается целиком в RAM при скачивании.
## Согласованность S3 и БД
1. API принимает и проверяет временный файл.
2. Отдельная транзакция сохраняет `pending` и назначенный object key до S3 PUT.
3. Новая транзакция блокирует строку, выполняет PUT и переводит её в `ready`.
4. Ответ 201 отправляется только после COMMIT. `pending` не показывается публично.
При обрыве передачи клиента до PUT остаётся только автоматически закрываемый временный
файл. При сбое PUT, падении процесса или неуспешном COMMIT остаётся учтённый `pending`.
Фоновый cleaner раз в минуту выбирает pending старше **1 часа**, блокируя строку
`FOR UPDATE SKIP LOCKED`, удаляет соответствующий S3 key и затем запись БД. Активная
транзакция пропускается. При ошибке S3 строка сохраняется для следующего прохода.
При сбое COMMIT очистка повторит безопасный DELETE отсутствующего объекта.
`ready` никогда не выбирается. Часовой grace period значительно больше таймаутов загрузки.
Это не распределённая транзакция: некоторое время возможны pending-метаданные и лишние байты.
Объекты, вручную записанные администратором без intent, вне модели очистки.
## Приватность и запуск хранилища
MinIO, PostgreSQL, Valkey и API не публикуют порты на хост. Только HAProxy доступен
на loopback. Консоль MinIO слушает 127.0.0.1 внутри его контейнера.
Идемпотентный `s3-init` создаёт bucket, запрещает anonymous и назначает отдельному API user
политику: ListBucket одного bucket и Get/Put/Delete исключительно в `originals/*`.
Root credentials доступны только MinIO и init, API их не получает.
Readiness зависит от PostgreSQL и MinIO. Valkey имеет fallback в PostgreSQL.
Секреты находятся в исключённом из Git каталоге `secrets/`.
## Проверки и доказательства
| Критерий | Проверка | Статус на момент подготовки |
|---|---|---|
| Структура и синтаксис | `make check`, AST всех новых Python-файлов | выполнено |
| Конфигурация | `docker compose config --quiet` | выполнено |
| Реальные PNG/JPEG, MIME, повреждение, пиксели | `python3 student-tests/unit.py` | 5 тестов пройдено |
| Обязательный API и checksum | `make test BASE_URL=http://localhost:8084` | требуется запуск Docker |
| Ошибки файлов и ACL, конкуренция ключа | `BASE_URL=http://localhost:8084 python3 student-tests/verify.py` | требуется запуск Docker |
| Приватность bucket и безопасная очистка | `student-tests/storage_checks.py` внутри API через общий runner | требуется запуск Docker |
| Обрыв загрузки и отказ S3 | `student-tests/outage.py` | требуется запуск Docker |
| Повторный bootstrap | повтор `docker compose run --rm s3-init` | требуется запуск Docker |
| Перезапуск всех сервисов | persistence + `scripts/restart.sh` | требуется запуск Docker |
| Остановка обслужившей реплики | failover + `scripts/stop-one.sh` | требуется запуск Docker |
| 10 одновременных загрузок и RSS | `student-tests/memory.py` | измерения ещё не получены |
`python3 scripts/verify.py` выполняет проверки и сохраняет реальные результаты в evidence,
останавливаясь при первом провале. Тест очистки создаёт только синтетические строки и объекты,
моделирует rollback финализации после PUT, проверяет grace period и сохранность ready.
Он удаляет только свои синтетические данные. До запуска убедитесь, что это учебный стенд.
Memory-тест создаёт PNG примерно 9 MiB и одновременно запускает 10 запросов.
Он пишет коды ответов, длительность, выборки VmRSS процессов API и измеренный максимум.
Отказы 503 из-за ограничения слотов учитываются явно; они не записываются как успешные загрузки.
Последовательная выборка `/proc/1/status` может пропустить короткий пик, поэтому результат —
максимум наблюдений, а не гарантированный абсолютный пик. Числа не подставлены заранее.
## Ограничения
Один host, HAProxy, PostgreSQL и MinIO остаются точками отказа. Это отказоустойчивость
к остановке одной API-реплики, не полноценная HA-система. При отказе S3 readiness
становится 503; после восстановления новые обращения повторяются клиентом.
Обрыв S3 посреди уже начатого download закрывает поток, поэтому клиент проверяет длину/checksum.
Нет UI, обработки изображений в очереди, presigned URL и исследовательских треков.
Проверка реальных контейнеров и образов остаётся обязательной перед сдачей.
+113
View File
@@ -0,0 +1,113 @@
"""Binary upload, private download, and durable upload intents."""
import asyncio
import hashlib
import tempfile
import uuid
import unicodedata
from urllib.parse import quote
from fastapi import Depends, Request
from fastapi.responses import StreamingResponse
from starlette.concurrency import run_in_threadpool
import storage
# Two decoded images per replica at most; no unbounded waiting queue.
slots = asyncio.Semaphore(2)
def valid_filename(name):
return (isinstance(name, str) and 1 <= len(name) <= 255 and bool(name.strip())
and not any(c in '/\\' or unicodedata.category(c) == 'Cc' for c in name))
def install(app, pool, current_user, visible_chat, query, wire, ApiError):
def public(row):
row = wire(row)
row.pop('object_key', None)
return dict(row, variants=[], error_code=None)
def visible_attachment(aid, actor):
try:
aid = str(uuid.UUID(aid))
except ValueError:
raise ApiError(400, 'invalid_request', 'Invalid attachment id')
rows = query("""SELECT a.* FROM attachments a JOIN members m ON m.chat_id=a.chat_id
WHERE a.id=%s AND m.user_id=%s AND a.status='ready'""", (aid, actor['id']))
if not rows:
raise ApiError(404, 'not_found', 'Attachment not found')
return rows[0]
def save(file, chat_id, actor, name, mime, size, digest):
try:
storage.validate_image(file, mime)
except storage.TooManyPixels:
raise ApiError(413, 'payload_too_large', 'Too many image pixels')
except storage.InvalidImage:
raise ApiError(415, 'unsupported_media_type', 'Invalid PNG/JPEG or MIME mismatch')
aid = str(uuid.uuid4())
key = 'originals/' + aid
# Commit intent before PUT. Crash/uncertain PUT or COMMIT is recoverable.
query("""INSERT INTO attachments(id,chat_id,owner_id,filename,object_key,content_type,size_bytes,sha256,status)
VALUES (%s,%s,%s,%s,%s,%s,%s,%s,'pending') RETURNING id""",
(aid,chat_id,actor['id'],name,key,mime,size,digest))
with pool.connection() as conn:
# Cleaner skips this row while PUT and commit are in flight.
conn.execute('SELECT id FROM attachments WHERE id=%s FOR UPDATE', (aid,)).fetchone()
storage.client.put_object(Bucket=storage.BUCKET, Key=key, Body=file,
ContentLength=size, ContentType=mime)
row = conn.execute("UPDATE attachments SET status='ready', updated_at=clock_timestamp() WHERE id=%s RETURNING *", (aid,)).fetchone()
return public(row)
@app.post('/api/v1/chats/{chat_id}/attachments', status_code=201)
async def upload(chat_id: str, request: Request, actor=Depends(current_user)):
await run_in_threadpool(visible_chat, chat_id, actor['id'])
name = request.query_params.get('filename')
if set(request.query_params) != {'filename'} or not valid_filename(name):
raise ApiError(400, 'invalid_request', 'Invalid filename')
mime = request.headers.get('content-type', '').split(';')[0].strip().lower()
if mime not in ('image/png', 'image/jpeg'):
raise ApiError(415, 'unsupported_media_type', 'PNG or JPEG required')
length = request.headers.get('content-length')
if length is not None:
if not length.isdigit():
raise ApiError(400, 'invalid_request', 'Invalid Content-Length')
if int(length) > storage.MAX_BYTES:
raise ApiError(413, 'payload_too_large', 'File too large')
try:
await asyncio.wait_for(slots.acquire(), timeout=0.5)
except asyncio.TimeoutError:
raise ApiError(503, 'unavailable', 'Upload capacity reached')
try:
with tempfile.TemporaryFile() as file:
size = 0
digest = hashlib.sha256()
try:
async with asyncio.timeout(30):
async for chunk in request.stream():
size += len(chunk)
if size > storage.MAX_BYTES:
raise ApiError(413, 'payload_too_large', 'File too large')
digest.update(chunk)
await run_in_threadpool(file.write, chunk)
except TimeoutError:
raise ApiError(400, 'invalid_request', 'Upload timed out')
return await run_in_threadpool(save, file, chat_id, actor, name, mime, size, digest.hexdigest())
finally:
slots.release()
@app.get('/api/v1/attachments/{aid}')
def metadata(aid: str, actor=Depends(current_user)):
return public(visible_attachment(aid, actor))
@app.get('/api/v1/attachments/{aid}/content')
def content(aid: str, variant: str = 'original', actor=Depends(current_user)):
row = visible_attachment(aid, actor)
if variant != 'original':
raise ApiError(404, 'not_found', 'Variant not found')
obj = storage.client.get_object(Bucket=storage.BUCKET, Key=row['object_key'])
def chunks():
try:
while data := obj['Body'].read(64 * 1024):
yield data
finally:
obj['Body'].close()
return StreamingResponse(chunks(), media_type=row['content_type'], headers={
'Content-Length': str(row['size_bytes']), 'X-Content-Type-Options': 'nosniff',
'Content-Disposition': "attachment; filename*=UTF-8''" + quote(row['filename'], safe='')})
+34
View File
@@ -0,0 +1,34 @@
"""Remove abandoned upload intents after an hour; never remove ready objects."""
import os
import time
import logging
import psycopg
from psycopg.rows import dict_row
import storage
def sweep():
count = 0
# Small transactions, no unbounded listing of the bucket into memory.
for _ in range(100):
with psycopg.connect(storage.secret('DATABASE_URL'), row_factory=dict_row,
connect_timeout=3, options='-c statement_timeout=15000') as conn:
row = conn.execute("""SELECT id,object_key FROM attachments
WHERE status='pending' AND created_at < clock_timestamp()-interval '1 hour'
ORDER BY created_at FOR UPDATE SKIP LOCKED LIMIT 1""").fetchone()
if not row:
return count
storage.client.delete_object(Bucket=storage.BUCKET, Key=row['object_key'])
conn.execute("DELETE FROM attachments WHERE id=%s AND status='pending'", (row['id'],))
count += 1
return count
if __name__ == '__main__':
while True:
try:
print('Removed abandoned uploads:', sweep(), flush=True)
except Exception:
logging.error('Cleanup failed; will retry next cycle')
if os.environ.get('CLEANUP_ONCE') == '1':
break
time.sleep(60)
+38
View File
@@ -0,0 +1,38 @@
"""Image checks without network access."""
import warnings
from PIL import Image, UnidentifiedImageError
MAX_BYTES = 10_485_760
MAX_PIXELS = 40_000_000
Image.MAX_IMAGE_PIXELS = MAX_PIXELS
class InvalidImage(ValueError):
pass
class TooManyPixels(ValueError):
pass
def validate_image(file, mime):
try:
with warnings.catch_warnings():
warnings.simplefilter('error', Image.DecompressionBombWarning)
file.seek(0)
with Image.open(file) as img:
if img.width * img.height > MAX_PIXELS:
raise TooManyPixels()
actual = {'PNG': 'image/png', 'JPEG': 'image/jpeg'}.get(img.format)
if actual != mime or actual is None:
raise InvalidImage()
if getattr(img, 'n_frames', 1) != 1:
raise InvalidImage()
img.verify()
file.seek(0)
with Image.open(file) as img:
img.load() # Detect truncated JPEG/pixel data, not only its header.
except (Image.DecompressionBombError, Image.DecompressionBombWarning) as exc:
raise TooManyPixels() from exc
except (UnidentifiedImageError, OSError, SyntaxError, ValueError) as exc:
if isinstance(exc, TooManyPixels):
raise
raise InvalidImage() from exc
finally:
file.seek(0)
+21
View File
@@ -0,0 +1,21 @@
"""Atomic, serialized, versioned migrations; changed applied files are rejected."""
import hashlib
import os
from pathlib import Path
import psycopg
with psycopg.connect(Path(os.environ["DATABASE_URL_FILE"]).read_text().strip()) as conn:
conn.execute("SELECT pg_advisory_xact_lock(20260002)")
conn.execute("CREATE TABLE IF NOT EXISTS schema_migrations(version text PRIMARY KEY, checksum text NOT NULL)")
for path in sorted(Path("migrations").glob("*.sql")):
sql = path.read_text()
digest = hashlib.sha256(sql.encode()).hexdigest()
row = conn.execute("SELECT checksum FROM schema_migrations WHERE version=%s", (path.name,)).fetchone()
if row:
if row[0] != digest:
raise RuntimeError("Applied migration changed: " + path.name)
continue
conn.execute(sql)
conn.execute("INSERT INTO schema_migrations VALUES (%s,%s)", (path.name,digest))
print("Applied", path.name)
print("Migrations OK")
+28
View File
@@ -0,0 +1,28 @@
annotated-types==0.8.0
anyio==4.15.1
argon2-cffi==23.1.0
argon2-cffi-bindings==26.1.0
cffi==2.1.1
click==8.5.0
fastapi==0.115.6
h11==0.16.0
idna==3.20
psycopg==3.2.6
psycopg-binary==3.2.6
psycopg-pool==3.3.3
pycparser==3.12
pydantic==2.14.1
pydantic_core==2.50.1
redis==5.2.1
starlette==0.41.3
typing-inspection==0.4.4
typing_extensions==4.16.0
uvicorn==0.34.0
boto3==1.35.99
botocore==1.35.99
s3transfer==0.10.4
jmespath==1.0.1
python-dateutil==2.9.0.post0
six==1.17.0
urllib3==2.3.0
Pillow==11.1.0
+502
View File
@@ -0,0 +1,502 @@
#!/usr/bin/env python3
"""Lab 4 PostgreSQL messenger, adapted from the author's lab 1."""
from __future__ import annotations
import base64
import binascii
import json
import logging
import os
import re
import hashlib
import secrets
from pathlib import Path
from contextlib import asynccontextmanager
import psycopg
from psycopg.rows import dict_row
from psycopg_pool import ConnectionPool, PoolTimeout, TooManyRequests
import redis
from argon2 import PasswordHasher
from argon2.exceptions import VerificationError
import uuid
from datetime import datetime, timezone
from typing import Any, Optional, Tuple
import uvicorn
from fastapi import Depends, FastAPI, Header, Query, Request
from fastapi.exceptions import RequestValidationError
from fastapi.responses import JSONResponse, Response
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
from starlette.exceptions import HTTPException as StarletteHTTPException
USERNAME_RE = re.compile(r"^[a-z0-9_]{3,32}$")
LOG = logging.getLogger("messenger")
INSTANCE_ID = os.environ.get("INSTANCE_ID", str(uuid.uuid4()))
def env_int(name: str, default: int, minimum: int = 0) -> int:
try:
value = int(os.environ.get(name, str(default)))
except ValueError as exc:
raise SystemExit("%s must be an integer" % name) from exc
if value < minimum:
raise SystemExit("%s must be at least %s" % (name, minimum))
return value
class ApiError(Exception):
def __init__(self, status: int, code: str, message: str) -> None:
super().__init__(message)
self.status = status
self.code = code
self.message = message
class StrictModel(BaseModel):
model_config = ConfigDict(extra="forbid")
class Health(StrictModel):
status: str = Field(pattern="^ok$")
class User(StrictModel):
id: uuid.UUID
username: str = Field(pattern=r"^[a-z0-9_]{3,32}$")
display_name: str = Field(min_length=1, max_length=100)
created_at: datetime
class CreateUser(StrictModel):
password: str = Field(min_length=12, max_length=128)
@field_validator("password")
@classmethod
def non_blank_password(cls, value: str) -> str:
if not value.strip():
raise ValueError("password must not be blank")
return value
username: str = Field(pattern=r"^[a-z0-9_]{3,32}$")
display_name: str = Field(min_length=1, max_length=100)
@field_validator("display_name")
@classmethod
def non_blank_display_name(cls, value: str) -> str:
if not value.strip():
raise ValueError("display_name must not be blank")
return value
class Chat(StrictModel):
id: uuid.UUID
title: str = Field(min_length=1, max_length=100)
member_ids: list[uuid.UUID] = Field(min_length=1, max_length=100)
created_at: datetime
class CreateChat(StrictModel):
title: str = Field(min_length=1, max_length=100)
member_ids: list[uuid.UUID] = Field(min_length=1, max_length=100)
@field_validator("title")
@classmethod
def non_blank_title(cls, value: str) -> str:
if not value.strip():
raise ValueError("title must not be blank")
return value
@field_validator("member_ids")
@classmethod
def unique_members(cls, value: list[uuid.UUID]) -> list[uuid.UUID]:
if len(set(value)) != len(value):
raise ValueError("member_ids must be unique")
return value
class Message(StrictModel):
id: uuid.UUID
chat_id: uuid.UUID
sender_id: uuid.UUID
text: str = Field(max_length=4000)
attachment_ids: list[uuid.UUID] = Field(default_factory=list, max_length=10)
created_at: datetime
class CreateMessage(StrictModel):
text: str = Field(max_length=4000)
attachment_ids: list[uuid.UUID] = Field(default_factory=list, max_length=10)
@model_validator(mode="after")
def valid_message(self):
if not self.text.strip() and not self.attachment_ids:
raise ValueError("message must not be empty")
if len(set(self.attachment_ids)) != len(self.attachment_ids):
raise ValueError("duplicate attachment")
return self
class ChatPage(StrictModel):
items: list[Chat]
next_cursor: Optional[str]
class MessagePage(StrictModel):
items: list[Message]
next_cursor: Optional[str]
class ErrorDetails(StrictModel):
code: str
message: str
request_id: str
class ErrorResponse(StrictModel):
error: ErrorDetails
def valid_uuid(value: Any) -> bool:
if not isinstance(value, str):
return False
try:
return str(uuid.UUID(value)) == value.lower()
except (ValueError, AttributeError):
return False
def make_cursor(scope: str, item: dict[str, Any]) -> str:
raw = json.dumps(
{"v": 1, "scope": scope, "created_at": item["created_at"], "id": item["id"]},
separators=(",", ":"),
).encode("utf-8")
return base64.urlsafe_b64encode(raw).rstrip(b"=").decode("ascii")
def read_cursor(value: str, scope: str) -> Tuple[str, str]:
try:
raw = base64.urlsafe_b64decode(value + "=" * (-len(value) % 4))
data = json.loads(raw)
if set(data) != {"v", "scope", "created_at", "id"}:
raise ValueError
if data["v"] != 1 or data["scope"] != scope or not valid_uuid(data["id"]):
raise ValueError
datetime.fromisoformat(data["created_at"].replace("Z", "+00:00"))
return data["created_at"], data["id"]
except (ValueError, TypeError, AttributeError, UnicodeError, json.JSONDecodeError, binascii.Error):
raise ApiError(400, "invalid_request", "Invalid cursor") from None
def secret(name):
return Path(os.environ[name + "_FILE"]).read_text().strip()
POOL = ConnectionPool(secret("DATABASE_URL"), min_size=1, max_size=8, timeout=3,
max_waiting=32, open=False, check=ConnectionPool.check_connection,
kwargs={"row_factory": dict_row, "connect_timeout": 3,
"options": "-c statement_timeout=3000 -c lock_timeout=2000"})
CACHE = redis.Redis.from_url(secret("CACHE_URL"), socket_connect_timeout=0.2,
socket_timeout=0.2, decode_responses=True)
HASHER = PasswordHasher(time_cost=2, memory_cost=19456, parallelism=1)
DUMMY_HASH = HASHER.hash(secrets.token_urlsafe(32))
TTL = env_int("SESSION_TTL_SECONDS", 3600, 1)
@asynccontextmanager
async def lifespan(app):
POOL.open()
try:
yield
finally:
POOL.close()
CACHE.close()
app = FastAPI(title="MTUSI Messenger — Lab 4", version="4.0.0", lifespan=lifespan)
def wire(row):
return {k: (v.isoformat(timespec="microseconds").replace("+00:00", "Z")
if isinstance(v, datetime) else str(v) if isinstance(v, uuid.UUID) else v)
for k, v in row.items()}
def query(sql, params=()):
with POOL.connection() as conn:
return [wire(row) for row in conn.execute(sql, params).fetchall()]
def profile(user_id):
key = "user:" + user_id
try:
value = CACHE.get(key)
if value:
return json.loads(value)
except (redis.RedisError, ValueError):
pass
rows = query("SELECT id, username, display_name, created_at FROM users WHERE id=%s", (user_id,))
if not rows:
raise ApiError(401, "unauthorized", "Invalid session")
user = rows[0]
try:
CACHE.setex(key, 60, json.dumps(user))
except redis.RedisError:
pass
return user
def token_hash(token):
return hashlib.sha256(token.encode()).hexdigest()
def bearer(authorization):
if not authorization or not authorization.startswith("Bearer "):
raise ApiError(401, "unauthorized", "Bearer session required")
token = authorization[7:]
if not re.fullmatch(r"[A-Za-z0-9_-]{43}", token):
raise ApiError(401, "unauthorized", "Invalid session")
return token_hash(token)
def current_user(authorization: Optional[str] = Header(default=None)):
digest = bearer(authorization)
rows = query("SELECT user_id FROM sessions WHERE token_hash=%s AND expires_at > clock_timestamp()", (digest,))
if not rows:
raise ApiError(401, "unauthorized", "Invalid session")
return profile(rows[0]["user_id"])
def visible_chat(chat_id, actor_id):
if not valid_uuid(chat_id):
raise ApiError(400, "invalid_request", "Invalid chat id")
rows = query("SELECT c.* FROM chats c JOIN members m ON m.chat_id=c.id WHERE c.id=%s AND m.user_id=%s", (chat_id, actor_id))
if not rows:
raise ApiError(404, "not_found", "Chat not found")
return chat_members(rows)[0]
def chat_members(rows):
if rows:
members = query("SELECT chat_id, user_id FROM members WHERE chat_id=ANY(%s::uuid[]) ORDER BY user_id", ([r["id"] for r in rows],))
grouped = {r["id"]: [] for r in rows}
for m in members:
grouped[m["chat_id"]].append(m["user_id"])
for row in rows:
row["member_ids"] = grouped[row["id"]]
return rows
def page_args(request, scope, limit, cursor):
if set(request.query_params).difference({"limit", "cursor"}):
raise ApiError(400, "invalid_request", "Invalid query parameters")
return limit, read_cursor(cursor, scope) if cursor is not None else None
def page(rows, limit, scope):
return {"items": rows[:limit], "next_cursor": make_cursor(scope, rows[limit-1]) if len(rows)>limit else None}
def request_id(request: Request) -> str:
return getattr(request.state, "request_id", str(uuid.uuid4()))
def error_response(request: Request, status: int, code: str, message: str) -> JSONResponse:
rid = request_id(request)
response = JSONResponse(
status_code=status,
content={"error": {"code": code, "message": message, "request_id": rid}},
)
response.headers["X-Request-Id"] = rid
response.headers["X-Instance-Id"] = INSTANCE_ID
if status == 401:
response.headers["WWW-Authenticate"] = ('Basic realm="messenger", charset="UTF-8"'
if request.url.path == "/api/v1/auth/sessions" else 'Bearer realm="messenger"')
return response
@app.middleware("http")
async def request_envelope(request: Request, call_next: Any) -> JSONResponse:
request.state.request_id = str(uuid.uuid4())
try:
response = await call_next(request)
except Exception:
LOG.error("request_id=%s unexpected error", request_id(request))
response = error_response(request, 500, "internal_error", "Internal server error")
response.headers["X-Request-Id"] = request_id(request)
response.headers["X-Instance-Id"] = INSTANCE_ID
return response
@app.exception_handler(ApiError)
async def api_error_handler(request: Request, error: ApiError) -> JSONResponse:
return error_response(request, error.status, error.code, error.message)
@app.exception_handler(RequestValidationError)
async def validation_error_handler(request: Request, _error: RequestValidationError) -> JSONResponse:
return error_response(request, 400, "invalid_request", "Request validation failed")
@app.exception_handler(StarletteHTTPException)
async def http_error_handler(request: Request, error: StarletteHTTPException) -> JSONResponse:
if error.status_code == 404:
return error_response(request, 404, "not_found", "Resource not found")
return error_response(request, error.status_code, "invalid_request", "Request failed")
async def require_json(request: Request) -> None:
content_type = request.headers.get("content-type", "").split(";", 1)[0].strip().lower()
if content_type != "application/json":
raise ApiError(415, "unsupported_media_type", "Content-Type must be application/json")
@app.exception_handler(psycopg.Error)
@app.exception_handler(PoolTimeout)
@app.exception_handler(TooManyRequests)
async def database_error(request, error):
return error_response(request, 503, "unavailable", "Database temporarily unavailable")
@app.get("/health/live", response_model=Health)
def health_live():
return {"status": "ok"}
@app.get("/health/ready", response_model=Health)
def health_ready():
query("SELECT 1 FROM schema_migrations LIMIT 1")
storage.client.head_bucket(Bucket=storage.BUCKET)
return {"status": "ok"}
@app.post("/api/v1/users", response_model=User, status_code=201, dependencies=[Depends(require_json)])
def create_user(payload: CreateUser):
password_hash = HASHER.hash(payload.password)
try:
return query("INSERT INTO users (id,username,display_name,password_hash) VALUES (%s,%s,%s,%s) RETURNING id,username,display_name,created_at",
(uuid.uuid4(), payload.username, payload.display_name, password_hash))[0]
except psycopg.errors.UniqueViolation:
raise ApiError(409, "conflict", "Username already exists") from None
@app.post("/api/v1/auth/sessions", status_code=201)
async def login(request: Request, authorization: Optional[str] = Header(default=None)):
# CPU/SQL work runs in FastAPI's worker pool, not in the event loop.
from starlette.concurrency import run_in_threadpool
if await request.body():
raise ApiError(400, "invalid_request", "Session request must not have a body")
return await run_in_threadpool(login_sync, authorization)
def login_sync(authorization):
try:
if not authorization or not authorization.startswith("Basic "):
raise ValueError
username, password = base64.b64decode(authorization[6:], validate=True).decode("utf-8").split(":", 1)
if not USERNAME_RE.fullmatch(username) or not 12 <= len(password) <= 128:
raise ValueError
except (ValueError, UnicodeError, binascii.Error):
raise ApiError(401, "unauthorized", "Invalid credentials") from None
users = query("SELECT * FROM users WHERE username=%s", (username,))
try:
HASHER.verify(users[0]["password_hash"] if users else DUMMY_HASH, password)
except VerificationError:
raise ApiError(401, "unauthorized", "Invalid credentials") from None
if not users:
raise ApiError(401, "unauthorized", "Invalid credentials")
token = secrets.token_urlsafe(32)
expiry = query("INSERT INTO sessions(token_hash,user_id,expires_at) VALUES (%s,%s,clock_timestamp() + %s * interval '1 second') RETURNING expires_at",
(token_hash(token), users[0]["id"], TTL))[0]["expires_at"]
user = users[0]
del user["password_hash"]
return {"token": token, "token_type": "Bearer", "expires_at": expiry, "user": user}
@app.delete("/api/v1/auth/sessions/current", status_code=204)
def logout(actor=Depends(current_user), authorization: Optional[str] = Header(default=None)):
query("DELETE FROM sessions WHERE token_hash=%s RETURNING user_id", (bearer(authorization),))
return Response(status_code=204)
@app.get("/api/v1/users/me", response_model=User)
def get_me(actor=Depends(current_user)):
return actor
@app.post("/api/v1/chats", response_model=Chat, status_code=201, dependencies=[Depends(require_json)])
def create_chat(payload: CreateChat, actor=Depends(current_user)):
members = [str(m) for m in payload.member_ids]
if actor["id"] not in members:
raise ApiError(400, "invalid_request", "The current user must be a member")
try:
with POOL.connection() as conn:
chat = wire(conn.execute("INSERT INTO chats(id,title) VALUES (%s,%s) RETURNING *", (uuid.uuid4(),payload.title)).fetchone())
with conn.cursor() as cur:
cur.executemany("INSERT INTO members(chat_id,user_id) VALUES (%s,%s)", [(chat["id"],m) for m in members])
chat["member_ids"] = members
return chat
except psycopg.errors.ForeignKeyViolation:
raise ApiError(400, "invalid_request", "All chat members must exist") from None
@app.get("/api/v1/chats", response_model=ChatPage)
def list_chats(request: Request, limit: int = Query(50,ge=1,le=100), cursor: Optional[str] = Query(None,min_length=1), actor=Depends(current_user)):
scope = "chats:" + actor["id"]
limit, after = page_args(request,scope,limit,cursor)
sql = "SELECT c.* FROM chats c JOIN members m ON c.id=m.chat_id WHERE m.user_id=%s"
params = [actor["id"]]
if after:
sql += " AND (c.created_at,c.id) > (%s,%s)"
params.extend(after)
rows = query(sql + " ORDER BY c.created_at,c.id LIMIT %s", params + [limit+1])
return page(chat_members(rows),limit,scope)
@app.get("/api/v1/chats/{chat_id}", response_model=Chat)
def get_chat(chat_id: str, actor=Depends(current_user)):
return visible_chat(chat_id,actor["id"])
@app.post("/api/v1/chats/{chat_id}/messages", response_model=Message, status_code=201, dependencies=[Depends(require_json)])
def create_message(chat_id: str, payload: CreateMessage, actor=Depends(current_user),
idempotency_key: Optional[str] = Header(default=None)):
visible_chat(chat_id, actor["id"])
if not idempotency_key or not re.fullmatch(r"[A-Za-z0-9._:-]{1,128}", idempotency_key):
raise ApiError(400, "invalid_request", "Invalid Idempotency-Key")
ids = [str(i) for i in payload.attachment_ids]
body = json.dumps({"text": payload.text, "attachment_ids": ids}, sort_keys=True)
scope = actor["id"] + ":" + chat_id + ":" + idempotency_key
lock = int.from_bytes(hashlib.sha256(scope.encode()).digest()[:8], "big", signed=True)
with POOL.connection() as conn:
conn.execute("SELECT pg_advisory_xact_lock(%s)", (lock,))
previous = conn.execute("SELECT * FROM idempotency WHERE user_id=%s AND chat_id=%s AND key=%s AND expires_at>clock_timestamp()",
(actor["id"], chat_id, idempotency_key)).fetchone()
if previous:
if previous["body"] != body:
raise ApiError(409, "conflict", "Key already used for a different message")
return previous["response"]
if ids:
found = conn.execute("SELECT id FROM attachments WHERE id=ANY(%s::uuid[]) AND chat_id=%s AND status='ready'", (ids, chat_id)).fetchall()
if len(found) != len(ids):
raise ApiError(404, "not_found", "Attachment not found")
msg = wire(conn.execute("INSERT INTO messages(id,chat_id,sender_id,text,attachment_ids) VALUES (%s,%s,%s,%s,%s::uuid[]) RETURNING *",
(uuid.uuid4(),chat_id,actor["id"],payload.text,ids)).fetchone())
msg["attachment_ids"] = ids
from psycopg.types.json import Jsonb
conn.execute("""INSERT INTO idempotency(user_id,chat_id,key,body,response) VALUES (%s,%s,%s,%s,%s)
ON CONFLICT(user_id,chat_id,key) DO UPDATE SET body=EXCLUDED.body,response=EXCLUDED.response,expires_at=clock_timestamp()+interval '24 hours'""",
(actor["id"],chat_id,idempotency_key,body,Jsonb(msg)))
return msg
@app.get("/api/v1/chats/{chat_id}/messages", response_model=MessagePage)
def list_messages(request: Request, chat_id: str, limit: int = Query(50,ge=1,le=100), cursor: Optional[str] = Query(None,min_length=1), actor=Depends(current_user)):
visible_chat(chat_id,actor["id"])
scope = "messages:" + chat_id
limit,after = page_args(request,scope,limit,cursor)
sql = "SELECT * FROM messages WHERE chat_id=%s"
params = [chat_id]
if after:
sql += " AND (created_at,id) > (%s,%s)"
params.extend(after)
return page(query(sql+" ORDER BY created_at,id LIMIT %s",params+[limit+1]),limit,scope)
import storage
from attachments import install
install(app, POOL, current_user, visible_chat, query, wire, ApiError)
@app.exception_handler(storage.StorageError)
@app.exception_handler(storage.ClientError)
async def storage_error(request, error):
return error_response(request, 503, "unavailable", "Storage temporarily unavailable")
if __name__ == "__main__":
uvicorn.run(app, host=os.environ.get("APP_HOST","0.0.0.0"), port=env_int("APP_PORT",8080,1))
+24
View File
@@ -0,0 +1,24 @@
"""Bounded S3 client and image validation shared by the API and cleaner."""
import os
import warnings
from pathlib import Path
import boto3
from botocore.config import Config
from botocore.exceptions import BotoCoreError as StorageError, ClientError
from PIL import Image, UnidentifiedImageError
MAX_BYTES = 10_485_760
MAX_PIXELS = 40_000_000
BUCKET = os.environ.get('S3_BUCKET', 'messenger-attachments')
Image.MAX_IMAGE_PIXELS = MAX_PIXELS
def secret(name):
return Path(os.environ[name + '_FILE']).read_text().strip()
client = boto3.client('s3', endpoint_url=os.environ.get('S3_ENDPOINT', 'http://s3:9000'),
region_name='us-east-1', aws_access_key_id=secret('S3_ACCESS_KEY_ID'),
aws_secret_access_key=secret('S3_SECRET_ACCESS_KEY'),
config=Config(connect_timeout=2, read_timeout=10, retries={'total_max_attempts': 1},
s3={'addressing_style': 'path'}))
from images import validate_image, InvalidImage, TooManyPixels
+101
View File
@@ -0,0 +1,101 @@
name: mtusi-lab4
x-app: &app
build: .
image: mtusi-messenger:lab4
mem_limit: 768m
cpus: 1.0
environment: &env
APP_HOST: 0.0.0.0
APP_PORT: 8080
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
DATABASE_URL_FILE: /run/secrets/database_url
CACHE_URL_FILE: /run/secrets/cache_url
S3_ENDPOINT: http://s3:9000
S3_BUCKET: messenger-attachments
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
secrets: [database_url, cache_url, s3_access_key_id, s3_secret_access_key]
stop_grace_period: 10s
services:
postgres:
image: postgres:16.6-bookworm
environment:
POSTGRES_USER: messenger
POSTGRES_DB: messenger
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
secrets: [postgres_password]
volumes: [postgres-data:/var/lib/postgresql/data]
healthcheck:
test: [CMD-SHELL, 'pg_isready -U messenger -d messenger']
interval: 2s
timeout: 3s
retries: 30
valkey:
image: valkey/valkey:8.0.2-alpine
command: [valkey-server, --appendonly, 'yes', --maxmemory, 64mb, --maxmemory-policy, allkeys-lru]
volumes: [valkey-data:/data]
s3:
image: minio/minio:RELEASE.2025-04-22T22-12-26Z
command: [server, /data, --console-address, '127.0.0.1:9001']
environment:
MINIO_ROOT_USER_FILE: /run/secrets/s3_root_user
MINIO_ROOT_PASSWORD_FILE: /run/secrets/s3_root_password
secrets: [s3_root_user, s3_root_password]
volumes: [s3-data:/data]
healthcheck:
test: [CMD, curl, -f, http://localhost:9000/minio/health/live]
interval: 3s
timeout: 3s
retries: 30
s3-init:
image: minio/mc:RELEASE.2025-04-16T18-13-26Z
entrypoint: [/bin/sh, /config/init.sh]
volumes: ['./infra/s3:/config:ro']
secrets: [s3_root_user, s3_root_password, s3_access_key_id, s3_secret_access_key]
depends_on:
s3: {condition: service_healthy}
migrate:
<<: *app
command: [python, app/migrate.py]
healthcheck: {disable: true}
depends_on:
postgres: {condition: service_healthy}
api1:
<<: *app
environment:
<<: *env
INSTANCE_ID: api1
depends_on: &deps
migrate: {condition: service_completed_successfully}
s3-init: {condition: service_completed_successfully}
valkey: {condition: service_started}
api2:
<<: *app
environment:
<<: *env
INSTANCE_ID: api2
depends_on: *deps
cleaner:
<<: *app
command: [python, app/cleanup.py]
healthcheck: {disable: true}
depends_on: *deps
lb:
image: haproxy:3.0.8-alpine
ports: ['127.0.0.1:${HOST_PORT:-8084}:8080']
volumes: ['./infra/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro']
depends_on:
api1: {condition: service_healthy}
api2: {condition: service_healthy}
volumes:
postgres-data:
valkey-data:
s3-data:
secrets:
postgres_password: {file: ./secrets/postgres_password}
database_url: {file: ./secrets/database_url}
cache_url: {file: ./secrets/cache_url}
s3_root_user: {file: ./secrets/s3_root_user}
s3_root_password: {file: ./secrets/s3_root_password}
s3_access_key_id: {file: ./secrets/s3_access_key_id}
s3_secret_access_key: {file: ./secrets/s3_secret_access_key}
+22
View File
@@ -0,0 +1,22 @@
$ make check
python3 scripts/check_contract.py
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
exit=0
$ docker compose config --quiet
exit=0
$ python3 student-tests/unit.py
test_mime_substitution (__main__.Images) ... ok
test_not_image (__main__.Images) ... ok
test_pixel_limit_before_decode (__main__.Images) ... ok
test_png_and_jpeg (__main__.Images) ... ok
test_truncated_pixels (__main__.Images) ... ok
----------------------------------------------------------------------
Ran 5 tests in 0.090s
OK
exit=0
PASS: AST parsed for 15 Python files.
+21
View File
@@ -0,0 +1,21 @@
global
log stdout format raw local0
maxconn 128
defaults
mode http
log global
timeout connect 2s
timeout client 40s
timeout server 40s
timeout http-request 10s
timeout queue 3s
frontend http
bind :8080
default_backend api
backend api
balance roundrobin
option httpchk GET /health/ready
http-check expect status 200
retries 0
server api1 api1:8080 check inter 1s fall 2 rise 1
server api2 api2:8080 check inter 1s fall 2 rise 1
+9
View File
@@ -0,0 +1,9 @@
#!/bin/sh
set -eu
mc alias set local http://s3:9000 "$(cat /run/secrets/s3_root_user)" "$(cat /run/secrets/s3_root_password)" >/dev/null
mc mb --ignore-existing local/messenger-attachments >/dev/null
mc anonymous set none local/messenger-attachments >/dev/null
mc admin user add local "$(cat /run/secrets/s3_access_key_id)" "$(cat /run/secrets/s3_secret_access_key)" >/dev/null
mc admin policy create local attachments /config/policy.json >/dev/null
mc admin policy attach local attachments --user "$(cat /run/secrets/s3_access_key_id)" >/dev/null
echo 'Private bucket and restricted API user ready'
+7
View File
@@ -0,0 +1,7 @@
{
"Version": "2012-10-17",
"Statement": [
{"Effect":"Allow","Action":["s3:ListBucket"],"Resource":["arn:aws:s3:::messenger-attachments"]},
{"Effect":"Allow","Action":["s3:GetObject","s3:PutObject","s3:DeleteObject"],"Resource":["arn:aws:s3:::messenger-attachments/originals/*"]}
]
}
+13
View File
@@ -0,0 +1,13 @@
CREATE TABLE users (
id uuid PRIMARY KEY, username varchar(32) NOT NULL UNIQUE CHECK (username ~ '^[a-z0-9_]{3,32}$'),
display_name varchar(100) NOT NULL CHECK (length(trim(display_name))>0),
password_hash text NOT NULL, created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE TABLE chats (id uuid PRIMARY KEY, title varchar(100) NOT NULL CHECK(length(trim(title))>0), created_at timestamptz NOT NULL DEFAULT clock_timestamp());
CREATE TABLE members (chat_id uuid REFERENCES chats(id), user_id uuid REFERENCES users(id), PRIMARY KEY(chat_id,user_id));
CREATE INDEX members_user_chat ON members(user_id,chat_id);
CREATE TABLE messages (id uuid PRIMARY KEY, chat_id uuid NOT NULL REFERENCES chats(id), sender_id uuid NOT NULL REFERENCES users(id), text varchar(4000) NOT NULL CHECK(length(trim(text))>0), created_at timestamptz NOT NULL DEFAULT clock_timestamp(), FOREIGN KEY(chat_id,sender_id) REFERENCES members(chat_id,user_id));
CREATE INDEX messages_history ON messages(chat_id,created_at,id);
CREATE INDEX chats_order ON chats(created_at,id);
CREATE TABLE sessions (token_hash char(64) PRIMARY KEY, user_id uuid NOT NULL REFERENCES users(id), expires_at timestamptz NOT NULL);
CREATE INDEX sessions_expiry ON sessions(expires_at);
+18
View File
@@ -0,0 +1,18 @@
ALTER TABLE messages DROP CONSTRAINT messages_text_check;
ALTER TABLE messages ADD COLUMN attachment_ids uuid[] NOT NULL DEFAULT '{}';
ALTER TABLE messages ADD CHECK (length(trim(text))>0 OR cardinality(attachment_ids)>0);
CREATE TABLE idempotency (
user_id uuid REFERENCES users(id), chat_id uuid REFERENCES chats(id), key varchar(128),
body text NOT NULL, response jsonb NOT NULL,
expires_at timestamptz NOT NULL DEFAULT clock_timestamp()+interval '24 hours',
PRIMARY KEY(user_id,chat_id,key)
);
CREATE TABLE attachments (
id uuid PRIMARY KEY, chat_id uuid NOT NULL REFERENCES chats(id), owner_id uuid NOT NULL REFERENCES users(id),
filename varchar(255) NOT NULL, object_key text NOT NULL UNIQUE,
content_type text NOT NULL, size_bytes bigint NOT NULL, sha256 char(64) NOT NULL,
status text NOT NULL CHECK(status IN ('pending','ready')),
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
updated_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE INDEX attachments_pending ON attachments(created_at) WHERE status='pending';
+32
View File
@@ -0,0 +1,32 @@
"""Generate local-only synthetic credentials once; never overwrite them."""
import os
import secrets
from pathlib import Path
root = Path(__file__).resolve().parents[1]
os.umask(0o077)
(root / "secrets").mkdir(exist_ok=True)
p = root / "secrets/postgres_password"
if not p.exists():
p.write_text(secrets.token_hex(32))
values = {"database_url": "postgresql://messenger:" + p.read_text().strip() + "@postgres:5432/messenger", "cache_url": "redis://valkey:6379/0"}
for name, value in values.items():
dest = root / "secrets" / name
if not dest.exists():
dest.write_text(value)
# Docker Desktop bind-mounted secrets need read permission for container UID 10001.
for name in ["postgres_password", "database_url", "cache_url"]:
(root / "secrets" / name).chmod(0o644)
env = root / ".env"
if not env.exists():
env.write_text("SESSION_TTL_SECONDS=3600\nHOST_PORT=8084\n")
print("Local configuration ready (secrets not printed)")
for name, value in {
's3_root_user': 'admin-' + secrets.token_hex(8),
's3_root_password': secrets.token_hex(32),
's3_access_key_id': 'api-' + secrets.token_hex(8),
's3_secret_access_key': secrets.token_hex(32),
}.items():
dest = root / 'secrets' / name
if not dest.exists():
dest.write_text(value)
dest.chmod(0o644)
+4
View File
@@ -0,0 +1,4 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")/.."
docker compose restart postgres valkey s3 api1 api2 cleaner lb
+7
View File
@@ -0,0 +1,7 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")/.."
case "${TARGET_INSTANCE_ID:-}" in
api1|api2) docker compose stop "$TARGET_INSTANCE_ID" ;;
*) echo 'Unknown target instance' >&2; exit 1 ;;
esac
+40
View File
@@ -0,0 +1,40 @@
"""Run all checks and save actual outputs. Stop on the first failure."""
import os
from pathlib import Path
import subprocess
import sys
root=Path(__file__).resolve().parents[1]
os.chdir(root)
os.environ.setdefault('BASE_URL','http://localhost:8084')
(root/'evidence').mkdir(exist_ok=True)
def run(name, args, stdin=None):
result=subprocess.run(args,input=stdin,text=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
(root/'evidence'/name).write_text(result.stdout)
print(result.stdout,end='')
if result.returncode:
raise SystemExit(result.returncode)
run('check.txt',['make','check'])
run('unit.txt',[sys.executable,'student-tests/unit.py'])
run('smoke.txt',['make','test','BASE_URL='+os.environ['BASE_URL']])
run('student-tests.txt',[sys.executable,'student-tests/verify.py'])
subprocess.run(['docker','compose','stop','cleaner'],check=True)
try:
run('storage-checks.txt',['docker','compose','exec','-T','api1','python','-'],
(root/'student-tests/storage_checks.py').read_text())
finally:
subprocess.run(['docker','compose','start','cleaner'],check=True)
run('outage.txt',[sys.executable,'student-tests/outage.py'])
run('bootstrap-repeat.txt',['docker','compose','run','--rm','s3-init'])
run('containers-before.txt',['docker','compose','ps','--format','json'])
run('persistence.txt',['make','test-persistence','ACTION_SCRIPT=scripts/restart.sh','BASE_URL='+os.environ['BASE_URL']])
run('containers-after.txt',['docker','compose','ps','--format','json'])
try:
run('failover.txt',['make','test-failover','ACTION_SCRIPT=scripts/stop-one.sh','BASE_URL='+os.environ['BASE_URL']])
finally:
subprocess.run(['docker','compose','start','api1','api2'],check=True)
# Compose waits for the API's actual readiness after restoring the replica.
subprocess.run(['docker','compose','up','-d','--wait','api1','api2'],check=True)
run('memory.txt',[sys.executable,'student-tests/memory.py'])
print('All requested checks passed; actual results are in evidence/.')
+57
View File
@@ -0,0 +1,57 @@
"""Start ten simultaneous uploads; record actual container RSS via /proc."""
from concurrent.futures import ThreadPoolExecutor
import json
import os
from pathlib import Path
import subprocess
import sys
import threading
import time
sys.path.insert(0,str(Path(__file__).resolve().parents[1]/'tests'))
from client import Client, check
from PIL import Image
import io
c=Client(timeout=45)
a,b,e,chat=c.bootstrap()
file=io.BytesIO()
# Incompressible ~9 MiB RGB PNG rather than ten tiny fixtures.
Image.frombytes('RGB',(1750,1750),os.urandom(1750*1750*3)).save(file,'PNG')
data=file.getvalue()
check(len(data)<=10485760,'fixture too large')
barrier=threading.Barrier(10)
finished=threading.Event()
samples=[]
def rss():
while not finished.is_set():
row={'seconds':round(time.monotonic()-start,3)}
for name in ['api1','api2']:
result=subprocess.run(['docker','compose','exec','-T',name,'python','-c',
"from pathlib import Path; print(next(x.split()[1] for x in Path('/proc/1/status').read_text().splitlines() if x.startswith('VmRSS:')))"] ,capture_output=True,text=True,check=True)
row[name+'_rss_kib']=int(result.stdout.strip())
samples.append(row)
finished.wait(0.1)
def upload(i):
barrier.wait()
response=Client(timeout=45).call('POST','/api/v1/chats/'+chat['id']+'/attachments?filename=load.png',
[201,503],actor=a,raw=data,headers={'Content-Type':'image/png'})
return {'status':response['status'],'seconds':round(response['elapsed'],3)}
start=time.monotonic()
with ThreadPoolExecutor(max_workers=1) as monitor:
future=monitor.submit(rss)
try:
with ThreadPoolExecutor(max_workers=10) as pool:
results=list(pool.map(upload,range(10)))
finally:
finished.set()
future.result()
check(samples,'no memory measurements')
report={'concurrency':10,'file_bytes':len(data),'results':results,'samples':samples,
'peak_rss_kib':{n:max(s[n+'_rss_kib'] for s in samples) for n in ['api1','api2']},
'sampling_note':'Sequential /proc sampling; very brief peaks may be missed. 503 is explicit admission control.'}
Path('evidence/memory.json').write_text(json.dumps(report,indent=2))
check(any(r['status']==201 for r in results),'no upload succeeded')
print(json.dumps({k:v for k,v in report.items() if k!='samples'},indent=2))
+66
View File
@@ -0,0 +1,66 @@
"""Exercise interrupted HTTP upload and unavailable S3 on this Compose project."""
import http.client
import json
import os
from pathlib import Path
import subprocess
import sys
import time
from urllib.parse import urlsplit
sys.path.insert(0,str(Path(__file__).resolve().parents[1]/'tests'))
from client import Client, check
c=Client(timeout=45)
a,b,e,chat=c.bootstrap()
att,png=c.attachment(a,chat)
base=urlsplit(c.base)
check(base.hostname in ('localhost','127.0.0.1'),'test is limited to the local lab')
# Close while the body is incomplete. No intent should be created before validation.
connection=http.client.HTTPConnection(base.hostname,base.port,timeout=5)
connection.putrequest('POST','/api/v1/chats/'+chat['id']+'/attachments?filename=interrupted.png')
connection.putheader('Authorization','Bearer '+a['token'])
connection.putheader('Content-Type','image/png')
connection.putheader('Content-Length',str(len(png)))
connection.endheaders()
connection.send(png[:16])
connection.close()
# Query only the synthetic chat through stdin; credentials/tokens are never printed.
probe="""
import sys
sys.path.insert(0,'/srv/app')
import psycopg,storage
with psycopg.connect(storage.secret('DATABASE_URL')) as db:
n=db.execute("SELECT count(*) FROM attachments WHERE chat_id=%s", (CHAT,)).fetchone()[0]
assert n==1, 'interrupted upload created metadata'
print('PASS: interrupted upload did not create an attachment')
""".replace('CHAT',repr(chat['id']))
subprocess.run(['docker','compose','exec','-T','api1','python','-'],input=probe,text=True,check=True)
# Direct request within api1 ensures the API's structured 503 is tested,
# independently of HAProxy taking all unready instances out of rotation.
probe="""
import urllib.request,urllib.error
req=urllib.request.Request('http://localhost:8080/api/v1/attachments/ATT/content',headers={'Authorization':'Bearer TOKEN'})
try:
urllib.request.urlopen(req,timeout=20)
raise AssertionError('unavailable S3 returned success')
except urllib.error.HTTPError as error:
assert error.code==503, error.code
print('PASS: unavailable S3 returns 503 from API')
""".replace('ATT',att['id']).replace('TOKEN',a['token'])
try:
subprocess.run(['docker','compose','stop','s3'],check=True)
subprocess.run(['docker','compose','exec','-T','api1','python','-'],input=probe,text=True,check=True)
finally:
subprocess.run(['docker','compose','up','-d','--wait','s3'],check=True)
for attempt in range(30):
try:
result=c.call('GET','/api/v1/attachments/'+att['id']+'/content',actor=b)
check(result['bytes']==png,'recovered file differs')
break
except AssertionError:
if attempt==29:
raise
time.sleep(1)
print('PASS: S3 recovery preserves the original bytes')
+60
View File
@@ -0,0 +1,60 @@
"""Run inside the API container: private bucket and cleanup failure recovery."""
import io
import json
from pathlib import Path
import sys
import uuid
import urllib.request
import urllib.error
sys.path.insert(0, '/srv/app')
import psycopg
import storage
from cleanup import sweep
prefix = uuid.uuid4().hex
ids = [str(uuid.uuid4()) for _ in range(3)]
keys = ['originals/'+i for i in ids]
user,chat = str(uuid.uuid4()),str(uuid.uuid4())
dsn=storage.secret('DATABASE_URL')
# Model a process stopping after S3 PUT but before final DB COMMIT.
# Also create a fresh intent and an old ready file; neither may be removed.
try:
with psycopg.connect(dsn) as db:
db.execute('INSERT INTO users(id,username,display_name,password_hash) VALUES (%s,%s,%s,%s)',
(user,'cleanup_'+prefix[:12],'Synthetic cleanup','not-a-login-hash'))
db.execute('INSERT INTO chats(id,title) VALUES (%s,%s)', (chat,'Synthetic cleanup'))
for i,(aid,key) in enumerate(zip(ids,keys)):
db.execute("""INSERT INTO attachments(id,chat_id,owner_id,filename,object_key,content_type,size_bytes,sha256,status,created_at)
VALUES (%s,%s,%s,'test.png',%s,'image/png',3,%s,%s,clock_timestamp()-%s*interval '1 hour')""",
(aid,chat,user,key,'0'*64,'ready' if i==2 else 'pending',2 if i!=1 else 0))
for key in keys:
storage.client.put_object(Bucket=storage.BUCKET,Key=key,Body=b'abc')
try:
urllib.request.urlopen('http://s3:9000/'+storage.BUCKET+'/'+keys[2],timeout=5)
raise AssertionError('Anonymous object access allowed')
except urllib.error.HTTPError as error:
assert error.code==403, error.code
# Roll back a simulated failed finalization; durable pending intent remains.
with psycopg.connect(dsn) as db:
db.execute("UPDATE attachments SET status='ready' WHERE id=%s", (ids[0],))
db.rollback()
sweep()
sweep() # Idempotent after completion.
with psycopg.connect(dsn) as db:
remaining={str(r[0]) for r in db.execute('SELECT id FROM attachments WHERE id=ANY(%s::uuid[])',(ids,))}
assert remaining==set(ids[1:]), remaining
try:
storage.client.head_object(Bucket=storage.BUCKET,Key=keys[0])
raise AssertionError('Abandoned object still exists')
except storage.ClientError as error:
assert error.response['ResponseMetadata']['HTTPStatusCode']==404
for key in keys[1:]:
storage.client.head_object(Bucket=storage.BUCKET,Key=key)
print('PASS: private bucket, failed finalization recovery, grace period, ready protection, repeated cleanup')
finally:
for key in keys:
storage.client.delete_object(Bucket=storage.BUCKET,Key=key)
with psycopg.connect(dsn) as db:
db.execute('DELETE FROM attachments WHERE id=ANY(%s::uuid[])',(ids,))
db.execute('DELETE FROM chats WHERE id=%s',(chat,))
db.execute('DELETE FROM users WHERE id=%s',(user,))
+50
View File
@@ -0,0 +1,50 @@
"""Offline checks of image content, size limits, and MIME detection."""
import io
from pathlib import Path
import struct
import sys
import unittest
import zlib
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'app'))
from PIL import Image
from images import validate_image, InvalidImage, TooManyPixels
class Images(unittest.TestCase):
def picture(self, fmt):
file = io.BytesIO()
Image.new('RGB', (320, 200), (20, 30, 40)).save(file, fmt)
file.seek(0)
return file
def test_png_and_jpeg(self):
for fmt, mime in [('PNG', 'image/png'), ('JPEG', 'image/jpeg')]:
file = self.picture(fmt)
before = file.getvalue()
validate_image(file, mime)
self.assertEqual(file.tell(), 0)
self.assertEqual(file.read(), before)
def test_mime_substitution(self):
with self.assertRaises(InvalidImage):
validate_image(self.picture('PNG'), 'image/jpeg')
def test_not_image(self):
for content in [b'', b'hello', b'<script>alert(1)</script>']:
with self.assertRaises(InvalidImage):
validate_image(io.BytesIO(content), 'image/png')
def test_truncated_pixels(self):
file = self.picture('JPEG')
with self.assertRaises(InvalidImage):
validate_image(io.BytesIO(file.getvalue()[:-20]), 'image/jpeg')
def test_pixel_limit_before_decode(self):
def chunk(kind, data):
return struct.pack('>I', len(data))+kind+data+struct.pack('>I', zlib.crc32(kind+data)&0xffffffff)
huge = b'\x89PNG\r\n\x1a\n'+chunk(b'IHDR',struct.pack('>IIBBBBB', 8000,8000,8,2,0,0,0))
huge += chunk(b'IDAT', zlib.compress(b'\x00'))+chunk(b'IEND', b'')
with self.assertRaises(TooManyPixels):
validate_image(io.BytesIO(huge), 'image/png')
if __name__ == '__main__':
unittest.main(verbosity=2)
+46
View File
@@ -0,0 +1,46 @@
"""Black-box grade-4 checks; creates only synthetic data on the local lab."""
from concurrent.futures import ThreadPoolExecutor
import hashlib
import json
from pathlib import Path
import sys
import threading
import uuid
from urllib.parse import quote
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'tests'))
from client import Client, check
c = Client(timeout=45)
a,b,e,chat = c.bootstrap()
base = '/api/v1/chats/' + chat['id']
att, png = c.attachment(a, chat)
for name in ['../x.png', 'a\\b.png', '\x00.png', '\n.png']:
c.call('POST', base+'/attachments?filename='+quote(name, safe=''), 400, actor=a,
raw=png, headers={'Content-Type':'image/png'})
for data,mime,status in [(b'x'*(10485760+1),'image/png',413),
(b'not an image','image/png',415), (png,'image/jpeg',415),
(png[:20],'image/png',415)]:
c.call('POST', base+'/attachments?filename=x.png', status, actor=a,
raw=data, headers={'Content-Type':mime})
other = c.call('POST','/api/v1/chats',201,actor=a,
body={'title':'Other chat','member_ids':[a['id']]},schema='Chat')['json']
for target,ids,actor in [(other,[att['id']],a), (chat,[str(uuid.uuid4())],a), (chat,[att['id']],e)]:
c.call('POST','/api/v1/chats/'+target['id']+'/messages',404,actor=actor,
body={'text':'','attachment_ids':ids},headers={'Idempotency-Key':uuid.uuid4().hex})
for suffix in ['', '/content']:
c.call('GET','/api/v1/attachments/'+att['id']+suffix,404,actor=e)
# A member other than the uploader can attach and download the same file.
c.message(b, chat, '', attachments=[att['id']])
key = uuid.uuid4().hex
barrier = threading.Barrier(10)
def replay(_):
barrier.wait()
return Client(timeout=45).message(a,chat,'one logical message',key=key)['json']
with ThreadPoolExecutor(max_workers=10) as pool:
messages = list(pool.map(replay,range(10)))
check(all(m == messages[0] for m in messages), 'concurrent replay duplicated messages')
c.call('POST',base+'/messages',409,actor=a,body={'text':'different'},headers={'Idempotency-Key':key})
# Default [] and explicit [] are equivalent.
c.call('POST',base+'/messages',201,actor=a,body={'text':'one logical message','attachment_ids':[]},
headers={'Idempotency-Key':key},schema='Message')
print('PASS: invalid files, ACL, cross-chat references, concurrent idempotency')