forked from mtusi2026/mtusi2026lab4
Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c8e1ef46d3 | ||
|
|
59fc7b96ec | ||
|
|
7fbdef3a6a |
+11
@@ -0,0 +1,11 @@
|
||||
FROM python:3.12.7-slim-bookworm
|
||||
WORKDIR /srv
|
||||
COPY app/requirements.txt app/requirements.txt
|
||||
RUN pip install --no-cache-dir -r app/requirements.txt && useradd --uid 10001 --create-home app
|
||||
COPY app app
|
||||
COPY migrations migrations
|
||||
USER 10001:10001
|
||||
ENV PYTHONUNBUFFERED=1 PYTHONDONTWRITEBYTECODE=1
|
||||
EXPOSE 8080
|
||||
HEALTHCHECK --interval=5s --timeout=4s --start-period=20s CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8080/health/ready', timeout=3)" || exit 1
|
||||
CMD ["python", "app/server.py"]
|
||||
@@ -1,37 +1,175 @@
|
||||
# Отчёт — лабораторная 4
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
Автор: Шурман А.Р., БПИ2302. Целевая оценка: **4**. Исследовательские треки на 5 не выполнялись.
|
||||
|
||||
## Запуск из чистого clone
|
||||
## Статус проверки
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
Полный прогон выполнен **12 ноября 2026 года** на macOS arm64 / Docker Desktop.
|
||||
`make check`, полная валидация OpenAPI, 5 unit-тестов и весь `scripts/verify.py`
|
||||
прошли. Реальные результаты находятся в `evidence/`: smoke, негативные сценарии,
|
||||
приватность bucket, безопасная очистка, обрыв upload, отказ S3, повторный bootstrap,
|
||||
persistence, failover и нагрузка. Контракт и публичные тесты не изменены.
|
||||
Первоначальный `evidence/offline.txt` сохранён как историческая проверка до запуска Docker.
|
||||
|
||||
## Архитектура и решения
|
||||
## Запуск
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab3, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
Нужны Docker с Compose, Python 3.9+ на хосте (для memory/unit — Pillow).
|
||||
API работает на Python 3.12.7. Новый стенд использует порт **8084**, отдельные volumes
|
||||
и Compose project `mtusi-lab4`, поэтому не занимает порт предыдущих работ.
|
||||
|
||||
## Доказательства критериев
|
||||
```sh
|
||||
cd mtusi2026lab4-fork
|
||||
python3 -m venv .venv
|
||||
.venv/bin/pip install -r requirements-dev.txt Pillow==11.1.0
|
||||
python3 scripts/bootstrap.py
|
||||
docker compose config --quiet
|
||||
docker compose up -d --build --wait
|
||||
BASE_URL=http://localhost:8084 .venv/bin/python scripts/verify.py
|
||||
make validate PYTHON=.venv/bin/python
|
||||
```
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
Команды выполняются из корня этой лабораторной. Если репозиторий склонирован отдельно,
|
||||
первую команду замените на переход в его каталог. `bootstrap.py` генерирует локальные
|
||||
секреты, не выводит и не перезаписывает их. Повторный запуск сохраняет данные.
|
||||
Для локальных тестов при отсутствии Pillow: отдельное venv и `pip install Pillow==11.1.0`.
|
||||
Полная проверка OpenAPI: `pip install -r requirements-dev.txt`, затем `make validate`.
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
API: http://localhost:8084; Swagger: http://localhost:8084/docs.
|
||||
Остановка без потери данных: `docker compose down` (без `-v`).
|
||||
Возобновление: `docker compose up -d --wait`.
|
||||
|
||||
## Эксперименты и отказы
|
||||
Готовые образы `minio/minio` и `minio/mc` оказались недоступны при проверке
|
||||
12 ноября 2026 года (Docker Hub: pull access denied; Quay: 401).
|
||||
Поэтому `infra/s3/Dockerfile` собирает официальные исходники закреплённых релизов
|
||||
[MinIO RELEASE.2025-04-22T22-12-26Z](https://github.com/minio/minio/tree/RELEASE.2025-04-22T22-12-26Z)
|
||||
и [mc RELEASE.2025-04-16T18-13-26Z](https://github.com/minio/mc/tree/RELEASE.2025-04-16T18-13-26Z)
|
||||
в Go 1.24.2. Лицензии включены в образы. Первый запуск требует интернет и время
|
||||
на скачивание Go-зависимостей и компиляцию; следующие сборки используют Docker cache.
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
## Архитектура
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
HAProxy → две FastAPI-реплики → общий PostgreSQL и приватный MinIO.
|
||||
Valkey кэширует публичные профили; сессии проверяются в PostgreSQL при каждом запросе.
|
||||
Исходная основа — собственная лабораторная 2: модели пользователей/чатов,
|
||||
аутентификация, SQL-хранилище и мигратор. Возможности лабораторной 3 добавлены здесь:
|
||||
балансировка, `X-Instance-Id`, общая таблица идемпотентности.
|
||||
Контракт и публичные тесты skeleton не изменены.
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
Каждая реплика возвращает `api1` или `api2`. HAProxy проверяет readiness каждую секунду,
|
||||
исключает реплику после двух неудач. Он не повторяет POST автоматически. При потере ответа
|
||||
клиент повторяет сообщение с тем же `Idempotency-Key`. Advisory lock в PostgreSQL
|
||||
сериализует один ключ пользователя/чата, вставка сообщения и результата атомарны.
|
||||
Повтор в течение 24 часов возвращает исходный объект; другой текст/список вложений — 409.
|
||||
Отсутствующий список вложений и `[]` равны. Просроченная запись ключа заменяется при повторном использовании.
|
||||
История читается из primary без кэша. Keyset cursor не обеспечивает snapshot между страницами.
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
Метаданные Attachment находятся в PostgreSQL, байты — только в S3. Временный файл API
|
||||
существует исключительно во время приёма и проверки запроса и закрывается по выходу.
|
||||
Имя пользователя не используется как S3 key. Ключ — `originals/<UUID>`.
|
||||
Скачивание идёт через API после проверки членства, без redirect и публичных URL.
|
||||
|
||||
## Валидация и ресурсы
|
||||
|
||||
Загрузка принимает сырые PNG/JPEG, максимум 10 MiB. Content-Length проверяется заранее,
|
||||
но фактические байты также считаются во время потока. Общий срок приёма — 30 секунд.
|
||||
Pillow проверяет формат, MIME, структуру и декодирование. Размер не выше 40 млн пикселей
|
||||
проверяется до загрузки пикселей. Анимированные PNG в этой реализации отклоняются.
|
||||
Оригинал сохраняется без преобразования; SHA-256 и размер вычисляются при чтении.
|
||||
Пустые/опасные filename, MIME-подмена и чужие вложения имеют ошибки контракта.
|
||||
|
||||
Одновременно каждая реплика обрабатывает максимум две загрузки. Ожидание слота до 0.5 s,
|
||||
затем явный 503. Две реплики ограничены по 768 MiB и 1 CPU. На диск пишется максимум
|
||||
10 MiB на активную загрузку, декодирование занимает ограниченную память до размера
|
||||
изображения. S3: connect timeout 2 s, read timeout 10 s, автоматических повторов нет.
|
||||
Download читает S3 чанками по 64 KiB. Оригинал не загружается целиком в RAM при скачивании.
|
||||
|
||||
## Согласованность S3 и БД
|
||||
|
||||
1. API принимает и проверяет временный файл.
|
||||
2. Отдельная транзакция сохраняет `pending` и назначенный object key до S3 PUT.
|
||||
3. Новая транзакция блокирует строку, выполняет PUT и переводит её в `ready`.
|
||||
4. Ответ 201 отправляется только после COMMIT. `pending` не показывается публично.
|
||||
|
||||
При обрыве передачи клиента до PUT остаётся только автоматически закрываемый временный
|
||||
файл. При сбое PUT, падении процесса или неуспешном COMMIT остаётся учтённый `pending`.
|
||||
Фоновый cleaner раз в минуту выбирает pending старше **1 часа**, блокируя строку
|
||||
`FOR UPDATE SKIP LOCKED`, удаляет соответствующий S3 key и затем запись БД. Активная
|
||||
транзакция пропускается. При ошибке S3 строка сохраняется для следующего прохода.
|
||||
При сбое COMMIT очистка повторит безопасный DELETE отсутствующего объекта.
|
||||
`ready` никогда не выбирается. Часовой grace period значительно больше таймаутов загрузки.
|
||||
Это не распределённая транзакция: некоторое время возможны pending-метаданные и лишние байты.
|
||||
Объекты, вручную записанные администратором без intent, вне модели очистки.
|
||||
|
||||
## Приватность и запуск хранилища
|
||||
|
||||
MinIO, PostgreSQL, Valkey и API не публикуют порты на хост. Только HAProxy доступен
|
||||
на loopback. Консоль MinIO слушает 127.0.0.1 внутри его контейнера.
|
||||
Идемпотентный `s3-init` создаёт bucket, запрещает anonymous и назначает отдельному API user
|
||||
политику: ListBucket одного bucket и Get/Put/Delete исключительно в `originals/*`.
|
||||
Root credentials доступны только MinIO и init, API их не получает.
|
||||
Readiness зависит от PostgreSQL и MinIO. Valkey имеет fallback в PostgreSQL.
|
||||
Секреты находятся в исключённом из Git каталоге `secrets/`.
|
||||
|
||||
| Роль / service | Версия | Готовность | Постоянные данные | Доступ / credentials |
|
||||
|---|---|---|---|---|
|
||||
| LB / `lb` | HAProxy 3.0.8-alpine | HTTP `/health/ready` обеих API | нет | только `127.0.0.1:8084` |
|
||||
| API / `api1`, `api2` | Python 3.12.7; `app/requirements.txt` | PostgreSQL + MinIO | нет | внутренняя сеть; DB/S3 через secrets |
|
||||
| БД / `postgres` | PostgreSQL 16.6-bookworm | `pg_isready` | `postgres-data` | внутренняя сеть; password secret |
|
||||
| Кэш / `valkey` | Valkey 8.0.2-alpine | некритичен, fallback в БД | `valkey-data` (AOF) | внутренняя сеть |
|
||||
| S3 / `s3` | MinIO RELEASE.2025-04-22T22-12-26Z | `/minio/health/live` | `s3-data` | внутренняя сеть; root secrets |
|
||||
| Bootstrap / `s3-init` | mc RELEASE.2025-04-16T18-13-26Z | успешное завершение | bucket/policy в S3 | root + API secrets |
|
||||
| Миграции / `migrate` | образ API | успешное завершение | `schema_migrations` в БД | secrets, advisory lock, checksum миграций |
|
||||
| Очистка / `cleaner` | образ API | повтор каждые 60 s | pending в БД | ограниченные API S3 credentials |
|
||||
|
||||
Все сервисы находятся в отдельной Compose-сети `mtusi-lab4_default`.
|
||||
Повторный `bootstrap.py` сохраняет секреты; повторные `migrate` и `s3-init`
|
||||
не удаляют данные и выполняются при `docker compose up`.
|
||||
|
||||
## Проверки и доказательства
|
||||
|
||||
| Критерий | Проверка | Результат 12.11.2026 |
|
||||
|---|---|---|
|
||||
| Структура и синтаксис | `make check`, AST всех новых Python-файлов | выполнено |
|
||||
| Конфигурация | `docker compose config --quiet` | выполнено |
|
||||
| Реальные PNG/JPEG, MIME, повреждение, пиксели | `python3 student-tests/unit.py` | 5 тестов пройдено |
|
||||
| Обязательный API и checksum | `make test BASE_URL=http://localhost:8084` | PASS; вывод в `evidence/` |
|
||||
| Ошибки файлов и ACL, конкуренция ключа | `BASE_URL=http://localhost:8084 python3 student-tests/verify.py` | PASS; вывод в `evidence/` |
|
||||
| Приватность bucket и безопасная очистка | `student-tests/storage_checks.py` внутри API через общий runner | PASS; вывод в `evidence/` |
|
||||
| Обрыв загрузки и отказ S3 | `student-tests/outage.py` | PASS; вывод в `evidence/` |
|
||||
| Повторный bootstrap | повтор `docker compose run --rm s3-init` | PASS; вывод в `evidence/` |
|
||||
| Перезапуск всех сервисов | persistence + `scripts/restart.sh` | PASS; вывод в `evidence/` |
|
||||
| Остановка обслужившей реплики | failover + `scripts/stop-one.sh` | PASS; 89 записей, 3 неудачные попытки, восстановление 2.8 s |
|
||||
| Остановка именно загрузившей реплики | `student-tests/attachment_failover.py` | PASS; upload api1 → download api2; bytes совпадают |
|
||||
| 10 одновременных загрузок и RSS | `student-tests/memory.py` | 8 × 201, 2 × 503; пик RSS 132796 / 133912 KiB |
|
||||
|
||||
`.venv/bin/python scripts/verify.py` выполняет проверки и сохраняет реальные результаты в evidence,
|
||||
останавливаясь при первом провале. Тест очистки создаёт только синтетические строки и объекты,
|
||||
моделирует rollback финализации после PUT, проверяет grace period и сохранность ready.
|
||||
Он удаляет только свои синтетические данные. До запуска убедитесь, что это учебный стенд.
|
||||
|
||||
Memory-тест создаёт PNG примерно 9 MiB и одновременно запускает 10 запросов.
|
||||
Он пишет коды ответов, длительность, выборки VmRSS процессов API и измеренный максимум.
|
||||
Отказы 503 из-за ограничения слотов учитываются явно; они не записываются как успешные загрузки.
|
||||
Последовательная выборка `/proc/1/status` может пропустить короткий пик, поэтому результат —
|
||||
максимум наблюдений, а не гарантированный абсолютный пик. Фактический эксперимент: 10 одновременных запросов по 9 201 866 байт,
|
||||
8 ответов 201 и 2 ответа 503; время ответа 0.427–0.680 s. Наблюдаемый пик RSS:
|
||||
api1 — 132 796 KiB (129.7 MiB), api2 — 133 912 KiB (130.8 MiB), при лимите
|
||||
768 MiB на реплику. Полные выборки сохранены в `evidence/memory.json`.
|
||||
Это проверка ограниченности ресурсов, а не обещание принять любую конкуренцию без 503.
|
||||
|
||||
## Ограничения
|
||||
|
||||
Один host, HAProxy, PostgreSQL и MinIO остаются точками отказа. Это отказоустойчивость
|
||||
к остановке одной API-реплики, не полноценная HA-система. При отказе S3 readiness
|
||||
становится 503; после восстановления новые обращения повторяются клиентом.
|
||||
Обрыв S3 посреди уже начатого download закрывает поток, поэтому клиент проверяет длину/checksum.
|
||||
Нет UI, обработки изображений в очереди, presigned URL и исследовательских треков.
|
||||
Проверка реальных контейнеров выполнена; при переносе на другую машину её нужно повторить.
|
||||
|
||||
## Источники и использование ИИ
|
||||
|
||||
Основа — собственные предыдущие лабораторные и skeleton преподавателя;
|
||||
API и критерии взяты из `contracts/` и `README.md`. MinIO/mc собираются из официальных
|
||||
исходников по ссылкам выше. При проверке и исправлении работы использован Codex:
|
||||
анализ кода, исправление сборки S3, дополнительные тесты, запуск стенда и оформление
|
||||
фактических результатов. Результаты получены запуском, а не сгенерированы.
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
"""Binary upload, private download, and durable upload intents."""
|
||||
import asyncio
|
||||
import hashlib
|
||||
import tempfile
|
||||
import uuid
|
||||
import unicodedata
|
||||
from urllib.parse import quote
|
||||
from fastapi import Depends, Request
|
||||
from fastapi.responses import StreamingResponse
|
||||
from starlette.concurrency import run_in_threadpool
|
||||
import storage
|
||||
|
||||
# Two decoded images per replica at most; no unbounded waiting queue.
|
||||
slots = asyncio.Semaphore(2)
|
||||
|
||||
def valid_filename(name):
|
||||
return (isinstance(name, str) and 1 <= len(name) <= 255 and bool(name.strip())
|
||||
and not any(c in '/\\' or unicodedata.category(c) == 'Cc' for c in name))
|
||||
|
||||
def install(app, pool, current_user, visible_chat, query, wire, ApiError):
|
||||
def public(row):
|
||||
row = wire(row)
|
||||
row.pop('object_key', None)
|
||||
return dict(row, variants=[], error_code=None)
|
||||
|
||||
def visible_attachment(aid, actor):
|
||||
try:
|
||||
aid = str(uuid.UUID(aid))
|
||||
except ValueError:
|
||||
raise ApiError(400, 'invalid_request', 'Invalid attachment id')
|
||||
rows = query("""SELECT a.* FROM attachments a JOIN members m ON m.chat_id=a.chat_id
|
||||
WHERE a.id=%s AND m.user_id=%s AND a.status='ready'""", (aid, actor['id']))
|
||||
if not rows:
|
||||
raise ApiError(404, 'not_found', 'Attachment not found')
|
||||
return rows[0]
|
||||
|
||||
def save(file, chat_id, actor, name, mime, size, digest):
|
||||
try:
|
||||
storage.validate_image(file, mime)
|
||||
except storage.TooManyPixels:
|
||||
raise ApiError(413, 'payload_too_large', 'Too many image pixels')
|
||||
except storage.InvalidImage:
|
||||
raise ApiError(415, 'unsupported_media_type', 'Invalid PNG/JPEG or MIME mismatch')
|
||||
aid = str(uuid.uuid4())
|
||||
key = 'originals/' + aid
|
||||
# Commit intent before PUT. Crash/uncertain PUT or COMMIT is recoverable.
|
||||
query("""INSERT INTO attachments(id,chat_id,owner_id,filename,object_key,content_type,size_bytes,sha256,status)
|
||||
VALUES (%s,%s,%s,%s,%s,%s,%s,%s,'pending') RETURNING id""",
|
||||
(aid,chat_id,actor['id'],name,key,mime,size,digest))
|
||||
with pool.connection() as conn:
|
||||
# Cleaner skips this row while PUT and commit are in flight.
|
||||
conn.execute('SELECT id FROM attachments WHERE id=%s FOR UPDATE', (aid,)).fetchone()
|
||||
storage.client.put_object(Bucket=storage.BUCKET, Key=key, Body=file,
|
||||
ContentLength=size, ContentType=mime)
|
||||
row = conn.execute("UPDATE attachments SET status='ready', updated_at=clock_timestamp() WHERE id=%s RETURNING *", (aid,)).fetchone()
|
||||
return public(row)
|
||||
|
||||
@app.post('/api/v1/chats/{chat_id}/attachments', status_code=201)
|
||||
async def upload(chat_id: str, request: Request, actor=Depends(current_user)):
|
||||
await run_in_threadpool(visible_chat, chat_id, actor['id'])
|
||||
name = request.query_params.get('filename')
|
||||
if set(request.query_params) != {'filename'} or not valid_filename(name):
|
||||
raise ApiError(400, 'invalid_request', 'Invalid filename')
|
||||
mime = request.headers.get('content-type', '').split(';')[0].strip().lower()
|
||||
if mime not in ('image/png', 'image/jpeg'):
|
||||
raise ApiError(415, 'unsupported_media_type', 'PNG or JPEG required')
|
||||
length = request.headers.get('content-length')
|
||||
if length is not None:
|
||||
if not length.isdigit():
|
||||
raise ApiError(400, 'invalid_request', 'Invalid Content-Length')
|
||||
if int(length) > storage.MAX_BYTES:
|
||||
raise ApiError(413, 'payload_too_large', 'File too large')
|
||||
try:
|
||||
await asyncio.wait_for(slots.acquire(), timeout=0.5)
|
||||
except asyncio.TimeoutError:
|
||||
raise ApiError(503, 'unavailable', 'Upload capacity reached')
|
||||
try:
|
||||
with tempfile.TemporaryFile() as file:
|
||||
size = 0
|
||||
digest = hashlib.sha256()
|
||||
try:
|
||||
async with asyncio.timeout(30):
|
||||
async for chunk in request.stream():
|
||||
size += len(chunk)
|
||||
if size > storage.MAX_BYTES:
|
||||
raise ApiError(413, 'payload_too_large', 'File too large')
|
||||
digest.update(chunk)
|
||||
await run_in_threadpool(file.write, chunk)
|
||||
except TimeoutError:
|
||||
raise ApiError(400, 'invalid_request', 'Upload timed out')
|
||||
return await run_in_threadpool(save, file, chat_id, actor, name, mime, size, digest.hexdigest())
|
||||
finally:
|
||||
slots.release()
|
||||
|
||||
@app.get('/api/v1/attachments/{aid}')
|
||||
def metadata(aid: str, actor=Depends(current_user)):
|
||||
return public(visible_attachment(aid, actor))
|
||||
|
||||
@app.get('/api/v1/attachments/{aid}/content')
|
||||
def content(aid: str, variant: str = 'original', actor=Depends(current_user)):
|
||||
row = visible_attachment(aid, actor)
|
||||
if variant != 'original':
|
||||
raise ApiError(404, 'not_found', 'Variant not found')
|
||||
obj = storage.client.get_object(Bucket=storage.BUCKET, Key=row['object_key'])
|
||||
def chunks():
|
||||
try:
|
||||
while data := obj['Body'].read(64 * 1024):
|
||||
yield data
|
||||
finally:
|
||||
obj['Body'].close()
|
||||
return StreamingResponse(chunks(), media_type=row['content_type'], headers={
|
||||
'Content-Length': str(row['size_bytes']), 'X-Content-Type-Options': 'nosniff',
|
||||
'Content-Disposition': "attachment; filename*=UTF-8''" + quote(row['filename'], safe='')})
|
||||
@@ -0,0 +1,34 @@
|
||||
"""Remove abandoned upload intents after an hour; never remove ready objects."""
|
||||
import os
|
||||
import time
|
||||
import logging
|
||||
import psycopg
|
||||
from psycopg.rows import dict_row
|
||||
import storage
|
||||
|
||||
|
||||
def sweep():
|
||||
count = 0
|
||||
# Small transactions, no unbounded listing of the bucket into memory.
|
||||
for _ in range(100):
|
||||
with psycopg.connect(storage.secret('DATABASE_URL'), row_factory=dict_row,
|
||||
connect_timeout=3, options='-c statement_timeout=15000') as conn:
|
||||
row = conn.execute("""SELECT id,object_key FROM attachments
|
||||
WHERE status='pending' AND created_at < clock_timestamp()-interval '1 hour'
|
||||
ORDER BY created_at FOR UPDATE SKIP LOCKED LIMIT 1""").fetchone()
|
||||
if not row:
|
||||
return count
|
||||
storage.client.delete_object(Bucket=storage.BUCKET, Key=row['object_key'])
|
||||
conn.execute("DELETE FROM attachments WHERE id=%s AND status='pending'", (row['id'],))
|
||||
count += 1
|
||||
return count
|
||||
|
||||
if __name__ == '__main__':
|
||||
while True:
|
||||
try:
|
||||
print('Removed abandoned uploads:', sweep(), flush=True)
|
||||
except Exception:
|
||||
logging.error('Cleanup failed; will retry next cycle')
|
||||
if os.environ.get('CLEANUP_ONCE') == '1':
|
||||
break
|
||||
time.sleep(60)
|
||||
@@ -0,0 +1,38 @@
|
||||
"""Image checks without network access."""
|
||||
import warnings
|
||||
from PIL import Image, UnidentifiedImageError
|
||||
MAX_BYTES = 10_485_760
|
||||
MAX_PIXELS = 40_000_000
|
||||
Image.MAX_IMAGE_PIXELS = MAX_PIXELS
|
||||
|
||||
class InvalidImage(ValueError):
|
||||
pass
|
||||
|
||||
class TooManyPixels(ValueError):
|
||||
pass
|
||||
|
||||
def validate_image(file, mime):
|
||||
try:
|
||||
with warnings.catch_warnings():
|
||||
warnings.simplefilter('error', Image.DecompressionBombWarning)
|
||||
file.seek(0)
|
||||
with Image.open(file) as img:
|
||||
if img.width * img.height > MAX_PIXELS:
|
||||
raise TooManyPixels()
|
||||
actual = {'PNG': 'image/png', 'JPEG': 'image/jpeg'}.get(img.format)
|
||||
if actual != mime or actual is None:
|
||||
raise InvalidImage()
|
||||
if getattr(img, 'n_frames', 1) != 1:
|
||||
raise InvalidImage()
|
||||
img.verify()
|
||||
file.seek(0)
|
||||
with Image.open(file) as img:
|
||||
img.load() # Detect truncated JPEG/pixel data, not only its header.
|
||||
except (Image.DecompressionBombError, Image.DecompressionBombWarning) as exc:
|
||||
raise TooManyPixels() from exc
|
||||
except (UnidentifiedImageError, OSError, SyntaxError, ValueError) as exc:
|
||||
if isinstance(exc, TooManyPixels):
|
||||
raise
|
||||
raise InvalidImage() from exc
|
||||
finally:
|
||||
file.seek(0)
|
||||
@@ -0,0 +1,21 @@
|
||||
"""Atomic, serialized, versioned migrations; changed applied files are rejected."""
|
||||
import hashlib
|
||||
import os
|
||||
from pathlib import Path
|
||||
import psycopg
|
||||
|
||||
with psycopg.connect(Path(os.environ["DATABASE_URL_FILE"]).read_text().strip()) as conn:
|
||||
conn.execute("SELECT pg_advisory_xact_lock(20260002)")
|
||||
conn.execute("CREATE TABLE IF NOT EXISTS schema_migrations(version text PRIMARY KEY, checksum text NOT NULL)")
|
||||
for path in sorted(Path("migrations").glob("*.sql")):
|
||||
sql = path.read_text()
|
||||
digest = hashlib.sha256(sql.encode()).hexdigest()
|
||||
row = conn.execute("SELECT checksum FROM schema_migrations WHERE version=%s", (path.name,)).fetchone()
|
||||
if row:
|
||||
if row[0] != digest:
|
||||
raise RuntimeError("Applied migration changed: " + path.name)
|
||||
continue
|
||||
conn.execute(sql)
|
||||
conn.execute("INSERT INTO schema_migrations VALUES (%s,%s)", (path.name,digest))
|
||||
print("Applied", path.name)
|
||||
print("Migrations OK")
|
||||
@@ -0,0 +1,28 @@
|
||||
annotated-types==0.8.0
|
||||
anyio==4.15.1
|
||||
argon2-cffi==23.1.0
|
||||
argon2-cffi-bindings==26.1.0
|
||||
cffi==2.1.1
|
||||
click==8.5.0
|
||||
fastapi==0.115.6
|
||||
h11==0.16.0
|
||||
idna==3.20
|
||||
psycopg==3.2.6
|
||||
psycopg-binary==3.2.6
|
||||
psycopg-pool==3.3.3
|
||||
pycparser==3.12
|
||||
pydantic==2.14.1
|
||||
pydantic_core==2.50.1
|
||||
redis==5.2.1
|
||||
starlette==0.41.3
|
||||
typing-inspection==0.4.4
|
||||
typing_extensions==4.16.0
|
||||
uvicorn==0.34.0
|
||||
boto3==1.35.99
|
||||
botocore==1.35.99
|
||||
s3transfer==0.10.4
|
||||
jmespath==1.0.1
|
||||
python-dateutil==2.9.0.post0
|
||||
six==1.17.0
|
||||
urllib3==2.3.0
|
||||
Pillow==11.1.0
|
||||
+502
@@ -0,0 +1,502 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Lab 4 PostgreSQL messenger, adapted from the author's lab 1."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import binascii
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import hashlib
|
||||
import secrets
|
||||
from pathlib import Path
|
||||
from contextlib import asynccontextmanager
|
||||
import psycopg
|
||||
from psycopg.rows import dict_row
|
||||
from psycopg_pool import ConnectionPool, PoolTimeout, TooManyRequests
|
||||
import redis
|
||||
from argon2 import PasswordHasher
|
||||
from argon2.exceptions import VerificationError
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any, Optional, Tuple
|
||||
|
||||
import uvicorn
|
||||
from fastapi import Depends, FastAPI, Header, Query, Request
|
||||
from fastapi.exceptions import RequestValidationError
|
||||
from fastapi.responses import JSONResponse, Response
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
|
||||
from starlette.exceptions import HTTPException as StarletteHTTPException
|
||||
|
||||
|
||||
USERNAME_RE = re.compile(r"^[a-z0-9_]{3,32}$")
|
||||
LOG = logging.getLogger("messenger")
|
||||
INSTANCE_ID = os.environ.get("INSTANCE_ID", str(uuid.uuid4()))
|
||||
|
||||
|
||||
def env_int(name: str, default: int, minimum: int = 0) -> int:
|
||||
try:
|
||||
value = int(os.environ.get(name, str(default)))
|
||||
except ValueError as exc:
|
||||
raise SystemExit("%s must be an integer" % name) from exc
|
||||
if value < minimum:
|
||||
raise SystemExit("%s must be at least %s" % (name, minimum))
|
||||
return value
|
||||
|
||||
|
||||
class ApiError(Exception):
|
||||
def __init__(self, status: int, code: str, message: str) -> None:
|
||||
super().__init__(message)
|
||||
self.status = status
|
||||
self.code = code
|
||||
self.message = message
|
||||
|
||||
|
||||
class StrictModel(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
|
||||
class Health(StrictModel):
|
||||
status: str = Field(pattern="^ok$")
|
||||
|
||||
|
||||
class User(StrictModel):
|
||||
id: uuid.UUID
|
||||
username: str = Field(pattern=r"^[a-z0-9_]{3,32}$")
|
||||
display_name: str = Field(min_length=1, max_length=100)
|
||||
created_at: datetime
|
||||
|
||||
|
||||
class CreateUser(StrictModel):
|
||||
password: str = Field(min_length=12, max_length=128)
|
||||
|
||||
@field_validator("password")
|
||||
@classmethod
|
||||
def non_blank_password(cls, value: str) -> str:
|
||||
if not value.strip():
|
||||
raise ValueError("password must not be blank")
|
||||
return value
|
||||
|
||||
username: str = Field(pattern=r"^[a-z0-9_]{3,32}$")
|
||||
display_name: str = Field(min_length=1, max_length=100)
|
||||
|
||||
@field_validator("display_name")
|
||||
@classmethod
|
||||
def non_blank_display_name(cls, value: str) -> str:
|
||||
if not value.strip():
|
||||
raise ValueError("display_name must not be blank")
|
||||
return value
|
||||
|
||||
|
||||
class Chat(StrictModel):
|
||||
id: uuid.UUID
|
||||
title: str = Field(min_length=1, max_length=100)
|
||||
member_ids: list[uuid.UUID] = Field(min_length=1, max_length=100)
|
||||
created_at: datetime
|
||||
|
||||
|
||||
class CreateChat(StrictModel):
|
||||
title: str = Field(min_length=1, max_length=100)
|
||||
member_ids: list[uuid.UUID] = Field(min_length=1, max_length=100)
|
||||
|
||||
@field_validator("title")
|
||||
@classmethod
|
||||
def non_blank_title(cls, value: str) -> str:
|
||||
if not value.strip():
|
||||
raise ValueError("title must not be blank")
|
||||
return value
|
||||
|
||||
@field_validator("member_ids")
|
||||
@classmethod
|
||||
def unique_members(cls, value: list[uuid.UUID]) -> list[uuid.UUID]:
|
||||
if len(set(value)) != len(value):
|
||||
raise ValueError("member_ids must be unique")
|
||||
return value
|
||||
|
||||
|
||||
class Message(StrictModel):
|
||||
id: uuid.UUID
|
||||
chat_id: uuid.UUID
|
||||
sender_id: uuid.UUID
|
||||
text: str = Field(max_length=4000)
|
||||
attachment_ids: list[uuid.UUID] = Field(default_factory=list, max_length=10)
|
||||
created_at: datetime
|
||||
|
||||
|
||||
class CreateMessage(StrictModel):
|
||||
text: str = Field(max_length=4000)
|
||||
attachment_ids: list[uuid.UUID] = Field(default_factory=list, max_length=10)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def valid_message(self):
|
||||
if not self.text.strip() and not self.attachment_ids:
|
||||
raise ValueError("message must not be empty")
|
||||
if len(set(self.attachment_ids)) != len(self.attachment_ids):
|
||||
raise ValueError("duplicate attachment")
|
||||
return self
|
||||
|
||||
|
||||
class ChatPage(StrictModel):
|
||||
items: list[Chat]
|
||||
next_cursor: Optional[str]
|
||||
|
||||
|
||||
class MessagePage(StrictModel):
|
||||
items: list[Message]
|
||||
next_cursor: Optional[str]
|
||||
|
||||
|
||||
class ErrorDetails(StrictModel):
|
||||
code: str
|
||||
message: str
|
||||
request_id: str
|
||||
|
||||
|
||||
class ErrorResponse(StrictModel):
|
||||
error: ErrorDetails
|
||||
|
||||
|
||||
def valid_uuid(value: Any) -> bool:
|
||||
if not isinstance(value, str):
|
||||
return False
|
||||
try:
|
||||
return str(uuid.UUID(value)) == value.lower()
|
||||
except (ValueError, AttributeError):
|
||||
return False
|
||||
|
||||
|
||||
def make_cursor(scope: str, item: dict[str, Any]) -> str:
|
||||
raw = json.dumps(
|
||||
{"v": 1, "scope": scope, "created_at": item["created_at"], "id": item["id"]},
|
||||
separators=(",", ":"),
|
||||
).encode("utf-8")
|
||||
return base64.urlsafe_b64encode(raw).rstrip(b"=").decode("ascii")
|
||||
|
||||
|
||||
def read_cursor(value: str, scope: str) -> Tuple[str, str]:
|
||||
try:
|
||||
raw = base64.urlsafe_b64decode(value + "=" * (-len(value) % 4))
|
||||
data = json.loads(raw)
|
||||
if set(data) != {"v", "scope", "created_at", "id"}:
|
||||
raise ValueError
|
||||
if data["v"] != 1 or data["scope"] != scope or not valid_uuid(data["id"]):
|
||||
raise ValueError
|
||||
datetime.fromisoformat(data["created_at"].replace("Z", "+00:00"))
|
||||
return data["created_at"], data["id"]
|
||||
except (ValueError, TypeError, AttributeError, UnicodeError, json.JSONDecodeError, binascii.Error):
|
||||
raise ApiError(400, "invalid_request", "Invalid cursor") from None
|
||||
|
||||
|
||||
|
||||
def secret(name):
|
||||
return Path(os.environ[name + "_FILE"]).read_text().strip()
|
||||
|
||||
POOL = ConnectionPool(secret("DATABASE_URL"), min_size=1, max_size=8, timeout=3,
|
||||
max_waiting=32, open=False, check=ConnectionPool.check_connection,
|
||||
kwargs={"row_factory": dict_row, "connect_timeout": 3,
|
||||
"options": "-c statement_timeout=3000 -c lock_timeout=2000"})
|
||||
CACHE = redis.Redis.from_url(secret("CACHE_URL"), socket_connect_timeout=0.2,
|
||||
socket_timeout=0.2, decode_responses=True)
|
||||
HASHER = PasswordHasher(time_cost=2, memory_cost=19456, parallelism=1)
|
||||
DUMMY_HASH = HASHER.hash(secrets.token_urlsafe(32))
|
||||
TTL = env_int("SESSION_TTL_SECONDS", 3600, 1)
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app):
|
||||
POOL.open()
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
POOL.close()
|
||||
CACHE.close()
|
||||
|
||||
app = FastAPI(title="MTUSI Messenger — Lab 4", version="4.0.0", lifespan=lifespan)
|
||||
|
||||
|
||||
def wire(row):
|
||||
return {k: (v.isoformat(timespec="microseconds").replace("+00:00", "Z")
|
||||
if isinstance(v, datetime) else str(v) if isinstance(v, uuid.UUID) else v)
|
||||
for k, v in row.items()}
|
||||
|
||||
|
||||
def query(sql, params=()):
|
||||
with POOL.connection() as conn:
|
||||
return [wire(row) for row in conn.execute(sql, params).fetchall()]
|
||||
|
||||
|
||||
def profile(user_id):
|
||||
key = "user:" + user_id
|
||||
try:
|
||||
value = CACHE.get(key)
|
||||
if value:
|
||||
return json.loads(value)
|
||||
except (redis.RedisError, ValueError):
|
||||
pass
|
||||
rows = query("SELECT id, username, display_name, created_at FROM users WHERE id=%s", (user_id,))
|
||||
if not rows:
|
||||
raise ApiError(401, "unauthorized", "Invalid session")
|
||||
user = rows[0]
|
||||
try:
|
||||
CACHE.setex(key, 60, json.dumps(user))
|
||||
except redis.RedisError:
|
||||
pass
|
||||
return user
|
||||
|
||||
|
||||
def token_hash(token):
|
||||
return hashlib.sha256(token.encode()).hexdigest()
|
||||
|
||||
|
||||
def bearer(authorization):
|
||||
if not authorization or not authorization.startswith("Bearer "):
|
||||
raise ApiError(401, "unauthorized", "Bearer session required")
|
||||
token = authorization[7:]
|
||||
if not re.fullmatch(r"[A-Za-z0-9_-]{43}", token):
|
||||
raise ApiError(401, "unauthorized", "Invalid session")
|
||||
return token_hash(token)
|
||||
|
||||
|
||||
def current_user(authorization: Optional[str] = Header(default=None)):
|
||||
digest = bearer(authorization)
|
||||
rows = query("SELECT user_id FROM sessions WHERE token_hash=%s AND expires_at > clock_timestamp()", (digest,))
|
||||
if not rows:
|
||||
raise ApiError(401, "unauthorized", "Invalid session")
|
||||
return profile(rows[0]["user_id"])
|
||||
|
||||
|
||||
def visible_chat(chat_id, actor_id):
|
||||
if not valid_uuid(chat_id):
|
||||
raise ApiError(400, "invalid_request", "Invalid chat id")
|
||||
rows = query("SELECT c.* FROM chats c JOIN members m ON m.chat_id=c.id WHERE c.id=%s AND m.user_id=%s", (chat_id, actor_id))
|
||||
if not rows:
|
||||
raise ApiError(404, "not_found", "Chat not found")
|
||||
return chat_members(rows)[0]
|
||||
|
||||
|
||||
def chat_members(rows):
|
||||
if rows:
|
||||
members = query("SELECT chat_id, user_id FROM members WHERE chat_id=ANY(%s::uuid[]) ORDER BY user_id", ([r["id"] for r in rows],))
|
||||
grouped = {r["id"]: [] for r in rows}
|
||||
for m in members:
|
||||
grouped[m["chat_id"]].append(m["user_id"])
|
||||
for row in rows:
|
||||
row["member_ids"] = grouped[row["id"]]
|
||||
return rows
|
||||
|
||||
|
||||
def page_args(request, scope, limit, cursor):
|
||||
if set(request.query_params).difference({"limit", "cursor"}):
|
||||
raise ApiError(400, "invalid_request", "Invalid query parameters")
|
||||
return limit, read_cursor(cursor, scope) if cursor is not None else None
|
||||
|
||||
|
||||
def page(rows, limit, scope):
|
||||
return {"items": rows[:limit], "next_cursor": make_cursor(scope, rows[limit-1]) if len(rows)>limit else None}
|
||||
def request_id(request: Request) -> str:
|
||||
return getattr(request.state, "request_id", str(uuid.uuid4()))
|
||||
|
||||
|
||||
def error_response(request: Request, status: int, code: str, message: str) -> JSONResponse:
|
||||
rid = request_id(request)
|
||||
response = JSONResponse(
|
||||
status_code=status,
|
||||
content={"error": {"code": code, "message": message, "request_id": rid}},
|
||||
)
|
||||
response.headers["X-Request-Id"] = rid
|
||||
response.headers["X-Instance-Id"] = INSTANCE_ID
|
||||
if status == 401:
|
||||
response.headers["WWW-Authenticate"] = ('Basic realm="messenger", charset="UTF-8"'
|
||||
if request.url.path == "/api/v1/auth/sessions" else 'Bearer realm="messenger"')
|
||||
return response
|
||||
|
||||
|
||||
@app.middleware("http")
|
||||
async def request_envelope(request: Request, call_next: Any) -> JSONResponse:
|
||||
request.state.request_id = str(uuid.uuid4())
|
||||
try:
|
||||
response = await call_next(request)
|
||||
except Exception:
|
||||
LOG.error("request_id=%s unexpected error", request_id(request))
|
||||
response = error_response(request, 500, "internal_error", "Internal server error")
|
||||
response.headers["X-Request-Id"] = request_id(request)
|
||||
response.headers["X-Instance-Id"] = INSTANCE_ID
|
||||
return response
|
||||
|
||||
|
||||
@app.exception_handler(ApiError)
|
||||
async def api_error_handler(request: Request, error: ApiError) -> JSONResponse:
|
||||
return error_response(request, error.status, error.code, error.message)
|
||||
|
||||
|
||||
@app.exception_handler(RequestValidationError)
|
||||
async def validation_error_handler(request: Request, _error: RequestValidationError) -> JSONResponse:
|
||||
return error_response(request, 400, "invalid_request", "Request validation failed")
|
||||
|
||||
|
||||
@app.exception_handler(StarletteHTTPException)
|
||||
async def http_error_handler(request: Request, error: StarletteHTTPException) -> JSONResponse:
|
||||
if error.status_code == 404:
|
||||
return error_response(request, 404, "not_found", "Resource not found")
|
||||
return error_response(request, error.status_code, "invalid_request", "Request failed")
|
||||
|
||||
|
||||
async def require_json(request: Request) -> None:
|
||||
content_type = request.headers.get("content-type", "").split(";", 1)[0].strip().lower()
|
||||
if content_type != "application/json":
|
||||
raise ApiError(415, "unsupported_media_type", "Content-Type must be application/json")
|
||||
|
||||
|
||||
|
||||
@app.exception_handler(psycopg.Error)
|
||||
@app.exception_handler(PoolTimeout)
|
||||
@app.exception_handler(TooManyRequests)
|
||||
async def database_error(request, error):
|
||||
return error_response(request, 503, "unavailable", "Database temporarily unavailable")
|
||||
|
||||
@app.get("/health/live", response_model=Health)
|
||||
def health_live():
|
||||
return {"status": "ok"}
|
||||
|
||||
@app.get("/health/ready", response_model=Health)
|
||||
def health_ready():
|
||||
query("SELECT 1 FROM schema_migrations LIMIT 1")
|
||||
storage.client.head_bucket(Bucket=storage.BUCKET)
|
||||
return {"status": "ok"}
|
||||
|
||||
@app.post("/api/v1/users", response_model=User, status_code=201, dependencies=[Depends(require_json)])
|
||||
def create_user(payload: CreateUser):
|
||||
password_hash = HASHER.hash(payload.password)
|
||||
try:
|
||||
return query("INSERT INTO users (id,username,display_name,password_hash) VALUES (%s,%s,%s,%s) RETURNING id,username,display_name,created_at",
|
||||
(uuid.uuid4(), payload.username, payload.display_name, password_hash))[0]
|
||||
except psycopg.errors.UniqueViolation:
|
||||
raise ApiError(409, "conflict", "Username already exists") from None
|
||||
|
||||
@app.post("/api/v1/auth/sessions", status_code=201)
|
||||
async def login(request: Request, authorization: Optional[str] = Header(default=None)):
|
||||
# CPU/SQL work runs in FastAPI's worker pool, not in the event loop.
|
||||
from starlette.concurrency import run_in_threadpool
|
||||
if await request.body():
|
||||
raise ApiError(400, "invalid_request", "Session request must not have a body")
|
||||
return await run_in_threadpool(login_sync, authorization)
|
||||
|
||||
|
||||
def login_sync(authorization):
|
||||
try:
|
||||
if not authorization or not authorization.startswith("Basic "):
|
||||
raise ValueError
|
||||
username, password = base64.b64decode(authorization[6:], validate=True).decode("utf-8").split(":", 1)
|
||||
if not USERNAME_RE.fullmatch(username) or not 12 <= len(password) <= 128:
|
||||
raise ValueError
|
||||
except (ValueError, UnicodeError, binascii.Error):
|
||||
raise ApiError(401, "unauthorized", "Invalid credentials") from None
|
||||
users = query("SELECT * FROM users WHERE username=%s", (username,))
|
||||
try:
|
||||
HASHER.verify(users[0]["password_hash"] if users else DUMMY_HASH, password)
|
||||
except VerificationError:
|
||||
raise ApiError(401, "unauthorized", "Invalid credentials") from None
|
||||
if not users:
|
||||
raise ApiError(401, "unauthorized", "Invalid credentials")
|
||||
token = secrets.token_urlsafe(32)
|
||||
expiry = query("INSERT INTO sessions(token_hash,user_id,expires_at) VALUES (%s,%s,clock_timestamp() + %s * interval '1 second') RETURNING expires_at",
|
||||
(token_hash(token), users[0]["id"], TTL))[0]["expires_at"]
|
||||
user = users[0]
|
||||
del user["password_hash"]
|
||||
return {"token": token, "token_type": "Bearer", "expires_at": expiry, "user": user}
|
||||
|
||||
@app.delete("/api/v1/auth/sessions/current", status_code=204)
|
||||
def logout(actor=Depends(current_user), authorization: Optional[str] = Header(default=None)):
|
||||
query("DELETE FROM sessions WHERE token_hash=%s RETURNING user_id", (bearer(authorization),))
|
||||
return Response(status_code=204)
|
||||
|
||||
@app.get("/api/v1/users/me", response_model=User)
|
||||
def get_me(actor=Depends(current_user)):
|
||||
return actor
|
||||
|
||||
@app.post("/api/v1/chats", response_model=Chat, status_code=201, dependencies=[Depends(require_json)])
|
||||
def create_chat(payload: CreateChat, actor=Depends(current_user)):
|
||||
members = [str(m) for m in payload.member_ids]
|
||||
if actor["id"] not in members:
|
||||
raise ApiError(400, "invalid_request", "The current user must be a member")
|
||||
try:
|
||||
with POOL.connection() as conn:
|
||||
chat = wire(conn.execute("INSERT INTO chats(id,title) VALUES (%s,%s) RETURNING *", (uuid.uuid4(),payload.title)).fetchone())
|
||||
with conn.cursor() as cur:
|
||||
cur.executemany("INSERT INTO members(chat_id,user_id) VALUES (%s,%s)", [(chat["id"],m) for m in members])
|
||||
chat["member_ids"] = members
|
||||
return chat
|
||||
except psycopg.errors.ForeignKeyViolation:
|
||||
raise ApiError(400, "invalid_request", "All chat members must exist") from None
|
||||
|
||||
@app.get("/api/v1/chats", response_model=ChatPage)
|
||||
def list_chats(request: Request, limit: int = Query(50,ge=1,le=100), cursor: Optional[str] = Query(None,min_length=1), actor=Depends(current_user)):
|
||||
scope = "chats:" + actor["id"]
|
||||
limit, after = page_args(request,scope,limit,cursor)
|
||||
sql = "SELECT c.* FROM chats c JOIN members m ON c.id=m.chat_id WHERE m.user_id=%s"
|
||||
params = [actor["id"]]
|
||||
if after:
|
||||
sql += " AND (c.created_at,c.id) > (%s,%s)"
|
||||
params.extend(after)
|
||||
rows = query(sql + " ORDER BY c.created_at,c.id LIMIT %s", params + [limit+1])
|
||||
return page(chat_members(rows),limit,scope)
|
||||
|
||||
@app.get("/api/v1/chats/{chat_id}", response_model=Chat)
|
||||
def get_chat(chat_id: str, actor=Depends(current_user)):
|
||||
return visible_chat(chat_id,actor["id"])
|
||||
|
||||
@app.post("/api/v1/chats/{chat_id}/messages", response_model=Message, status_code=201, dependencies=[Depends(require_json)])
|
||||
def create_message(chat_id: str, payload: CreateMessage, actor=Depends(current_user),
|
||||
idempotency_key: Optional[str] = Header(default=None)):
|
||||
visible_chat(chat_id, actor["id"])
|
||||
if not idempotency_key or not re.fullmatch(r"[A-Za-z0-9._:-]{1,128}", idempotency_key):
|
||||
raise ApiError(400, "invalid_request", "Invalid Idempotency-Key")
|
||||
ids = [str(i) for i in payload.attachment_ids]
|
||||
body = json.dumps({"text": payload.text, "attachment_ids": ids}, sort_keys=True)
|
||||
scope = actor["id"] + ":" + chat_id + ":" + idempotency_key
|
||||
lock = int.from_bytes(hashlib.sha256(scope.encode()).digest()[:8], "big", signed=True)
|
||||
with POOL.connection() as conn:
|
||||
conn.execute("SELECT pg_advisory_xact_lock(%s)", (lock,))
|
||||
previous = conn.execute("SELECT * FROM idempotency WHERE user_id=%s AND chat_id=%s AND key=%s AND expires_at>clock_timestamp()",
|
||||
(actor["id"], chat_id, idempotency_key)).fetchone()
|
||||
if previous:
|
||||
if previous["body"] != body:
|
||||
raise ApiError(409, "conflict", "Key already used for a different message")
|
||||
return previous["response"]
|
||||
if ids:
|
||||
found = conn.execute("SELECT id FROM attachments WHERE id=ANY(%s::uuid[]) AND chat_id=%s AND status='ready'", (ids, chat_id)).fetchall()
|
||||
if len(found) != len(ids):
|
||||
raise ApiError(404, "not_found", "Attachment not found")
|
||||
msg = wire(conn.execute("INSERT INTO messages(id,chat_id,sender_id,text,attachment_ids) VALUES (%s,%s,%s,%s,%s::uuid[]) RETURNING *",
|
||||
(uuid.uuid4(),chat_id,actor["id"],payload.text,ids)).fetchone())
|
||||
msg["attachment_ids"] = ids
|
||||
from psycopg.types.json import Jsonb
|
||||
conn.execute("""INSERT INTO idempotency(user_id,chat_id,key,body,response) VALUES (%s,%s,%s,%s,%s)
|
||||
ON CONFLICT(user_id,chat_id,key) DO UPDATE SET body=EXCLUDED.body,response=EXCLUDED.response,expires_at=clock_timestamp()+interval '24 hours'""",
|
||||
(actor["id"],chat_id,idempotency_key,body,Jsonb(msg)))
|
||||
return msg
|
||||
|
||||
@app.get("/api/v1/chats/{chat_id}/messages", response_model=MessagePage)
|
||||
def list_messages(request: Request, chat_id: str, limit: int = Query(50,ge=1,le=100), cursor: Optional[str] = Query(None,min_length=1), actor=Depends(current_user)):
|
||||
visible_chat(chat_id,actor["id"])
|
||||
scope = "messages:" + chat_id
|
||||
limit,after = page_args(request,scope,limit,cursor)
|
||||
sql = "SELECT * FROM messages WHERE chat_id=%s"
|
||||
params = [chat_id]
|
||||
if after:
|
||||
sql += " AND (created_at,id) > (%s,%s)"
|
||||
params.extend(after)
|
||||
return page(query(sql+" ORDER BY created_at,id LIMIT %s",params+[limit+1]),limit,scope)
|
||||
|
||||
import storage
|
||||
from attachments import install
|
||||
install(app, POOL, current_user, visible_chat, query, wire, ApiError)
|
||||
|
||||
@app.exception_handler(storage.StorageError)
|
||||
@app.exception_handler(storage.ClientError)
|
||||
async def storage_error(request, error):
|
||||
return error_response(request, 503, "unavailable", "Storage temporarily unavailable")
|
||||
|
||||
if __name__ == "__main__":
|
||||
uvicorn.run(app, host=os.environ.get("APP_HOST","0.0.0.0"), port=env_int("APP_PORT",8080,1))
|
||||
@@ -0,0 +1,24 @@
|
||||
"""Bounded S3 client and image validation shared by the API and cleaner."""
|
||||
import os
|
||||
import warnings
|
||||
from pathlib import Path
|
||||
import boto3
|
||||
from botocore.config import Config
|
||||
from botocore.exceptions import BotoCoreError as StorageError, ClientError
|
||||
from PIL import Image, UnidentifiedImageError
|
||||
|
||||
MAX_BYTES = 10_485_760
|
||||
MAX_PIXELS = 40_000_000
|
||||
BUCKET = os.environ.get('S3_BUCKET', 'messenger-attachments')
|
||||
Image.MAX_IMAGE_PIXELS = MAX_PIXELS
|
||||
|
||||
def secret(name):
|
||||
return Path(os.environ[name + '_FILE']).read_text().strip()
|
||||
|
||||
client = boto3.client('s3', endpoint_url=os.environ.get('S3_ENDPOINT', 'http://s3:9000'),
|
||||
region_name='us-east-1', aws_access_key_id=secret('S3_ACCESS_KEY_ID'),
|
||||
aws_secret_access_key=secret('S3_SECRET_ACCESS_KEY'),
|
||||
config=Config(connect_timeout=2, read_timeout=10, retries={'total_max_attempts': 1},
|
||||
s3={'addressing_style': 'path'}))
|
||||
|
||||
from images import validate_image, InvalidImage, TooManyPixels
|
||||
+107
@@ -0,0 +1,107 @@
|
||||
name: mtusi-lab4
|
||||
x-app: &app
|
||||
build: .
|
||||
image: mtusi-messenger:lab4
|
||||
mem_limit: 768m
|
||||
cpus: 1.0
|
||||
environment: &env
|
||||
APP_HOST: 0.0.0.0
|
||||
APP_PORT: 8080
|
||||
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
CACHE_URL_FILE: /run/secrets/cache_url
|
||||
S3_ENDPOINT: http://s3:9000
|
||||
S3_BUCKET: messenger-attachments
|
||||
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
|
||||
secrets: [database_url, cache_url, s3_access_key_id, s3_secret_access_key]
|
||||
stop_grace_period: 10s
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16.6-bookworm
|
||||
environment:
|
||||
POSTGRES_USER: messenger
|
||||
POSTGRES_DB: messenger
|
||||
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
|
||||
secrets: [postgres_password]
|
||||
volumes: [postgres-data:/var/lib/postgresql/data]
|
||||
healthcheck:
|
||||
test: [CMD-SHELL, 'pg_isready -U messenger -d messenger']
|
||||
interval: 2s
|
||||
timeout: 3s
|
||||
retries: 30
|
||||
valkey:
|
||||
image: valkey/valkey:8.0.2-alpine
|
||||
command: [valkey-server, --appendonly, 'yes', --maxmemory, 64mb, --maxmemory-policy, allkeys-lru]
|
||||
volumes: [valkey-data:/data]
|
||||
s3:
|
||||
build:
|
||||
context: ./infra/s3
|
||||
target: server
|
||||
image: mtusi-minio:RELEASE.2025-04-22T22-12-26Z
|
||||
command: [server, /data, --console-address, '127.0.0.1:9001']
|
||||
environment:
|
||||
MINIO_ROOT_USER_FILE: /run/secrets/s3_root_user
|
||||
MINIO_ROOT_PASSWORD_FILE: /run/secrets/s3_root_password
|
||||
secrets: [s3_root_user, s3_root_password]
|
||||
volumes: [s3-data:/data]
|
||||
healthcheck:
|
||||
test: [CMD, curl, -f, http://localhost:9000/minio/health/live]
|
||||
interval: 3s
|
||||
timeout: 3s
|
||||
retries: 30
|
||||
s3-init:
|
||||
build:
|
||||
context: ./infra/s3
|
||||
target: client
|
||||
image: mtusi-mc:RELEASE.2025-04-16T18-13-26Z
|
||||
entrypoint: [/bin/sh, /config/init.sh]
|
||||
volumes: ['./infra/s3:/config:ro']
|
||||
secrets: [s3_root_user, s3_root_password, s3_access_key_id, s3_secret_access_key]
|
||||
depends_on:
|
||||
s3: {condition: service_healthy}
|
||||
migrate:
|
||||
<<: *app
|
||||
command: [python, app/migrate.py]
|
||||
healthcheck: {disable: true}
|
||||
depends_on:
|
||||
postgres: {condition: service_healthy}
|
||||
api1:
|
||||
<<: *app
|
||||
environment:
|
||||
<<: *env
|
||||
INSTANCE_ID: api1
|
||||
depends_on: &deps
|
||||
migrate: {condition: service_completed_successfully}
|
||||
s3-init: {condition: service_completed_successfully}
|
||||
valkey: {condition: service_started}
|
||||
api2:
|
||||
<<: *app
|
||||
environment:
|
||||
<<: *env
|
||||
INSTANCE_ID: api2
|
||||
depends_on: *deps
|
||||
cleaner:
|
||||
<<: *app
|
||||
command: [python, app/cleanup.py]
|
||||
healthcheck: {disable: true}
|
||||
depends_on: *deps
|
||||
lb:
|
||||
image: haproxy:3.0.8-alpine
|
||||
ports: ['127.0.0.1:${HOST_PORT:-8084}:8080']
|
||||
volumes: ['./infra/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro']
|
||||
depends_on:
|
||||
api1: {condition: service_healthy}
|
||||
api2: {condition: service_healthy}
|
||||
volumes:
|
||||
postgres-data:
|
||||
valkey-data:
|
||||
s3-data:
|
||||
secrets:
|
||||
postgres_password: {file: ./secrets/postgres_password}
|
||||
database_url: {file: ./secrets/database_url}
|
||||
cache_url: {file: ./secrets/cache_url}
|
||||
s3_root_user: {file: ./secrets/s3_root_user}
|
||||
s3_root_password: {file: ./secrets/s3_root_password}
|
||||
s3_access_key_id: {file: ./secrets/s3_access_key_id}
|
||||
s3_secret_access_key: {file: ./secrets/s3_secret_access_key}
|
||||
@@ -0,0 +1,39 @@
|
||||
Container mtusi-lab4-api1-1 Stopping
|
||||
Container mtusi-lab4-api1-1 Stopped
|
||||
Container mtusi-lab4-valkey-1 Running
|
||||
Container mtusi-lab4-s3-1 Running
|
||||
Container mtusi-lab4-api2-1 Running
|
||||
Container mtusi-lab4-postgres-1 Running
|
||||
Container mtusi-lab4-postgres-1 Waiting
|
||||
Container mtusi-lab4-s3-1 Waiting
|
||||
Container mtusi-lab4-s3-1 Healthy
|
||||
Container mtusi-lab4-postgres-1 Healthy
|
||||
Container mtusi-lab4-s3-init-1 Starting
|
||||
Container mtusi-lab4-migrate-1 Starting
|
||||
Container mtusi-lab4-s3-init-1 Started
|
||||
Container mtusi-lab4-migrate-1 Started
|
||||
Container mtusi-lab4-migrate-1 Waiting
|
||||
Container mtusi-lab4-s3-init-1 Waiting
|
||||
Container mtusi-lab4-migrate-1 Waiting
|
||||
Container mtusi-lab4-s3-init-1 Waiting
|
||||
Container mtusi-lab4-s3-init-1 Exited
|
||||
Container mtusi-lab4-migrate-1 Exited
|
||||
Container mtusi-lab4-migrate-1 Exited
|
||||
Container mtusi-lab4-s3-init-1 Exited
|
||||
Container mtusi-lab4-api1-1 Starting
|
||||
Container mtusi-lab4-api1-1 Started
|
||||
Container mtusi-lab4-migrate-1 Waiting
|
||||
Container mtusi-lab4-api1-1 Waiting
|
||||
Container mtusi-lab4-s3-init-1 Waiting
|
||||
Container mtusi-lab4-valkey-1 Waiting
|
||||
Container mtusi-lab4-s3-1 Waiting
|
||||
Container mtusi-lab4-api2-1 Waiting
|
||||
Container mtusi-lab4-postgres-1 Waiting
|
||||
Container mtusi-lab4-postgres-1 Healthy
|
||||
Container mtusi-lab4-api2-1 Healthy
|
||||
Container mtusi-lab4-s3-init-1 Exited
|
||||
Container mtusi-lab4-migrate-1 Exited
|
||||
Container mtusi-lab4-valkey-1 Healthy
|
||||
Container mtusi-lab4-s3-1 Healthy
|
||||
Container mtusi-lab4-api1-1 Healthy
|
||||
PASS: upload=api1, download=api2, sha256=99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
|
||||
@@ -0,0 +1,6 @@
|
||||
Container mtusi-lab4-s3-1 Running
|
||||
Container mtusi-lab4-s3-1 Waiting
|
||||
Container mtusi-lab4-s3-1 Healthy
|
||||
Container mtusi-lab4-s3-init-run-6bff1f84d1f3 Creating
|
||||
Container mtusi-lab4-s3-init-run-6bff1f84d1f3 Created
|
||||
Private bucket and restricted API user ready
|
||||
@@ -0,0 +1,2 @@
|
||||
python3 scripts/check_contract.py
|
||||
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||
@@ -0,0 +1,7 @@
|
||||
{"Service": "api1", "ID": "dfb599607e63", "Image": "mtusi-messenger:lab4", "State": "running", "Health": "healthy", "Status": "Up 9 seconds (healthy)", "Ports": "8080/tcp"}
|
||||
{"Service": "api2", "ID": "0f6254d1d00b", "Image": "mtusi-messenger:lab4", "State": "running", "Health": "healthy", "Status": "Up 9 seconds (healthy)", "Ports": "8080/tcp"}
|
||||
{"Service": "cleaner", "ID": "91c5a5e1aecb", "Image": "mtusi-messenger:lab4", "State": "running", "Health": "", "Status": "Up Less than a second", "Ports": "8080/tcp"}
|
||||
{"Service": "lb", "ID": "2b763a6b13e7", "Image": "haproxy:3.0.8-alpine", "State": "running", "Health": "", "Status": "Up 10 seconds", "Ports": "127.0.0.1:8084->8080/tcp"}
|
||||
{"Service": "postgres", "ID": "1f7074602484", "Image": "postgres:16.6-bookworm", "State": "running", "Health": "healthy", "Status": "Up 10 seconds (healthy)", "Ports": "5432/tcp"}
|
||||
{"Service": "s3", "ID": "52d093feabad", "Image": "mtusi-minio:RELEASE.2025-04-22T22-12-26Z", "State": "running", "Health": "healthy", "Status": "Up 10 seconds (healthy)", "Ports": ""}
|
||||
{"Service": "valkey", "ID": "54a602e6f545", "Image": "valkey/valkey:8.0.2-alpine", "State": "running", "Health": "", "Status": "Up 10 seconds", "Ports": "6379/tcp"}
|
||||
@@ -0,0 +1,7 @@
|
||||
{"Service": "api1", "ID": "dfb599607e63", "Image": "mtusi-messenger:lab4", "State": "running", "Health": "healthy", "Status": "Up 42 seconds (healthy)", "Ports": "8080/tcp"}
|
||||
{"Service": "api2", "ID": "0f6254d1d00b", "Image": "mtusi-messenger:lab4", "State": "running", "Health": "healthy", "Status": "Up 42 seconds (healthy)", "Ports": "8080/tcp"}
|
||||
{"Service": "cleaner", "ID": "91c5a5e1aecb", "Image": "mtusi-messenger:lab4", "State": "running", "Health": "", "Status": "Up 8 seconds", "Ports": "8080/tcp"}
|
||||
{"Service": "lb", "ID": "2b763a6b13e7", "Image": "haproxy:3.0.8-alpine", "State": "running", "Health": "", "Status": "Up 36 seconds", "Ports": "127.0.0.1:8084->8080/tcp"}
|
||||
{"Service": "postgres", "ID": "1f7074602484", "Image": "postgres:16.6-bookworm", "State": "running", "Health": "healthy", "Status": "Up 46 seconds (healthy)", "Ports": "5432/tcp"}
|
||||
{"Service": "s3", "ID": "52d093feabad", "Image": "mtusi-minio:RELEASE.2025-04-22T22-12-26Z", "State": "running", "Health": "healthy", "Status": "Up 4 seconds (healthy)", "Ports": ""}
|
||||
{"Service": "valkey", "ID": "54a602e6f545", "Image": "valkey/valkey:8.0.2-alpine", "State": "running", "Health": "", "Status": "Up 46 seconds", "Ports": "6379/tcp"}
|
||||
@@ -0,0 +1,4 @@
|
||||
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://localhost:8084"
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||
{"observed_instances_before": 2, "successful_writes": 89, "failed_attempts": 3, "first_recovery_seconds": 2.8}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
@@ -0,0 +1,68 @@
|
||||
{
|
||||
"concurrency": 10,
|
||||
"file_bytes": 9201866,
|
||||
"results": [
|
||||
{
|
||||
"status": 503,
|
||||
"seconds": 0.597
|
||||
},
|
||||
{
|
||||
"status": 201,
|
||||
"seconds": 0.489
|
||||
},
|
||||
{
|
||||
"status": 201,
|
||||
"seconds": 0.68
|
||||
},
|
||||
{
|
||||
"status": 201,
|
||||
"seconds": 0.624
|
||||
},
|
||||
{
|
||||
"status": 201,
|
||||
"seconds": 0.427
|
||||
},
|
||||
{
|
||||
"status": 201,
|
||||
"seconds": 0.673
|
||||
},
|
||||
{
|
||||
"status": 503,
|
||||
"seconds": 0.57
|
||||
},
|
||||
{
|
||||
"status": 201,
|
||||
"seconds": 0.493
|
||||
},
|
||||
{
|
||||
"status": 201,
|
||||
"seconds": 0.455
|
||||
},
|
||||
{
|
||||
"status": 201,
|
||||
"seconds": 0.645
|
||||
}
|
||||
],
|
||||
"samples": [
|
||||
{
|
||||
"seconds": 0.0,
|
||||
"api1_rss_kib": 107580,
|
||||
"api2_rss_kib": 109104
|
||||
},
|
||||
{
|
||||
"seconds": 0.309,
|
||||
"api1_rss_kib": 128832,
|
||||
"api2_rss_kib": 133908
|
||||
},
|
||||
{
|
||||
"seconds": 0.639,
|
||||
"api1_rss_kib": 132796,
|
||||
"api2_rss_kib": 133912
|
||||
}
|
||||
],
|
||||
"peak_rss_kib": {
|
||||
"api1": 132796,
|
||||
"api2": 133912
|
||||
},
|
||||
"sampling_note": "Sequential /proc sampling; very brief peaks may be missed. 503 is explicit admission control."
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"concurrency": 10,
|
||||
"file_bytes": 9201866,
|
||||
"results": [
|
||||
{
|
||||
"status": 503,
|
||||
"seconds": 0.597
|
||||
},
|
||||
{
|
||||
"status": 201,
|
||||
"seconds": 0.489
|
||||
},
|
||||
{
|
||||
"status": 201,
|
||||
"seconds": 0.68
|
||||
},
|
||||
{
|
||||
"status": 201,
|
||||
"seconds": 0.624
|
||||
},
|
||||
{
|
||||
"status": 201,
|
||||
"seconds": 0.427
|
||||
},
|
||||
{
|
||||
"status": 201,
|
||||
"seconds": 0.673
|
||||
},
|
||||
{
|
||||
"status": 503,
|
||||
"seconds": 0.57
|
||||
},
|
||||
{
|
||||
"status": 201,
|
||||
"seconds": 0.493
|
||||
},
|
||||
{
|
||||
"status": 201,
|
||||
"seconds": 0.455
|
||||
},
|
||||
{
|
||||
"status": 201,
|
||||
"seconds": 0.645
|
||||
}
|
||||
],
|
||||
"peak_rss_kib": {
|
||||
"api1": 132796,
|
||||
"api2": 133912
|
||||
},
|
||||
"sampling_note": "Sequential /proc sampling; very brief peaks may be missed. 503 is explicit admission control."
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
$ make check
|
||||
python3 scripts/check_contract.py
|
||||
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||
exit=0
|
||||
|
||||
$ docker compose config --quiet
|
||||
exit=0
|
||||
|
||||
$ python3 student-tests/unit.py
|
||||
test_mime_substitution (__main__.Images) ... ok
|
||||
test_not_image (__main__.Images) ... ok
|
||||
test_pixel_limit_before_decode (__main__.Images) ... ok
|
||||
test_png_and_jpeg (__main__.Images) ... ok
|
||||
test_truncated_pixels (__main__.Images) ... ok
|
||||
|
||||
----------------------------------------------------------------------
|
||||
Ran 5 tests in 0.090s
|
||||
|
||||
OK
|
||||
exit=0
|
||||
|
||||
PASS: AST parsed for 15 Python files.
|
||||
@@ -0,0 +1,9 @@
|
||||
PASS: interrupted upload did not create an attachment
|
||||
Container mtusi-lab4-s3-1 Stopping
|
||||
Container mtusi-lab4-s3-1 Stopped
|
||||
PASS: unavailable S3 returns 503 from API
|
||||
Container mtusi-lab4-s3-1 Starting
|
||||
Container mtusi-lab4-s3-1 Started
|
||||
Container mtusi-lab4-s3-1 Waiting
|
||||
Container mtusi-lab4-s3-1 Healthy
|
||||
PASS: S3 recovery preserves the original bytes
|
||||
@@ -0,0 +1,3 @@
|
||||
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:8084"
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
@@ -0,0 +1,2 @@
|
||||
python3 tests/smoke.py --base-url "http://localhost:8084"
|
||||
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
|
||||
@@ -0,0 +1 @@
|
||||
PASS: private bucket, failed finalization recovery, active lock protection, grace period, ready protection, repeated cleanup
|
||||
@@ -0,0 +1 @@
|
||||
PASS: JPEG round-trip, invalid files, ACL, cross-chat references, concurrent idempotency
|
||||
@@ -0,0 +1,10 @@
|
||||
test_mime_substitution (__main__.Images) ... ok
|
||||
test_not_image (__main__.Images) ... ok
|
||||
test_pixel_limit_before_decode (__main__.Images) ... ok
|
||||
test_png_and_jpeg (__main__.Images) ... ok
|
||||
test_truncated_pixels (__main__.Images) ... ok
|
||||
|
||||
----------------------------------------------------------------------
|
||||
Ran 5 tests in 0.714s
|
||||
|
||||
OK
|
||||
@@ -0,0 +1,4 @@
|
||||
.venv/bin/python scripts/validate_openapi.py
|
||||
/Users/sleepy13/Desktop/projects/vp/mtusi2026lab4/.venv/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
|
||||
warnings.warn(
|
||||
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
|
||||
@@ -0,0 +1,21 @@
|
||||
global
|
||||
log stdout format raw local0
|
||||
maxconn 128
|
||||
defaults
|
||||
mode http
|
||||
log global
|
||||
timeout connect 2s
|
||||
timeout client 40s
|
||||
timeout server 40s
|
||||
timeout http-request 10s
|
||||
timeout queue 3s
|
||||
frontend http
|
||||
bind :8080
|
||||
default_backend api
|
||||
backend api
|
||||
balance roundrobin
|
||||
option httpchk GET /health/ready
|
||||
http-check expect status 200
|
||||
retries 0
|
||||
server api1 api1:8080 check inter 1s fall 2 rise 1
|
||||
server api2 api2:8080 check inter 1s fall 2 rise 1
|
||||
@@ -0,0 +1,23 @@
|
||||
# Build the pinned upstream releases: public MinIO registry images are unavailable.
|
||||
FROM golang:1.24.2-bookworm AS minio-build
|
||||
WORKDIR /src
|
||||
RUN curl -fsSL https://codeload.github.com/minio/minio/tar.gz/refs/tags/RELEASE.2025-04-22T22-12-26Z | tar xz --strip-components=1
|
||||
RUN CGO_ENABLED=0 go build -trimpath -o /out/minio .
|
||||
|
||||
FROM golang:1.24.2-bookworm AS mc-build
|
||||
WORKDIR /src
|
||||
RUN curl -fsSL https://codeload.github.com/minio/mc/tar.gz/refs/tags/RELEASE.2025-04-16T18-13-26Z | tar xz --strip-components=1
|
||||
RUN CGO_ENABLED=0 go build -trimpath -o /out/mc .
|
||||
|
||||
FROM debian:bookworm-20250407-slim AS runtime
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates curl && rm -rf /var/lib/apt/lists/*
|
||||
|
||||
FROM runtime AS server
|
||||
COPY --from=minio-build /out/minio /usr/local/bin/minio
|
||||
COPY --from=minio-build /src/LICENSE /usr/share/licenses/minio/LICENSE
|
||||
ENTRYPOINT ["minio"]
|
||||
|
||||
FROM runtime AS client
|
||||
COPY --from=mc-build /out/mc /usr/local/bin/mc
|
||||
COPY --from=mc-build /src/LICENSE /usr/share/licenses/mc/LICENSE
|
||||
ENTRYPOINT ["mc"]
|
||||
@@ -0,0 +1,9 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
mc alias set local http://s3:9000 "$(cat /run/secrets/s3_root_user)" "$(cat /run/secrets/s3_root_password)" >/dev/null
|
||||
mc mb --ignore-existing local/messenger-attachments >/dev/null
|
||||
mc anonymous set none local/messenger-attachments >/dev/null
|
||||
mc admin user add local "$(cat /run/secrets/s3_access_key_id)" "$(cat /run/secrets/s3_secret_access_key)" >/dev/null
|
||||
mc admin policy create local attachments /config/policy.json >/dev/null
|
||||
mc admin policy attach local attachments --user "$(cat /run/secrets/s3_access_key_id)" >/dev/null
|
||||
echo 'Private bucket and restricted API user ready'
|
||||
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{"Effect":"Allow","Action":["s3:ListBucket"],"Resource":["arn:aws:s3:::messenger-attachments"]},
|
||||
{"Effect":"Allow","Action":["s3:GetObject","s3:PutObject","s3:DeleteObject"],"Resource":["arn:aws:s3:::messenger-attachments/originals/*"]}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
CREATE TABLE users (
|
||||
id uuid PRIMARY KEY, username varchar(32) NOT NULL UNIQUE CHECK (username ~ '^[a-z0-9_]{3,32}$'),
|
||||
display_name varchar(100) NOT NULL CHECK (length(trim(display_name))>0),
|
||||
password_hash text NOT NULL, created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
CREATE TABLE chats (id uuid PRIMARY KEY, title varchar(100) NOT NULL CHECK(length(trim(title))>0), created_at timestamptz NOT NULL DEFAULT clock_timestamp());
|
||||
CREATE TABLE members (chat_id uuid REFERENCES chats(id), user_id uuid REFERENCES users(id), PRIMARY KEY(chat_id,user_id));
|
||||
CREATE INDEX members_user_chat ON members(user_id,chat_id);
|
||||
CREATE TABLE messages (id uuid PRIMARY KEY, chat_id uuid NOT NULL REFERENCES chats(id), sender_id uuid NOT NULL REFERENCES users(id), text varchar(4000) NOT NULL CHECK(length(trim(text))>0), created_at timestamptz NOT NULL DEFAULT clock_timestamp(), FOREIGN KEY(chat_id,sender_id) REFERENCES members(chat_id,user_id));
|
||||
CREATE INDEX messages_history ON messages(chat_id,created_at,id);
|
||||
CREATE INDEX chats_order ON chats(created_at,id);
|
||||
CREATE TABLE sessions (token_hash char(64) PRIMARY KEY, user_id uuid NOT NULL REFERENCES users(id), expires_at timestamptz NOT NULL);
|
||||
CREATE INDEX sessions_expiry ON sessions(expires_at);
|
||||
@@ -0,0 +1,18 @@
|
||||
ALTER TABLE messages DROP CONSTRAINT messages_text_check;
|
||||
ALTER TABLE messages ADD COLUMN attachment_ids uuid[] NOT NULL DEFAULT '{}';
|
||||
ALTER TABLE messages ADD CHECK (length(trim(text))>0 OR cardinality(attachment_ids)>0);
|
||||
CREATE TABLE idempotency (
|
||||
user_id uuid REFERENCES users(id), chat_id uuid REFERENCES chats(id), key varchar(128),
|
||||
body text NOT NULL, response jsonb NOT NULL,
|
||||
expires_at timestamptz NOT NULL DEFAULT clock_timestamp()+interval '24 hours',
|
||||
PRIMARY KEY(user_id,chat_id,key)
|
||||
);
|
||||
CREATE TABLE attachments (
|
||||
id uuid PRIMARY KEY, chat_id uuid NOT NULL REFERENCES chats(id), owner_id uuid NOT NULL REFERENCES users(id),
|
||||
filename varchar(255) NOT NULL, object_key text NOT NULL UNIQUE,
|
||||
content_type text NOT NULL, size_bytes bigint NOT NULL, sha256 char(64) NOT NULL,
|
||||
status text NOT NULL CHECK(status IN ('pending','ready')),
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
updated_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
CREATE INDEX attachments_pending ON attachments(created_at) WHERE status='pending';
|
||||
@@ -0,0 +1,32 @@
|
||||
"""Generate local-only synthetic credentials once; never overwrite them."""
|
||||
import os
|
||||
import secrets
|
||||
from pathlib import Path
|
||||
root = Path(__file__).resolve().parents[1]
|
||||
os.umask(0o077)
|
||||
(root / "secrets").mkdir(exist_ok=True)
|
||||
p = root / "secrets/postgres_password"
|
||||
if not p.exists():
|
||||
p.write_text(secrets.token_hex(32))
|
||||
values = {"database_url": "postgresql://messenger:" + p.read_text().strip() + "@postgres:5432/messenger", "cache_url": "redis://valkey:6379/0"}
|
||||
for name, value in values.items():
|
||||
dest = root / "secrets" / name
|
||||
if not dest.exists():
|
||||
dest.write_text(value)
|
||||
# Docker Desktop bind-mounted secrets need read permission for container UID 10001.
|
||||
for name in ["postgres_password", "database_url", "cache_url"]:
|
||||
(root / "secrets" / name).chmod(0o644)
|
||||
env = root / ".env"
|
||||
if not env.exists():
|
||||
env.write_text("SESSION_TTL_SECONDS=3600\nHOST_PORT=8084\n")
|
||||
print("Local configuration ready (secrets not printed)")
|
||||
for name, value in {
|
||||
's3_root_user': 'admin-' + secrets.token_hex(8),
|
||||
's3_root_password': secrets.token_hex(32),
|
||||
's3_access_key_id': 'api-' + secrets.token_hex(8),
|
||||
's3_secret_access_key': secrets.token_hex(32),
|
||||
}.items():
|
||||
dest = root / 'secrets' / name
|
||||
if not dest.exists():
|
||||
dest.write_text(value)
|
||||
dest.chmod(0o644)
|
||||
Executable
+4
@@ -0,0 +1,4 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
docker compose restart postgres valkey s3 api1 api2 cleaner lb
|
||||
Executable
+7
@@ -0,0 +1,7 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
case "${TARGET_INSTANCE_ID:-}" in
|
||||
api1|api2) docker compose stop "$TARGET_INSTANCE_ID" ;;
|
||||
*) echo 'Unknown target instance' >&2; exit 1 ;;
|
||||
esac
|
||||
@@ -0,0 +1,48 @@
|
||||
"""Run all checks and save actual outputs. Stop on the first failure."""
|
||||
import os
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
root=Path(__file__).resolve().parents[1]
|
||||
os.chdir(root)
|
||||
os.environ.setdefault('BASE_URL','http://localhost:8084')
|
||||
(root/'evidence').mkdir(exist_ok=True)
|
||||
|
||||
def run(name, args, stdin=None):
|
||||
result=subprocess.run(args,input=stdin,text=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
|
||||
output = result.stdout
|
||||
if name.startswith('containers-') and result.returncode == 0:
|
||||
fields = ('Service', 'ID', 'Image', 'State', 'Health', 'Status', 'Ports')
|
||||
output = '\n'.join(json.dumps({key: row.get(key) for key in fields})
|
||||
for row in map(json.loads, output.splitlines())) + '\n'
|
||||
output = '\n'.join(line.rstrip() for line in output.splitlines()) + '\n'
|
||||
(root/'evidence'/name).write_text(output)
|
||||
print(output,end='')
|
||||
if result.returncode:
|
||||
raise SystemExit(result.returncode)
|
||||
|
||||
run('check.txt',['make','check'])
|
||||
run('unit.txt',[sys.executable,'student-tests/unit.py'])
|
||||
run('smoke.txt',['make','test','BASE_URL='+os.environ['BASE_URL']])
|
||||
run('student-tests.txt',[sys.executable,'student-tests/verify.py'])
|
||||
subprocess.run(['docker','compose','stop','cleaner'],check=True)
|
||||
try:
|
||||
run('storage-checks.txt',['docker','compose','exec','-T','api1','python','-'],
|
||||
(root/'student-tests/storage_checks.py').read_text())
|
||||
finally:
|
||||
subprocess.run(['docker','compose','start','cleaner'],check=True)
|
||||
run('outage.txt',[sys.executable,'student-tests/outage.py'])
|
||||
run('bootstrap-repeat.txt',['docker','compose','run','--rm','s3-init'])
|
||||
run('containers-before.txt',['docker','compose','ps','--format','json'])
|
||||
run('persistence.txt',['make','test-persistence','ACTION_SCRIPT=scripts/restart.sh','BASE_URL='+os.environ['BASE_URL']])
|
||||
run('containers-after.txt',['docker','compose','ps','--format','json'])
|
||||
try:
|
||||
run('failover.txt',['make','test-failover','ACTION_SCRIPT=scripts/stop-one.sh','BASE_URL='+os.environ['BASE_URL']])
|
||||
finally:
|
||||
subprocess.run(['docker','compose','start','api1','api2'],check=True)
|
||||
# Compose waits for the API's actual readiness after restoring the replica.
|
||||
subprocess.run(['docker','compose','up','-d','--wait','api1','api2'],check=True)
|
||||
run('attachment-failover.txt',[sys.executable,'student-tests/attachment_failover.py'])
|
||||
run('memory.txt',[sys.executable,'student-tests/memory.py'])
|
||||
print('All requested checks passed; actual results are in evidence/.')
|
||||
@@ -0,0 +1,34 @@
|
||||
"""Stop the exact replica that accepted the upload, then compare original bytes."""
|
||||
import hashlib
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'tests'))
|
||||
from client import Client, Failure, check
|
||||
|
||||
c = Client(timeout=10)
|
||||
a, b, _, chat = c.bootstrap()
|
||||
png = (Path(__file__).resolve().parents[1] / 'tests/fixtures/sample.png').read_bytes()
|
||||
result = c.call('POST', '/api/v1/chats/' + chat['id'] + '/attachments?filename=failover.png',
|
||||
201, actor=a, raw=png, headers={'Content-Type': 'image/png'}, schema='Attachment')
|
||||
target = result['headers']['X-Instance-Id']
|
||||
check(target in ('api1', 'api2'), 'unknown upload replica')
|
||||
try:
|
||||
subprocess.run(['docker', 'compose', 'stop', target], check=True)
|
||||
deadline = time.monotonic() + 20
|
||||
while True:
|
||||
try:
|
||||
downloaded = c.call('GET', '/api/v1/attachments/' + result['json']['id'] + '/content', actor=b)
|
||||
break
|
||||
except Failure:
|
||||
if time.monotonic() >= deadline:
|
||||
raise
|
||||
time.sleep(0.5)
|
||||
check(downloaded['headers']['X-Instance-Id'] != target, 'uploader still serves download')
|
||||
check(downloaded['bytes'] == png, 'original changed after uploader stopped')
|
||||
print('PASS: upload=' + target + ', download=' + downloaded['headers']['X-Instance-Id']
|
||||
+ ', sha256=' + hashlib.sha256(png).hexdigest())
|
||||
finally:
|
||||
subprocess.run(['docker', 'compose', 'up', '-d', '--wait', 'api1', 'api2'], check=True)
|
||||
@@ -0,0 +1,57 @@
|
||||
"""Start ten simultaneous uploads; record actual container RSS via /proc."""
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
sys.path.insert(0,str(Path(__file__).resolve().parents[1]/'tests'))
|
||||
from client import Client, check
|
||||
from PIL import Image
|
||||
import io
|
||||
|
||||
c=Client(timeout=45)
|
||||
a,b,e,chat=c.bootstrap()
|
||||
file=io.BytesIO()
|
||||
# Incompressible ~9 MiB RGB PNG rather than ten tiny fixtures.
|
||||
Image.frombytes('RGB',(1750,1750),os.urandom(1750*1750*3)).save(file,'PNG')
|
||||
data=file.getvalue()
|
||||
check(len(data)<=10485760,'fixture too large')
|
||||
barrier=threading.Barrier(10)
|
||||
finished=threading.Event()
|
||||
samples=[]
|
||||
|
||||
def rss():
|
||||
while not finished.is_set():
|
||||
row={'seconds':round(time.monotonic()-start,3)}
|
||||
for name in ['api1','api2']:
|
||||
result=subprocess.run(['docker','compose','exec','-T',name,'python','-c',
|
||||
"from pathlib import Path; print(next(x.split()[1] for x in Path('/proc/1/status').read_text().splitlines() if x.startswith('VmRSS:')))"] ,capture_output=True,text=True,check=True)
|
||||
row[name+'_rss_kib']=int(result.stdout.strip())
|
||||
samples.append(row)
|
||||
finished.wait(0.1)
|
||||
|
||||
def upload(i):
|
||||
barrier.wait()
|
||||
response=Client(timeout=45).call('POST','/api/v1/chats/'+chat['id']+'/attachments?filename=load.png',
|
||||
[201,503],actor=a,raw=data,headers={'Content-Type':'image/png'})
|
||||
return {'status':response['status'],'seconds':round(response['elapsed'],3)}
|
||||
|
||||
start=time.monotonic()
|
||||
with ThreadPoolExecutor(max_workers=1) as monitor:
|
||||
future=monitor.submit(rss)
|
||||
try:
|
||||
with ThreadPoolExecutor(max_workers=10) as pool:
|
||||
results=list(pool.map(upload,range(10)))
|
||||
finally:
|
||||
finished.set()
|
||||
future.result()
|
||||
check(samples,'no memory measurements')
|
||||
report={'concurrency':10,'file_bytes':len(data),'results':results,'samples':samples,
|
||||
'peak_rss_kib':{n:max(s[n+'_rss_kib'] for s in samples) for n in ['api1','api2']},
|
||||
'sampling_note':'Sequential /proc sampling; very brief peaks may be missed. 503 is explicit admission control.'}
|
||||
Path('evidence/memory.json').write_text(json.dumps(report,indent=2))
|
||||
check(any(r['status']==201 for r in results),'no upload succeeded')
|
||||
print(json.dumps({k:v for k,v in report.items() if k!='samples'},indent=2))
|
||||
@@ -0,0 +1,66 @@
|
||||
"""Exercise interrupted HTTP upload and unavailable S3 on this Compose project."""
|
||||
import http.client
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from urllib.parse import urlsplit
|
||||
sys.path.insert(0,str(Path(__file__).resolve().parents[1]/'tests'))
|
||||
from client import Client, check
|
||||
c=Client(timeout=45)
|
||||
a,b,e,chat=c.bootstrap()
|
||||
att,png=c.attachment(a,chat)
|
||||
base=urlsplit(c.base)
|
||||
check(base.hostname in ('localhost','127.0.0.1'),'test is limited to the local lab')
|
||||
|
||||
# Close while the body is incomplete. No intent should be created before validation.
|
||||
connection=http.client.HTTPConnection(base.hostname,base.port,timeout=5)
|
||||
connection.putrequest('POST','/api/v1/chats/'+chat['id']+'/attachments?filename=interrupted.png')
|
||||
connection.putheader('Authorization','Bearer '+a['token'])
|
||||
connection.putheader('Content-Type','image/png')
|
||||
connection.putheader('Content-Length',str(len(png)))
|
||||
connection.endheaders()
|
||||
connection.send(png[:16])
|
||||
connection.close()
|
||||
|
||||
# Query only the synthetic chat through stdin; credentials/tokens are never printed.
|
||||
probe="""
|
||||
import sys
|
||||
sys.path.insert(0,'/srv/app')
|
||||
import psycopg,storage
|
||||
with psycopg.connect(storage.secret('DATABASE_URL')) as db:
|
||||
n=db.execute("SELECT count(*) FROM attachments WHERE chat_id=%s", (CHAT,)).fetchone()[0]
|
||||
assert n==1, 'interrupted upload created metadata'
|
||||
print('PASS: interrupted upload did not create an attachment')
|
||||
""".replace('CHAT',repr(chat['id']))
|
||||
subprocess.run(['docker','compose','exec','-T','api1','python','-'],input=probe,text=True,check=True)
|
||||
|
||||
# Direct request within api1 ensures the API's structured 503 is tested,
|
||||
# independently of HAProxy taking all unready instances out of rotation.
|
||||
probe="""
|
||||
import urllib.request,urllib.error
|
||||
req=urllib.request.Request('http://localhost:8080/api/v1/attachments/ATT/content',headers={'Authorization':'Bearer TOKEN'})
|
||||
try:
|
||||
urllib.request.urlopen(req,timeout=20)
|
||||
raise AssertionError('unavailable S3 returned success')
|
||||
except urllib.error.HTTPError as error:
|
||||
assert error.code==503, error.code
|
||||
print('PASS: unavailable S3 returns 503 from API')
|
||||
""".replace('ATT',att['id']).replace('TOKEN',a['token'])
|
||||
try:
|
||||
subprocess.run(['docker','compose','stop','s3'],check=True)
|
||||
subprocess.run(['docker','compose','exec','-T','api1','python','-'],input=probe,text=True,check=True)
|
||||
finally:
|
||||
subprocess.run(['docker','compose','up','-d','--wait','s3'],check=True)
|
||||
for attempt in range(30):
|
||||
try:
|
||||
result=c.call('GET','/api/v1/attachments/'+att['id']+'/content',actor=b)
|
||||
check(result['bytes']==png,'recovered file differs')
|
||||
break
|
||||
except AssertionError:
|
||||
if attempt==29:
|
||||
raise
|
||||
time.sleep(1)
|
||||
print('PASS: S3 recovery preserves the original bytes')
|
||||
@@ -0,0 +1,66 @@
|
||||
"""Run inside the API container: private bucket and cleanup failure recovery."""
|
||||
import io
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import uuid
|
||||
import urllib.request
|
||||
import urllib.error
|
||||
sys.path.insert(0, '/srv/app')
|
||||
import psycopg
|
||||
import storage
|
||||
from cleanup import sweep
|
||||
|
||||
prefix = uuid.uuid4().hex
|
||||
ids = [str(uuid.uuid4()) for _ in range(3)]
|
||||
keys = ['originals/'+i for i in ids]
|
||||
user,chat = str(uuid.uuid4()),str(uuid.uuid4())
|
||||
dsn=storage.secret('DATABASE_URL')
|
||||
# Model a process stopping after S3 PUT but before final DB COMMIT.
|
||||
# Also create a fresh intent and an old ready file; neither may be removed.
|
||||
try:
|
||||
with psycopg.connect(dsn) as db:
|
||||
db.execute('INSERT INTO users(id,username,display_name,password_hash) VALUES (%s,%s,%s,%s)',
|
||||
(user,'cleanup_'+prefix[:12],'Synthetic cleanup','not-a-login-hash'))
|
||||
db.execute('INSERT INTO chats(id,title) VALUES (%s,%s)', (chat,'Synthetic cleanup'))
|
||||
for i,(aid,key) in enumerate(zip(ids,keys)):
|
||||
db.execute("""INSERT INTO attachments(id,chat_id,owner_id,filename,object_key,content_type,size_bytes,sha256,status,created_at)
|
||||
VALUES (%s,%s,%s,'test.png',%s,'image/png',3,%s,%s,clock_timestamp()-%s*interval '1 hour')""",
|
||||
(aid,chat,user,key,'0'*64,'ready' if i==2 else 'pending',2 if i!=1 else 0))
|
||||
for key in keys:
|
||||
storage.client.put_object(Bucket=storage.BUCKET,Key=key,Body=b'abc')
|
||||
try:
|
||||
urllib.request.urlopen('http://s3:9000/'+storage.BUCKET+'/'+keys[2],timeout=5)
|
||||
raise AssertionError('Anonymous object access allowed')
|
||||
except urllib.error.HTTPError as error:
|
||||
assert error.code==403, error.code
|
||||
# Roll back a simulated failed finalization; durable pending intent remains.
|
||||
with psycopg.connect(dsn) as db:
|
||||
db.execute("UPDATE attachments SET status='ready' WHERE id=%s", (ids[0],))
|
||||
db.rollback()
|
||||
# Even an old intent must survive while a live finalization holds its lock.
|
||||
with psycopg.connect(dsn) as active:
|
||||
active.execute('SELECT id FROM attachments WHERE id=%s FOR UPDATE', (ids[0],))
|
||||
sweep()
|
||||
storage.client.head_object(Bucket=storage.BUCKET, Key=keys[0])
|
||||
assert active.execute('SELECT status FROM attachments WHERE id=%s', (ids[0],)).fetchone()[0] == 'pending'
|
||||
sweep()
|
||||
sweep() # Idempotent after completion.
|
||||
with psycopg.connect(dsn) as db:
|
||||
remaining={str(r[0]) for r in db.execute('SELECT id FROM attachments WHERE id=ANY(%s::uuid[])',(ids,))}
|
||||
assert remaining==set(ids[1:]), remaining
|
||||
try:
|
||||
storage.client.head_object(Bucket=storage.BUCKET,Key=keys[0])
|
||||
raise AssertionError('Abandoned object still exists')
|
||||
except storage.ClientError as error:
|
||||
assert error.response['ResponseMetadata']['HTTPStatusCode']==404
|
||||
for key in keys[1:]:
|
||||
storage.client.head_object(Bucket=storage.BUCKET,Key=key)
|
||||
print('PASS: private bucket, failed finalization recovery, active lock protection, grace period, ready protection, repeated cleanup')
|
||||
finally:
|
||||
for key in keys:
|
||||
storage.client.delete_object(Bucket=storage.BUCKET,Key=key)
|
||||
with psycopg.connect(dsn) as db:
|
||||
db.execute('DELETE FROM attachments WHERE id=ANY(%s::uuid[])',(ids,))
|
||||
db.execute('DELETE FROM chats WHERE id=%s',(chat,))
|
||||
db.execute('DELETE FROM users WHERE id=%s',(user,))
|
||||
@@ -0,0 +1,50 @@
|
||||
"""Offline checks of image content, size limits, and MIME detection."""
|
||||
import io
|
||||
from pathlib import Path
|
||||
import struct
|
||||
import sys
|
||||
import unittest
|
||||
import zlib
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'app'))
|
||||
from PIL import Image
|
||||
from images import validate_image, InvalidImage, TooManyPixels
|
||||
|
||||
class Images(unittest.TestCase):
|
||||
def picture(self, fmt):
|
||||
file = io.BytesIO()
|
||||
Image.new('RGB', (320, 200), (20, 30, 40)).save(file, fmt)
|
||||
file.seek(0)
|
||||
return file
|
||||
|
||||
def test_png_and_jpeg(self):
|
||||
for fmt, mime in [('PNG', 'image/png'), ('JPEG', 'image/jpeg')]:
|
||||
file = self.picture(fmt)
|
||||
before = file.getvalue()
|
||||
validate_image(file, mime)
|
||||
self.assertEqual(file.tell(), 0)
|
||||
self.assertEqual(file.read(), before)
|
||||
|
||||
def test_mime_substitution(self):
|
||||
with self.assertRaises(InvalidImage):
|
||||
validate_image(self.picture('PNG'), 'image/jpeg')
|
||||
|
||||
def test_not_image(self):
|
||||
for content in [b'', b'hello', b'<script>alert(1)</script>']:
|
||||
with self.assertRaises(InvalidImage):
|
||||
validate_image(io.BytesIO(content), 'image/png')
|
||||
|
||||
def test_truncated_pixels(self):
|
||||
file = self.picture('JPEG')
|
||||
with self.assertRaises(InvalidImage):
|
||||
validate_image(io.BytesIO(file.getvalue()[:-20]), 'image/jpeg')
|
||||
|
||||
def test_pixel_limit_before_decode(self):
|
||||
def chunk(kind, data):
|
||||
return struct.pack('>I', len(data))+kind+data+struct.pack('>I', zlib.crc32(kind+data)&0xffffffff)
|
||||
huge = b'\x89PNG\r\n\x1a\n'+chunk(b'IHDR',struct.pack('>IIBBBBB', 8000,8000,8,2,0,0,0))
|
||||
huge += chunk(b'IDAT', zlib.compress(b'\x00'))+chunk(b'IEND', b'')
|
||||
with self.assertRaises(TooManyPixels):
|
||||
validate_image(io.BytesIO(huge), 'image/png')
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main(verbosity=2)
|
||||
@@ -0,0 +1,59 @@
|
||||
"""Black-box grade-4 checks; creates only synthetic data on the local lab."""
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import threading
|
||||
import uuid
|
||||
from PIL import Image
|
||||
from urllib.parse import quote
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'tests'))
|
||||
from client import Client, check
|
||||
|
||||
c = Client(timeout=45)
|
||||
a,b,e,chat = c.bootstrap()
|
||||
base = '/api/v1/chats/' + chat['id']
|
||||
att, png = c.attachment(a, chat)
|
||||
# Exercise the second supported format through both stores, not only Pillow.
|
||||
jpeg_file = io.BytesIO()
|
||||
Image.new('RGB', (320, 200), (20, 30, 40)).save(jpeg_file, 'JPEG')
|
||||
jpeg = jpeg_file.getvalue()
|
||||
jpeg_att = c.call('POST', base+'/attachments?filename=photo.jpg', 201, actor=a,
|
||||
raw=jpeg, headers={'Content-Type': 'image/jpeg'}, schema='Attachment')['json']
|
||||
check(jpeg_att['sha256'] == hashlib.sha256(jpeg).hexdigest(), 'JPEG checksum differs')
|
||||
check(jpeg_att['size_bytes'] == len(jpeg), 'JPEG length differs')
|
||||
jpeg_download = c.call('GET', '/api/v1/attachments/'+jpeg_att['id']+'/content', actor=b)
|
||||
check(jpeg_download['bytes'] == jpeg, 'JPEG original differs')
|
||||
check(jpeg_download['headers'].get_content_type() == 'image/jpeg', 'JPEG MIME differs')
|
||||
for name in ['../x.png', 'a\\b.png', '\x00.png', '\n.png']:
|
||||
c.call('POST', base+'/attachments?filename='+quote(name, safe=''), 400, actor=a,
|
||||
raw=png, headers={'Content-Type':'image/png'})
|
||||
for data,mime,status in [(b'x'*(10485760+1),'image/png',413),
|
||||
(b'not an image','image/png',415), (png,'image/jpeg',415),
|
||||
(png[:20],'image/png',415)]:
|
||||
c.call('POST', base+'/attachments?filename=x.png', status, actor=a,
|
||||
raw=data, headers={'Content-Type':mime})
|
||||
other = c.call('POST','/api/v1/chats',201,actor=a,
|
||||
body={'title':'Other chat','member_ids':[a['id']]},schema='Chat')['json']
|
||||
for target,ids,actor in [(other,[att['id']],a), (chat,[str(uuid.uuid4())],a), (chat,[att['id']],e)]:
|
||||
c.call('POST','/api/v1/chats/'+target['id']+'/messages',404,actor=actor,
|
||||
body={'text':'','attachment_ids':ids},headers={'Idempotency-Key':uuid.uuid4().hex})
|
||||
for suffix in ['', '/content']:
|
||||
c.call('GET','/api/v1/attachments/'+att['id']+suffix,404,actor=e)
|
||||
# A member other than the uploader can attach and download the same file.
|
||||
c.message(b, chat, '', attachments=[att['id']])
|
||||
key = uuid.uuid4().hex
|
||||
barrier = threading.Barrier(10)
|
||||
def replay(_):
|
||||
barrier.wait()
|
||||
return Client(timeout=45).message(a,chat,'one logical message',key=key)['json']
|
||||
with ThreadPoolExecutor(max_workers=10) as pool:
|
||||
messages = list(pool.map(replay,range(10)))
|
||||
check(all(m == messages[0] for m in messages), 'concurrent replay duplicated messages')
|
||||
c.call('POST',base+'/messages',409,actor=a,body={'text':'different'},headers={'Idempotency-Key':key})
|
||||
# Default [] and explicit [] are equivalent.
|
||||
c.call('POST',base+'/messages',201,actor=a,body={'text':'one logical message','attachment_ids':[]},
|
||||
headers={'Idempotency-Key':key},schema='Message')
|
||||
print('PASS: JPEG round-trip, invalid files, ACL, cross-chat references, concurrent idempotency')
|
||||
Reference in New Issue
Block a user