forked from mtusi2026/mtusi2026lab5
lab 5 + kafka message broker
This commit is contained in:
+13
@@ -0,0 +1,13 @@
|
|||||||
|
FROM python:3.12.11-slim-bookworm
|
||||||
|
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 PIP_DISABLE_PIP_VERSION_CHECK=1
|
||||||
|
WORKDIR /app
|
||||||
|
RUN addgroup --system app && adduser --system --ingroup app app
|
||||||
|
COPY requirements.txt ./requirements.txt
|
||||||
|
RUN python -m pip install --no-cache-dir -r requirements.txt
|
||||||
|
COPY src ./src
|
||||||
|
COPY migrations ./migrations
|
||||||
|
COPY scripts ./scripts
|
||||||
|
USER app
|
||||||
|
EXPOSE 8080
|
||||||
|
STOPSIGNAL SIGTERM
|
||||||
|
CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8080", "--timeout-graceful-shutdown", "10", "--proxy-headers"]
|
||||||
+113
@@ -0,0 +1,113 @@
|
|||||||
|
# Отчёт - лабораторная 5
|
||||||
|
|
||||||
|
- Автор, группа: Алексей Миронов, БВТ2302
|
||||||
|
- Целевая оценка: **4** (трек 5 не заявляется)
|
||||||
|
- Commit/PR: TODO после публикации
|
||||||
|
- Источники, библиотеки, помощь ИИ: FastAPI 0.116.1, psycopg 3.2.9, boto3 1.40.20, Pillow 11.3.0, confluent-kafka 2.10.1 (librdkafka), Apache Kafka 3.9.0 (KRaft), PostgreSQL 17.6, Redis 7.4.2, HAProxy 3.2.5, MinIO-совместимый S3. Код lab5 собран на базе lab4; использовалась помощь ИИ
|
||||||
|
|
||||||
|
## Запуск из чистого clone
|
||||||
|
|
||||||
|
Требования: Docker Compose v2, Python 3.9+ для тестов. Секреты - в локальном `.env` (в git не коммитится; значения синтетические, шаблон - `.env.example`).
|
||||||
|
|
||||||
|
```sh
|
||||||
|
make check
|
||||||
|
make validate
|
||||||
|
docker compose --env-file .env config --quiet
|
||||||
|
docker compose --env-file .env up -d --build # postgres, redis, s3, kafka -> init/migrate -> api x2, worker, dispatcher, sweeper, haproxy
|
||||||
|
# дождаться готовности (до ~60 с):
|
||||||
|
until curl -fs http://localhost:8080/health/ready; do sleep 2; done
|
||||||
|
make test # публичный smoke
|
||||||
|
python3 student-tests/level4_failures.py all # сценарии сбоев уровня 4
|
||||||
|
make test-persistence ACTION_SCRIPT=scripts/restart.sh
|
||||||
|
make test-failover ACTION_SCRIPT=scripts/stop-one.sh
|
||||||
|
```
|
||||||
|
Безопасная остановка: `docker compose stop` (volumes сохраняются). Намеренное удаление учебных данных: `docker compose down -v`. Перезапуск всех хранилищ: `scripts/restart.sh`; остановка одной API-реплики: `scripts/stop-one.sh` (hostname = `X-Instance-Id`). Оператор: failed-реестр и replay - `docker compose exec worker python scripts/replay_failed.py --list | --job-id <uuid> --reason <text>`.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## Архитектура и решения
|
||||||
|
|
||||||
|
```
|
||||||
|
Клиент -> HAProxy -> API x2 --(1) PUT staging/<random> --> S3 (private bucket)
|
||||||
|
\--(2) ONE TX: attachments(queued) + attachment_jobs + outbox --> PostgreSQL => 202
|
||||||
|
dispatcher: outbox -> Kafka (acks=all, idempotent producer, delivery report) ; reaper (lease)
|
||||||
|
worker (consumer group): Kafka message(IDs only) -> claim+lease в PG -> staging -> imgproc (child, limits) -> final S3 -> TX ready -> offset commit
|
||||||
|
```
|
||||||
|
- **Источник истины** - PostgreSQL (`attachments.status`, `attachment_jobs`, `attachment_variants`, `outbox`, `job_events`). Kafka - транспорт at-least-once: сообщение содержит только ID (`jobs.schema.json`), без байтов/ключей/URL.
|
||||||
|
- **Staging** - приватный префикс `staging/` того же приватного bucket; honest-граница: staging в S3, файл уже полностью принят и сохранён до 202. Оригинал воркер публикует в `attachments/<id>/v1/original`, варианты - `.../v1/{thumbnail,crop,optimized}`. Ключ эффекта `(attachment_id, pipeline_version, variant)` = PK `attachment_variants` + детерминированный S3-ключ.
|
||||||
|
- **Kafka**: топик `attachment-jobs` (3 партиции, key = attachment_id, RF=1, min.insync=1, volume `kafka_data`). Группа `attachment-workers`, `enable.auto.commit=false`. Publisher confirm = delivery report продюсера при `acks=all` (приём брокером); consumer ack = commit offset **после** фиксации результата в PG.
|
||||||
|
- **Outbox**: строка outbox вставляется в ту же транзакцию, что и attachment/job. Relay берёт строки `FOR UPDATE SKIP LOCKED`, публикует, ставит `published_at` только по delivery report. Падение между COMMIT и publish -> строка осталась, relay доставит позже; между publish и mark -> дубль сообщения (допустим, идемпотентность на потребителе).
|
||||||
|
- **Retry**: временные ошибки (S3/сеть/timeout/неожиданный код дочернего процесса) -> `queued`, `next_attempt_at = now + backoff` (экспонента 2·2^(n-1), cap 60 с, equal jitter [d/2, d]), задержка реализована **отложенной строкой outbox** (`available_at`). Постоянные (`invalid_image`, `image_too_large`, `image_too_complex`, `integrity_error`, `staging_missing`) -> сразу `failed`. `JOB_MAX_ATTEMPTS=5`, исчерпание -> `retry_exhausted`. Каждый failed сохраняется в реестре PostgreSQL (`attachment_jobs`) и журнале `job_events`.
|
||||||
|
- **Lease**: `lease_until = now + JOB_TIMEOUT(60) + grace(15)`. Dispatcher возвращает `processing` с истёкшим lease в `queued` (новая попытка) или в `failed`. Повторная доставка при живом lease - no-op (конкурентные дубли не запускаются), при готовом/failed - no-op.
|
||||||
|
- **Конкурентность/prefetch**: `consume(num_messages=WORKER_CONCURRENCY)` + ThreadPool = не более 2 job одновременно; `queued.max.messages.kbytes=1024`. Лимиты обработки: байты 10 MiB (API, до чтения тела), пиксели 40 млн (из заголовка до декодирования), время `JOB_TIMEOUT_SECONDS` (kill дочернего процесса), память `RLIMIT_AS=IMG_MEMORY_MB` (1024 МБ) + `mem_limit` контейнера, CPU `RLIMIT_CPU`.
|
||||||
|
- **Варианты**: thumbnail JPEG q85 (вписан в 256×256, без увеличения, EXIF-orientation применён, alpha на белом; 320×200->256×160); crop JPEG q85, центрированный квадрат 128×128; optimized WebP q80 (вписан в 1280×1280 без увеличения, alpha сохраняется, EXIF/ICC не переносятся). Ready - только когда записаны все три и COMMIT выполнен одной транзакцией.
|
||||||
|
- **Очистка staging**: воркер удаляет staging после COMMIT; sweeper (grace 1 ч) удаляет сироты `staging/` и `attachments/<id>/` без строки БД; staging failed-вложений хранится `STAGING_RETENTION_SECONDS` (24 ч), после чего replay невозможен (нужна повторная загрузка).
|
||||||
|
- **Readiness API**: PostgreSQL, Redis (сессии), S3. Kafka и worker **не** критичны: приём работает через outbox.
|
||||||
|
- **Перенесено из lab4** без изменений: пользователи, Argon2id, сессии в Redis, чаты, сообщения, keyset-пагинация, идемпотентность (advisory lock + атомарная запись), HAProxy, bucket/политики s3_init (+`staging/*`), sweeper. **Изменено**: 201->202, статусы и варианты в метаданных, 409 для content не-ready и для прикрепления failed, новые таблицы (миграция 004), сервисы kafka/worker/dispatcher.
|
||||||
|
- **Соответствие .env.example**: `BROKER_URL_FILE` -> `KAFKA_BOOTSTRAP_SERVERS` (без паролей, PLAINTEXT во внутренней сети backend), `WORKER_CONCURRENCY`, `JOB_MAX_ATTEMPTS`, `JOB_TIMEOUT_SECONDS`, `STAGING_RETENTION_SECONDS`, `MAX_*` - те же имена. Secrets в `_FILE`-виде не используются (учебное упрощение lab4: переменные окружения).
|
||||||
|
|
||||||
|
| Роль | Сервис | Версия | Health | Volume |
|
||||||
|
| --- | --- | --- | --- | --- |
|
||||||
|
| load_balancer | load_balancer | haproxy 3.2.5-alpine | `haproxy -c` / backend check `/health/ready` | - |
|
||||||
|
| api | api (2 реплики) | свой образ | `/health/ready` | нет (stateless) |
|
||||||
|
| postgres | postgres | 17.6-alpine | pg_isready | postgres_data |
|
||||||
|
| sessions | redis | 7.4.2-alpine, AOF | redis-cli ping | redis_data |
|
||||||
|
| s3 | s3 (+s3_init) | pgsty/silo (MinIO-совм.) | mc ready | s3_data |
|
||||||
|
| broker | kafka (+kafka_init) | apache/kafka 3.9.0 | kafka-broker-api-versions | kafka_data |
|
||||||
|
| worker / dispatcher / sweeper | worker, dispatcher, s3_sweeper | свой образ | перезапуск `unless-stopped` | - |
|
||||||
|
|
||||||
|
## Доказательства критериев
|
||||||
|
|
||||||
|
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| Структура проекта и синтаксис Python | `make check` | `PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.` | `evidence/evidence.txt` |
|
||||||
|
| OpenAPI 3.1 и JSON Schema | `make validate` | `PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.` Контракты прошли валидацию. | `evidence/evidence.txt` |
|
||||||
|
| Корректность конфигурации Compose | `docker compose --env-file .env config --quiet` | Команда завершилась без вывода ошибки. | `evidence/evidence.txt` |
|
||||||
|
| Готовность приложения | `curl -fs http://localhost:8080/health/ready` в цикле ожидания | Получен ответ `{"status":"ok"}`. | `evidence/evidence.txt` |
|
||||||
|
| Публичный smoke | `make test` | `PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.` | `evidence/evidence.txt` |
|
||||||
|
| 202 и устойчивый приём; worker остановлен -> upload -> рестарт API -> worker -> ready | `python3 student-tests/level4_failures.py all` - сценарий `worker_down` | `PASS worker_down`; события: `accepted,claimed,ready`. | `evidence/evidence.txt` |
|
||||||
|
| Crash между COMMIT и publish; восстановление доставки через outbox | `python3 student-tests/level4_failures.py all` - сценарий `commit_publish_gap` | `PASS commit_publish_gap`; события: `accepted,claimed,ready`. | `evidence/evidence.txt` |
|
||||||
|
| Crash после записи в S3 до фиксации результата/ack; восстановление по lease | `python3 student-tests/level4_failures.py all` - сценарий `crash_after_s3` | `PASS crash_after_s3`; события: `accepted,claimed,lease_expired,claimed,ready`. Повторный claim после истечения lease завершился `ready`. | `evidence/evidence.txt` |
|
||||||
|
| Retry при временной недоступности S3 | `python3 student-tests/level4_failures.py all` - сценарий `s3_outage` | `PASS s3_outage`; события: `accepted,claimed,retry_scheduled,claimed,retry_scheduled,claimed,retry_scheduled,claimed,ready`. Зафиксированы три планирования повторной попытки, после чего задача завершилась успешно. | `evidence/evidence.txt` |
|
||||||
|
| Реестр failed и операторский replay | `python3 student-tests/level4_failures.py all` - сценарий `replay` | Команда вывела `REPLAYED job=... attachment=...`; сценарий завершился `PASS replay`. Последовательность событий: `accepted,claimed,failed,replay,claimed,ready`. | `evidence/evidence.txt` |
|
||||||
|
| Сохранность данных после перезапуска хранилищ и сохранение существующей сессии | `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | `PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.` Тест подтвердил сохранность проверенных данных и существующей сессии | `evidence/evidence.txt` |
|
||||||
|
| Работа при остановке одной API-реплики | `make test-failover ACTION_SCRIPT=scripts/stop-one.sh` | Перед воздействием наблюдались 2 экземпляра API; выполнено 88 успешных записей, 2 неуспешные попытки; первое восстановление - 3.153 с. Итог: `PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.` | `evidence/evidence.txt` |
|
||||||
|
| Отдельные broker и worker, durable-сообщения, at-least-once, ack только после устойчивого эффекта | Реализованы отдельные сервисы Kafka broker и worker в compose.yaml; Kafka использует persistent volume kafka_data. В src/dispatcher.py producer настроен с acks=all и idempotence, outbox помечается опубликованным только после подтверждения доставки. В src/worker.py автоматический commit offset отключён: сообщение подтверждается только после записи вариантов в S3 и фиксации состояния ready в PostgreSQL. | worker_down, commit_publish_gap, crash_after_s3 - PASS. В crash_after_s3 после lease_expired задача была повторно получена и завершилась состоянием ready. | Реализация подтверждается compose.yaml, src/dispatcher.py, src/worker.py; порядок подтверждения дополнительно проверен fault-injection сценариями. Kafka работает с RF=1: persistent volume обеспечивает сохранение данных при перезапуске, но репликационной защиты от потери единственного брокера нет. |
|
||||||
|
| Crop 128×128, optimized WebP и ограничения обработки по bytes/pixels/time/memory | В src/imgproc.py реализованы центрированный crop 128×128 JPEG q85 и optimized WebP q80 с ограничением размера 1280×1280; метаданные исходного изображения не переносятся. В src/main.py ограничен размер входного файла (10 MiB): проверяется Content-Length и фактический объём потокового тела; количество пикселей ограничено 40 000 000 до декодирования. В src/worker.py обработка выполняется с ограничением времени 60 секунд. В src/imgproc.py используются RLIMIT_CPU и RLIMIT_AS=1024 MiB; в compose.yaml для worker задан mem_limit: 1g. | Функция check_variants() в student-tests/level4_failures.py проверяет наличие crop 128×128 и WebP (image/webp). Она используется в сценариях worker_down, commit_publish_gap, crash_after_s3, replay, s3_outage; все пять сценариев завершились PASS. | Все преобразования и ограничения обработки реализованы в исходном коде и конфигурации; размеры crop и формат WebP дополнительно проверяются автоматическими сценариями. |
|
||||||
|
| Lease для зависшего processing, ограниченные concurrency/prefetch и очистка staging | В src/jobs.py задан срок lease на основе JOB_TIMEOUT_SECONDS и LEASE_GRACE_SECONDS (60+15 секунд). В src/dispatcher.py просроченные задачи processing возвращаются в queued с повторной публикацией через outbox. В src/worker.py ограничены параллелизм (WORKER_CONCURRENCY=2) и количество получаемых сообщений (consume(num_messages=CONCURRENCY)); также заданы ограничения Kafka fetch/prefetch. После фиксации ready worker удаляет staging-файл. scripts/s3_sweep.py удаляет осиротевшие staging-объекты после проверки ссылок в PostgreSQL; grace period - 3600 секунд, период запуска - 600 секунд, срок хранения staging для failed - 86400 секунд. | crash_after_s3 - PASS: после lease_expired задача повторно получена (claimed) и завершилась ready. | Восстановление по lease подтверждено fault-injection тестом; ограничения concurrency/prefetch и механизм очистки staging подтверждаются реализацией в src/worker.py, src/jobs.py, src/dispatcher.py, scripts/s3_sweep.py. |
|
||||||
|
|
||||||
|
## Эксперименты и отказы
|
||||||
|
|
||||||
|
Среда проведения экспериментов:
|
||||||
|
- ОС: Microsoft Windows 11 Pro 10.0.22631 (build 22631).
|
||||||
|
- CPU: Intel Core i5-8350U, 4 ядра / 8 потоков.
|
||||||
|
- RAM: 15.9 GB.
|
||||||
|
- Docker Compose: v2.32.4-desktop.1.
|
||||||
|
- Docker Client/Engine: 27.5.1; Docker Desktop 4.38.0; контекст `desktop-linux`.
|
||||||
|
|
||||||
|
Перед функциональными сценариями выполнены `make check`, `make validate` и `docker compose --env-file .env config --quiet`. Первые две команды завершились с `PASS`; в их выводе отдельно указано, что они не тестируют работающее приложение. Compose config завершился без сообщения об ошибке. После запуска стенда readiness endpoint вернул `{"status":"ok"}`. Публичный smoke (`make test`) завершился с `PASS`.
|
||||||
|
|
||||||
|
### Сценарии отказов уровня 4
|
||||||
|
|
||||||
|
Команда `python3 student-tests/level4_failures.py all` завершилась успешно для всех пяти сценариев:
|
||||||
|
|
||||||
|
| Воздействие | Наблюдаемые события / результат | Вывод |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Остановка worker, приём задания и последующий запуск worker (`worker_down`) | `accepted,claimed,ready`; `PASS worker_down` | Задание обработано после восстановления worker. |
|
||||||
|
| Разрыв между фиксацией задания в PostgreSQL и публикацией (`commit_publish_gap`) | `accepted,claimed,ready`; `PASS commit_publish_gap` | Задание в итоге доставлено и обработано без ручного вмешательства. |
|
||||||
|
| Аварийное завершение после записи в S3 (`crash_after_s3`) | `accepted,claimed,lease_expired,claimed,ready`; `PASS crash_after_s3` | После истечения lease задача была повторно захвачена и завершена. |
|
||||||
|
| Повторный запуск задания из failed-реестра (`replay`) | `accepted,claimed,failed,replay,claimed,ready`; команда вывела `REPLAYED`; `PASS replay` | Failed-задание прошло операторский replay и завершилось успешно. |
|
||||||
|
| Недоступность S3 во время обработки (`s3_outage`) | `accepted,claimed,retry_scheduled,claimed,retry_scheduled,claimed,retry_scheduled,claimed,ready`; `PASS s3_outage` | Зафиксированы три запланированные повторные попытки; после восстановления S3 задача завершилась успешно. |
|
||||||
|
|
||||||
|
### Сохранность данных и отказ API-реплики
|
||||||
|
|
||||||
|
| Воздействие | Наблюдаемый результат | Время восстановления / потеря данных |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Перезапуск хранилищ: `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | `PASS: persistence preserved confirmed data and existing session.` | Проверенные подтверждённые данные и существующая сессия сохранены. |
|
||||||
|
| Остановка одной API-реплики: `make test-failover ACTION_SCRIPT=scripts/stop-one.sh` | До воздействия наблюдались 2 экземпляра; 88 успешных записей, 2 неуспешные попытки; `PASS: failover preserved confirmed data and existing session.` | Первое восстановление - 3.153 с; тест сообщил о сохранении подтверждённых данных и существующей сессии. |
|
||||||
|
|
||||||
|
## Ограничения и следующие шаги
|
||||||
|
|
||||||
|
Один брокер Kafka (RF=1): при потере диска kafka_data сообщения могут быть потеряны; автоматического восстановления потерянного брокерного лога нет. PostgreSQL остаётся источником истины для состояния job, но восстановление доставки после потери самого лога не гарантируется. Лимит памяти - RLIMIT_AS + mem_limit, не cgroup-метрика RSS по job. Валидация изображения (декодирование) при upload выполняется в API (перенос lab4), resize - только в worker. Replay доступен оператору через CLI, не через публичный API (контракт не менялся). Трек 5 не реализован; нагрузочные измерения pipeline не проводились.
|
||||||
+314
@@ -0,0 +1,314 @@
|
|||||||
|
name: mtusi-lab5
|
||||||
|
services:
|
||||||
|
postgres:
|
||||||
|
image: postgres:17.6-alpine
|
||||||
|
environment:
|
||||||
|
POSTGRES_DB: ${POSTGRES_DB:-messenger}
|
||||||
|
POSTGRES_USER: ${POSTGRES_USER:-messenger}
|
||||||
|
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}
|
||||||
|
volumes:
|
||||||
|
- postgres_data:/var/lib/postgresql/data
|
||||||
|
networks: [backend]
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
|
||||||
|
interval: 5s
|
||||||
|
timeout: 3s
|
||||||
|
retries: 20
|
||||||
|
cpus: 1.0
|
||||||
|
mem_limit: 768m
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
redis:
|
||||||
|
image: redis:7.4.2-alpine
|
||||||
|
command: ["redis-server", "--appendonly", "yes", "--appendfsync", "everysec"]
|
||||||
|
volumes:
|
||||||
|
- redis_data:/data
|
||||||
|
networks: [backend]
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "redis-cli", "ping"]
|
||||||
|
interval: 5s
|
||||||
|
timeout: 3s
|
||||||
|
retries: 20
|
||||||
|
cpus: 0.25
|
||||||
|
mem_limit: 256m
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
s3:
|
||||||
|
image: pgsty/silo:RELEASE.2026-10-15T00-00-00Z
|
||||||
|
command: ["server", "/data", "--address", ":9000"]
|
||||||
|
environment:
|
||||||
|
MINIO_ROOT_USER: ${MINIO_ROOT_USER:?Set MINIO_ROOT_USER in .env}
|
||||||
|
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:?Set MINIO_ROOT_PASSWORD in .env}
|
||||||
|
MINIO_BROWSER: "off" # admin web console disabled; no ports are published at all
|
||||||
|
volumes:
|
||||||
|
- s3_data:/data
|
||||||
|
networks: [backend]
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "mc", "ready", "local"]
|
||||||
|
interval: 5s
|
||||||
|
timeout: 5s
|
||||||
|
retries: 20
|
||||||
|
cpus: 1.0
|
||||||
|
mem_limit: 512m
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
s3_init:
|
||||||
|
image: pgsty/silo:RELEASE.2026-10-15T00-00-00Z
|
||||||
|
entrypoint: ["/bin/sh", "/infra/s3-init.sh"]
|
||||||
|
environment:
|
||||||
|
S3_ENDPOINT: http://s3:9000
|
||||||
|
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
|
||||||
|
MINIO_ROOT_USER: ${MINIO_ROOT_USER:?Set MINIO_ROOT_USER in .env}
|
||||||
|
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:?Set MINIO_ROOT_PASSWORD in .env}
|
||||||
|
S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID:?Set S3_ACCESS_KEY_ID in .env}
|
||||||
|
S3_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY:?Set S3_SECRET_ACCESS_KEY in .env}
|
||||||
|
SWEEPER_S3_ACCESS_KEY_ID: ${SWEEPER_S3_ACCESS_KEY_ID:?Set SWEEPER_S3_ACCESS_KEY_ID in .env}
|
||||||
|
SWEEPER_S3_SECRET_ACCESS_KEY: ${SWEEPER_S3_SECRET_ACCESS_KEY:?Set SWEEPER_S3_SECRET_ACCESS_KEY in .env}
|
||||||
|
volumes:
|
||||||
|
- ./infra/s3-init.sh:/infra/s3-init.sh:ro
|
||||||
|
networks: [backend]
|
||||||
|
depends_on:
|
||||||
|
s3:
|
||||||
|
condition: service_started
|
||||||
|
restart: "no"
|
||||||
|
|
||||||
|
kafka:
|
||||||
|
image: apache/kafka:3.9.0
|
||||||
|
environment:
|
||||||
|
KAFKA_NODE_ID: "1"
|
||||||
|
KAFKA_PROCESS_ROLES: broker,controller
|
||||||
|
KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
|
||||||
|
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
|
||||||
|
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
|
||||||
|
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
|
||||||
|
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
|
||||||
|
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
|
||||||
|
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: "1"
|
||||||
|
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: "1"
|
||||||
|
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: "1"
|
||||||
|
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: "0"
|
||||||
|
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
|
||||||
|
KAFKA_LOG_DIRS: /var/lib/kafka/data
|
||||||
|
KAFKA_HEAP_OPTS: "-Xms256m -Xmx512m"
|
||||||
|
volumes:
|
||||||
|
- kafka_data:/var/lib/kafka/data
|
||||||
|
networks: [backend]
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD-SHELL", "/opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 >/dev/null 2>&1"]
|
||||||
|
interval: 5s
|
||||||
|
timeout: 10s
|
||||||
|
retries: 30
|
||||||
|
start_period: 15s
|
||||||
|
cpus: 1.0
|
||||||
|
mem_limit: 1g
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
kafka_init:
|
||||||
|
image: apache/kafka:3.9.0
|
||||||
|
entrypoint: ["/bin/sh", "-c"]
|
||||||
|
command:
|
||||||
|
- |
|
||||||
|
set -e
|
||||||
|
T=/opt/kafka/bin/kafka-topics.sh
|
||||||
|
$$T --bootstrap-server kafka:9092 --create --if-not-exists --topic $${JOBS_TOPIC} --partitions 3 --replication-factor 1 --config min.insync.replicas=1 --config retention.ms=604800000
|
||||||
|
$$T --bootstrap-server kafka:9092 --list
|
||||||
|
environment:
|
||||||
|
JOBS_TOPIC: ${JOBS_TOPIC:-attachment-jobs}
|
||||||
|
networks: [backend]
|
||||||
|
depends_on:
|
||||||
|
kafka:
|
||||||
|
condition: service_healthy
|
||||||
|
restart: "no"
|
||||||
|
|
||||||
|
migrate:
|
||||||
|
build: .
|
||||||
|
image: mtusi-lab5-api:1.0.0
|
||||||
|
command: ["python", "scripts/migrate.py"]
|
||||||
|
environment:
|
||||||
|
PGHOST: postgres
|
||||||
|
PGPORT: "5432"
|
||||||
|
PGDATABASE: ${POSTGRES_DB:-messenger}
|
||||||
|
PGUSER: ${POSTGRES_USER:-messenger}
|
||||||
|
PGPASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}
|
||||||
|
networks: [backend]
|
||||||
|
depends_on:
|
||||||
|
postgres:
|
||||||
|
condition: service_healthy
|
||||||
|
restart: "no"
|
||||||
|
cpus: 0.5
|
||||||
|
mem_limit: 384m
|
||||||
|
|
||||||
|
api:
|
||||||
|
build: .
|
||||||
|
image: mtusi-lab5-api:1.0.0
|
||||||
|
environment:
|
||||||
|
APP_HOST: 0.0.0.0
|
||||||
|
APP_PORT: "8080"
|
||||||
|
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
|
||||||
|
DB_POOL_MIN_SIZE: ${DB_POOL_MIN_SIZE:-1}
|
||||||
|
DB_POOL_MAX_SIZE: ${DB_POOL_MAX_SIZE:-10}
|
||||||
|
DB_POOL_TIMEOUT_SECONDS: ${DB_POOL_TIMEOUT_SECONDS:-5}
|
||||||
|
PGHOST: postgres
|
||||||
|
PGPORT: "5432"
|
||||||
|
PGDATABASE: ${POSTGRES_DB:-messenger}
|
||||||
|
PGUSER: ${POSTGRES_USER:-messenger}
|
||||||
|
PGPASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}
|
||||||
|
REDIS_URL: redis://redis:6379/0
|
||||||
|
S3_ENDPOINT: http://s3:9000
|
||||||
|
S3_REGION: ${S3_REGION:-us-east-1}
|
||||||
|
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
|
||||||
|
S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID:?Set S3_ACCESS_KEY_ID in .env}
|
||||||
|
S3_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY:?Set S3_SECRET_ACCESS_KEY in .env}
|
||||||
|
MAX_UPLOAD_BYTES: ${MAX_UPLOAD_BYTES:-10485760}
|
||||||
|
MAX_IMAGE_PIXELS: ${MAX_IMAGE_PIXELS:-40000000}
|
||||||
|
IMAGE_DECODE_CONCURRENCY: ${IMAGE_DECODE_CONCURRENCY:-1}
|
||||||
|
JOB_MAX_ATTEMPTS: ${JOB_MAX_ATTEMPTS:-5}
|
||||||
|
deploy:
|
||||||
|
replicas: 2
|
||||||
|
expose: ["8080"]
|
||||||
|
networks: [backend]
|
||||||
|
depends_on:
|
||||||
|
migrate:
|
||||||
|
condition: service_completed_successfully
|
||||||
|
redis:
|
||||||
|
condition: service_healthy
|
||||||
|
s3_init:
|
||||||
|
condition: service_completed_successfully
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=2)"]
|
||||||
|
interval: 3s
|
||||||
|
timeout: 3s
|
||||||
|
retries: 3
|
||||||
|
start_period: 5s
|
||||||
|
stop_grace_period: 15s
|
||||||
|
cpus: 1.0
|
||||||
|
mem_limit: 512m
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
worker:
|
||||||
|
build: .
|
||||||
|
image: mtusi-lab5-api:1.0.0
|
||||||
|
command: ["python", "-m", "src.worker"]
|
||||||
|
environment:
|
||||||
|
PGHOST: postgres
|
||||||
|
PGPORT: "5432"
|
||||||
|
PGDATABASE: ${POSTGRES_DB:-messenger}
|
||||||
|
PGUSER: ${POSTGRES_USER:-messenger}
|
||||||
|
PGPASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}
|
||||||
|
S3_ENDPOINT: http://s3:9000
|
||||||
|
S3_REGION: ${S3_REGION:-us-east-1}
|
||||||
|
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
|
||||||
|
S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID:?Set S3_ACCESS_KEY_ID in .env}
|
||||||
|
S3_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY:?Set S3_SECRET_ACCESS_KEY in .env}
|
||||||
|
KAFKA_BOOTSTRAP_SERVERS: kafka:9092
|
||||||
|
JOBS_TOPIC: ${JOBS_TOPIC:-attachment-jobs}
|
||||||
|
JOB_MAX_ATTEMPTS: ${JOB_MAX_ATTEMPTS:-5}
|
||||||
|
JOB_TIMEOUT_SECONDS: ${JOB_TIMEOUT_SECONDS:-60}
|
||||||
|
LEASE_GRACE_SECONDS: ${LEASE_GRACE_SECONDS:-15}
|
||||||
|
JOB_BACKOFF_BASE_SECONDS: ${JOB_BACKOFF_BASE_SECONDS:-2}
|
||||||
|
JOB_BACKOFF_CAP_SECONDS: ${JOB_BACKOFF_CAP_SECONDS:-60}
|
||||||
|
WORKER_CONCURRENCY: ${WORKER_CONCURRENCY:-2}
|
||||||
|
IMG_MEMORY_MB: ${IMG_MEMORY_MB:-1024}
|
||||||
|
MAX_IMAGE_PIXELS: ${MAX_IMAGE_PIXELS:-40000000}
|
||||||
|
FAULT_WORKER_EXIT_AFTER_S3: ${FAULT_WORKER_EXIT_AFTER_S3:-0}
|
||||||
|
networks: [backend]
|
||||||
|
depends_on:
|
||||||
|
migrate:
|
||||||
|
condition: service_completed_successfully
|
||||||
|
kafka_init:
|
||||||
|
condition: service_completed_successfully
|
||||||
|
s3_init:
|
||||||
|
condition: service_completed_successfully
|
||||||
|
stop_grace_period: 30s
|
||||||
|
cpus: 1.0
|
||||||
|
mem_limit: 1g
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
dispatcher:
|
||||||
|
build: .
|
||||||
|
image: mtusi-lab5-api:1.0.0
|
||||||
|
command: ["python", "-m", "src.dispatcher"]
|
||||||
|
environment:
|
||||||
|
PGHOST: postgres
|
||||||
|
PGPORT: "5432"
|
||||||
|
PGDATABASE: ${POSTGRES_DB:-messenger}
|
||||||
|
PGUSER: ${POSTGRES_USER:-messenger}
|
||||||
|
PGPASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}
|
||||||
|
KAFKA_BOOTSTRAP_SERVERS: kafka:9092
|
||||||
|
JOBS_TOPIC: ${JOBS_TOPIC:-attachment-jobs}
|
||||||
|
JOB_MAX_ATTEMPTS: ${JOB_MAX_ATTEMPTS:-5}
|
||||||
|
JOB_TIMEOUT_SECONDS: ${JOB_TIMEOUT_SECONDS:-60}
|
||||||
|
LEASE_GRACE_SECONDS: ${LEASE_GRACE_SECONDS:-15}
|
||||||
|
JOB_BACKOFF_BASE_SECONDS: ${JOB_BACKOFF_BASE_SECONDS:-2}
|
||||||
|
JOB_BACKOFF_CAP_SECONDS: ${JOB_BACKOFF_CAP_SECONDS:-60}
|
||||||
|
FAULT_DISPATCHER_EXIT_BEFORE_PUBLISH: ${FAULT_DISPATCHER_EXIT_BEFORE_PUBLISH:-0}
|
||||||
|
networks: [backend]
|
||||||
|
depends_on:
|
||||||
|
migrate:
|
||||||
|
condition: service_completed_successfully
|
||||||
|
kafka_init:
|
||||||
|
condition: service_completed_successfully
|
||||||
|
s3_init:
|
||||||
|
condition: service_completed_successfully
|
||||||
|
cpus: 0.25
|
||||||
|
mem_limit: 192m
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
s3_sweeper:
|
||||||
|
build: .
|
||||||
|
image: mtusi-lab5-api:1.0.0
|
||||||
|
command: ["python", "scripts/s3_sweep.py", "--loop", "--apply"]
|
||||||
|
environment:
|
||||||
|
PGHOST: postgres
|
||||||
|
PGPORT: "5432"
|
||||||
|
PGDATABASE: ${POSTGRES_DB:-messenger}
|
||||||
|
PGUSER: ${POSTGRES_USER:-messenger}
|
||||||
|
PGPASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}
|
||||||
|
S3_ENDPOINT: http://s3:9000
|
||||||
|
S3_REGION: ${S3_REGION:-us-east-1}
|
||||||
|
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
|
||||||
|
S3_ACCESS_KEY_ID: ${SWEEPER_S3_ACCESS_KEY_ID:?Set SWEEPER_S3_ACCESS_KEY_ID in .env}
|
||||||
|
S3_SECRET_ACCESS_KEY: ${SWEEPER_S3_SECRET_ACCESS_KEY:?Set SWEEPER_S3_SECRET_ACCESS_KEY in .env}
|
||||||
|
SWEEP_GRACE_SECONDS: ${SWEEP_GRACE_SECONDS:-3600}
|
||||||
|
SWEEP_INTERVAL_SECONDS: ${SWEEP_INTERVAL_SECONDS:-600}
|
||||||
|
STAGING_RETENTION_SECONDS: ${STAGING_RETENTION_SECONDS:-86400}
|
||||||
|
networks: [backend]
|
||||||
|
depends_on:
|
||||||
|
migrate:
|
||||||
|
condition: service_completed_successfully
|
||||||
|
s3_init:
|
||||||
|
condition: service_completed_successfully
|
||||||
|
cpus: 0.25
|
||||||
|
mem_limit: 128m
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
load_balancer:
|
||||||
|
image: haproxy:3.2.5-alpine
|
||||||
|
ports:
|
||||||
|
- "127.0.0.1:${APP_PORT:-8080}:8080"
|
||||||
|
volumes:
|
||||||
|
- ./infra/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
|
||||||
|
networks: [frontend, backend]
|
||||||
|
depends_on:
|
||||||
|
api:
|
||||||
|
condition: service_healthy
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "haproxy", "-c", "-f", "/usr/local/etc/haproxy/haproxy.cfg"]
|
||||||
|
interval: 10s
|
||||||
|
timeout: 3s
|
||||||
|
retries: 3
|
||||||
|
cpus: 0.5
|
||||||
|
mem_limit: 128m
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
networks:
|
||||||
|
frontend:
|
||||||
|
driver: bridge
|
||||||
|
backend:
|
||||||
|
driver: bridge
|
||||||
|
internal: true # no route to/from the host or internet: S3, PostgreSQL, Redis, Kafka and API are reachable only inside
|
||||||
|
volumes:
|
||||||
|
postgres_data:
|
||||||
|
redis_data:
|
||||||
|
s3_data:
|
||||||
|
kafka_data:
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
thereal_alyoname@DESKTOP-8Q26HPT:/mnt/c/Users/User/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork$
|
||||||
|
make check
|
||||||
|
python3 scripts/check_contract.py
|
||||||
|
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||||
|
|
||||||
|
|
||||||
|
thereal_alyoname@DESKTOP-8Q26HPT:/mnt/c/Users/User/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork$
|
||||||
|
make validate
|
||||||
|
python3 scripts/validate_openapi.py
|
||||||
|
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
|
||||||
|
thereal_alyoname@DESKTOP-8Q26HPT:/mnt/c/Users/User/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork$
|
||||||
|
|
||||||
|
|
||||||
|
thereal_alyoname@DESKTOP-8Q26HPT:/mnt/c/Users/User/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork$
|
||||||
|
docker compose --env-file .env config --quiet
|
||||||
|
thereal_alyoname@DESKTOP-8Q26HPT:/mnt/c/Users/User/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork$
|
||||||
|
|
||||||
|
|
||||||
|
User@DESKTOP-8Q26HPT MINGW64 ~/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork (solution/bvt2302-mironov)
|
||||||
|
$ docker compose --env-file .env up --build
|
||||||
|
[+] Building 21.1s (8/12) docker:desktop-linux
|
||||||
|
=> => transferring context: 208.14kB 0.0s
|
||||||
|
=> CACHED [migrate 2/8] WORKDIR /app 0.0s
|
||||||
|
=> CACHED [migrate 3/8] RUN addgroup --system app && adduser --system --ingroup app app 0.0s
|
||||||
|
=> [migrate 4/8] COPY requirements.txt ./requirements.txt 0.0s
|
||||||
|
=> [migrate 5/8] RUN python -m pip install --no-cache-dir -r requirements.txt
|
||||||
|
|
||||||
|
(проверка доступности приложения)
|
||||||
|
thereal_alyoname@DESKTOP-8Q26HPT:/mnt/c/Users/User/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork$
|
||||||
|
until curl -fs http://localhost:8080/health/ready; do sleep 2; done
|
||||||
|
{"status":"ok"}thereal_alyoname@DESKTOP-8Q26HPT:/mnt/c/Users/User/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork$
|
||||||
|
|
||||||
|
|
||||||
|
thereal_alyoname@DESKTOP-8Q26HPT:/mnt/c/Users/User/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork$
|
||||||
|
make test
|
||||||
|
python3 tests/smoke.py --base-url "http://localhost:8080"
|
||||||
|
PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.
|
||||||
|
thereal_alyoname@DESKTOP-8Q26HPT:/mnt/c/Users/User/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork$
|
||||||
|
|
||||||
|
(проверки на уровень 4)
|
||||||
|
thereal_alyoname@DESKTOP-8Q26HPT:/mnt/c/Users/User/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork$
|
||||||
|
python3 student-tests/level4_failures.py all
|
||||||
|
|
||||||
|
PASS worker_down: e05d5f76-4b0f-4b33-a81e-e3f6ab4bc72f events: accepted,claimed,ready
|
||||||
|
|
||||||
|
PASS commit_publish_gap: df3506d1-b835-49c7-8b0c-bf150273d750 events: accepted,claimed,ready
|
||||||
|
|
||||||
|
PASS crash_after_s3: 9f2b8d69-04f2-49f6-bd6c-dca100a9fe87 events: accepted,claimed,lease_expired,claimed,ready
|
||||||
|
|
||||||
|
REPLAYED job=5ea63d39-7ba8-4f9b-a9f9-506cc7e69034 attachment=b5a4228a-654f-4212-9b4d-9bd429d85020
|
||||||
|
PASS replay: b5a4228a-654f-4212-9b4d-9bd429d85020 events: accepted,claimed,failed,replay,claimed,ready
|
||||||
|
|
||||||
|
PASS s3_outage: e4ec52dc-cc63-4c2e-babe-d00b1e70de16 events: accepted,claimed,retry_scheduled,claimed,retry_scheduled,claimed,retry_scheduled,claimed,ready
|
||||||
|
thereal_alyoname@DESKTOP-8Q26HPT:/mnt/c/Users/User/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork$
|
||||||
|
|
||||||
|
(тест на сохранность данных сервисов)
|
||||||
|
thereal_alyoname@DESKTOP-8Q26HPT:/mnt/c/Users/User/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork$
|
||||||
|
make test-persistence ACTION_SCRIPT=scripts/restart.sh
|
||||||
|
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:8080"
|
||||||
|
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||||
|
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||||
|
|
||||||
|
(тест на работоспособность приложения во время того, как опущена одна реплика апи)
|
||||||
|
thereal_alyoname@DESKTOP-8Q26HPT:/mnt/c/Users/User/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork$
|
||||||
|
make test-failover ACTION_SCRIPT=scripts/stop-one.sh
|
||||||
|
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://localhost:8080"
|
||||||
|
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||||
|
{"observed_instances_before": 2, "successful_writes": 88, "failed_attempts": 2, "first_recovery_seconds": 3.153}
|
||||||
|
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||||
|
thereal_alyoname@DESKTOP-8Q26HPT:/mnt/c/Users/User/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork$
|
||||||
|
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
global
|
||||||
|
log stdout format raw local0
|
||||||
|
maxconn 1024
|
||||||
|
|
||||||
|
defaults
|
||||||
|
log global
|
||||||
|
mode http
|
||||||
|
option httplog
|
||||||
|
option http-keep-alive
|
||||||
|
timeout connect 2s
|
||||||
|
timeout client 30s
|
||||||
|
timeout server 30s
|
||||||
|
timeout check 2s
|
||||||
|
|
||||||
|
resolvers docker
|
||||||
|
nameserver docker-dns 127.0.0.11:53
|
||||||
|
resolve_retries 3
|
||||||
|
timeout resolve 1s
|
||||||
|
timeout retry 1s
|
||||||
|
hold valid 2s
|
||||||
|
hold obsolete 2s
|
||||||
|
|
||||||
|
frontend public_http
|
||||||
|
bind :8080
|
||||||
|
default_backend api_replicas
|
||||||
|
|
||||||
|
backend api_replicas
|
||||||
|
balance roundrobin
|
||||||
|
option httpchk GET /health/ready
|
||||||
|
http-check expect status 200
|
||||||
|
default-server inter 1s fall 2 rise 1
|
||||||
|
server-template api 10 api:8080 check resolvers docker init-addr libc,none
|
||||||
@@ -0,0 +1,56 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# Idempotent bootstrap of the private bucket, the restricted application user
|
||||||
|
# and the separate sweeper user (safe to re-run).
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
: "${S3_ENDPOINT:?}" "${S3_BUCKET:?}" "${MINIO_ROOT_USER:?}" "${MINIO_ROOT_PASSWORD:?}"
|
||||||
|
: "${S3_ACCESS_KEY_ID:?}" "${S3_SECRET_ACCESS_KEY:?}"
|
||||||
|
: "${SWEEPER_S3_ACCESS_KEY_ID:?}" "${SWEEPER_S3_SECRET_ACCESS_KEY:?}"
|
||||||
|
|
||||||
|
[ "$S3_ACCESS_KEY_ID" != "$MINIO_ROOT_USER" ] || { echo "app key must differ from root user" >&2; exit 1; }
|
||||||
|
[ "$SWEEPER_S3_ACCESS_KEY_ID" != "$MINIO_ROOT_USER" ] || { echo "sweeper key must differ from root user" >&2; exit 1; }
|
||||||
|
[ "$S3_ACCESS_KEY_ID" != "$SWEEPER_S3_ACCESS_KEY_ID" ] || { echo "app and sweeper keys must differ" >&2; exit 1; }
|
||||||
|
|
||||||
|
i=0
|
||||||
|
until mc alias set local "$S3_ENDPOINT" "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" >/dev/null 2>&1; do
|
||||||
|
i=$((i+1)); [ "$i" -lt 60 ] || { echo "S3 not reachable" >&2; exit 1; }; sleep 1
|
||||||
|
done
|
||||||
|
|
||||||
|
mc mb --ignore-existing "local/$S3_BUCKET"
|
||||||
|
mc anonymous set none "local/$S3_BUCKET"
|
||||||
|
printf ok | mc pipe "local/$S3_BUCKET/attachments/.ready"
|
||||||
|
|
||||||
|
|
||||||
|
# --- app policy: exact keys only, NO ListBucket ---
|
||||||
|
cat > /tmp/policy-app.json <<POLICY
|
||||||
|
{"Version":"2012-10-17","Statement":[
|
||||||
|
{"Effect":"Allow","Action":["s3:PutObject","s3:GetObject","s3:DeleteObject"],"Resource":["arn:aws:s3:::$S3_BUCKET/attachments/*","arn:aws:s3:::$S3_BUCKET/staging/*"]}]}
|
||||||
|
POLICY
|
||||||
|
|
||||||
|
# --- sweeper policy: ListBucket for enumeration + Get/Delete, NO PutObject ---
|
||||||
|
cat > /tmp/policy-sweeper.json <<POLICY
|
||||||
|
{"Version":"2012-10-17","Statement":[
|
||||||
|
{"Effect":"Allow","Action":["s3:ListBucket","s3:GetBucketLocation"],"Resource":["arn:aws:s3:::$S3_BUCKET"]},
|
||||||
|
{"Effect":"Allow","Action":["s3:GetObject","s3:DeleteObject"],"Resource":["arn:aws:s3:::$S3_BUCKET/attachments/*","arn:aws:s3:::$S3_BUCKET/staging/*"]}]}
|
||||||
|
POLICY
|
||||||
|
|
||||||
|
# MinIO's `policy create` does not overwrite an existing policy, so remove first.
|
||||||
|
mc admin policy remove local messenger-app --force >/dev/null 2>&1 || true
|
||||||
|
mc admin policy remove local messenger-sweeper --force >/dev/null 2>&1 || true
|
||||||
|
mc admin policy create local messenger-app /tmp/policy-app.json
|
||||||
|
mc admin policy create local messenger-sweeper /tmp/policy-sweeper.json
|
||||||
|
|
||||||
|
# `user add` is idempotent (re-running updates the secret).
|
||||||
|
mc admin user add local "$S3_ACCESS_KEY_ID" "$S3_SECRET_ACCESS_KEY"
|
||||||
|
mc admin user add local "$SWEEPER_S3_ACCESS_KEY_ID" "$SWEEPER_S3_SECRET_ACCESS_KEY"
|
||||||
|
|
||||||
|
out=$(mc admin policy attach local messenger-app --user "$S3_ACCESS_KEY_ID" 2>&1) || {
|
||||||
|
echo "$out" | grep -qi "already" || { echo "$out" >&2; exit 1; }
|
||||||
|
}
|
||||||
|
out=$(mc admin policy attach local messenger-sweeper --user "$SWEEPER_S3_ACCESS_KEY_ID" 2>&1) || {
|
||||||
|
echo "$out" | grep -qi "already" || { echo "$out" >&2; exit 1; }
|
||||||
|
}
|
||||||
|
|
||||||
|
echo "PASS: bucket $S3_BUCKET private"
|
||||||
|
echo " app=$S3_ACCESS_KEY_ID (Put/Get/Delete on attachments/* and staging/*, no ListBucket)"
|
||||||
|
echo " sweeper=$SWEEPER_S3_ACCESS_KEY_ID (ListBucket + Get/Delete, no PutObject)"
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
CREATE TABLE IF NOT EXISTS users (
|
||||||
|
id UUID PRIMARY KEY,
|
||||||
|
username VARCHAR(32) NOT NULL UNIQUE,
|
||||||
|
display_name VARCHAR(100) NOT NULL,
|
||||||
|
password_hash TEXT NOT NULL,
|
||||||
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||||
|
CONSTRAINT users_username_format CHECK (username ~ '^[a-z0-9_]{3,32}$'),
|
||||||
|
CONSTRAINT users_display_name_nonblank CHECK (length(btrim(display_name)) > 0)
|
||||||
|
);
|
||||||
|
CREATE TABLE IF NOT EXISTS chats (
|
||||||
|
id UUID PRIMARY KEY,
|
||||||
|
title VARCHAR(100) NOT NULL,
|
||||||
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||||
|
CONSTRAINT chats_title_nonblank CHECK (length(btrim(title)) > 0)
|
||||||
|
);
|
||||||
|
CREATE TABLE IF NOT EXISTS chat_members (
|
||||||
|
chat_id UUID NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||||
|
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||||
|
position INTEGER NOT NULL DEFAULT 0,
|
||||||
|
PRIMARY KEY (chat_id, user_id)
|
||||||
|
);
|
||||||
|
ALTER TABLE chat_members ADD COLUMN IF NOT EXISTS position INTEGER NOT NULL DEFAULT 0;
|
||||||
|
CREATE INDEX IF NOT EXISTS chat_members_user_chat_idx ON chat_members(user_id, chat_id);
|
||||||
|
CREATE TABLE IF NOT EXISTS messages (
|
||||||
|
id UUID PRIMARY KEY,
|
||||||
|
chat_id UUID NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||||
|
sender_id UUID NOT NULL REFERENCES users(id),
|
||||||
|
text VARCHAR(4000) NOT NULL,
|
||||||
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||||
|
CONSTRAINT messages_text_nonblank CHECK (length(btrim(text)) > 0)
|
||||||
|
);
|
||||||
|
CREATE INDEX IF NOT EXISTS messages_chat_created_id_idx ON messages(chat_id, created_at, id);
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
CREATE TABLE IF NOT EXISTS idempotency_records (
|
||||||
|
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||||
|
chat_id UUID NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||||
|
idempotency_key VARCHAR(128) NOT NULL,
|
||||||
|
request_hash CHAR(64) NOT NULL,
|
||||||
|
response_body JSONB NOT NULL,
|
||||||
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||||
|
PRIMARY KEY (user_id, chat_id, idempotency_key),
|
||||||
|
CONSTRAINT idempotency_key_format CHECK (idempotency_key ~ '^[A-Za-z0-9._:-]{1,128}$')
|
||||||
|
);
|
||||||
|
CREATE INDEX IF NOT EXISTS idempotency_records_created_at_idx ON idempotency_records(created_at);
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
CREATE TABLE IF NOT EXISTS attachments (
|
||||||
|
id UUID PRIMARY KEY,
|
||||||
|
chat_id UUID NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||||
|
owner_id UUID NOT NULL REFERENCES users(id),
|
||||||
|
filename VARCHAR(255) NOT NULL,
|
||||||
|
content_type TEXT NOT NULL,
|
||||||
|
size_bytes BIGINT NOT NULL,
|
||||||
|
sha256 CHAR(64) NOT NULL,
|
||||||
|
s3_key TEXT NOT NULL UNIQUE,
|
||||||
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||||
|
CONSTRAINT attachments_filename_nonblank CHECK (length(btrim(filename)) > 0),
|
||||||
|
CONSTRAINT attachments_content_type CHECK (content_type IN ('image/png', 'image/jpeg')),
|
||||||
|
CONSTRAINT attachments_size CHECK (size_bytes BETWEEN 1 AND 10485760),
|
||||||
|
CONSTRAINT attachments_sha256 CHECK (sha256 ~ '^[a-f0-9]{64}$')
|
||||||
|
);
|
||||||
|
CREATE INDEX IF NOT EXISTS attachments_chat_idx ON attachments(chat_id);
|
||||||
|
CREATE TABLE IF NOT EXISTS message_attachments (
|
||||||
|
message_id UUID NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
|
||||||
|
attachment_id UUID NOT NULL REFERENCES attachments(id),
|
||||||
|
position INTEGER NOT NULL,
|
||||||
|
PRIMARY KEY (message_id, attachment_id)
|
||||||
|
);
|
||||||
|
CREATE INDEX IF NOT EXISTS message_attachments_attachment_idx ON message_attachments(attachment_id);
|
||||||
|
ALTER TABLE messages DROP CONSTRAINT IF EXISTS messages_text_nonblank;
|
||||||
@@ -0,0 +1,58 @@
|
|||||||
|
ALTER TABLE attachments ADD COLUMN IF NOT EXISTS status TEXT NOT NULL DEFAULT 'ready';
|
||||||
|
ALTER TABLE attachments ADD COLUMN IF NOT EXISTS error_code TEXT;
|
||||||
|
ALTER TABLE attachments ADD COLUMN IF NOT EXISTS updated_at TIMESTAMPTZ NOT NULL DEFAULT now();
|
||||||
|
ALTER TABLE attachments ADD COLUMN IF NOT EXISTS staging_key TEXT;
|
||||||
|
ALTER TABLE attachments ADD COLUMN IF NOT EXISTS pipeline_version INTEGER NOT NULL DEFAULT 1;
|
||||||
|
ALTER TABLE attachments DROP CONSTRAINT IF EXISTS attachments_status;
|
||||||
|
ALTER TABLE attachments ADD CONSTRAINT attachments_status CHECK (status IN ('queued', 'processing', 'ready', 'failed'));
|
||||||
|
CREATE INDEX IF NOT EXISTS attachments_staging_idx ON attachments(staging_key) WHERE staging_key IS NOT NULL;
|
||||||
|
CREATE TABLE IF NOT EXISTS attachment_variants (
|
||||||
|
attachment_id UUID NOT NULL REFERENCES attachments(id) ON DELETE CASCADE,
|
||||||
|
pipeline_version INTEGER NOT NULL,
|
||||||
|
name TEXT NOT NULL,
|
||||||
|
s3_key TEXT NOT NULL,
|
||||||
|
content_type TEXT NOT NULL,
|
||||||
|
size_bytes BIGINT NOT NULL,
|
||||||
|
width INTEGER NOT NULL,
|
||||||
|
height INTEGER NOT NULL,
|
||||||
|
sha256 CHAR(64) NOT NULL,
|
||||||
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||||
|
PRIMARY KEY (attachment_id, pipeline_version, name),
|
||||||
|
CONSTRAINT variants_name CHECK (name IN ('thumbnail', 'crop', 'optimized'))
|
||||||
|
);
|
||||||
|
CREATE TABLE IF NOT EXISTS attachment_jobs (
|
||||||
|
job_id UUID PRIMARY KEY,
|
||||||
|
attachment_id UUID NOT NULL REFERENCES attachments(id) ON DELETE CASCADE,
|
||||||
|
pipeline_version INTEGER NOT NULL DEFAULT 1,
|
||||||
|
status TEXT NOT NULL,
|
||||||
|
attempt INTEGER NOT NULL DEFAULT 0,
|
||||||
|
max_attempts INTEGER NOT NULL,
|
||||||
|
next_attempt_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||||
|
lease_until TIMESTAMPTZ,
|
||||||
|
worker_id TEXT,
|
||||||
|
error_code TEXT,
|
||||||
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||||
|
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||||
|
CONSTRAINT jobs_status CHECK (status IN ('queued', 'processing', 'ready', 'failed')),
|
||||||
|
CONSTRAINT jobs_effect_key UNIQUE (attachment_id, pipeline_version)
|
||||||
|
);
|
||||||
|
CREATE INDEX IF NOT EXISTS attachment_jobs_status_idx ON attachment_jobs(status, lease_until);
|
||||||
|
CREATE TABLE IF NOT EXISTS job_events (
|
||||||
|
id BIGSERIAL PRIMARY KEY,
|
||||||
|
job_id UUID NOT NULL REFERENCES attachment_jobs(job_id) ON DELETE CASCADE,
|
||||||
|
at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||||
|
event TEXT NOT NULL,
|
||||||
|
detail TEXT
|
||||||
|
);
|
||||||
|
CREATE INDEX IF NOT EXISTS job_events_job_idx ON job_events(job_id, id);
|
||||||
|
CREATE TABLE IF NOT EXISTS outbox (
|
||||||
|
id BIGSERIAL PRIMARY KEY,
|
||||||
|
job_id UUID NOT NULL REFERENCES attachment_jobs(job_id) ON DELETE CASCADE,
|
||||||
|
topic TEXT NOT NULL,
|
||||||
|
msg_key TEXT NOT NULL,
|
||||||
|
payload JSONB NOT NULL,
|
||||||
|
available_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||||
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||||
|
published_at TIMESTAMPTZ
|
||||||
|
);
|
||||||
|
CREATE INDEX IF NOT EXISTS outbox_pending_idx ON outbox(available_at, id) WHERE published_at IS NULL;
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
fastapi==0.116.1
|
||||||
|
uvicorn[standard]==0.35.0
|
||||||
|
psycopg[pool,binary]==3.2.9
|
||||||
|
argon2-cffi==25.1.0
|
||||||
|
redis==6.4.0
|
||||||
|
boto3==1.40.20
|
||||||
|
Pillow==11.3.0
|
||||||
|
confluent-kafka==2.10.1
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Apply SQL migrations once, before any API replica is started."""
|
||||||
|
from pathlib import Path
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
import psycopg
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
CONNINFO = os.getenv("DATABASE_URL", "")
|
||||||
|
|
||||||
|
def main():
|
||||||
|
last_error = None
|
||||||
|
for _ in range(30):
|
||||||
|
try:
|
||||||
|
with psycopg.connect(CONNINFO, connect_timeout=3) as conn:
|
||||||
|
conn.execute("CREATE TABLE IF NOT EXISTS schema_migrations (version TEXT PRIMARY KEY, applied_at TIMESTAMPTZ NOT NULL DEFAULT now())")
|
||||||
|
conn.commit()
|
||||||
|
for path in sorted((ROOT / "migrations").glob("*.sql")):
|
||||||
|
version = path.name
|
||||||
|
if conn.execute("SELECT 1 FROM schema_migrations WHERE version=%s", (version,)).fetchone():
|
||||||
|
continue
|
||||||
|
sql = path.read_text(encoding="utf-8")
|
||||||
|
with conn.transaction():
|
||||||
|
for statement in sql.split(";"):
|
||||||
|
if statement.strip():
|
||||||
|
conn.execute(statement)
|
||||||
|
conn.execute("INSERT INTO schema_migrations(version) VALUES (%s)", (version,))
|
||||||
|
print(f"APPLIED {version}", flush=True)
|
||||||
|
print("PASS: migrations complete", flush=True)
|
||||||
|
return
|
||||||
|
except Exception as exc:
|
||||||
|
last_error = exc
|
||||||
|
time.sleep(1)
|
||||||
|
raise SystemExit(f"Migration failed after retries: {type(last_error).__name__}")
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Operator tool: failed-job registry and SAFE explicit replay (lab5, level 4).
|
||||||
|
|
||||||
|
docker compose exec worker python scripts/replay_failed.py --list
|
||||||
|
docker compose exec worker python scripts/replay_failed.py --job-id <uuid> --reason "ticket-123"
|
||||||
|
|
||||||
|
Replay never executes a duplicate of a successful job: it requires status=failed, that the staging source still exists,
|
||||||
|
and that the effect (all variants of the pipeline version) is NOT already present. Everything happens in ONE transaction
|
||||||
|
(state change + audit event + outbox row), so the new attempt is delivered by the normal dispatcher path.
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
|
||||||
|
import psycopg
|
||||||
|
|
||||||
|
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||||
|
from src import jobs # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
p = argparse.ArgumentParser()
|
||||||
|
p.add_argument("--list", action="store_true")
|
||||||
|
p.add_argument("--job-id")
|
||||||
|
p.add_argument("--reason", default="operator replay")
|
||||||
|
a = p.parse_args()
|
||||||
|
with psycopg.connect(connect_timeout=5) as conn:
|
||||||
|
if a.list or not a.job_id:
|
||||||
|
for r in conn.execute("SELECT job_id, attachment_id, error_code, attempt, updated_at FROM attachment_jobs WHERE status='failed' ORDER BY updated_at"):
|
||||||
|
print(*r, sep="\t")
|
||||||
|
return 0
|
||||||
|
row = conn.execute("SELECT j.status, j.attachment_id, j.pipeline_version, a.staging_key FROM attachment_jobs j "
|
||||||
|
"JOIN attachments a ON a.id=j.attachment_id WHERE j.job_id=%s FOR UPDATE OF j", (a.job_id,)).fetchone()
|
||||||
|
if row is None:
|
||||||
|
print("REFUSED: unknown job"); return 1
|
||||||
|
status, att_id, pv, staging_key = row
|
||||||
|
if status != "failed":
|
||||||
|
print(f"REFUSED: job is {status}, only failed jobs can be replayed"); return 1
|
||||||
|
done = conn.execute("SELECT count(*) FROM attachment_variants WHERE attachment_id=%s AND pipeline_version=%s", (att_id, pv)).fetchone()[0]
|
||||||
|
if done >= len(jobs.VARIANT_ORDER):
|
||||||
|
print("REFUSED: the effect already exists (all variants present)"); return 1
|
||||||
|
if not staging_key:
|
||||||
|
print("REFUSED: staging source expired (retention), upload the file again"); return 1
|
||||||
|
conn.execute("UPDATE attachment_jobs SET status='queued', attempt=0, error_code=NULL, lease_until=NULL, next_attempt_at=now(), updated_at=now() WHERE job_id=%s", (a.job_id,))
|
||||||
|
conn.execute("UPDATE attachments SET status='queued', error_code=NULL, updated_at=now() WHERE id=%s", (att_id,))
|
||||||
|
jobs.add_event(conn, a.job_id, "replay", a.reason[:200])
|
||||||
|
jobs.enqueue(conn, a.job_id, att_id, pv, 1)
|
||||||
|
print(f"REPLAYED job={a.job_id} attachment={att_id}")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
set -eu
|
||||||
|
# Restart ALL state stores (PostgreSQL, Redis, S3, Kafka), then API, dispatcher and worker; named volumes are preserved.
|
||||||
|
docker compose restart postgres redis s3 kafka
|
||||||
|
docker compose restart api dispatcher worker
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
set -eu
|
||||||
|
# Sequential replacement-equivalent exercise for Compose (which has no native rolling-update primitive).
|
||||||
|
# Run a load generator in another terminal while this script executes.
|
||||||
|
replicas="$(docker compose ps -q --all api)"
|
||||||
|
[ -n "$replicas" ] || { echo "No API containers found; start the stack first" >&2; exit 1; }
|
||||||
|
for id in $replicas; do
|
||||||
|
# Remove one replica; the LB excludes it after health-check failure.
|
||||||
|
docker stop --time 10 "$id" >/dev/null || true
|
||||||
|
docker rm -f "$id" >/dev/null
|
||||||
|
docker compose up -d --no-deps --scale api=2 api
|
||||||
|
# Wait for at least two healthy API containers before moving to the next one.
|
||||||
|
tries=0
|
||||||
|
while [ "$tries" -lt 30 ]; do
|
||||||
|
healthy=0
|
||||||
|
for candidate in $(docker compose ps -q api); do
|
||||||
|
status=$(docker inspect --format '{{.State.Health.Status}}' "$candidate" 2>/dev/null || echo starting)
|
||||||
|
[ "$status" = healthy ] && healthy=$((healthy+1))
|
||||||
|
done
|
||||||
|
[ "$healthy" -ge 2 ] && break
|
||||||
|
tries=$((tries+1)); sleep 1
|
||||||
|
done
|
||||||
|
[ "$healthy" -ge 2 ] || { echo "Fewer than two healthy replicas after replacement" >&2; exit 1; }
|
||||||
|
done
|
||||||
@@ -0,0 +1,74 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Orphan sweeper (lab5): staging/ and attachments/ cleanup, retention of failed staging (see run_once).
|
||||||
|
|
||||||
|
Safety: only objects older than --grace-seconds are candidates (default 3600, far above the few seconds between
|
||||||
|
PUT and COMMIT), and the DB is re-checked immediately before each delete. Default mode is --dry-run.
|
||||||
|
"""
|
||||||
|
import argparse, os, sys, time
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
import boto3, psycopg
|
||||||
|
from botocore.config import Config
|
||||||
|
|
||||||
|
def run_once(s3, conn, bucket, grace, apply, retention):
|
||||||
|
"""Lab5 cleanup rules (documented in REPORT.md):
|
||||||
|
1) staging/<k> without any attachments.staging_key row older than grace = orphan of a crash between PUT and COMMIT -> delete
|
||||||
|
(staging of ready attachments is deleted by the worker and its staging_key set to NULL, so a leftover is also an orphan);
|
||||||
|
2) attachments/<id>/... whose attachment row no longer exists -> delete;
|
||||||
|
3) staging of FAILED attachments older than retention -> delete object and NULL staging_key (replay is then impossible).
|
||||||
|
Current work (queued/processing, or failed within retention) is never touched; the DB is re-checked right before each delete."""
|
||||||
|
now = datetime.now(timezone.utc)
|
||||||
|
seen = old = orphans = deleted = 0
|
||||||
|
for prefix in ("staging/", "attachments/"):
|
||||||
|
for page in s3.get_paginator("list_objects_v2").paginate(Bucket=bucket, Prefix=prefix):
|
||||||
|
for obj in page.get("Contents", []):
|
||||||
|
seen += 1
|
||||||
|
key = obj["Key"]
|
||||||
|
if key == "attachments/.ready" or (now - obj["LastModified"]).total_seconds() < grace:
|
||||||
|
continue # fresh: may still be between PUT and COMMIT
|
||||||
|
old += 1
|
||||||
|
if prefix == "staging/":
|
||||||
|
referenced = lambda: conn.execute("SELECT 1 FROM attachments WHERE staging_key=%s", (key,)).fetchone()
|
||||||
|
else:
|
||||||
|
att = key.split("/")[1] if key.count("/") >= 2 else ""
|
||||||
|
referenced = lambda: conn.execute("SELECT 1 FROM attachments WHERE id::text=%s", (att,)).fetchone()
|
||||||
|
if referenced():
|
||||||
|
continue
|
||||||
|
orphans += 1
|
||||||
|
print(("DELETE " if apply else "WOULD-DELETE ") + key, flush=True)
|
||||||
|
if apply and not referenced():
|
||||||
|
s3.delete_object(Bucket=bucket, Key=key); deleted += 1
|
||||||
|
expired = conn.execute("SELECT id, staging_key FROM attachments WHERE status='failed' AND staging_key IS NOT NULL "
|
||||||
|
"AND updated_at < now() - make_interval(secs=>%s)", (retention,)).fetchall()
|
||||||
|
for att_id, skey in expired:
|
||||||
|
print(("EXPIRE-STAGING " if apply else "WOULD-EXPIRE-STAGING ") + skey, flush=True)
|
||||||
|
if apply:
|
||||||
|
s3.delete_object(Bucket=bucket, Key=skey)
|
||||||
|
conn.execute("UPDATE attachments SET staging_key=NULL WHERE id=%s", (att_id,))
|
||||||
|
deleted += 1
|
||||||
|
print(f"sweep: seen={seen} older_than_grace={old} orphans={orphans} failed_expired={len(expired)} deleted={deleted} apply={apply}", flush=True)
|
||||||
|
return orphans, deleted
|
||||||
|
|
||||||
|
def main():
|
||||||
|
p = argparse.ArgumentParser()
|
||||||
|
p.add_argument("--apply", action="store_true", help="really delete (default: dry-run)")
|
||||||
|
p.add_argument("--grace-seconds", type=int, default=int(os.getenv("SWEEP_GRACE_SECONDS", "3600")))
|
||||||
|
p.add_argument("--loop", action="store_true"); p.add_argument("--interval", type=int, default=int(os.getenv("SWEEP_INTERVAL_SECONDS", "600")))
|
||||||
|
p.add_argument("--retention-seconds", type=int, default=int(os.getenv("STAGING_RETENTION_SECONDS", "86400")))
|
||||||
|
a = p.parse_args()
|
||||||
|
if a.grace_seconds < 0: sys.exit("grace must be >= 0")
|
||||||
|
s3 = boto3.client("s3", endpoint_url=os.environ["S3_ENDPOINT"], region_name=os.getenv("S3_REGION", "us-east-1"),
|
||||||
|
aws_access_key_id=os.environ["S3_ACCESS_KEY_ID"], aws_secret_access_key=os.environ["S3_SECRET_ACCESS_KEY"],
|
||||||
|
config=Config(signature_version="s3v4", s3={"addressing_style": "path"}, connect_timeout=3, read_timeout=15, retries={"max_attempts": 2}))
|
||||||
|
bucket = os.environ["S3_BUCKET"]
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
with psycopg.connect(connect_timeout=5, autocommit=True) as conn:
|
||||||
|
run_once(s3, conn, bucket, a.grace_seconds, a.apply, a.retention_seconds)
|
||||||
|
except Exception as exc: # keep looping through outages
|
||||||
|
print("sweep failed:", type(exc).__name__, flush=True)
|
||||||
|
if not a.loop: raise
|
||||||
|
if not a.loop: return
|
||||||
|
time.sleep(a.interval)
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
set -eu
|
||||||
|
: "${TARGET_INSTANCE_ID:?TARGET_INSTANCE_ID must be supplied by tests/resilience.py}"
|
||||||
|
found=""
|
||||||
|
for id in $(docker compose ps -q api); do
|
||||||
|
host=$(docker inspect --format '{{.Config.Hostname}}' "$id")
|
||||||
|
if [ "$host" = "$TARGET_INSTANCE_ID" ]; then
|
||||||
|
found="$id"
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
if [ -z "$found" ]; then
|
||||||
|
echo "No running API container matches TARGET_INSTANCE_ID" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
# Stop exactly one API replica and intentionally leave it stopped for the failover check.
|
||||||
|
docker stop --time 10 "$found" >/dev/null
|
||||||
@@ -0,0 +1,88 @@
|
|||||||
|
"""Lab 5 dispatcher: transactional-outbox relay + lease reaper.
|
||||||
|
|
||||||
|
* relay : outbox rows (committed together with the state change) -> Kafka (acks=all, idempotent producer, delivery report)
|
||||||
|
* reaper : processing job with an expired lease (worker died) -> queued again (or failed when attempts are exhausted)
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import psycopg
|
||||||
|
from confluent_kafka import Producer
|
||||||
|
|
||||||
|
from . import jobs
|
||||||
|
|
||||||
|
logging.basicConfig(level=os.getenv("LOG_LEVEL", "INFO"), format="%(asctime)s %(levelname)s dispatcher %(message)s")
|
||||||
|
log = logging.getLogger("dispatcher")
|
||||||
|
POLL_SECONDS = float(os.getenv("DISPATCH_POLL_SECONDS", "1"))
|
||||||
|
BATCH = int(os.getenv("OUTBOX_BATCH", "100"))
|
||||||
|
FAULT_FLAG = Path("/tmp/dispatcher_fault_done")
|
||||||
|
FAULT_EXIT_BEFORE_PUBLISH = os.getenv("FAULT_DISPATCHER_EXIT_BEFORE_PUBLISH", "0") == "1" # test hook: crash between COMMIT and publish
|
||||||
|
|
||||||
|
|
||||||
|
def relay(conn, producer: Producer) -> int:
|
||||||
|
with conn.transaction():
|
||||||
|
rows = conn.execute("SELECT id, topic, msg_key, payload::text FROM outbox WHERE published_at IS NULL AND available_at <= now() "
|
||||||
|
"ORDER BY id LIMIT %s FOR UPDATE SKIP LOCKED", (BATCH,)).fetchall()
|
||||||
|
if not rows:
|
||||||
|
return 0
|
||||||
|
if FAULT_EXIT_BEFORE_PUBLISH and not FAULT_FLAG.exists():
|
||||||
|
FAULT_FLAG.touch()
|
||||||
|
log.error("FAULT: exiting after reading the outbox, before publishing (COMMIT happened, publish did not)")
|
||||||
|
os._exit(137)
|
||||||
|
delivered, errors = set(), {}
|
||||||
|
|
||||||
|
for oid, topic, key, payload in rows:
|
||||||
|
def cb(err, msg, oid=oid):
|
||||||
|
(errors.__setitem__(oid, str(err)) if err else delivered.add(oid)) # publisher confirm arrives here
|
||||||
|
producer.produce(topic, key=key.encode(), value=payload.encode(), on_delivery=cb)
|
||||||
|
producer.flush(15)
|
||||||
|
if delivered:
|
||||||
|
conn.execute("UPDATE outbox SET published_at=now() WHERE id = ANY(%s)", (list(delivered),))
|
||||||
|
if errors:
|
||||||
|
log.warning("%d outbox rows not confirmed by the broker, will retry", len(errors))
|
||||||
|
return len(delivered)
|
||||||
|
|
||||||
|
|
||||||
|
def reap(conn) -> int:
|
||||||
|
n = 0
|
||||||
|
with conn.transaction():
|
||||||
|
for job_id, att_id, attempt, max_attempts, pv in conn.execute(
|
||||||
|
"SELECT job_id, attachment_id, attempt, max_attempts, pipeline_version FROM attachment_jobs "
|
||||||
|
"WHERE status='processing' AND lease_until < now() FOR UPDATE SKIP LOCKED").fetchall():
|
||||||
|
if attempt >= max_attempts:
|
||||||
|
jobs.fail_job(conn, job_id, att_id, "retry_exhausted", attempt)
|
||||||
|
else:
|
||||||
|
conn.execute("UPDATE attachment_jobs SET status='queued', lease_until=NULL, next_attempt_at=now(), updated_at=now() WHERE job_id=%s", (job_id,))
|
||||||
|
conn.execute("UPDATE attachments SET status='queued', updated_at=now() WHERE id=%s", (att_id,))
|
||||||
|
jobs.add_event(conn, job_id, "lease_expired", f"attempt={attempt}")
|
||||||
|
jobs.enqueue(conn, job_id, att_id, pv, attempt + 1)
|
||||||
|
n += 1
|
||||||
|
return n
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
producer = Producer({"bootstrap.servers": jobs.KAFKA_BOOTSTRAP, "acks": "all", "enable.idempotence": True,
|
||||||
|
"message.timeout.ms": 15000, "linger.ms": 5})
|
||||||
|
log.info("dispatcher started (poll=%.1fs)", POLL_SECONDS)
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
with psycopg.connect(connect_timeout=5, autocommit=True, options="-c statement_timeout=20000") as conn:
|
||||||
|
while True:
|
||||||
|
done = reap(conn)
|
||||||
|
sent = relay(conn, producer)
|
||||||
|
if done or sent:
|
||||||
|
log.info("reaped=%d published=%d", done, sent)
|
||||||
|
if not sent:
|
||||||
|
time.sleep(POLL_SECONDS)
|
||||||
|
except psycopg.Error as exc:
|
||||||
|
log.warning("database unavailable: %s", type(exc).__name__); time.sleep(2)
|
||||||
|
except Exception as exc:
|
||||||
|
log.warning("dispatcher iteration failed: %s", type(exc).__name__); time.sleep(2)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,97 @@
|
|||||||
|
"""Image pipeline v1. Runs in a CHILD process with hard CPU/address-space limits (called by worker.py).
|
||||||
|
|
||||||
|
python -m src.imgproc <input-file> <output-dir>
|
||||||
|
|
||||||
|
Writes thumbnail.jpg, crop.jpg, optimized.webp and prints JSON metadata on stdout.
|
||||||
|
Exit codes: 0 ok, 2 invalid image, 3 too many pixels, 4 out of memory, other = unexpected (treated as transient).
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import resource
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from PIL import Image, ImageOps
|
||||||
|
|
||||||
|
MAX_IMAGE_PIXELS = int(os.getenv("MAX_IMAGE_PIXELS", "40000000"))
|
||||||
|
THUMB_BOX, CROP_SIZE, OPT_BOX = 256, 128, 1280
|
||||||
|
JPEG_QUALITY, WEBP_QUALITY = 85, 80 # documented in REPORT.md
|
||||||
|
EXIT_INVALID, EXIT_TOO_LARGE, EXIT_OOM = 2, 3, 4
|
||||||
|
|
||||||
|
|
||||||
|
def fit_box(w: int, h: int, box: int) -> tuple[int, int]:
|
||||||
|
"""Fit (w,h) into box x box, keep aspect ratio, never upscale, floor, min 1 px. Integer math: 320x200 -> 256x160."""
|
||||||
|
if w <= box and h <= box:
|
||||||
|
return w, h
|
||||||
|
if w >= h:
|
||||||
|
return box, max(1, h * box // w)
|
||||||
|
return max(1, w * box // h), box
|
||||||
|
|
||||||
|
|
||||||
|
def flatten_on_white(img: Image.Image) -> Image.Image:
|
||||||
|
"""RGB image; any alpha is composited on white (JPEG has no alpha)."""
|
||||||
|
if img.mode in ("RGBA", "LA") or (img.mode == "P" and "transparency" in img.info):
|
||||||
|
rgba = img.convert("RGBA")
|
||||||
|
bg = Image.new("RGB", rgba.size, (255, 255, 255))
|
||||||
|
bg.paste(rgba, mask=rgba.getchannel("A"))
|
||||||
|
return bg
|
||||||
|
return img.convert("RGB")
|
||||||
|
|
||||||
|
|
||||||
|
def save(img: Image.Image, path: Path, fmt: str, **opts) -> dict:
|
||||||
|
img.save(path, fmt, **opts) # no exif=/icc_profile= passed => metadata is stripped
|
||||||
|
data = path.read_bytes()
|
||||||
|
return {"size_bytes": len(data), "width": img.width, "height": img.height, "sha256": hashlib.sha256(data).hexdigest()}
|
||||||
|
|
||||||
|
|
||||||
|
def process(src: Path, out: Path) -> dict:
|
||||||
|
with Image.open(src) as img:
|
||||||
|
if img.format not in ("PNG", "JPEG"):
|
||||||
|
raise ValueError("unsupported format")
|
||||||
|
if img.width < 1 or img.height < 1:
|
||||||
|
raise ValueError("empty image")
|
||||||
|
if img.width * img.height > MAX_IMAGE_PIXELS: # decided from the header, before any pixel buffer exists
|
||||||
|
raise OverflowError("too many pixels")
|
||||||
|
img.load()
|
||||||
|
base = ImageOps.exif_transpose(img) # apply EXIF orientation once; result has no orientation tag
|
||||||
|
base.load()
|
||||||
|
result = {}
|
||||||
|
# thumbnail: JPEG, fit 256x256, no upscale, alpha -> white
|
||||||
|
flat = flatten_on_white(base)
|
||||||
|
t = flat.resize(fit_box(*flat.size, THUMB_BOX), Image.LANCZOS) if fit_box(*flat.size, THUMB_BOX) != flat.size else flat
|
||||||
|
result["thumbnail"] = {"content_type": "image/jpeg", **save(t, out / "thumbnail.jpg", "JPEG", quality=JPEG_QUALITY, optimize=True)}
|
||||||
|
# crop: JPEG, centered square 128x128 (small sources may be upscaled)
|
||||||
|
c = ImageOps.fit(flat, (CROP_SIZE, CROP_SIZE), Image.LANCZOS, centering=(0.5, 0.5))
|
||||||
|
result["crop"] = {"content_type": "image/jpeg", **save(c, out / "crop.jpg", "JPEG", quality=JPEG_QUALITY, optimize=True)}
|
||||||
|
# optimized: WebP, fit 1280x1280, no upscale, keeps alpha, lossy q=80, no EXIF/ICC
|
||||||
|
o = base if base.mode in ("RGB", "RGBA") else (base.convert("RGBA") if "A" in base.getbands() or base.mode == "P" else base.convert("RGB"))
|
||||||
|
size = fit_box(*o.size, OPT_BOX)
|
||||||
|
if size != o.size:
|
||||||
|
o = o.resize(size, Image.LANCZOS)
|
||||||
|
result["optimized"] = {"content_type": "image/webp", **save(o, out / "optimized.webp", "WEBP", quality=WEBP_QUALITY, method=4)}
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
mem_mb, cpu_s = int(os.getenv("IMG_MEMORY_MB", "1024")), int(os.getenv("IMG_CPU_SECONDS", "60"))
|
||||||
|
resource.setrlimit(resource.RLIMIT_AS, (mem_mb * 1024 * 1024, mem_mb * 1024 * 1024))
|
||||||
|
resource.setrlimit(resource.RLIMIT_CPU, (cpu_s, cpu_s + 5))
|
||||||
|
Image.MAX_IMAGE_PIXELS = MAX_IMAGE_PIXELS
|
||||||
|
try:
|
||||||
|
print(json.dumps(process(Path(sys.argv[1]), Path(sys.argv[2]))))
|
||||||
|
return 0
|
||||||
|
except OverflowError:
|
||||||
|
return EXIT_TOO_LARGE
|
||||||
|
except Image.DecompressionBombError:
|
||||||
|
return EXIT_TOO_LARGE
|
||||||
|
except MemoryError:
|
||||||
|
return EXIT_OOM
|
||||||
|
except (ValueError, OSError, SyntaxError, Image.UnidentifiedImageError):
|
||||||
|
return EXIT_INVALID
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
+81
@@ -0,0 +1,81 @@
|
|||||||
|
"""Shared constants and helpers for the API, the worker and the dispatcher (lab 5)."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import random
|
||||||
|
import uuid
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
|
||||||
|
import boto3
|
||||||
|
from botocore.config import Config as BotoConfig
|
||||||
|
|
||||||
|
KAFKA_BOOTSTRAP = os.getenv("KAFKA_BOOTSTRAP_SERVERS", "kafka:9092")
|
||||||
|
JOBS_TOPIC = os.getenv("JOBS_TOPIC", "attachment-jobs")
|
||||||
|
PIPELINE_VERSION = 1
|
||||||
|
JOB_TYPE = "attachment.process.v1"
|
||||||
|
|
||||||
|
JOB_MAX_ATTEMPTS = int(os.getenv("JOB_MAX_ATTEMPTS", "5"))
|
||||||
|
JOB_TIMEOUT_SECONDS = int(os.getenv("JOB_TIMEOUT_SECONDS", "60"))
|
||||||
|
LEASE_GRACE_SECONDS = int(os.getenv("LEASE_GRACE_SECONDS", "15"))
|
||||||
|
LEASE_SECONDS = JOB_TIMEOUT_SECONDS + LEASE_GRACE_SECONDS # lease > hard job timeout: a live worker never loses it
|
||||||
|
BACKOFF_BASE_SECONDS = float(os.getenv("JOB_BACKOFF_BASE_SECONDS", "2"))
|
||||||
|
BACKOFF_CAP_SECONDS = float(os.getenv("JOB_BACKOFF_CAP_SECONDS", "60"))
|
||||||
|
|
||||||
|
S3_ENDPOINT = os.getenv("S3_ENDPOINT", "http://s3:9000")
|
||||||
|
S3_REGION = os.getenv("S3_REGION", "us-east-1")
|
||||||
|
S3_BUCKET = os.getenv("S3_BUCKET", "messenger-attachments")
|
||||||
|
STAGING_PREFIX = "staging/"
|
||||||
|
FINAL_PREFIX = "attachments/"
|
||||||
|
|
||||||
|
VARIANT_ORDER = ("thumbnail", "crop", "optimized")
|
||||||
|
|
||||||
|
|
||||||
|
def final_key(attachment_id, pipeline_version: int, name: str) -> str:
|
||||||
|
"""Deterministic key: the effect key (attachment_id, pipeline_version, variant) maps 1:1 to an S3 object."""
|
||||||
|
return f"{FINAL_PREFIX}{attachment_id}/v{pipeline_version}/{name}"
|
||||||
|
|
||||||
|
|
||||||
|
def make_s3():
|
||||||
|
return boto3.client(
|
||||||
|
"s3", endpoint_url=S3_ENDPOINT, region_name=S3_REGION,
|
||||||
|
aws_access_key_id=os.getenv("S3_ACCESS_KEY_ID", ""), aws_secret_access_key=os.getenv("S3_SECRET_ACCESS_KEY", ""),
|
||||||
|
config=BotoConfig(signature_version="s3v4", s3={"addressing_style": "path"}, connect_timeout=2, read_timeout=20,
|
||||||
|
retries={"max_attempts": 2, "mode": "standard"},
|
||||||
|
request_checksum_calculation="when_required", response_checksum_validation="when_required"))
|
||||||
|
|
||||||
|
|
||||||
|
def backoff_seconds(attempt: int) -> float:
|
||||||
|
"""Exponential backoff with 'equal jitter': delay in [d/2, d], d = min(cap, base * 2^(attempt-1))."""
|
||||||
|
d = min(BACKOFF_CAP_SECONDS, BACKOFF_BASE_SECONDS * (2 ** max(0, attempt - 1)))
|
||||||
|
return random.uniform(d / 2, d)
|
||||||
|
|
||||||
|
|
||||||
|
def iso_now() -> str:
|
||||||
|
return datetime.now(timezone.utc).isoformat(timespec="seconds").replace("+00:00", "Z")
|
||||||
|
|
||||||
|
|
||||||
|
def add_event(conn, job_id, event: str, detail: str | None = None) -> None:
|
||||||
|
conn.execute("INSERT INTO job_events(job_id,event,detail) VALUES (%s,%s,%s)", (job_id, event, detail))
|
||||||
|
|
||||||
|
|
||||||
|
def enqueue(conn, job_id, attachment_id, pipeline_version: int, attempt: int, delay_seconds: float = 0.0) -> None:
|
||||||
|
"""Record the intent to publish a job message in the SAME transaction as the state change (transactional outbox).
|
||||||
|
|
||||||
|
The message carries only IDs (jobs.schema.json): no bytes, no credentials, no URLs.
|
||||||
|
"""
|
||||||
|
payload = {"schema_version": 1, "job_id": str(job_id), "type": JOB_TYPE, "attachment_id": str(attachment_id),
|
||||||
|
"pipeline_version": pipeline_version, "attempt": attempt, "created_at": iso_now()}
|
||||||
|
conn.execute(
|
||||||
|
"INSERT INTO outbox(job_id,topic,msg_key,payload,available_at) VALUES (%s,%s,%s,%s::jsonb, now() + make_interval(secs => %s))",
|
||||||
|
(job_id, JOBS_TOPIC, str(attachment_id), json.dumps(payload), delay_seconds))
|
||||||
|
|
||||||
|
|
||||||
|
def new_job_id() -> uuid.UUID:
|
||||||
|
return uuid.uuid4()
|
||||||
|
|
||||||
|
|
||||||
|
def fail_job(conn, job_id, att_id, code: str, attempt: int) -> None:
|
||||||
|
conn.execute("UPDATE attachment_jobs SET status='failed', error_code=%s, lease_until=NULL, updated_at=now() WHERE job_id=%s", (code, job_id))
|
||||||
|
conn.execute("UPDATE attachments SET status='failed', error_code=%s, updated_at=now() WHERE id=%s", (code, att_id))
|
||||||
|
add_event(conn, job_id, "failed", code)
|
||||||
+772
@@ -0,0 +1,772 @@
|
|||||||
|
"""Lab 5 REST API: FastAPI + PostgreSQL + Redis sessions + idempotency + S3 attachments with ASYNC processing (Kafka + outbox)."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import anyio
|
||||||
|
import base64
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import secrets
|
||||||
|
import socket
|
||||||
|
import tempfile
|
||||||
|
import unicodedata
|
||||||
|
import uuid
|
||||||
|
from datetime import datetime, timedelta, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
from urllib.parse import quote
|
||||||
|
|
||||||
|
import psycopg
|
||||||
|
from psycopg_pool import ConnectionPool, PoolTimeout
|
||||||
|
import anyio
|
||||||
|
import boto3
|
||||||
|
import redis
|
||||||
|
from botocore.config import Config as BotoConfig
|
||||||
|
from botocore.exceptions import BotoCoreError, ClientError
|
||||||
|
from PIL import Image
|
||||||
|
from redis.exceptions import RedisError
|
||||||
|
from argon2 import PasswordHasher
|
||||||
|
from argon2.exceptions import InvalidHashError, VerificationError, VerifyMismatchError
|
||||||
|
from fastapi import FastAPI, Request
|
||||||
|
from fastapi.exceptions import RequestValidationError
|
||||||
|
from fastapi.responses import JSONResponse, Response, StreamingResponse
|
||||||
|
from starlette.concurrency import iterate_in_threadpool, run_in_threadpool
|
||||||
|
from starlette.requests import ClientDisconnect
|
||||||
|
from pydantic import BaseModel, ConfigDict, Field, ValidationError
|
||||||
|
from starlette.exceptions import HTTPException as StarletteHTTPException
|
||||||
|
|
||||||
|
from . import jobs
|
||||||
|
|
||||||
|
logging.basicConfig(level=os.getenv("LOG_LEVEL", "INFO"))
|
||||||
|
log = logging.getLogger("messenger")
|
||||||
|
DATABASE_URL = os.getenv("DATABASE_URL", "")
|
||||||
|
INSTANCE_ID = os.getenv("INSTANCE_ID") or socket.gethostname()
|
||||||
|
IDEMPOTENCY_KEY_RE = re.compile(r"^[A-Za-z0-9._:-]{1,128}$")
|
||||||
|
REDIS_URL = os.getenv("REDIS_URL", "redis://localhost:6379/0")
|
||||||
|
SESSION_TTL_SECONDS = int(os.getenv("SESSION_TTL_SECONDS", "3600"))
|
||||||
|
if SESSION_TTL_SECONDS < 1:
|
||||||
|
raise RuntimeError("SESSION_TTL_SECONDS must be positive")
|
||||||
|
# Redis uses its connection pool; session keys contain only a hash of the opaque token.
|
||||||
|
session_store = redis.Redis.from_url(REDIS_URL, decode_responses=True, socket_connect_timeout=3, socket_timeout=3, health_check_interval=30)
|
||||||
|
DB_POOL_MIN_SIZE = int(os.getenv("DB_POOL_MIN_SIZE", "1"))
|
||||||
|
DB_POOL_MAX_SIZE = int(os.getenv("DB_POOL_MAX_SIZE", "10"))
|
||||||
|
DB_POOL_TIMEOUT_SECONDS = float(os.getenv("DB_POOL_TIMEOUT_SECONDS", "5"))
|
||||||
|
if DB_POOL_MIN_SIZE < 1 or DB_POOL_MAX_SIZE < DB_POOL_MIN_SIZE:
|
||||||
|
raise RuntimeError("Invalid PostgreSQL pool size configuration")
|
||||||
|
# Every pooled connection gets bounded connect/query/idle-transaction timeouts.
|
||||||
|
db_pool = ConnectionPool(
|
||||||
|
conninfo=DATABASE_URL,
|
||||||
|
min_size=DB_POOL_MIN_SIZE,
|
||||||
|
max_size=DB_POOL_MAX_SIZE,
|
||||||
|
timeout=DB_POOL_TIMEOUT_SECONDS,
|
||||||
|
kwargs={
|
||||||
|
"connect_timeout": 5,
|
||||||
|
"options": "-c statement_timeout=5000 -c idle_in_transaction_session_timeout=10000",
|
||||||
|
},
|
||||||
|
open=False,
|
||||||
|
name="messenger-postgres",
|
||||||
|
)
|
||||||
|
|
||||||
|
# ---- Lab 4: S3 / attachment limits (names follow .env.example of the lab) ----
|
||||||
|
S3_ENDPOINT = os.getenv("S3_ENDPOINT", "http://s3:9000")
|
||||||
|
S3_REGION = os.getenv("S3_REGION", "us-east-1")
|
||||||
|
S3_BUCKET = os.getenv("S3_BUCKET", "messenger-attachments")
|
||||||
|
S3_ACCESS_KEY_ID = os.getenv("S3_ACCESS_KEY_ID", "")
|
||||||
|
S3_SECRET_ACCESS_KEY = os.getenv("S3_SECRET_ACCESS_KEY", "")
|
||||||
|
S3_KEY_PREFIX = "attachments/"
|
||||||
|
STAGING_PREFIX = jobs.STAGING_PREFIX # lab5: durable staging (private S3 prefix) for bytes accepted but not yet processed
|
||||||
|
MAX_UPLOAD_BYTES = int(os.getenv("MAX_UPLOAD_BYTES", "10485760"))
|
||||||
|
MAX_IMAGE_PIXELS = int(os.getenv("MAX_IMAGE_PIXELS", "40000000"))
|
||||||
|
UPLOAD_IDLE_TIMEOUT_SECONDS = float(os.getenv("UPLOAD_IDLE_TIMEOUT_SECONDS", "10"))
|
||||||
|
UPLOAD_TOTAL_TIMEOUT_SECONDS = float(os.getenv("UPLOAD_TOTAL_TIMEOUT_SECONDS", "60"))
|
||||||
|
IMAGE_DECODE_CONCURRENCY = int(os.getenv("IMAGE_DECODE_CONCURRENCY", "1"))
|
||||||
|
SPOOL_MEMORY_BYTES = 1024 * 1024 # larger bodies roll over to an anonymous temp file, never to RAM
|
||||||
|
CHUNK_BYTES = 64 * 1024
|
||||||
|
# Pillow's own bomb guard is a second line of defence behind the explicit check below.
|
||||||
|
Image.MAX_IMAGE_PIXELS = MAX_IMAGE_PIXELS
|
||||||
|
# Credentials are the restricted application user created by s3_init, never the MinIO root user.
|
||||||
|
_s3_common = dict(endpoint_url=S3_ENDPOINT, region_name=S3_REGION, aws_access_key_id=S3_ACCESS_KEY_ID, aws_secret_access_key=S3_SECRET_ACCESS_KEY)
|
||||||
|
s3 = boto3.client("s3", **_s3_common, config=BotoConfig(
|
||||||
|
signature_version="s3v4", s3={"addressing_style": "path"}, connect_timeout=2, read_timeout=20,
|
||||||
|
retries={"max_attempts": 2, "mode": "standard"}, max_pool_connections=32,
|
||||||
|
request_checksum_calculation="when_required", response_checksum_validation="when_required"))
|
||||||
|
# Readiness probe client: one attempt and tight timeouts so HAProxy's 2 s check never hangs on it.
|
||||||
|
s3_probe = boto3.client("s3", **_s3_common, config=BotoConfig(
|
||||||
|
signature_version="s3v4", s3={"addressing_style": "path"}, connect_timeout=1, read_timeout=1,
|
||||||
|
retries={"max_attempts": 1}, request_checksum_calculation="when_required", response_checksum_validation="when_required"))
|
||||||
|
decode_limiter = anyio.CapacityLimiter(IMAGE_DECODE_CONCURRENCY)
|
||||||
|
IMAGE_TYPES = {"PNG": "image/png", "JPEG": "image/jpeg"}
|
||||||
|
USERNAME_RE = re.compile(r"^[a-z0-9_]{3,32}$")
|
||||||
|
PASSWORDS = PasswordHasher()
|
||||||
|
|
||||||
|
class ApiError(Exception):
|
||||||
|
def __init__(self, status: int, code: str, message: str, headers: dict[str, str] | None = None):
|
||||||
|
self.status, self.code, self.message, self.headers = status, code, message, headers or {}
|
||||||
|
|
||||||
|
def fail(status: int, code: str, message: str, headers: dict[str, str] | None = None):
|
||||||
|
raise ApiError(status, code, message, headers)
|
||||||
|
|
||||||
|
class StrictModel(BaseModel):
|
||||||
|
model_config = ConfigDict(extra="forbid", strict=True)
|
||||||
|
|
||||||
|
class RegisterBody(StrictModel):
|
||||||
|
username: str
|
||||||
|
display_name: str
|
||||||
|
password: str
|
||||||
|
|
||||||
|
class ChatBody(StrictModel):
|
||||||
|
title: str
|
||||||
|
member_ids: list[str] = Field(min_length=1, max_length=100)
|
||||||
|
|
||||||
|
class MessageBody(StrictModel):
|
||||||
|
text: str
|
||||||
|
attachment_ids: list[str] = Field(default_factory=list, max_length=10)
|
||||||
|
|
||||||
|
app = FastAPI(docs_url=None, redoc_url=None, openapi_url=None)
|
||||||
|
|
||||||
|
@app.middleware("http")
|
||||||
|
async def request_id_middleware(request: Request, call_next):
|
||||||
|
request.state.request_id = uuid.uuid4().hex
|
||||||
|
try:
|
||||||
|
response = await call_next(request)
|
||||||
|
except Exception:
|
||||||
|
log.exception("Unhandled API exception request_id=%s", request.state.request_id)
|
||||||
|
response = JSONResponse(status_code=500, content={"error": {"code": "internal_error", "message": "Internal server error", "request_id": request.state.request_id}})
|
||||||
|
response.headers["X-Request-Id"] = request.state.request_id
|
||||||
|
response.headers["X-Instance-Id"] = INSTANCE_ID
|
||||||
|
return response
|
||||||
|
|
||||||
|
def error_response(status: int, code: str, message: str, request_id: str, headers: dict[str, str] | None = None):
|
||||||
|
return JSONResponse(status_code=status, content={"error": {"code": code, "message": message, "request_id": request_id}}, headers=headers or {})
|
||||||
|
|
||||||
|
@app.exception_handler(ApiError)
|
||||||
|
async def api_error_handler(request: Request, exc: ApiError):
|
||||||
|
return error_response(exc.status, exc.code, exc.message, request.state.request_id, exc.headers)
|
||||||
|
|
||||||
|
@app.exception_handler(PoolTimeout)
|
||||||
|
async def database_pool_timeout_handler(request: Request, exc: PoolTimeout):
|
||||||
|
log.warning("Database connection pool timed out request_id=%s", request.state.request_id)
|
||||||
|
return error_response(503, "unavailable", "Required storage is unavailable", request.state.request_id)
|
||||||
|
|
||||||
|
@app.exception_handler(psycopg.Error)
|
||||||
|
async def database_error_handler(request: Request, exc: psycopg.Error):
|
||||||
|
log.warning("Database operation failed request_id=%s", request.state.request_id)
|
||||||
|
return error_response(503, "unavailable", "Required storage is unavailable", request.state.request_id)
|
||||||
|
|
||||||
|
@app.exception_handler(RedisError)
|
||||||
|
async def redis_error_handler(request: Request, exc: RedisError):
|
||||||
|
log.warning("Session store operation failed request_id=%s", request.state.request_id)
|
||||||
|
return error_response(503, "unavailable", "Required storage is unavailable", request.state.request_id)
|
||||||
|
|
||||||
|
@app.exception_handler(BotoCoreError)
|
||||||
|
async def s3_core_error_handler(request: Request, exc: BotoCoreError):
|
||||||
|
log.warning("Object storage failure request_id=%s type=%s", request.state.request_id, type(exc).__name__)
|
||||||
|
return error_response(503, "unavailable", "Required storage is unavailable", request.state.request_id)
|
||||||
|
|
||||||
|
@app.exception_handler(ClientError)
|
||||||
|
async def s3_client_error_handler(request: Request, exc: ClientError):
|
||||||
|
log.warning("Object storage error request_id=%s code=%s", request.state.request_id, exc.response.get("Error", {}).get("Code"))
|
||||||
|
return error_response(503, "unavailable", "Required storage is unavailable", request.state.request_id)
|
||||||
|
|
||||||
|
@app.exception_handler(RequestValidationError)
|
||||||
|
async def validation_error_handler(request: Request, exc: RequestValidationError):
|
||||||
|
return error_response(400, "invalid_request", "Invalid request", request.state.request_id)
|
||||||
|
|
||||||
|
@app.exception_handler(StarletteHTTPException)
|
||||||
|
async def http_error_handler(request: Request, exc: StarletteHTTPException):
|
||||||
|
status = exc.status_code
|
||||||
|
code = "not_found" if status == 404 else "invalid_request"
|
||||||
|
message = "Object not found" if status == 404 else "Invalid request"
|
||||||
|
headers = dict(exc.headers or {})
|
||||||
|
return error_response(status, code, message, request.state.request_id, headers)
|
||||||
|
|
||||||
|
def db():
|
||||||
|
"""Borrow a bounded pooled connection; context exit commits/rolls back and returns it."""
|
||||||
|
return db_pool.connection(timeout=DB_POOL_TIMEOUT_SECONDS)
|
||||||
|
|
||||||
|
@app.on_event("startup")
|
||||||
|
def startup():
|
||||||
|
# Migrations run once in the dedicated `migrate` service before API replicas.
|
||||||
|
import time
|
||||||
|
db_pool.open(wait=False)
|
||||||
|
last_error = None
|
||||||
|
for _ in range(30):
|
||||||
|
try:
|
||||||
|
with db() as conn:
|
||||||
|
conn.execute("SELECT 1")
|
||||||
|
conn.execute("SELECT 1 FROM idempotency_records LIMIT 0")
|
||||||
|
conn.execute("SELECT 1 FROM outbox LIMIT 0")
|
||||||
|
session_store.ping()
|
||||||
|
s3.head_object(Bucket=S3_BUCKET, Key=S3_KEY_PREFIX + ".ready")
|
||||||
|
log.info("PostgreSQL, Redis and S3 bucket are ready; instance=%s", INSTANCE_ID)
|
||||||
|
return
|
||||||
|
except Exception as exc:
|
||||||
|
last_error = exc
|
||||||
|
time.sleep(1)
|
||||||
|
raise RuntimeError("PostgreSQL schema, Redis or S3 bucket unavailable during startup") from last_error
|
||||||
|
|
||||||
|
@app.on_event("shutdown")
|
||||||
|
def shutdown():
|
||||||
|
session_store.close()
|
||||||
|
db_pool.close()
|
||||||
|
|
||||||
|
def now_utc() -> datetime:
|
||||||
|
return datetime.now(timezone.utc)
|
||||||
|
|
||||||
|
def iso(value: datetime) -> str:
|
||||||
|
return value.astimezone(timezone.utc).isoformat(timespec="microseconds").replace("+00:00", "Z")
|
||||||
|
|
||||||
|
def user_dict(row: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
return {"id": str(row["id"]), "username": row["username"], "display_name": row["display_name"], "created_at": iso(row["created_at"])}
|
||||||
|
|
||||||
|
def chat_dict(row: dict[str, Any], member_ids: list[str]) -> dict[str, Any]:
|
||||||
|
return {"id": str(row["id"]), "title": row["title"], "member_ids": member_ids, "created_at": iso(row["created_at"])}
|
||||||
|
|
||||||
|
def message_dict(row: dict[str, Any], attachment_ids: list[str] | None = None) -> dict[str, Any]:
|
||||||
|
return {"id": str(row["id"]), "chat_id": str(row["chat_id"]), "sender_id": str(row["sender_id"]), "text": row["text"], "created_at": iso(row["created_at"]), "attachment_ids": attachment_ids or []}
|
||||||
|
|
||||||
|
def validate_unicode(value: str, label: str) -> str:
|
||||||
|
"""Reject unpaired UTF-16 surrogates before sending text to PostgreSQL/KDF."""
|
||||||
|
try:
|
||||||
|
value.encode("utf-8")
|
||||||
|
except UnicodeEncodeError:
|
||||||
|
fail(400, "invalid_request", "Invalid Unicode in " + label)
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def parse_uuid(value: str, label: str = "UUID") -> uuid.UUID:
|
||||||
|
try:
|
||||||
|
# Keep the canonical UUID representation used by the Lab 1 API.
|
||||||
|
if not isinstance(value, str) or len(value) != 36:
|
||||||
|
raise ValueError("non-canonical UUID")
|
||||||
|
parsed = uuid.UUID(value)
|
||||||
|
if str(parsed) != value:
|
||||||
|
raise ValueError("non-canonical UUID")
|
||||||
|
return parsed
|
||||||
|
except (ValueError, TypeError, AttributeError):
|
||||||
|
fail(400, "invalid_request", "Invalid " + label)
|
||||||
|
|
||||||
|
def validate_body_type(request: Request):
|
||||||
|
content_type = request.headers.get("content-type", "").split(";", 1)[0].strip().lower()
|
||||||
|
if content_type != "application/json":
|
||||||
|
fail(415, "unsupported_media_type", "Content-Type must be application/json")
|
||||||
|
|
||||||
|
async def parse_body(request: Request, model):
|
||||||
|
validate_body_type(request)
|
||||||
|
try:
|
||||||
|
raw = await request.body()
|
||||||
|
data = json.loads(raw.decode("utf-8"))
|
||||||
|
return model.model_validate(data)
|
||||||
|
except (UnicodeDecodeError, json.JSONDecodeError, ValidationError, TypeError):
|
||||||
|
fail(400, "invalid_request", "Invalid JSON request body")
|
||||||
|
|
||||||
|
def challenge(scheme: str) -> dict[str, str]:
|
||||||
|
if scheme == "Basic":
|
||||||
|
return {"WWW-Authenticate": 'Basic realm="messenger", charset="UTF-8"'}
|
||||||
|
return {"WWW-Authenticate": 'Bearer realm="messenger"'}
|
||||||
|
|
||||||
|
def current_user(request: Request) -> dict[str, Any]:
|
||||||
|
auth = request.headers.get("authorization", "")
|
||||||
|
if not auth.startswith("Bearer ") or not auth[7:].strip():
|
||||||
|
fail(401, "unauthorized", "Valid Bearer session required", challenge("Bearer"))
|
||||||
|
token_hash = hashlib.sha256(auth[7:].strip().encode("utf-8")).hexdigest()
|
||||||
|
session_json = session_store.get("session:" + token_hash)
|
||||||
|
if session_json is None:
|
||||||
|
fail(401, "unauthorized", "Valid Bearer session required", challenge("Bearer"))
|
||||||
|
try:
|
||||||
|
session = json.loads(session_json)
|
||||||
|
user_id = uuid.UUID(session["user_id"])
|
||||||
|
expires_at = datetime.fromisoformat(session["expires_at"].replace("Z", "+00:00"))
|
||||||
|
except (ValueError, KeyError, TypeError, json.JSONDecodeError):
|
||||||
|
# Corrupt session data is invalidated instead of trusted.
|
||||||
|
session_store.delete("session:" + token_hash)
|
||||||
|
fail(401, "unauthorized", "Valid Bearer session required", challenge("Bearer"))
|
||||||
|
if expires_at <= now_utc():
|
||||||
|
session_store.delete("session:" + token_hash)
|
||||||
|
fail(401, "unauthorized", "Valid Bearer session required", challenge("Bearer"))
|
||||||
|
with db() as conn:
|
||||||
|
row = conn.execute("SELECT id,username,display_name,created_at FROM users WHERE id=%s", (user_id,)).fetchone()
|
||||||
|
if row is None:
|
||||||
|
session_store.delete("session:" + token_hash)
|
||||||
|
fail(401, "unauthorized", "Valid Bearer session required", challenge("Bearer"))
|
||||||
|
return {"id": row[0], "username": row[1], "display_name": row[2], "created_at": row[3], "expires_at": expires_at, "user": {"id": str(row[0]), "username": row[1], "display_name": row[2], "created_at": iso(row[3])}, "token_hash": token_hash}
|
||||||
|
|
||||||
|
def ensure_member(actor_id: uuid.UUID, chat_id: uuid.UUID):
|
||||||
|
with db() as conn:
|
||||||
|
row = conn.execute("SELECT 1 FROM chats c JOIN chat_members m ON m.chat_id=c.id WHERE c.id=%s AND m.user_id=%s", (chat_id, actor_id)).fetchone()
|
||||||
|
if row is None:
|
||||||
|
fail(404, "not_found", "Object not found")
|
||||||
|
|
||||||
|
def parse_page_args(limit: int | None, cursor: str | None, resource: str):
|
||||||
|
actual_limit = 50 if limit is None else limit
|
||||||
|
if isinstance(actual_limit, bool) or actual_limit < 1 or actual_limit > 100:
|
||||||
|
fail(400, "invalid_request", "limit must be an integer from 1 to 100")
|
||||||
|
after = None
|
||||||
|
if cursor is not None:
|
||||||
|
if not cursor:
|
||||||
|
fail(400, "invalid_request", "Invalid cursor")
|
||||||
|
try:
|
||||||
|
decoded = base64.urlsafe_b64decode(cursor + "=" * (-len(cursor) % 4)).decode("utf-8")
|
||||||
|
obj = json.loads(decoded)
|
||||||
|
if obj.get("resource") != resource:
|
||||||
|
fail(400, "invalid_request", "Cursor belongs to another resource")
|
||||||
|
after = (datetime.fromisoformat(obj["created_at"].replace("Z", "+00:00")), uuid.UUID(obj["id"]))
|
||||||
|
except ApiError:
|
||||||
|
raise
|
||||||
|
except Exception:
|
||||||
|
fail(400, "invalid_request", "Invalid cursor")
|
||||||
|
return actual_limit, after
|
||||||
|
|
||||||
|
def make_cursor(resource: str, created_at: datetime, item_id: uuid.UUID) -> str:
|
||||||
|
raw = json.dumps({"resource": resource, "created_at": iso(created_at), "id": str(item_id)}, separators=(",", ":")).encode()
|
||||||
|
return base64.urlsafe_b64encode(raw).decode().rstrip("=")
|
||||||
|
|
||||||
|
def json_rows(cursor, query: str, params=()):
|
||||||
|
with db() as conn:
|
||||||
|
cur = conn.execute(query, params)
|
||||||
|
names = [d.name for d in cur.description]
|
||||||
|
return [dict(zip(names, row)) for row in cur.fetchall()]
|
||||||
|
|
||||||
|
@app.get("/health/live")
|
||||||
|
def live():
|
||||||
|
return {"status": "ok"}
|
||||||
|
|
||||||
|
@app.get("/health/ready")
|
||||||
|
def ready():
|
||||||
|
try:
|
||||||
|
with db() as conn:
|
||||||
|
conn.execute("SELECT 1")
|
||||||
|
session_store.ping()
|
||||||
|
s3_probe.head_object(Bucket=S3_BUCKET, Key=S3_KEY_PREFIX + ".ready") # S3 is critical: uploads/downloads are part of the mandatory path
|
||||||
|
return {"status": "ok"}
|
||||||
|
except (psycopg.Error, PoolTimeout, RedisError, BotoCoreError, ClientError):
|
||||||
|
fail(503, "unavailable", "Required storage is unavailable")
|
||||||
|
|
||||||
|
@app.post("/api/v1/users", status_code=201)
|
||||||
|
async def register(request: Request):
|
||||||
|
body = await parse_body(request, RegisterBody)
|
||||||
|
if not USERNAME_RE.fullmatch(body.username):
|
||||||
|
fail(400, "invalid_request", "Invalid username")
|
||||||
|
validate_unicode(body.display_name, "display_name")
|
||||||
|
validate_unicode(body.password, "password")
|
||||||
|
if not body.display_name.strip() or len(body.display_name) > 100:
|
||||||
|
fail(400, "invalid_request", "Invalid display_name")
|
||||||
|
if not 12 <= len(body.password) <= 128:
|
||||||
|
fail(400, "invalid_request", "Password must contain 12 to 128 Unicode code points")
|
||||||
|
user_id = uuid.uuid4()
|
||||||
|
password_hash = PASSWORDS.hash(body.password)
|
||||||
|
try:
|
||||||
|
with db() as conn:
|
||||||
|
row = conn.execute("INSERT INTO users(id,username,display_name,password_hash) VALUES (%s,%s,%s,%s) RETURNING id,username,display_name,created_at", (user_id, body.username, body.display_name, password_hash)).fetchone()
|
||||||
|
except psycopg.errors.UniqueViolation:
|
||||||
|
fail(409, "conflict", "Username already exists")
|
||||||
|
return user_dict(dict(zip(("id", "username", "display_name", "created_at"), row)))
|
||||||
|
|
||||||
|
@app.get("/api/v1/users/me")
|
||||||
|
def me(request: Request):
|
||||||
|
return current_user(request)["user"]
|
||||||
|
|
||||||
|
@app.post("/api/v1/auth/sessions", status_code=201)
|
||||||
|
async def create_session(request: Request):
|
||||||
|
if await request.body():
|
||||||
|
fail(400, "invalid_request", "Session creation does not accept a request body")
|
||||||
|
auth = request.headers.get("authorization", "")
|
||||||
|
if not auth[:6].lower() == "basic ":
|
||||||
|
fail(401, "unauthorized", "Valid Basic credentials required", challenge("Basic"))
|
||||||
|
try:
|
||||||
|
decoded = base64.b64decode(auth[6:], validate=True).decode("utf-8")
|
||||||
|
username, password = decoded.split(":", 1)
|
||||||
|
except Exception:
|
||||||
|
fail(401, "unauthorized", "Invalid credentials", challenge("Basic"))
|
||||||
|
try:
|
||||||
|
validate_unicode(password, "password")
|
||||||
|
except ApiError:
|
||||||
|
fail(401, "unauthorized", "Invalid credentials", challenge("Basic"))
|
||||||
|
if not password or len(password) < 12 or len(password) > 128:
|
||||||
|
fail(401, "unauthorized", "Invalid credentials", challenge("Basic"))
|
||||||
|
with db() as conn:
|
||||||
|
row = conn.execute("SELECT id,username,display_name,password_hash,created_at FROM users WHERE username=%s", (username,)).fetchone()
|
||||||
|
if row is None:
|
||||||
|
fail(401, "unauthorized", "Invalid credentials", challenge("Basic"))
|
||||||
|
try:
|
||||||
|
PASSWORDS.verify(row[3], password)
|
||||||
|
except (VerifyMismatchError, VerificationError, InvalidHashError):
|
||||||
|
fail(401, "unauthorized", "Invalid credentials", challenge("Basic"))
|
||||||
|
token = secrets.token_urlsafe(32)
|
||||||
|
token_hash = hashlib.sha256(token.encode()).hexdigest()
|
||||||
|
expires_at = now_utc() + timedelta(seconds=SESSION_TTL_SECONDS)
|
||||||
|
session_json = json.dumps({"user_id": str(row[0]), "expires_at": iso(expires_at)}, separators=(",", ":"))
|
||||||
|
# Redis TTL enforces expiry; AOF + named volume make active sessions survive stack restarts.
|
||||||
|
session_store.setex("session:" + token_hash, SESSION_TTL_SECONDS, session_json)
|
||||||
|
user = user_dict({"id": row[0], "username": row[1], "display_name": row[2], "created_at": row[4]})
|
||||||
|
return {"token": token, "token_type": "Bearer", "expires_at": iso(expires_at), "user": user}
|
||||||
|
|
||||||
|
@app.delete("/api/v1/auth/sessions/current", status_code=204)
|
||||||
|
def revoke_session(request: Request):
|
||||||
|
actor = current_user(request)
|
||||||
|
session_store.delete("session:" + actor["token_hash"])
|
||||||
|
return Response(status_code=204)
|
||||||
|
|
||||||
|
@app.post("/api/v1/chats", status_code=201)
|
||||||
|
async def create_chat(request: Request):
|
||||||
|
actor = current_user(request)
|
||||||
|
body = await parse_body(request, ChatBody)
|
||||||
|
validate_unicode(body.title, "title")
|
||||||
|
if not body.title.strip() or len(body.title) > 100:
|
||||||
|
fail(400, "invalid_request", "Invalid title")
|
||||||
|
ids = [parse_uuid(value, "member UUID") for value in body.member_ids]
|
||||||
|
if len(ids) != len(set(ids)) or actor["id"] not in ids:
|
||||||
|
fail(400, "invalid_request", "Members must be unique and include the caller")
|
||||||
|
chat_id = uuid.uuid4()
|
||||||
|
try:
|
||||||
|
with db() as conn:
|
||||||
|
existing = conn.execute("SELECT id FROM users WHERE id = ANY(%s)", (ids,)).fetchall()
|
||||||
|
if len(existing) != len(ids):
|
||||||
|
fail(400, "invalid_request", "Unknown chat member")
|
||||||
|
row = conn.execute("INSERT INTO chats(id,title) VALUES (%s,%s) RETURNING id,title,created_at", (chat_id, body.title)).fetchone()
|
||||||
|
# psycopg 3: executemany exists on cursors, not on Connection (fix of a latent lab3 bug)
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.executemany("INSERT INTO chat_members(chat_id,user_id,position) VALUES (%s,%s,%s)", [(chat_id, uid, position) for position, uid in enumerate(ids)])
|
||||||
|
except psycopg.errors.ForeignKeyViolation:
|
||||||
|
fail(400, "invalid_request", "Unknown chat member")
|
||||||
|
return chat_dict({"id": row[0], "title": row[1], "created_at": row[2]}, [str(x) for x in ids])
|
||||||
|
|
||||||
|
@app.get("/api/v1/chats")
|
||||||
|
def list_chats(request: Request, limit: int | None = None, cursor: str | None = None):
|
||||||
|
actor = current_user(request)
|
||||||
|
chat_list_resource = "chats:" + str(actor["id"])
|
||||||
|
actual_limit, after = parse_page_args(limit, cursor, chat_list_resource)
|
||||||
|
params: list[Any] = [actor["id"]]
|
||||||
|
where = "m.user_id=%s"
|
||||||
|
if after:
|
||||||
|
where += " AND (c.created_at,c.id)>(%s,%s)"
|
||||||
|
params.extend(after)
|
||||||
|
params.append(actual_limit + 1)
|
||||||
|
rows = json_rows(None, f"SELECT c.id,c.title,c.created_at FROM chats c JOIN chat_members m ON m.chat_id=c.id WHERE {where} ORDER BY c.created_at,c.id LIMIT %s", tuple(params))
|
||||||
|
more = len(rows) > actual_limit
|
||||||
|
rows = rows[:actual_limit]
|
||||||
|
items = []
|
||||||
|
for row in rows:
|
||||||
|
members = json_rows(None, "SELECT user_id FROM chat_members WHERE chat_id=%s ORDER BY position,user_id", (row["id"],))
|
||||||
|
items.append(chat_dict(row, [str(m["user_id"]) for m in members]))
|
||||||
|
next_cursor = make_cursor(chat_list_resource, rows[-1]["created_at"], rows[-1]["id"]) if more and rows else None
|
||||||
|
return {"items": items, "next_cursor": next_cursor}
|
||||||
|
|
||||||
|
@app.get("/api/v1/chats/{chat_id}")
|
||||||
|
def get_chat(chat_id: str, request: Request):
|
||||||
|
actor = current_user(request)
|
||||||
|
parsed = parse_uuid(chat_id, "chat UUID")
|
||||||
|
ensure_member(actor["id"], parsed)
|
||||||
|
rows = json_rows(None, "SELECT id,title,created_at FROM chats WHERE id=%s", (parsed,))
|
||||||
|
members = json_rows(None, "SELECT user_id FROM chat_members WHERE chat_id=%s ORDER BY position,user_id", (parsed,))
|
||||||
|
return chat_dict(rows[0], [str(m["user_id"]) for m in members])
|
||||||
|
|
||||||
|
@app.post("/api/v1/chats/{chat_id}/messages", status_code=201)
|
||||||
|
async def create_message(chat_id: str, request: Request):
|
||||||
|
actor = current_user(request)
|
||||||
|
parsed_chat = parse_uuid(chat_id, "chat UUID")
|
||||||
|
ensure_member(actor["id"], parsed_chat)
|
||||||
|
body = await parse_body(request, MessageBody)
|
||||||
|
validate_unicode(body.text, "text")
|
||||||
|
attachment_ids = [parse_uuid(value, "attachment UUID") for value in body.attachment_ids]
|
||||||
|
if len(attachment_ids) != len(set(attachment_ids)):
|
||||||
|
fail(400, "invalid_request", "attachment_ids must be unique")
|
||||||
|
if len(body.text) > 4000 or (not body.text.strip() and not attachment_ids):
|
||||||
|
fail(400, "invalid_request", "Message needs 1 to 4000 non-whitespace code points of text or at least one attachment")
|
||||||
|
|
||||||
|
key = request.headers.get("Idempotency-Key", "")
|
||||||
|
if not IDEMPOTENCY_KEY_RE.fullmatch(key):
|
||||||
|
fail(400, "invalid_request", "Idempotency-Key must contain 1 to 128 ASCII letters, digits, '.', '_', ':', or '-'")
|
||||||
|
|
||||||
|
# Hash semantic JSON rather than raw bytes, so insignificant JSON formatting does not matter.
|
||||||
|
# attachment_ids is part of the hash only when non-empty: absent == [] and lab3 records stay valid.
|
||||||
|
canonical = {"text": body.text}
|
||||||
|
if attachment_ids:
|
||||||
|
canonical["attachment_ids"] = [str(x) for x in attachment_ids]
|
||||||
|
canonical_body = json.dumps(canonical, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
|
||||||
|
request_hash = hashlib.sha256(canonical_body.encode("utf-8")).hexdigest()
|
||||||
|
scope = f"{actor['id']}|POST|/api/v1/chats/{parsed_chat}/messages|{key}"
|
||||||
|
|
||||||
|
# The advisory transaction lock serializes the same key across every API replica.
|
||||||
|
# Message and idempotency result are committed atomically in the same PostgreSQL transaction.
|
||||||
|
with db() as conn:
|
||||||
|
conn.execute("SELECT pg_advisory_xact_lock(hashtextextended(%s, 0))", (scope,))
|
||||||
|
existing = conn.execute(
|
||||||
|
"SELECT request_hash, response_body FROM idempotency_records "
|
||||||
|
"WHERE user_id=%s AND chat_id=%s AND idempotency_key=%s AND created_at >= now() - interval '24 hours'",
|
||||||
|
(actor["id"], parsed_chat, key),
|
||||||
|
).fetchone()
|
||||||
|
if existing is not None:
|
||||||
|
if existing[0] != request_hash:
|
||||||
|
fail(409, "conflict", "Idempotency-Key was already used with a different request body")
|
||||||
|
replay = dict(existing[1])
|
||||||
|
replay.setdefault("attachment_ids", [])
|
||||||
|
return JSONResponse(status_code=201, content=replay)
|
||||||
|
|
||||||
|
# Keys are guaranteed for 24 hours; expired records may be replaced by a new attempt.
|
||||||
|
conn.execute(
|
||||||
|
"DELETE FROM idempotency_records WHERE user_id=%s AND chat_id=%s AND idempotency_key=%s AND created_at < now() - interval '24 hours'",
|
||||||
|
(actor["id"], parsed_chat, key),
|
||||||
|
)
|
||||||
|
if attachment_ids:
|
||||||
|
# Foreign, unknown and other-chat attachments are indistinguishable: all -> 404.
|
||||||
|
found = conn.execute("SELECT count(*) FROM attachments WHERE id = ANY(%s) AND chat_id=%s", (attachment_ids, parsed_chat)).fetchone()[0]
|
||||||
|
if found != len(attachment_ids):
|
||||||
|
fail(404, "not_found", "Object not found")
|
||||||
|
# lab5: queued/processing attachments may be attached, failed ones may not
|
||||||
|
if conn.execute("SELECT 1 FROM attachments WHERE id = ANY(%s) AND status='failed' LIMIT 1", (attachment_ids,)).fetchone():
|
||||||
|
fail(409, "conflict", "Attachment processing failed")
|
||||||
|
row = conn.execute(
|
||||||
|
"INSERT INTO messages(id,chat_id,sender_id,text) VALUES (%s,%s,%s,%s) "
|
||||||
|
"RETURNING id,chat_id,sender_id,text,created_at",
|
||||||
|
(uuid.uuid4(), parsed_chat, actor["id"], body.text),
|
||||||
|
).fetchone()
|
||||||
|
result = message_dict(dict(zip(("id", "chat_id", "sender_id", "text", "created_at"), row)), [str(x) for x in attachment_ids])
|
||||||
|
if attachment_ids:
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.executemany("INSERT INTO message_attachments(message_id,attachment_id,position) VALUES (%s,%s,%s)", [(row[0], aid, i) for i, aid in enumerate(attachment_ids)])
|
||||||
|
conn.execute(
|
||||||
|
"INSERT INTO idempotency_records(user_id,chat_id,idempotency_key,request_hash,response_body) "
|
||||||
|
"VALUES (%s,%s,%s,%s,%s::jsonb)",
|
||||||
|
(actor["id"], parsed_chat, key, request_hash, json.dumps(result, ensure_ascii=False)),
|
||||||
|
)
|
||||||
|
return result
|
||||||
|
|
||||||
|
@app.get("/api/v1/chats/{chat_id}/messages")
|
||||||
|
def list_messages(chat_id: str, request: Request, limit: int | None = None, cursor: str | None = None):
|
||||||
|
actor = current_user(request)
|
||||||
|
parsed_chat = parse_uuid(chat_id, "chat UUID")
|
||||||
|
ensure_member(actor["id"], parsed_chat)
|
||||||
|
actual_limit, after = parse_page_args(limit, cursor, "messages:" + str(parsed_chat))
|
||||||
|
params: list[Any] = [parsed_chat]
|
||||||
|
where = "chat_id=%s"
|
||||||
|
if after:
|
||||||
|
where += " AND (created_at,id)>(%s,%s)"
|
||||||
|
params.extend(after)
|
||||||
|
params.append(actual_limit + 1)
|
||||||
|
rows = json_rows(None, f"SELECT id,chat_id,sender_id,text,created_at FROM messages WHERE {where} ORDER BY created_at,id LIMIT %s", tuple(params))
|
||||||
|
more = len(rows) > actual_limit
|
||||||
|
rows = rows[:actual_limit]
|
||||||
|
links: dict[Any, list[str]] = {}
|
||||||
|
if rows:
|
||||||
|
link_rows = json_rows(None, "SELECT message_id, attachment_id FROM message_attachments WHERE message_id = ANY(%s) ORDER BY message_id, position", ([r["id"] for r in rows],))
|
||||||
|
for link in link_rows:
|
||||||
|
links.setdefault(link["message_id"], []).append(str(link["attachment_id"]))
|
||||||
|
items = [message_dict(row, links.get(row["id"])) for row in rows]
|
||||||
|
next_cursor = make_cursor("messages:" + str(parsed_chat), rows[-1]["created_at"], rows[-1]["id"]) if more and rows else None
|
||||||
|
return {"items": items, "next_cursor": next_cursor}
|
||||||
|
|
||||||
|
# ======================= Lab 4: attachments =======================
|
||||||
|
|
||||||
|
def attachment_dict(row: dict[str, Any], variants: list[dict[str, Any]] | None = None) -> dict[str, Any]:
|
||||||
|
return {"id": str(row["id"]), "chat_id": str(row["chat_id"]), "owner_id": str(row["owner_id"]), "filename": row["filename"],
|
||||||
|
"content_type": row["content_type"], "size_bytes": row["size_bytes"], "sha256": row["sha256"], "status": row["status"],
|
||||||
|
"variants": variants or [], "error_code": row["error_code"] if row["status"] == "failed" else None,
|
||||||
|
"created_at": iso(row["created_at"]), "updated_at": iso(row["updated_at"])}
|
||||||
|
|
||||||
|
def validate_filename(values: list[str]) -> str:
|
||||||
|
"""filename is display-only: it never reaches a path or an S3 key, but dangerous values are still rejected."""
|
||||||
|
if len(values) != 1:
|
||||||
|
fail(400, "invalid_request", "Exactly one filename query parameter is required")
|
||||||
|
name = values[0]
|
||||||
|
validate_unicode(name, "filename")
|
||||||
|
if not 1 <= len(name) <= 255 or not name.strip() or name in (".", ".."):
|
||||||
|
fail(400, "invalid_request", "Invalid filename")
|
||||||
|
if "/" in name or "\\" in name or any(unicodedata.category(ch) == "Cc" for ch in name):
|
||||||
|
fail(400, "invalid_request", "Invalid filename")
|
||||||
|
return name
|
||||||
|
|
||||||
|
def sniff_image_type(head: bytes) -> str | None:
|
||||||
|
if head.startswith(b"\x89PNG\r\n\x1a\n"):
|
||||||
|
return "image/png"
|
||||||
|
if head.startswith(b"\xff\xd8\xff"):
|
||||||
|
return "image/jpeg"
|
||||||
|
return None
|
||||||
|
|
||||||
|
async def read_body_to_spool(request: Request, declared: str):
|
||||||
|
"""Stream the request body into a bounded spool while hashing; abort as soon as a limit is broken.
|
||||||
|
|
||||||
|
RAM per upload stays ~1 MiB: bodies above SPOOL_MEMORY_BYTES roll over to an anonymous (already unlinked) temp file.
|
||||||
|
"""
|
||||||
|
spool = tempfile.SpooledTemporaryFile(max_size=SPOOL_MEMORY_BYTES)
|
||||||
|
digest, total, head = hashlib.sha256(), 0, b""
|
||||||
|
try:
|
||||||
|
iterator = request.stream().__aiter__()
|
||||||
|
with anyio.fail_after(UPLOAD_TOTAL_TIMEOUT_SECONDS):
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
chunk = await asyncio.wait_for(iterator.__anext__(), UPLOAD_IDLE_TIMEOUT_SECONDS)
|
||||||
|
except StopAsyncIteration:
|
||||||
|
break
|
||||||
|
if not chunk:
|
||||||
|
continue
|
||||||
|
total += len(chunk)
|
||||||
|
if total > MAX_UPLOAD_BYTES:
|
||||||
|
fail(413, "payload_too_large", "Attachment exceeds the size limit")
|
||||||
|
if len(head) < 8:
|
||||||
|
head += chunk[: 8 - len(head)]
|
||||||
|
if len(head) >= 8 and sniff_image_type(head) != declared:
|
||||||
|
fail(415, "unsupported_media_type", "Content does not match the declared image type")
|
||||||
|
digest.update(chunk)
|
||||||
|
spool.write(chunk)
|
||||||
|
except ApiError:
|
||||||
|
spool.close()
|
||||||
|
raise
|
||||||
|
except (TimeoutError, asyncio.TimeoutError):
|
||||||
|
spool.close()
|
||||||
|
fail(400, "invalid_request", "Upload timed out")
|
||||||
|
except BaseException: # ClientDisconnect, cancellation: nothing was written to S3 yet
|
||||||
|
spool.close()
|
||||||
|
raise
|
||||||
|
if total == 0 or sniff_image_type(head) != declared:
|
||||||
|
spool.close()
|
||||||
|
fail(415, "unsupported_media_type", "Content is not a supported image")
|
||||||
|
return spool, total, digest.hexdigest()
|
||||||
|
|
||||||
|
def decode_image_checked(spool, declared: str) -> None:
|
||||||
|
"""Runs in a worker thread: header -> pixel limit -> full decode. Raises ApiError for every user mistake."""
|
||||||
|
try:
|
||||||
|
spool.seek(0)
|
||||||
|
with Image.open(spool) as img:
|
||||||
|
if IMAGE_TYPES.get(img.format or "") != declared:
|
||||||
|
fail(415, "unsupported_media_type", "Content does not match the declared image type")
|
||||||
|
width, height = img.size
|
||||||
|
if width < 1 or height < 1:
|
||||||
|
fail(415, "unsupported_media_type", "Corrupt image")
|
||||||
|
if width * height > MAX_IMAGE_PIXELS: # decided from the header, before any pixel buffer exists
|
||||||
|
fail(413, "payload_too_large", "Image has too many pixels")
|
||||||
|
img.load() # full decode detects truncated/corrupt data
|
||||||
|
except ApiError:
|
||||||
|
raise
|
||||||
|
except Image.DecompressionBombError:
|
||||||
|
fail(413, "payload_too_large", "Image has too many pixels")
|
||||||
|
except MemoryError:
|
||||||
|
fail(503, "unavailable", "Temporarily unable to process the image")
|
||||||
|
except Exception: # Pillow raises OSError/SyntaxError/ValueError/struct.error/... for corrupt input
|
||||||
|
fail(415, "unsupported_media_type", "Corrupt or unsupported image")
|
||||||
|
finally:
|
||||||
|
spool.seek(0)
|
||||||
|
|
||||||
|
def s3_put(key: str, spool, size: int, content_type: str) -> None:
|
||||||
|
spool.seek(0)
|
||||||
|
s3.put_object(Bucket=S3_BUCKET, Key=key, Body=spool, ContentLength=size, ContentType=content_type)
|
||||||
|
|
||||||
|
def s3_delete_quietly(key: str) -> None:
|
||||||
|
try:
|
||||||
|
s3.delete_object(Bucket=S3_BUCKET, Key=key)
|
||||||
|
except Exception:
|
||||||
|
# The periodic sweeper (scripts/s3_sweep.py) removes it after the grace period.
|
||||||
|
log.warning("Compensation delete failed; orphan left for sweeper key=%s", key)
|
||||||
|
|
||||||
|
async def compensate(key: str) -> None:
|
||||||
|
with anyio.CancelScope(shield=True):
|
||||||
|
await run_in_threadpool(s3_delete_quietly, key)
|
||||||
|
|
||||||
|
@app.post("/api/v1/chats/{chat_id}/attachments", status_code=202)
|
||||||
|
async def upload_attachment(chat_id: str, request: Request):
|
||||||
|
actor = current_user(request)
|
||||||
|
parsed_chat = parse_uuid(chat_id, "chat UUID")
|
||||||
|
ensure_member(actor["id"], parsed_chat)
|
||||||
|
filename = validate_filename(request.query_params.getlist("filename"))
|
||||||
|
declared = request.headers.get("content-type", "").split(";", 1)[0].strip().lower()
|
||||||
|
if declared not in ("image/png", "image/jpeg"):
|
||||||
|
fail(415, "unsupported_media_type", "Content-Type must be image/png or image/jpeg")
|
||||||
|
length = request.headers.get("content-length")
|
||||||
|
if length is not None:
|
||||||
|
if not length.isascii() or not length.isdigit():
|
||||||
|
fail(400, "invalid_request", "Invalid Content-Length")
|
||||||
|
if int(length) > MAX_UPLOAD_BYTES: # reject before reading a single body byte
|
||||||
|
fail(413, "payload_too_large", "Attachment exceeds the size limit")
|
||||||
|
spool, size, sha256 = await read_body_to_spool(request, declared)
|
||||||
|
key = STAGING_PREFIX + secrets.token_hex(16) # server-chosen random STAGING key, unrelated to filename or attachment id
|
||||||
|
try:
|
||||||
|
await anyio.to_thread.run_sync(decode_image_checked, spool, declared, limiter=decode_limiter)
|
||||||
|
await run_in_threadpool(s3_put, key, spool, size, declared)
|
||||||
|
except ApiError: # user error found by validation, before any S3 write
|
||||||
|
spool.close()
|
||||||
|
raise
|
||||||
|
except BaseException:
|
||||||
|
spool.close()
|
||||||
|
await compensate(key) # a PUT whose reply was lost may still have stored the object
|
||||||
|
raise
|
||||||
|
spool.close()
|
||||||
|
attachment_id, job_id = uuid.uuid4(), uuid.uuid4()
|
||||||
|
final_original = jobs.final_key(attachment_id, jobs.PIPELINE_VERSION, "original")
|
||||||
|
try:
|
||||||
|
# ONE transaction: metadata + job + outbox row. 202 is returned only after this COMMIT (durable handoff).
|
||||||
|
with db() as conn:
|
||||||
|
row = conn.execute(
|
||||||
|
"INSERT INTO attachments(id,chat_id,owner_id,filename,content_type,size_bytes,sha256,s3_key,staging_key,status,pipeline_version) "
|
||||||
|
"VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,'queued',%s) "
|
||||||
|
"RETURNING id,chat_id,owner_id,filename,content_type,size_bytes,sha256,status,error_code,created_at,updated_at",
|
||||||
|
(attachment_id, parsed_chat, actor["id"], filename, declared, size, sha256, final_original, key, jobs.PIPELINE_VERSION)).fetchone()
|
||||||
|
conn.execute("INSERT INTO attachment_jobs(job_id,attachment_id,pipeline_version,status,max_attempts) VALUES (%s,%s,%s,'queued',%s)",
|
||||||
|
(job_id, attachment_id, jobs.PIPELINE_VERSION, jobs.JOB_MAX_ATTEMPTS))
|
||||||
|
jobs.add_event(conn, job_id, "accepted")
|
||||||
|
jobs.enqueue(conn, job_id, attachment_id, jobs.PIPELINE_VERSION, 1)
|
||||||
|
except BaseException:
|
||||||
|
await compensate(key) # staging PUT succeeded but the COMMIT did not: remove the unaccounted object
|
||||||
|
raise
|
||||||
|
cols = ("id", "chat_id", "owner_id", "filename", "content_type", "size_bytes", "sha256", "status", "error_code", "created_at", "updated_at")
|
||||||
|
return JSONResponse(status_code=202, content=attachment_dict(dict(zip(cols, row))))
|
||||||
|
|
||||||
|
ATT_COLS = ("id", "chat_id", "owner_id", "filename", "content_type", "size_bytes", "sha256", "s3_key", "status", "error_code", "created_at", "updated_at")
|
||||||
|
|
||||||
|
def load_attachment(attachment_id: str, request: Request) -> dict[str, Any]:
|
||||||
|
actor = current_user(request)
|
||||||
|
parsed = parse_uuid(attachment_id, "attachment UUID")
|
||||||
|
rows = json_rows(None, "SELECT a.id,a.chat_id,a.owner_id,a.filename,a.content_type,a.size_bytes,a.sha256,a.s3_key,a.status,a.error_code,a.created_at,a.updated_at "
|
||||||
|
"FROM attachments a JOIN chat_members m ON m.chat_id=a.chat_id WHERE a.id=%s AND m.user_id=%s", (parsed, actor["id"]))
|
||||||
|
if not rows:
|
||||||
|
fail(404, "not_found", "Object not found")
|
||||||
|
return rows[0]
|
||||||
|
|
||||||
|
def variant_rows(attachment_id, pipeline_version: int = jobs.PIPELINE_VERSION) -> list[dict[str, Any]]:
|
||||||
|
rows = json_rows(None, "SELECT name,s3_key,content_type,size_bytes,width,height,sha256 FROM attachment_variants WHERE attachment_id=%s AND pipeline_version=%s", (attachment_id, pipeline_version))
|
||||||
|
return sorted(rows, key=lambda r: jobs.VARIANT_ORDER.index(r["name"]))
|
||||||
|
|
||||||
|
@app.get("/api/v1/attachments/{attachment_id}")
|
||||||
|
def get_attachment(attachment_id: str, request: Request):
|
||||||
|
row = load_attachment(attachment_id, request)
|
||||||
|
variants = []
|
||||||
|
if row["status"] == "ready": # variants are published only together with ready (one COMMIT in the worker)
|
||||||
|
variants = [{k: v[k] for k in ("name", "content_type", "size_bytes", "width", "height", "sha256")} for v in variant_rows(row["id"])]
|
||||||
|
return attachment_dict(row, variants)
|
||||||
|
|
||||||
|
@app.get("/api/v1/attachments/{attachment_id}/content")
|
||||||
|
def download_attachment(attachment_id: str, request: Request, variant: str = "original"):
|
||||||
|
row = load_attachment(attachment_id, request)
|
||||||
|
if variant != "original" and variant not in jobs.VARIANT_ORDER:
|
||||||
|
fail(400, "invalid_request", "Unknown variant")
|
||||||
|
if row["status"] != "ready":
|
||||||
|
fail(409, "conflict", "Attachment is not ready") # queued, processing and failed: no content
|
||||||
|
key, content_type, size = row["s3_key"], row["content_type"], row["size_bytes"]
|
||||||
|
if variant != "original":
|
||||||
|
found = [v for v in variant_rows(row["id"]) if v["name"] == variant]
|
||||||
|
if not found:
|
||||||
|
fail(404, "not_found", "Object not found")
|
||||||
|
key, content_type, size = found[0]["s3_key"], found[0]["content_type"], found[0]["size_bytes"]
|
||||||
|
try:
|
||||||
|
obj = s3.get_object(Bucket=S3_BUCKET, Key=key)
|
||||||
|
except ClientError as exc:
|
||||||
|
if exc.response.get("Error", {}).get("Code") in ("NoSuchKey", "404"):
|
||||||
|
log.error("INTEGRITY: metadata without object attachment=%s variant=%s", row["id"], variant)
|
||||||
|
fail(500, "internal_error", "Internal server error")
|
||||||
|
raise
|
||||||
|
body = obj["Body"]
|
||||||
|
async def stream():
|
||||||
|
try:
|
||||||
|
async for chunk in iterate_in_threadpool(body.iter_chunks(CHUNK_BYTES)):
|
||||||
|
yield chunk
|
||||||
|
finally:
|
||||||
|
body.close()
|
||||||
|
headers = {"Content-Length": str(size), "X-Content-Type-Options": "nosniff", "Cache-Control": "private, no-store",
|
||||||
|
"Content-Disposition": "inline; filename*=UTF-8''" + quote(row["filename"], safe="")}
|
||||||
|
return StreamingResponse(stream(), media_type=content_type, headers=headers)
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
import uvicorn
|
||||||
|
uvicorn.run("main:app", host=os.getenv("APP_HOST", "0.0.0.0"), port=int(os.getenv("APP_PORT", "8080")), timeout_graceful_shutdown=5)
|
||||||
+230
@@ -0,0 +1,230 @@
|
|||||||
|
"""Lab 5 worker: Kafka consumer -> claim (lease) in PostgreSQL -> staging -> pipeline -> final S3 + COMMIT -> offset commit.
|
||||||
|
|
||||||
|
PostgreSQL is the source of truth for job state; Kafka is the at-least-once transport. The Kafka offset is committed
|
||||||
|
only AFTER the durable effect (final objects + COMMIT of the result) or after a durable decision (retry scheduled / failed).
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import signal
|
||||||
|
import socket
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import time
|
||||||
|
import uuid
|
||||||
|
from concurrent.futures import ThreadPoolExecutor
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import psycopg
|
||||||
|
from botocore.exceptions import BotoCoreError, ClientError
|
||||||
|
from confluent_kafka import Consumer, TopicPartition
|
||||||
|
|
||||||
|
from . import jobs
|
||||||
|
|
||||||
|
logging.basicConfig(level=os.getenv("LOG_LEVEL", "INFO"), format="%(asctime)s %(levelname)s worker %(message)s")
|
||||||
|
log = logging.getLogger("worker")
|
||||||
|
WORKER_ID = os.getenv("INSTANCE_ID") or socket.gethostname()
|
||||||
|
CONCURRENCY = int(os.getenv("WORKER_CONCURRENCY", "2"))
|
||||||
|
FAULT_EXIT_AFTER_S3 = os.getenv("FAULT_WORKER_EXIT_AFTER_S3", "0") == "1" # test hook: crash after the S3 write, before COMMIT/ack
|
||||||
|
IMG_MEMORY_MB = os.getenv("IMG_MEMORY_MB", "1024")
|
||||||
|
CODE_BY_EXIT = {2: "invalid_image", 3: "image_too_large", 4: "image_too_complex"} # permanent errors of imgproc
|
||||||
|
STOP = False
|
||||||
|
|
||||||
|
|
||||||
|
class Permanent(Exception):
|
||||||
|
def __init__(self, code: str):
|
||||||
|
super().__init__(code)
|
||||||
|
self.code = code
|
||||||
|
|
||||||
|
|
||||||
|
class Transient(Exception):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def connect():
|
||||||
|
return psycopg.connect(connect_timeout=5, options="-c statement_timeout=10000")
|
||||||
|
|
||||||
|
|
||||||
|
def parse_job(raw: bytes) -> dict | None:
|
||||||
|
"""Validate against jobs.schema.json (the subset that matters). Poison messages are logged and skipped."""
|
||||||
|
try:
|
||||||
|
m = json.loads(raw)
|
||||||
|
assert m["schema_version"] == 1 and m["type"] == jobs.JOB_TYPE
|
||||||
|
uuid.UUID(m["job_id"]); uuid.UUID(m["attachment_id"])
|
||||||
|
assert isinstance(m["pipeline_version"], int) and m["pipeline_version"] >= 1
|
||||||
|
return m
|
||||||
|
except Exception:
|
||||||
|
log.error("poison message skipped (schema violation), size=%d", len(raw))
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def claim(msg: dict) -> dict | None:
|
||||||
|
"""queued -> processing with a lease. Returns None when this delivery must be a no-op (ack)."""
|
||||||
|
with connect() as conn:
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT status, attempt, max_attempts, attachment_id, pipeline_version, "
|
||||||
|
"(next_attempt_at <= now()) AS due, (lease_until IS NOT NULL AND lease_until < now()) AS lease_expired "
|
||||||
|
"FROM attachment_jobs WHERE job_id=%s FOR UPDATE", (msg["job_id"],)).fetchone()
|
||||||
|
if row is None:
|
||||||
|
log.warning("job %s unknown, skipped", msg["job_id"]); return None
|
||||||
|
status, attempt, max_attempts, att_id, pv, due, expired = row
|
||||||
|
if pv != msg["pipeline_version"]:
|
||||||
|
return None # stale job of another pipeline version never overwrites the newer one
|
||||||
|
if status in ("ready", "failed"):
|
||||||
|
log.info("job %s already %s: duplicate delivery is a no-op", msg["job_id"], status); return None
|
||||||
|
if status == "queued" and not due:
|
||||||
|
return None # early duplicate: the delayed outbox row will deliver it later
|
||||||
|
if status == "processing" and not expired:
|
||||||
|
log.info("job %s is leased by another worker: skip (dispatcher recovers it if that worker dies)", msg["job_id"]); return None
|
||||||
|
if attempt >= max_attempts: # lease expired and no attempts left
|
||||||
|
jobs.fail_job(conn, msg["job_id"], att_id, "retry_exhausted", attempt)
|
||||||
|
return None
|
||||||
|
attempt += 1
|
||||||
|
conn.execute("UPDATE attachment_jobs SET status='processing', attempt=%s, worker_id=%s, "
|
||||||
|
"lease_until=now()+make_interval(secs=>%s), updated_at=now() WHERE job_id=%s",
|
||||||
|
(attempt, WORKER_ID, jobs.LEASE_SECONDS, msg["job_id"]))
|
||||||
|
conn.execute("UPDATE attachments SET status='processing', updated_at=now() WHERE id=%s", (att_id,))
|
||||||
|
jobs.add_event(conn, msg["job_id"], "claimed", f"attempt={attempt} worker={WORKER_ID}")
|
||||||
|
a = conn.execute("SELECT sha256, staging_key, s3_key, content_type, filename FROM attachments WHERE id=%s", (att_id,)).fetchone()
|
||||||
|
return {"job_id": msg["job_id"], "attachment_id": att_id, "pv": pv, "attempt": attempt, "max_attempts": max_attempts,
|
||||||
|
"sha256": a[0].strip(), "staging_key": a[1], "original_key": a[2], "content_type": a[3]}
|
||||||
|
|
||||||
|
|
||||||
|
def run_pipeline(s3, c: dict) -> dict:
|
||||||
|
"""Download staging -> verify -> child process with limits -> idempotent PUTs of deterministic keys."""
|
||||||
|
if not c["staging_key"]:
|
||||||
|
raise Permanent("staging_missing")
|
||||||
|
with tempfile.TemporaryDirectory(prefix="job-") as tmp:
|
||||||
|
src, out = Path(tmp) / "in.bin", Path(tmp) / "out"
|
||||||
|
out.mkdir()
|
||||||
|
try:
|
||||||
|
s3.download_file(jobs.S3_BUCKET, c["staging_key"], str(src))
|
||||||
|
except ClientError as e:
|
||||||
|
if e.response.get("Error", {}).get("Code") in ("NoSuchKey", "404"):
|
||||||
|
raise Permanent("staging_missing")
|
||||||
|
raise
|
||||||
|
if hashlib.sha256(src.read_bytes()).hexdigest() != c["sha256"]:
|
||||||
|
raise Permanent("integrity_error") # bytes differ from what the API accepted
|
||||||
|
try:
|
||||||
|
r = subprocess.run([sys.executable, "-m", "src.imgproc", str(src), str(out)], capture_output=True, text=True,
|
||||||
|
timeout=jobs.JOB_TIMEOUT_SECONDS, env={"PATH": os.environ.get("PATH", ""), "IMG_MEMORY_MB": IMG_MEMORY_MB,
|
||||||
|
"IMG_CPU_SECONDS": str(jobs.JOB_TIMEOUT_SECONDS),
|
||||||
|
"MAX_IMAGE_PIXELS": os.getenv("MAX_IMAGE_PIXELS", "40000000")}) # no secrets in the child
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
raise Transient("job timeout") # child is killed by subprocess.run
|
||||||
|
if r.returncode in CODE_BY_EXIT:
|
||||||
|
raise Permanent(CODE_BY_EXIT[r.returncode])
|
||||||
|
if r.returncode != 0:
|
||||||
|
raise Transient(f"imgproc exit {r.returncode}")
|
||||||
|
meta = json.loads(r.stdout)
|
||||||
|
files = {"thumbnail": "thumbnail.jpg", "crop": "crop.jpg", "optimized": "optimized.webp"}
|
||||||
|
# 1) original into the final prefix (same key every time => repeated PUT is harmless)
|
||||||
|
s3.upload_file(str(src), jobs.S3_BUCKET, c["original_key"], ExtraArgs={"ContentType": c["content_type"]})
|
||||||
|
# 2) variants
|
||||||
|
for name in jobs.VARIANT_ORDER:
|
||||||
|
key = jobs.final_key(c["attachment_id"], c["pv"], name)
|
||||||
|
s3.upload_file(str(out / files[name]), jobs.S3_BUCKET, key, ExtraArgs={"ContentType": meta[name]["content_type"]})
|
||||||
|
meta[name]["s3_key"] = key
|
||||||
|
return meta
|
||||||
|
|
||||||
|
|
||||||
|
def commit_ready(c: dict, meta: dict) -> bool:
|
||||||
|
"""Atomically publish the result. False if this attempt no longer owns the job (lease lost to a newer attempt)."""
|
||||||
|
with connect() as conn:
|
||||||
|
row = conn.execute("SELECT status, attempt FROM attachment_jobs WHERE job_id=%s FOR UPDATE", (c["job_id"],)).fetchone()
|
||||||
|
if row is None or row[0] != "processing" or row[1] != c["attempt"]:
|
||||||
|
log.warning("job %s: attempt %s lost ownership, result not committed", c["job_id"], c["attempt"]); return False
|
||||||
|
for name in jobs.VARIANT_ORDER:
|
||||||
|
v = meta[name]
|
||||||
|
conn.execute("INSERT INTO attachment_variants(attachment_id,pipeline_version,name,s3_key,content_type,size_bytes,width,height,sha256) "
|
||||||
|
"VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s) ON CONFLICT (attachment_id,pipeline_version,name) DO NOTHING",
|
||||||
|
(c["attachment_id"], c["pv"], name, v["s3_key"], v["content_type"], v["size_bytes"], v["width"], v["height"], v["sha256"]))
|
||||||
|
conn.execute("UPDATE attachment_jobs SET status='ready', error_code=NULL, lease_until=NULL, updated_at=now() WHERE job_id=%s", (c["job_id"],))
|
||||||
|
conn.execute("UPDATE attachments SET status='ready', error_code=NULL, staging_key=NULL, updated_at=now() WHERE id=%s", (c["attachment_id"],))
|
||||||
|
jobs.add_event(conn, c["job_id"], "ready", f"attempt={c['attempt']}")
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def record_failure(c: dict, exc: Exception) -> None:
|
||||||
|
"""Permanent -> failed in the PostgreSQL registry. Transient -> queued with backoff+jitter (delayed outbox row) or failed when attempts are exhausted."""
|
||||||
|
with connect() as conn:
|
||||||
|
row = conn.execute("SELECT status, attempt FROM attachment_jobs WHERE job_id=%s FOR UPDATE", (c["job_id"],)).fetchone()
|
||||||
|
if row is None or row[0] != "processing" or row[1] != c["attempt"]:
|
||||||
|
return
|
||||||
|
if isinstance(exc, Permanent):
|
||||||
|
jobs.fail_job(conn, c["job_id"], c["attachment_id"], exc.code, c["attempt"])
|
||||||
|
elif c["attempt"] >= c["max_attempts"]:
|
||||||
|
jobs.fail_job(conn, c["job_id"], c["attachment_id"], "retry_exhausted", c["attempt"])
|
||||||
|
else:
|
||||||
|
delay = jobs.backoff_seconds(c["attempt"])
|
||||||
|
conn.execute("UPDATE attachment_jobs SET status='queued', lease_until=NULL, next_attempt_at=now()+make_interval(secs=>%s), "
|
||||||
|
"error_code=NULL, updated_at=now() WHERE job_id=%s", (delay, c["job_id"]))
|
||||||
|
conn.execute("UPDATE attachments SET status='queued', updated_at=now() WHERE id=%s", (c["attachment_id"],))
|
||||||
|
jobs.add_event(conn, c["job_id"], "retry_scheduled", f"attempt={c['attempt']} delay={delay:.1f}s reason={type(exc).__name__}")
|
||||||
|
jobs.enqueue(conn, c["job_id"], c["attachment_id"], c["pv"], c["attempt"] + 1, delay_seconds=delay)
|
||||||
|
|
||||||
|
|
||||||
|
def handle(s3, raw: bytes) -> None:
|
||||||
|
msg = parse_job(raw)
|
||||||
|
if msg is None:
|
||||||
|
return
|
||||||
|
c = claim(msg)
|
||||||
|
if c is None:
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
meta = run_pipeline(s3, c)
|
||||||
|
if FAULT_EXIT_AFTER_S3 and c["attempt"] == 1:
|
||||||
|
log.error("FAULT: exiting after S3 write, before COMMIT and offset commit")
|
||||||
|
os._exit(137)
|
||||||
|
if commit_ready(c, meta):
|
||||||
|
try:
|
||||||
|
s3.delete_object(Bucket=jobs.S3_BUCKET, Key=c["staging_key"]) # staging cleanup after the safe COMMIT
|
||||||
|
except Exception:
|
||||||
|
log.warning("staging cleanup failed; sweeper will remove it")
|
||||||
|
except (Permanent, Transient, ClientError, BotoCoreError, OSError, json.JSONDecodeError) as exc:
|
||||||
|
log.warning("job %s attempt %s failed: %s", c["job_id"], c["attempt"], type(exc).__name__)
|
||||||
|
record_failure(c, exc)
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
def request_stop(*_):
|
||||||
|
global STOP
|
||||||
|
STOP = True
|
||||||
|
signal.signal(signal.SIGTERM, request_stop)
|
||||||
|
signal.signal(signal.SIGINT, request_stop)
|
||||||
|
s3 = jobs.make_s3()
|
||||||
|
consumer = Consumer({
|
||||||
|
"bootstrap.servers": jobs.KAFKA_BOOTSTRAP, "group.id": "attachment-workers",
|
||||||
|
"enable.auto.commit": False, "auto.offset.reset": "earliest", "isolation.level": "read_committed",
|
||||||
|
"max.poll.interval.ms": 600000, "session.timeout.ms": 30000,
|
||||||
|
"queued.max.messages.kbytes": 1024, "fetch.max.bytes": 1048576, # bounded client-side prefetch
|
||||||
|
})
|
||||||
|
consumer.subscribe([jobs.JOBS_TOPIC])
|
||||||
|
log.info("worker %s started, concurrency=%d, max_attempts=%d, lease=%ds", WORKER_ID, CONCURRENCY, jobs.JOB_MAX_ATTEMPTS, jobs.LEASE_SECONDS)
|
||||||
|
with ThreadPoolExecutor(max_workers=CONCURRENCY) as pool:
|
||||||
|
while not STOP:
|
||||||
|
msgs = consumer.consume(num_messages=CONCURRENCY, timeout=1.0) # at most CONCURRENCY jobs in flight
|
||||||
|
batch = []
|
||||||
|
for m in msgs:
|
||||||
|
if m.error():
|
||||||
|
log.warning("kafka error: %s", m.error()); continue
|
||||||
|
batch.append(m)
|
||||||
|
if not batch:
|
||||||
|
continue
|
||||||
|
futures = [pool.submit(handle, s3, m.value()) for m in batch]
|
||||||
|
for f in futures:
|
||||||
|
f.result() # unexpected exception => process exits, offsets stay uncommitted, Kafka redelivers
|
||||||
|
# ACK boundary: commit the highest processed offset+1 per partition, only after every job of the batch is durable
|
||||||
|
top = {}
|
||||||
|
for m in batch:
|
||||||
|
top[(m.topic(), m.partition())] = max(top.get((m.topic(), m.partition()), -1), m.offset() + 1)
|
||||||
|
consumer.commit(offsets=[TopicPartition(t, p, o) for (t, p), o in top.items()], asynchronous=False)
|
||||||
|
consumer.close()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,176 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Lab5 level-4 failure scenarios against a RUNNING compose stack (stdlib only; run from the repo root).
|
||||||
|
|
||||||
|
python3 student-tests/level4_failures.py all # or: worker_down | commit_publish_gap | crash_after_s3 | replay | s3_outage
|
||||||
|
|
||||||
|
Each scenario drives `docker compose` (never `down -v`: volumes stay) and prints PASS/FAIL with the observed facts.
|
||||||
|
"""
|
||||||
|
import base64, json, os, subprocess, sys, time, urllib.error, urllib.request, uuid
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
BASE = os.getenv("BASE_URL", "http://localhost:8080")
|
||||||
|
PNG = (Path(__file__).resolve().parents[1] / "tests/fixtures/sample.png").read_bytes()
|
||||||
|
DC = ["docker", "compose", "--env-file", ".env"]
|
||||||
|
|
||||||
|
|
||||||
|
def sh(*args, env=None, check=True):
|
||||||
|
r = subprocess.run(list(args), capture_output=True, text=True, env={**os.environ, **(env or {})})
|
||||||
|
if check and r.returncode:
|
||||||
|
raise SystemExit(f"command failed: {' '.join(args)}\n{r.stderr[-500:]}")
|
||||||
|
return r.stdout.strip()
|
||||||
|
|
||||||
|
|
||||||
|
def sql(query):
|
||||||
|
return sh(*DC, "exec", "-T", "postgres", "sh", "-c", f'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -At -c "{query}"')
|
||||||
|
|
||||||
|
|
||||||
|
def http(method, path, token=None, body=None, headers=None, raw=None):
|
||||||
|
h = dict(headers or {})
|
||||||
|
if token:
|
||||||
|
h["Authorization"] = "Bearer " + token
|
||||||
|
data = raw if raw is not None else (json.dumps(body).encode() if body is not None else None)
|
||||||
|
if body is not None:
|
||||||
|
h["Content-Type"] = "application/json"
|
||||||
|
req = urllib.request.Request(BASE + path, data=data, method=method, headers=h)
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(req, timeout=30) as r:
|
||||||
|
payload = r.read()
|
||||||
|
return r.status, (json.loads(payload) if payload and "json" in r.headers.get("Content-Type", "") else payload)
|
||||||
|
except urllib.error.HTTPError as e:
|
||||||
|
payload = e.read()
|
||||||
|
try:
|
||||||
|
return e.code, json.loads(payload)
|
||||||
|
except ValueError:
|
||||||
|
return e.code, payload
|
||||||
|
|
||||||
|
|
||||||
|
def setup():
|
||||||
|
name = "u" + uuid.uuid4().hex[:10]
|
||||||
|
pw = "correct-horse-battery"
|
||||||
|
st, user = http("POST", "/api/v1/users", body={"username": name, "display_name": "Tester", "password": pw})
|
||||||
|
assert st == 201, (st, user)
|
||||||
|
cred = base64.b64encode(f"{name}:{pw}".encode()).decode()
|
||||||
|
st, sess = http("POST", "/api/v1/auth/sessions", headers={"Authorization": "Basic " + cred})
|
||||||
|
token = sess["token"]
|
||||||
|
st, chat = http("POST", "/api/v1/chats", token, {"title": "lab5", "member_ids": [user["id"]]})
|
||||||
|
return token, chat["id"]
|
||||||
|
|
||||||
|
|
||||||
|
def upload(token, chat):
|
||||||
|
st, att = http("POST", f"/api/v1/chats/{chat}/attachments?filename=sample.png", token, raw=PNG, headers={"Content-Type": "image/png"})
|
||||||
|
assert st == 202 and att["status"] == "queued", (st, att)
|
||||||
|
return att["id"]
|
||||||
|
|
||||||
|
|
||||||
|
def wait(token, aid, want, timeout=90):
|
||||||
|
end, last = time.time() + timeout, None
|
||||||
|
while time.time() < end:
|
||||||
|
st, att = http("GET", f"/api/v1/attachments/{aid}", token)
|
||||||
|
last = att
|
||||||
|
if st == 200 and att["status"] == want:
|
||||||
|
return att
|
||||||
|
time.sleep(1)
|
||||||
|
raise AssertionError(f"attachment {aid} did not reach {want}; last={last}")
|
||||||
|
|
||||||
|
|
||||||
|
def wait_ready_api(timeout=90):
|
||||||
|
end = time.time() + timeout
|
||||||
|
while time.time() < end:
|
||||||
|
if http("GET", "/health/ready")[0] == 200:
|
||||||
|
return
|
||||||
|
time.sleep(1)
|
||||||
|
raise AssertionError("API not ready")
|
||||||
|
|
||||||
|
|
||||||
|
def check_variants(att):
|
||||||
|
v = {x["name"]: x for x in att["variants"]}
|
||||||
|
assert (v["thumbnail"]["width"], v["thumbnail"]["height"], v["thumbnail"]["content_type"]) == (256, 160, "image/jpeg"), v["thumbnail"]
|
||||||
|
assert (v["crop"]["width"], v["crop"]["height"]) == (128, 128), v["crop"]
|
||||||
|
assert v["optimized"]["content_type"] == "image/webp"
|
||||||
|
|
||||||
|
|
||||||
|
def events(aid):
|
||||||
|
return sql(f"SELECT string_agg(event, ',' ORDER BY id) FROM job_events e JOIN attachment_jobs j USING (job_id) WHERE j.attachment_id='{aid}'")
|
||||||
|
|
||||||
|
|
||||||
|
def worker_down():
|
||||||
|
token, chat = setup()
|
||||||
|
sh(*DC, "stop", "worker")
|
||||||
|
aid = upload(token, chat)
|
||||||
|
st, _ = http("GET", f"/api/v1/attachments/{aid}/content", token)
|
||||||
|
assert st == 409, st # content of a queued attachment
|
||||||
|
st, msg = http("POST", f"/api/v1/chats/{chat}/messages", token, {"text": "hi", "attachment_ids": [aid]}, {"Idempotency-Key": "k1"})
|
||||||
|
assert st == 201, (st, msg) # other features keep working; queued attachment can be attached
|
||||||
|
sh(*DC, "restart", "api"); wait_ready_api()
|
||||||
|
assert http("GET", f"/api/v1/attachments/{aid}", token)[1]["status"] == "queued" # accepted work survived the API restart
|
||||||
|
sh(*DC, "up", "-d", "worker")
|
||||||
|
att = wait(token, aid, "ready"); check_variants(att)
|
||||||
|
print("PASS worker_down:", aid, "events:", events(aid))
|
||||||
|
|
||||||
|
|
||||||
|
def commit_publish_gap():
|
||||||
|
token, chat = setup()
|
||||||
|
sh(*DC, "up", "-d", "dispatcher", env={"FAULT_DISPATCHER_EXIT_BEFORE_PUBLISH": "1"}, check=True)
|
||||||
|
sh(*DC, "exec", "-T", "dispatcher", "rm", "-f", "/tmp/dispatcher_fault_done", check=False)
|
||||||
|
sh(*DC, "restart", "dispatcher") # fresh container process: the fault fires once, on the first outbox batch
|
||||||
|
time.sleep(3)
|
||||||
|
aid = upload(token, chat) # COMMIT happens; the dispatcher dies before publishing
|
||||||
|
att = wait(token, aid, "ready", 90); check_variants(att)
|
||||||
|
sh(*DC, "up", "-d", "dispatcher") # back to the normal configuration (.env: fault off)
|
||||||
|
print("PASS commit_publish_gap:", aid, "events:", events(aid))
|
||||||
|
|
||||||
|
|
||||||
|
def crash_after_s3():
|
||||||
|
token, chat = setup()
|
||||||
|
env = {"FAULT_WORKER_EXIT_AFTER_S3": "1", "JOB_TIMEOUT_SECONDS": "10", "LEASE_GRACE_SECONDS": "5"}
|
||||||
|
sh(*DC, "up", "-d", "worker", env=env)
|
||||||
|
time.sleep(5)
|
||||||
|
aid = upload(token, chat)
|
||||||
|
att = wait(token, aid, "ready", 150); check_variants(att)
|
||||||
|
n = sql(f"SELECT count(*) FROM attachment_variants WHERE attachment_id='{aid}'")
|
||||||
|
assert n == "3", n # the retry completed the SAME result, no second set of variants
|
||||||
|
sh(*DC, "up", "-d", "worker") # restore normal worker config
|
||||||
|
print("PASS crash_after_s3:", aid, "events:", events(aid))
|
||||||
|
|
||||||
|
|
||||||
|
def replay():
|
||||||
|
token, chat = setup()
|
||||||
|
sh(*DC, "stop", "worker")
|
||||||
|
aid = upload(token, chat)
|
||||||
|
good = sql(f"SELECT sha256 FROM attachments WHERE id='{aid}'")
|
||||||
|
sql(f"UPDATE attachments SET sha256=repeat('0',64) WHERE id='{aid}'") # simulate corrupted/tampered staging bytes
|
||||||
|
sh(*DC, "up", "-d", "worker")
|
||||||
|
att = wait(token, aid, "failed", 60)
|
||||||
|
assert att["error_code"] == "integrity_error", att # permanent error: failed at once, no retries
|
||||||
|
st, _ = http("POST", f"/api/v1/chats/{chat}/messages", token, {"text": "", "attachment_ids": [aid]}, {"Idempotency-Key": "k2"})
|
||||||
|
assert st == 409, st
|
||||||
|
sql(f"UPDATE attachments SET sha256='{good}' WHERE id='{aid}'") # operator fixes the cause
|
||||||
|
job = sql(f"SELECT job_id FROM attachment_jobs WHERE attachment_id='{aid}'")
|
||||||
|
print(sh(*DC, "exec", "-T", "worker", "python", "scripts/replay_failed.py", "--job-id", job, "--reason", "student-test"))
|
||||||
|
att = wait(token, aid, "ready", 60); check_variants(att)
|
||||||
|
again = sh(*DC, "exec", "-T", "worker", "python", "scripts/replay_failed.py", "--job-id", job, check=False)
|
||||||
|
assert "REFUSED" in again, again # replay of a successful job is refused
|
||||||
|
print("PASS replay:", aid, "events:", events(aid))
|
||||||
|
|
||||||
|
|
||||||
|
def s3_outage():
|
||||||
|
token, chat = setup()
|
||||||
|
sh(*DC, "stop", "worker")
|
||||||
|
aid = upload(token, chat)
|
||||||
|
sh(*DC, "stop", "s3")
|
||||||
|
sh(*DC, "up", "-d", "--no-deps", "worker")
|
||||||
|
time.sleep(15)
|
||||||
|
ev = events(aid)
|
||||||
|
assert "retry_scheduled" in ev, ev # temporary error => backoff with jitter, not failed
|
||||||
|
sh(*DC, "start", "s3"); wait_ready_api()
|
||||||
|
att = wait(token, aid, "ready", 120); check_variants(att)
|
||||||
|
print("PASS s3_outage:", aid, "events:", events(aid))
|
||||||
|
|
||||||
|
|
||||||
|
SCENARIOS = {"worker_down": worker_down, "commit_publish_gap": commit_publish_gap, "crash_after_s3": crash_after_s3, "replay": replay, "s3_outage": s3_outage}
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
which = sys.argv[1] if len(sys.argv) > 1 else "all"
|
||||||
|
for name, fn in SCENARIOS.items():
|
||||||
|
if which in ("all", name):
|
||||||
|
fn()
|
||||||
Reference in New Issue
Block a user