lab 5 + kafka message broker

This commit is contained in:
2026-11-07 19:45:45 +03:00
parent 276dbdad8a
commit d26ef6e660
54 changed files with 7203 additions and 4817 deletions
+13
View File
@@ -0,0 +1,13 @@
FROM python:3.12.11-slim-bookworm
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 PIP_DISABLE_PIP_VERSION_CHECK=1
WORKDIR /app
RUN addgroup --system app && adduser --system --ingroup app app
COPY requirements.txt ./requirements.txt
RUN python -m pip install --no-cache-dir -r requirements.txt
COPY src ./src
COPY migrations ./migrations
COPY scripts ./scripts
USER app
EXPOSE 8080
STOPSIGNAL SIGTERM
CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8080", "--timeout-graceful-shutdown", "10", "--proxy-headers"]
+113
View File
@@ -0,0 +1,113 @@
# Отчёт - лабораторная 5
- Автор, группа: Алексей Миронов, БВТ2302
- Целевая оценка: **4** (трек 5 не заявляется)
- Commit/PR: TODO после публикации
- Источники, библиотеки, помощь ИИ: FastAPI 0.116.1, psycopg 3.2.9, boto3 1.40.20, Pillow 11.3.0, confluent-kafka 2.10.1 (librdkafka), Apache Kafka 3.9.0 (KRaft), PostgreSQL 17.6, Redis 7.4.2, HAProxy 3.2.5, MinIO-совместимый S3. Код lab5 собран на базе lab4; использовалась помощь ИИ
## Запуск из чистого clone
Требования: Docker Compose v2, Python 3.9+ для тестов. Секреты - в локальном `.env` (в git не коммитится; значения синтетические, шаблон - `.env.example`).
```sh
make check
make validate
docker compose --env-file .env config --quiet
docker compose --env-file .env up -d --build # postgres, redis, s3, kafka -> init/migrate -> api x2, worker, dispatcher, sweeper, haproxy
# дождаться готовности (до ~60 с):
until curl -fs http://localhost:8080/health/ready; do sleep 2; done
make test # публичный smoke
python3 student-tests/level4_failures.py all # сценарии сбоев уровня 4
make test-persistence ACTION_SCRIPT=scripts/restart.sh
make test-failover ACTION_SCRIPT=scripts/stop-one.sh
```
Безопасная остановка: `docker compose stop` (volumes сохраняются). Намеренное удаление учебных данных: `docker compose down -v`. Перезапуск всех хранилищ: `scripts/restart.sh`; остановка одной API-реплики: `scripts/stop-one.sh` (hostname = `X-Instance-Id`). Оператор: failed-реестр и replay - `docker compose exec worker python scripts/replay_failed.py --list | --job-id <uuid> --reason <text>`.
## Архитектура и решения
```
Клиент -> HAProxy -> API x2 --(1) PUT staging/<random> --> S3 (private bucket)
\--(2) ONE TX: attachments(queued) + attachment_jobs + outbox --> PostgreSQL => 202
dispatcher: outbox -> Kafka (acks=all, idempotent producer, delivery report) ; reaper (lease)
worker (consumer group): Kafka message(IDs only) -> claim+lease в PG -> staging -> imgproc (child, limits) -> final S3 -> TX ready -> offset commit
```
- **Источник истины** - PostgreSQL (`attachments.status`, `attachment_jobs`, `attachment_variants`, `outbox`, `job_events`). Kafka - транспорт at-least-once: сообщение содержит только ID (`jobs.schema.json`), без байтов/ключей/URL.
- **Staging** - приватный префикс `staging/` того же приватного bucket; honest-граница: staging в S3, файл уже полностью принят и сохранён до 202. Оригинал воркер публикует в `attachments/<id>/v1/original`, варианты - `.../v1/{thumbnail,crop,optimized}`. Ключ эффекта `(attachment_id, pipeline_version, variant)` = PK `attachment_variants` + детерминированный S3-ключ.
- **Kafka**: топик `attachment-jobs` (3 партиции, key = attachment_id, RF=1, min.insync=1, volume `kafka_data`). Группа `attachment-workers`, `enable.auto.commit=false`. Publisher confirm = delivery report продюсера при `acks=all` (приём брокером); consumer ack = commit offset **после** фиксации результата в PG.
- **Outbox**: строка outbox вставляется в ту же транзакцию, что и attachment/job. Relay берёт строки `FOR UPDATE SKIP LOCKED`, публикует, ставит `published_at` только по delivery report. Падение между COMMIT и publish -> строка осталась, relay доставит позже; между publish и mark -> дубль сообщения (допустим, идемпотентность на потребителе).
- **Retry**: временные ошибки (S3/сеть/timeout/неожиданный код дочернего процесса) -> `queued`, `next_attempt_at = now + backoff` (экспонента 2·2^(n-1), cap 60 с, equal jitter [d/2, d]), задержка реализована **отложенной строкой outbox** (`available_at`). Постоянные (`invalid_image`, `image_too_large`, `image_too_complex`, `integrity_error`, `staging_missing`) -> сразу `failed`. `JOB_MAX_ATTEMPTS=5`, исчерпание -> `retry_exhausted`. Каждый failed сохраняется в реестре PostgreSQL (`attachment_jobs`) и журнале `job_events`.
- **Lease**: `lease_until = now + JOB_TIMEOUT(60) + grace(15)`. Dispatcher возвращает `processing` с истёкшим lease в `queued` (новая попытка) или в `failed`. Повторная доставка при живом lease - no-op (конкурентные дубли не запускаются), при готовом/failed - no-op.
- **Конкурентность/prefetch**: `consume(num_messages=WORKER_CONCURRENCY)` + ThreadPool = не более 2 job одновременно; `queued.max.messages.kbytes=1024`. Лимиты обработки: байты 10 MiB (API, до чтения тела), пиксели 40 млн (из заголовка до декодирования), время `JOB_TIMEOUT_SECONDS` (kill дочернего процесса), память `RLIMIT_AS=IMG_MEMORY_MB` (1024 МБ) + `mem_limit` контейнера, CPU `RLIMIT_CPU`.
- **Варианты**: thumbnail JPEG q85 (вписан в 256×256, без увеличения, EXIF-orientation применён, alpha на белом; 320×200->256×160); crop JPEG q85, центрированный квадрат 128×128; optimized WebP q80 (вписан в 1280×1280 без увеличения, alpha сохраняется, EXIF/ICC не переносятся). Ready - только когда записаны все три и COMMIT выполнен одной транзакцией.
- **Очистка staging**: воркер удаляет staging после COMMIT; sweeper (grace 1 ч) удаляет сироты `staging/` и `attachments/<id>/` без строки БД; staging failed-вложений хранится `STAGING_RETENTION_SECONDS` (24 ч), после чего replay невозможен (нужна повторная загрузка).
- **Readiness API**: PostgreSQL, Redis (сессии), S3. Kafka и worker **не** критичны: приём работает через outbox.
- **Перенесено из lab4** без изменений: пользователи, Argon2id, сессии в Redis, чаты, сообщения, keyset-пагинация, идемпотентность (advisory lock + атомарная запись), HAProxy, bucket/политики s3_init (+`staging/*`), sweeper. **Изменено**: 201->202, статусы и варианты в метаданных, 409 для content не-ready и для прикрепления failed, новые таблицы (миграция 004), сервисы kafka/worker/dispatcher.
- **Соответствие .env.example**: `BROKER_URL_FILE` -> `KAFKA_BOOTSTRAP_SERVERS` (без паролей, PLAINTEXT во внутренней сети backend), `WORKER_CONCURRENCY`, `JOB_MAX_ATTEMPTS`, `JOB_TIMEOUT_SECONDS`, `STAGING_RETENTION_SECONDS`, `MAX_*` - те же имена. Secrets в `_FILE`-виде не используются (учебное упрощение lab4: переменные окружения).
| Роль | Сервис | Версия | Health | Volume |
| --- | --- | --- | --- | --- |
| load_balancer | load_balancer | haproxy 3.2.5-alpine | `haproxy -c` / backend check `/health/ready` | - |
| api | api (2 реплики) | свой образ | `/health/ready` | нет (stateless) |
| postgres | postgres | 17.6-alpine | pg_isready | postgres_data |
| sessions | redis | 7.4.2-alpine, AOF | redis-cli ping | redis_data |
| s3 | s3 (+s3_init) | pgsty/silo (MinIO-совм.) | mc ready | s3_data |
| broker | kafka (+kafka_init) | apache/kafka 3.9.0 | kafka-broker-api-versions | kafka_data |
| worker / dispatcher / sweeper | worker, dispatcher, s3_sweeper | свой образ | перезапуск `unless-stopped` | - |
## Доказательства критериев
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Структура проекта и синтаксис Python | `make check` | `PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.` | `evidence/evidence.txt` |
| OpenAPI 3.1 и JSON Schema | `make validate` | `PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.` Контракты прошли валидацию. | `evidence/evidence.txt` |
| Корректность конфигурации Compose | `docker compose --env-file .env config --quiet` | Команда завершилась без вывода ошибки. | `evidence/evidence.txt` |
| Готовность приложения | `curl -fs http://localhost:8080/health/ready` в цикле ожидания | Получен ответ `{"status":"ok"}`. | `evidence/evidence.txt` |
| Публичный smoke | `make test` | `PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.` | `evidence/evidence.txt` |
| 202 и устойчивый приём; worker остановлен -> upload -> рестарт API -> worker -> ready | `python3 student-tests/level4_failures.py all` - сценарий `worker_down` | `PASS worker_down`; события: `accepted,claimed,ready`. | `evidence/evidence.txt` |
| Crash между COMMIT и publish; восстановление доставки через outbox | `python3 student-tests/level4_failures.py all` - сценарий `commit_publish_gap` | `PASS commit_publish_gap`; события: `accepted,claimed,ready`. | `evidence/evidence.txt` |
| Crash после записи в S3 до фиксации результата/ack; восстановление по lease | `python3 student-tests/level4_failures.py all` - сценарий `crash_after_s3` | `PASS crash_after_s3`; события: `accepted,claimed,lease_expired,claimed,ready`. Повторный claim после истечения lease завершился `ready`. | `evidence/evidence.txt` |
| Retry при временной недоступности S3 | `python3 student-tests/level4_failures.py all` - сценарий `s3_outage` | `PASS s3_outage`; события: `accepted,claimed,retry_scheduled,claimed,retry_scheduled,claimed,retry_scheduled,claimed,ready`. Зафиксированы три планирования повторной попытки, после чего задача завершилась успешно. | `evidence/evidence.txt` |
| Реестр failed и операторский replay | `python3 student-tests/level4_failures.py all` - сценарий `replay` | Команда вывела `REPLAYED job=... attachment=...`; сценарий завершился `PASS replay`. Последовательность событий: `accepted,claimed,failed,replay,claimed,ready`. | `evidence/evidence.txt` |
| Сохранность данных после перезапуска хранилищ и сохранение существующей сессии | `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | `PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.` Тест подтвердил сохранность проверенных данных и существующей сессии | `evidence/evidence.txt` |
| Работа при остановке одной API-реплики | `make test-failover ACTION_SCRIPT=scripts/stop-one.sh` | Перед воздействием наблюдались 2 экземпляра API; выполнено 88 успешных записей, 2 неуспешные попытки; первое восстановление - 3.153 с. Итог: `PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.` | `evidence/evidence.txt` |
| Отдельные broker и worker, durable-сообщения, at-least-once, ack только после устойчивого эффекта | Реализованы отдельные сервисы Kafka broker и worker в compose.yaml; Kafka использует persistent volume kafka_data. В src/dispatcher.py producer настроен с acks=all и idempotence, outbox помечается опубликованным только после подтверждения доставки. В src/worker.py автоматический commit offset отключён: сообщение подтверждается только после записи вариантов в S3 и фиксации состояния ready в PostgreSQL. | worker_down, commit_publish_gap, crash_after_s3 - PASS. В crash_after_s3 после lease_expired задача была повторно получена и завершилась состоянием ready. | Реализация подтверждается compose.yaml, src/dispatcher.py, src/worker.py; порядок подтверждения дополнительно проверен fault-injection сценариями. Kafka работает с RF=1: persistent volume обеспечивает сохранение данных при перезапуске, но репликационной защиты от потери единственного брокера нет. |
| Crop 128×128, optimized WebP и ограничения обработки по bytes/pixels/time/memory | В src/imgproc.py реализованы центрированный crop 128×128 JPEG q85 и optimized WebP q80 с ограничением размера 1280×1280; метаданные исходного изображения не переносятся. В src/main.py ограничен размер входного файла (10 MiB): проверяется Content-Length и фактический объём потокового тела; количество пикселей ограничено 40 000 000 до декодирования. В src/worker.py обработка выполняется с ограничением времени 60 секунд. В src/imgproc.py используются RLIMIT_CPU и RLIMIT_AS=1024 MiB; в compose.yaml для worker задан mem_limit: 1g. | Функция check_variants() в student-tests/level4_failures.py проверяет наличие crop 128×128 и WebP (image/webp). Она используется в сценариях worker_down, commit_publish_gap, crash_after_s3, replay, s3_outage; все пять сценариев завершились PASS. | Все преобразования и ограничения обработки реализованы в исходном коде и конфигурации; размеры crop и формат WebP дополнительно проверяются автоматическими сценариями. |
| Lease для зависшего processing, ограниченные concurrency/prefetch и очистка staging | В src/jobs.py задан срок lease на основе JOB_TIMEOUT_SECONDS и LEASE_GRACE_SECONDS (60+15 секунд). В src/dispatcher.py просроченные задачи processing возвращаются в queued с повторной публикацией через outbox. В src/worker.py ограничены параллелизм (WORKER_CONCURRENCY=2) и количество получаемых сообщений (consume(num_messages=CONCURRENCY)); также заданы ограничения Kafka fetch/prefetch. После фиксации ready worker удаляет staging-файл. scripts/s3_sweep.py удаляет осиротевшие staging-объекты после проверки ссылок в PostgreSQL; grace period - 3600 секунд, период запуска - 600 секунд, срок хранения staging для failed - 86400 секунд. | crash_after_s3 - PASS: после lease_expired задача повторно получена (claimed) и завершилась ready. | Восстановление по lease подтверждено fault-injection тестом; ограничения concurrency/prefetch и механизм очистки staging подтверждаются реализацией в src/worker.py, src/jobs.py, src/dispatcher.py, scripts/s3_sweep.py. |
## Эксперименты и отказы
Среда проведения экспериментов:
- ОС: Microsoft Windows 11 Pro 10.0.22631 (build 22631).
- CPU: Intel Core i5-8350U, 4 ядра / 8 потоков.
- RAM: 15.9 GB.
- Docker Compose: v2.32.4-desktop.1.
- Docker Client/Engine: 27.5.1; Docker Desktop 4.38.0; контекст `desktop-linux`.
Перед функциональными сценариями выполнены `make check`, `make validate` и `docker compose --env-file .env config --quiet`. Первые две команды завершились с `PASS`; в их выводе отдельно указано, что они не тестируют работающее приложение. Compose config завершился без сообщения об ошибке. После запуска стенда readiness endpoint вернул `{"status":"ok"}`. Публичный smoke (`make test`) завершился с `PASS`.
### Сценарии отказов уровня 4
Команда `python3 student-tests/level4_failures.py all` завершилась успешно для всех пяти сценариев:
| Воздействие | Наблюдаемые события / результат | Вывод |
| --- | --- | --- |
| Остановка worker, приём задания и последующий запуск worker (`worker_down`) | `accepted,claimed,ready`; `PASS worker_down` | Задание обработано после восстановления worker. |
| Разрыв между фиксацией задания в PostgreSQL и публикацией (`commit_publish_gap`) | `accepted,claimed,ready`; `PASS commit_publish_gap` | Задание в итоге доставлено и обработано без ручного вмешательства. |
| Аварийное завершение после записи в S3 (`crash_after_s3`) | `accepted,claimed,lease_expired,claimed,ready`; `PASS crash_after_s3` | После истечения lease задача была повторно захвачена и завершена. |
| Повторный запуск задания из failed-реестра (`replay`) | `accepted,claimed,failed,replay,claimed,ready`; команда вывела `REPLAYED`; `PASS replay` | Failed-задание прошло операторский replay и завершилось успешно. |
| Недоступность S3 во время обработки (`s3_outage`) | `accepted,claimed,retry_scheduled,claimed,retry_scheduled,claimed,retry_scheduled,claimed,ready`; `PASS s3_outage` | Зафиксированы три запланированные повторные попытки; после восстановления S3 задача завершилась успешно. |
### Сохранность данных и отказ API-реплики
| Воздействие | Наблюдаемый результат | Время восстановления / потеря данных |
| --- | --- | --- |
| Перезапуск хранилищ: `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | `PASS: persistence preserved confirmed data and existing session.` | Проверенные подтверждённые данные и существующая сессия сохранены. |
| Остановка одной API-реплики: `make test-failover ACTION_SCRIPT=scripts/stop-one.sh` | До воздействия наблюдались 2 экземпляра; 88 успешных записей, 2 неуспешные попытки; `PASS: failover preserved confirmed data and existing session.` | Первое восстановление - 3.153 с; тест сообщил о сохранении подтверждённых данных и существующей сессии. |
## Ограничения и следующие шаги
Один брокер Kafka (RF=1): при потере диска kafka_data сообщения могут быть потеряны; автоматического восстановления потерянного брокерного лога нет. PostgreSQL остаётся источником истины для состояния job, но восстановление доставки после потери самого лога не гарантируется. Лимит памяти - RLIMIT_AS + mem_limit, не cgroup-метрика RSS по job. Валидация изображения (декодирование) при upload выполняется в API (перенос lab4), resize - только в worker. Replay доступен оператору через CLI, не через публичный API (контракт не менялся). Трек 5 не реализован; нагрузочные измерения pipeline не проводились.
+314
View File
@@ -0,0 +1,314 @@
name: mtusi-lab5
services:
postgres:
image: postgres:17.6-alpine
environment:
POSTGRES_DB: ${POSTGRES_DB:-messenger}
POSTGRES_USER: ${POSTGRES_USER:-messenger}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}
volumes:
- postgres_data:/var/lib/postgresql/data
networks: [backend]
healthcheck:
test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
interval: 5s
timeout: 3s
retries: 20
cpus: 1.0
mem_limit: 768m
restart: unless-stopped
redis:
image: redis:7.4.2-alpine
command: ["redis-server", "--appendonly", "yes", "--appendfsync", "everysec"]
volumes:
- redis_data:/data
networks: [backend]
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 3s
retries: 20
cpus: 0.25
mem_limit: 256m
restart: unless-stopped
s3:
image: pgsty/silo:RELEASE.2026-10-15T00-00-00Z
command: ["server", "/data", "--address", ":9000"]
environment:
MINIO_ROOT_USER: ${MINIO_ROOT_USER:?Set MINIO_ROOT_USER in .env}
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:?Set MINIO_ROOT_PASSWORD in .env}
MINIO_BROWSER: "off" # admin web console disabled; no ports are published at all
volumes:
- s3_data:/data
networks: [backend]
healthcheck:
test: ["CMD", "mc", "ready", "local"]
interval: 5s
timeout: 5s
retries: 20
cpus: 1.0
mem_limit: 512m
restart: unless-stopped
s3_init:
image: pgsty/silo:RELEASE.2026-10-15T00-00-00Z
entrypoint: ["/bin/sh", "/infra/s3-init.sh"]
environment:
S3_ENDPOINT: http://s3:9000
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
MINIO_ROOT_USER: ${MINIO_ROOT_USER:?Set MINIO_ROOT_USER in .env}
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:?Set MINIO_ROOT_PASSWORD in .env}
S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID:?Set S3_ACCESS_KEY_ID in .env}
S3_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY:?Set S3_SECRET_ACCESS_KEY in .env}
SWEEPER_S3_ACCESS_KEY_ID: ${SWEEPER_S3_ACCESS_KEY_ID:?Set SWEEPER_S3_ACCESS_KEY_ID in .env}
SWEEPER_S3_SECRET_ACCESS_KEY: ${SWEEPER_S3_SECRET_ACCESS_KEY:?Set SWEEPER_S3_SECRET_ACCESS_KEY in .env}
volumes:
- ./infra/s3-init.sh:/infra/s3-init.sh:ro
networks: [backend]
depends_on:
s3:
condition: service_started
restart: "no"
kafka:
image: apache/kafka:3.9.0
environment:
KAFKA_NODE_ID: "1"
KAFKA_PROCESS_ROLES: broker,controller
KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: "1"
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: "1"
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: "1"
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: "0"
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
KAFKA_LOG_DIRS: /var/lib/kafka/data
KAFKA_HEAP_OPTS: "-Xms256m -Xmx512m"
volumes:
- kafka_data:/var/lib/kafka/data
networks: [backend]
healthcheck:
test: ["CMD-SHELL", "/opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 >/dev/null 2>&1"]
interval: 5s
timeout: 10s
retries: 30
start_period: 15s
cpus: 1.0
mem_limit: 1g
restart: unless-stopped
kafka_init:
image: apache/kafka:3.9.0
entrypoint: ["/bin/sh", "-c"]
command:
- |
set -e
T=/opt/kafka/bin/kafka-topics.sh
$$T --bootstrap-server kafka:9092 --create --if-not-exists --topic $${JOBS_TOPIC} --partitions 3 --replication-factor 1 --config min.insync.replicas=1 --config retention.ms=604800000
$$T --bootstrap-server kafka:9092 --list
environment:
JOBS_TOPIC: ${JOBS_TOPIC:-attachment-jobs}
networks: [backend]
depends_on:
kafka:
condition: service_healthy
restart: "no"
migrate:
build: .
image: mtusi-lab5-api:1.0.0
command: ["python", "scripts/migrate.py"]
environment:
PGHOST: postgres
PGPORT: "5432"
PGDATABASE: ${POSTGRES_DB:-messenger}
PGUSER: ${POSTGRES_USER:-messenger}
PGPASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}
networks: [backend]
depends_on:
postgres:
condition: service_healthy
restart: "no"
cpus: 0.5
mem_limit: 384m
api:
build: .
image: mtusi-lab5-api:1.0.0
environment:
APP_HOST: 0.0.0.0
APP_PORT: "8080"
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
DB_POOL_MIN_SIZE: ${DB_POOL_MIN_SIZE:-1}
DB_POOL_MAX_SIZE: ${DB_POOL_MAX_SIZE:-10}
DB_POOL_TIMEOUT_SECONDS: ${DB_POOL_TIMEOUT_SECONDS:-5}
PGHOST: postgres
PGPORT: "5432"
PGDATABASE: ${POSTGRES_DB:-messenger}
PGUSER: ${POSTGRES_USER:-messenger}
PGPASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}
REDIS_URL: redis://redis:6379/0
S3_ENDPOINT: http://s3:9000
S3_REGION: ${S3_REGION:-us-east-1}
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID:?Set S3_ACCESS_KEY_ID in .env}
S3_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY:?Set S3_SECRET_ACCESS_KEY in .env}
MAX_UPLOAD_BYTES: ${MAX_UPLOAD_BYTES:-10485760}
MAX_IMAGE_PIXELS: ${MAX_IMAGE_PIXELS:-40000000}
IMAGE_DECODE_CONCURRENCY: ${IMAGE_DECODE_CONCURRENCY:-1}
JOB_MAX_ATTEMPTS: ${JOB_MAX_ATTEMPTS:-5}
deploy:
replicas: 2
expose: ["8080"]
networks: [backend]
depends_on:
migrate:
condition: service_completed_successfully
redis:
condition: service_healthy
s3_init:
condition: service_completed_successfully
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=2)"]
interval: 3s
timeout: 3s
retries: 3
start_period: 5s
stop_grace_period: 15s
cpus: 1.0
mem_limit: 512m
restart: unless-stopped
worker:
build: .
image: mtusi-lab5-api:1.0.0
command: ["python", "-m", "src.worker"]
environment:
PGHOST: postgres
PGPORT: "5432"
PGDATABASE: ${POSTGRES_DB:-messenger}
PGUSER: ${POSTGRES_USER:-messenger}
PGPASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}
S3_ENDPOINT: http://s3:9000
S3_REGION: ${S3_REGION:-us-east-1}
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID:?Set S3_ACCESS_KEY_ID in .env}
S3_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY:?Set S3_SECRET_ACCESS_KEY in .env}
KAFKA_BOOTSTRAP_SERVERS: kafka:9092
JOBS_TOPIC: ${JOBS_TOPIC:-attachment-jobs}
JOB_MAX_ATTEMPTS: ${JOB_MAX_ATTEMPTS:-5}
JOB_TIMEOUT_SECONDS: ${JOB_TIMEOUT_SECONDS:-60}
LEASE_GRACE_SECONDS: ${LEASE_GRACE_SECONDS:-15}
JOB_BACKOFF_BASE_SECONDS: ${JOB_BACKOFF_BASE_SECONDS:-2}
JOB_BACKOFF_CAP_SECONDS: ${JOB_BACKOFF_CAP_SECONDS:-60}
WORKER_CONCURRENCY: ${WORKER_CONCURRENCY:-2}
IMG_MEMORY_MB: ${IMG_MEMORY_MB:-1024}
MAX_IMAGE_PIXELS: ${MAX_IMAGE_PIXELS:-40000000}
FAULT_WORKER_EXIT_AFTER_S3: ${FAULT_WORKER_EXIT_AFTER_S3:-0}
networks: [backend]
depends_on:
migrate:
condition: service_completed_successfully
kafka_init:
condition: service_completed_successfully
s3_init:
condition: service_completed_successfully
stop_grace_period: 30s
cpus: 1.0
mem_limit: 1g
restart: unless-stopped
dispatcher:
build: .
image: mtusi-lab5-api:1.0.0
command: ["python", "-m", "src.dispatcher"]
environment:
PGHOST: postgres
PGPORT: "5432"
PGDATABASE: ${POSTGRES_DB:-messenger}
PGUSER: ${POSTGRES_USER:-messenger}
PGPASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}
KAFKA_BOOTSTRAP_SERVERS: kafka:9092
JOBS_TOPIC: ${JOBS_TOPIC:-attachment-jobs}
JOB_MAX_ATTEMPTS: ${JOB_MAX_ATTEMPTS:-5}
JOB_TIMEOUT_SECONDS: ${JOB_TIMEOUT_SECONDS:-60}
LEASE_GRACE_SECONDS: ${LEASE_GRACE_SECONDS:-15}
JOB_BACKOFF_BASE_SECONDS: ${JOB_BACKOFF_BASE_SECONDS:-2}
JOB_BACKOFF_CAP_SECONDS: ${JOB_BACKOFF_CAP_SECONDS:-60}
FAULT_DISPATCHER_EXIT_BEFORE_PUBLISH: ${FAULT_DISPATCHER_EXIT_BEFORE_PUBLISH:-0}
networks: [backend]
depends_on:
migrate:
condition: service_completed_successfully
kafka_init:
condition: service_completed_successfully
s3_init:
condition: service_completed_successfully
cpus: 0.25
mem_limit: 192m
restart: unless-stopped
s3_sweeper:
build: .
image: mtusi-lab5-api:1.0.0
command: ["python", "scripts/s3_sweep.py", "--loop", "--apply"]
environment:
PGHOST: postgres
PGPORT: "5432"
PGDATABASE: ${POSTGRES_DB:-messenger}
PGUSER: ${POSTGRES_USER:-messenger}
PGPASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}
S3_ENDPOINT: http://s3:9000
S3_REGION: ${S3_REGION:-us-east-1}
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
S3_ACCESS_KEY_ID: ${SWEEPER_S3_ACCESS_KEY_ID:?Set SWEEPER_S3_ACCESS_KEY_ID in .env}
S3_SECRET_ACCESS_KEY: ${SWEEPER_S3_SECRET_ACCESS_KEY:?Set SWEEPER_S3_SECRET_ACCESS_KEY in .env}
SWEEP_GRACE_SECONDS: ${SWEEP_GRACE_SECONDS:-3600}
SWEEP_INTERVAL_SECONDS: ${SWEEP_INTERVAL_SECONDS:-600}
STAGING_RETENTION_SECONDS: ${STAGING_RETENTION_SECONDS:-86400}
networks: [backend]
depends_on:
migrate:
condition: service_completed_successfully
s3_init:
condition: service_completed_successfully
cpus: 0.25
mem_limit: 128m
restart: unless-stopped
load_balancer:
image: haproxy:3.2.5-alpine
ports:
- "127.0.0.1:${APP_PORT:-8080}:8080"
volumes:
- ./infra/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
networks: [frontend, backend]
depends_on:
api:
condition: service_healthy
healthcheck:
test: ["CMD", "haproxy", "-c", "-f", "/usr/local/etc/haproxy/haproxy.cfg"]
interval: 10s
timeout: 3s
retries: 3
cpus: 0.5
mem_limit: 128m
restart: unless-stopped
networks:
frontend:
driver: bridge
backend:
driver: bridge
internal: true # no route to/from the host or internet: S3, PostgreSQL, Redis, Kafka and API are reachable only inside
volumes:
postgres_data:
redis_data:
s3_data:
kafka_data:
+71
View File
@@ -0,0 +1,71 @@
thereal_alyoname@DESKTOP-8Q26HPT:/mnt/c/Users/User/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork$
make check
python3 scripts/check_contract.py
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
thereal_alyoname@DESKTOP-8Q26HPT:/mnt/c/Users/User/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork$
make validate
python3 scripts/validate_openapi.py
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
thereal_alyoname@DESKTOP-8Q26HPT:/mnt/c/Users/User/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork$
thereal_alyoname@DESKTOP-8Q26HPT:/mnt/c/Users/User/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork$
docker compose --env-file .env config --quiet
thereal_alyoname@DESKTOP-8Q26HPT:/mnt/c/Users/User/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork$
User@DESKTOP-8Q26HPT MINGW64 ~/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork (solution/bvt2302-mironov)
$ docker compose --env-file .env up --build
[+] Building 21.1s (8/12) docker:desktop-linux
=> => transferring context: 208.14kB 0.0s
=> CACHED [migrate 2/8] WORKDIR /app 0.0s
=> CACHED [migrate 3/8] RUN addgroup --system app && adduser --system --ingroup app app 0.0s
=> [migrate 4/8] COPY requirements.txt ./requirements.txt 0.0s
=> [migrate 5/8] RUN python -m pip install --no-cache-dir -r requirements.txt
(проверка доступности приложения)
thereal_alyoname@DESKTOP-8Q26HPT:/mnt/c/Users/User/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork$
until curl -fs http://localhost:8080/health/ready; do sleep 2; done
{"status":"ok"}thereal_alyoname@DESKTOP-8Q26HPT:/mnt/c/Users/User/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork$
thereal_alyoname@DESKTOP-8Q26HPT:/mnt/c/Users/User/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork$
make test
python3 tests/smoke.py --base-url "http://localhost:8080"
PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.
thereal_alyoname@DESKTOP-8Q26HPT:/mnt/c/Users/User/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork$
(проверки на уровень 4)
thereal_alyoname@DESKTOP-8Q26HPT:/mnt/c/Users/User/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork$
python3 student-tests/level4_failures.py all
PASS worker_down: e05d5f76-4b0f-4b33-a81e-e3f6ab4bc72f events: accepted,claimed,ready
PASS commit_publish_gap: df3506d1-b835-49c7-8b0c-bf150273d750 events: accepted,claimed,ready
PASS crash_after_s3: 9f2b8d69-04f2-49f6-bd6c-dca100a9fe87 events: accepted,claimed,lease_expired,claimed,ready
REPLAYED job=5ea63d39-7ba8-4f9b-a9f9-506cc7e69034 attachment=b5a4228a-654f-4212-9b4d-9bd429d85020
PASS replay: b5a4228a-654f-4212-9b4d-9bd429d85020 events: accepted,claimed,failed,replay,claimed,ready
PASS s3_outage: e4ec52dc-cc63-4c2e-babe-d00b1e70de16 events: accepted,claimed,retry_scheduled,claimed,retry_scheduled,claimed,retry_scheduled,claimed,ready
thereal_alyoname@DESKTOP-8Q26HPT:/mnt/c/Users/User/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork$
(тест на сохранность данных сервисов)
thereal_alyoname@DESKTOP-8Q26HPT:/mnt/c/Users/User/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork$
make test-persistence ACTION_SCRIPT=scripts/restart.sh
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:8080"
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
(тест на работоспособность приложения во время того, как опущена одна реплика апи)
thereal_alyoname@DESKTOP-8Q26HPT:/mnt/c/Users/User/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork$
make test-failover ACTION_SCRIPT=scripts/stop-one.sh
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://localhost:8080"
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 88, "failed_attempts": 2, "first_recovery_seconds": 3.153}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
thereal_alyoname@DESKTOP-8Q26HPT:/mnt/c/Users/User/Desktop/MTUCI/4_course/highload_apps/lab5/mtusi2026lab5-fork$
+32
View File
@@ -0,0 +1,32 @@
global
log stdout format raw local0
maxconn 1024
defaults
log global
mode http
option httplog
option http-keep-alive
timeout connect 2s
timeout client 30s
timeout server 30s
timeout check 2s
resolvers docker
nameserver docker-dns 127.0.0.11:53
resolve_retries 3
timeout resolve 1s
timeout retry 1s
hold valid 2s
hold obsolete 2s
frontend public_http
bind :8080
default_backend api_replicas
backend api_replicas
balance roundrobin
option httpchk GET /health/ready
http-check expect status 200
default-server inter 1s fall 2 rise 1
server-template api 10 api:8080 check resolvers docker init-addr libc,none
+56
View File
@@ -0,0 +1,56 @@
#!/bin/sh
# Idempotent bootstrap of the private bucket, the restricted application user
# and the separate sweeper user (safe to re-run).
set -eu
: "${S3_ENDPOINT:?}" "${S3_BUCKET:?}" "${MINIO_ROOT_USER:?}" "${MINIO_ROOT_PASSWORD:?}"
: "${S3_ACCESS_KEY_ID:?}" "${S3_SECRET_ACCESS_KEY:?}"
: "${SWEEPER_S3_ACCESS_KEY_ID:?}" "${SWEEPER_S3_SECRET_ACCESS_KEY:?}"
[ "$S3_ACCESS_KEY_ID" != "$MINIO_ROOT_USER" ] || { echo "app key must differ from root user" >&2; exit 1; }
[ "$SWEEPER_S3_ACCESS_KEY_ID" != "$MINIO_ROOT_USER" ] || { echo "sweeper key must differ from root user" >&2; exit 1; }
[ "$S3_ACCESS_KEY_ID" != "$SWEEPER_S3_ACCESS_KEY_ID" ] || { echo "app and sweeper keys must differ" >&2; exit 1; }
i=0
until mc alias set local "$S3_ENDPOINT" "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" >/dev/null 2>&1; do
i=$((i+1)); [ "$i" -lt 60 ] || { echo "S3 not reachable" >&2; exit 1; }; sleep 1
done
mc mb --ignore-existing "local/$S3_BUCKET"
mc anonymous set none "local/$S3_BUCKET"
printf ok | mc pipe "local/$S3_BUCKET/attachments/.ready"
# --- app policy: exact keys only, NO ListBucket ---
cat > /tmp/policy-app.json <<POLICY
{"Version":"2012-10-17","Statement":[
{"Effect":"Allow","Action":["s3:PutObject","s3:GetObject","s3:DeleteObject"],"Resource":["arn:aws:s3:::$S3_BUCKET/attachments/*","arn:aws:s3:::$S3_BUCKET/staging/*"]}]}
POLICY
# --- sweeper policy: ListBucket for enumeration + Get/Delete, NO PutObject ---
cat > /tmp/policy-sweeper.json <<POLICY
{"Version":"2012-10-17","Statement":[
{"Effect":"Allow","Action":["s3:ListBucket","s3:GetBucketLocation"],"Resource":["arn:aws:s3:::$S3_BUCKET"]},
{"Effect":"Allow","Action":["s3:GetObject","s3:DeleteObject"],"Resource":["arn:aws:s3:::$S3_BUCKET/attachments/*","arn:aws:s3:::$S3_BUCKET/staging/*"]}]}
POLICY
# MinIO's `policy create` does not overwrite an existing policy, so remove first.
mc admin policy remove local messenger-app --force >/dev/null 2>&1 || true
mc admin policy remove local messenger-sweeper --force >/dev/null 2>&1 || true
mc admin policy create local messenger-app /tmp/policy-app.json
mc admin policy create local messenger-sweeper /tmp/policy-sweeper.json
# `user add` is idempotent (re-running updates the secret).
mc admin user add local "$S3_ACCESS_KEY_ID" "$S3_SECRET_ACCESS_KEY"
mc admin user add local "$SWEEPER_S3_ACCESS_KEY_ID" "$SWEEPER_S3_SECRET_ACCESS_KEY"
out=$(mc admin policy attach local messenger-app --user "$S3_ACCESS_KEY_ID" 2>&1) || {
echo "$out" | grep -qi "already" || { echo "$out" >&2; exit 1; }
}
out=$(mc admin policy attach local messenger-sweeper --user "$SWEEPER_S3_ACCESS_KEY_ID" 2>&1) || {
echo "$out" | grep -qi "already" || { echo "$out" >&2; exit 1; }
}
echo "PASS: bucket $S3_BUCKET private"
echo " app=$S3_ACCESS_KEY_ID (Put/Get/Delete on attachments/* and staging/*, no ListBucket)"
echo " sweeper=$SWEEPER_S3_ACCESS_KEY_ID (ListBucket + Get/Delete, no PutObject)"
+32
View File
@@ -0,0 +1,32 @@
CREATE TABLE IF NOT EXISTS users (
id UUID PRIMARY KEY,
username VARCHAR(32) NOT NULL UNIQUE,
display_name VARCHAR(100) NOT NULL,
password_hash TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT users_username_format CHECK (username ~ '^[a-z0-9_]{3,32}$'),
CONSTRAINT users_display_name_nonblank CHECK (length(btrim(display_name)) > 0)
);
CREATE TABLE IF NOT EXISTS chats (
id UUID PRIMARY KEY,
title VARCHAR(100) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT chats_title_nonblank CHECK (length(btrim(title)) > 0)
);
CREATE TABLE IF NOT EXISTS chat_members (
chat_id UUID NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
position INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (chat_id, user_id)
);
ALTER TABLE chat_members ADD COLUMN IF NOT EXISTS position INTEGER NOT NULL DEFAULT 0;
CREATE INDEX IF NOT EXISTS chat_members_user_chat_idx ON chat_members(user_id, chat_id);
CREATE TABLE IF NOT EXISTS messages (
id UUID PRIMARY KEY,
chat_id UUID NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
sender_id UUID NOT NULL REFERENCES users(id),
text VARCHAR(4000) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT messages_text_nonblank CHECK (length(btrim(text)) > 0)
);
CREATE INDEX IF NOT EXISTS messages_chat_created_id_idx ON messages(chat_id, created_at, id);
+11
View File
@@ -0,0 +1,11 @@
CREATE TABLE IF NOT EXISTS idempotency_records (
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
chat_id UUID NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
idempotency_key VARCHAR(128) NOT NULL,
request_hash CHAR(64) NOT NULL,
response_body JSONB NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (user_id, chat_id, idempotency_key),
CONSTRAINT idempotency_key_format CHECK (idempotency_key ~ '^[A-Za-z0-9._:-]{1,128}$')
);
CREATE INDEX IF NOT EXISTS idempotency_records_created_at_idx ON idempotency_records(created_at);
+24
View File
@@ -0,0 +1,24 @@
CREATE TABLE IF NOT EXISTS attachments (
id UUID PRIMARY KEY,
chat_id UUID NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
owner_id UUID NOT NULL REFERENCES users(id),
filename VARCHAR(255) NOT NULL,
content_type TEXT NOT NULL,
size_bytes BIGINT NOT NULL,
sha256 CHAR(64) NOT NULL,
s3_key TEXT NOT NULL UNIQUE,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT attachments_filename_nonblank CHECK (length(btrim(filename)) > 0),
CONSTRAINT attachments_content_type CHECK (content_type IN ('image/png', 'image/jpeg')),
CONSTRAINT attachments_size CHECK (size_bytes BETWEEN 1 AND 10485760),
CONSTRAINT attachments_sha256 CHECK (sha256 ~ '^[a-f0-9]{64}$')
);
CREATE INDEX IF NOT EXISTS attachments_chat_idx ON attachments(chat_id);
CREATE TABLE IF NOT EXISTS message_attachments (
message_id UUID NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
attachment_id UUID NOT NULL REFERENCES attachments(id),
position INTEGER NOT NULL,
PRIMARY KEY (message_id, attachment_id)
);
CREATE INDEX IF NOT EXISTS message_attachments_attachment_idx ON message_attachments(attachment_id);
ALTER TABLE messages DROP CONSTRAINT IF EXISTS messages_text_nonblank;
+58
View File
@@ -0,0 +1,58 @@
ALTER TABLE attachments ADD COLUMN IF NOT EXISTS status TEXT NOT NULL DEFAULT 'ready';
ALTER TABLE attachments ADD COLUMN IF NOT EXISTS error_code TEXT;
ALTER TABLE attachments ADD COLUMN IF NOT EXISTS updated_at TIMESTAMPTZ NOT NULL DEFAULT now();
ALTER TABLE attachments ADD COLUMN IF NOT EXISTS staging_key TEXT;
ALTER TABLE attachments ADD COLUMN IF NOT EXISTS pipeline_version INTEGER NOT NULL DEFAULT 1;
ALTER TABLE attachments DROP CONSTRAINT IF EXISTS attachments_status;
ALTER TABLE attachments ADD CONSTRAINT attachments_status CHECK (status IN ('queued', 'processing', 'ready', 'failed'));
CREATE INDEX IF NOT EXISTS attachments_staging_idx ON attachments(staging_key) WHERE staging_key IS NOT NULL;
CREATE TABLE IF NOT EXISTS attachment_variants (
attachment_id UUID NOT NULL REFERENCES attachments(id) ON DELETE CASCADE,
pipeline_version INTEGER NOT NULL,
name TEXT NOT NULL,
s3_key TEXT NOT NULL,
content_type TEXT NOT NULL,
size_bytes BIGINT NOT NULL,
width INTEGER NOT NULL,
height INTEGER NOT NULL,
sha256 CHAR(64) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (attachment_id, pipeline_version, name),
CONSTRAINT variants_name CHECK (name IN ('thumbnail', 'crop', 'optimized'))
);
CREATE TABLE IF NOT EXISTS attachment_jobs (
job_id UUID PRIMARY KEY,
attachment_id UUID NOT NULL REFERENCES attachments(id) ON DELETE CASCADE,
pipeline_version INTEGER NOT NULL DEFAULT 1,
status TEXT NOT NULL,
attempt INTEGER NOT NULL DEFAULT 0,
max_attempts INTEGER NOT NULL,
next_attempt_at TIMESTAMPTZ NOT NULL DEFAULT now(),
lease_until TIMESTAMPTZ,
worker_id TEXT,
error_code TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT jobs_status CHECK (status IN ('queued', 'processing', 'ready', 'failed')),
CONSTRAINT jobs_effect_key UNIQUE (attachment_id, pipeline_version)
);
CREATE INDEX IF NOT EXISTS attachment_jobs_status_idx ON attachment_jobs(status, lease_until);
CREATE TABLE IF NOT EXISTS job_events (
id BIGSERIAL PRIMARY KEY,
job_id UUID NOT NULL REFERENCES attachment_jobs(job_id) ON DELETE CASCADE,
at TIMESTAMPTZ NOT NULL DEFAULT now(),
event TEXT NOT NULL,
detail TEXT
);
CREATE INDEX IF NOT EXISTS job_events_job_idx ON job_events(job_id, id);
CREATE TABLE IF NOT EXISTS outbox (
id BIGSERIAL PRIMARY KEY,
job_id UUID NOT NULL REFERENCES attachment_jobs(job_id) ON DELETE CASCADE,
topic TEXT NOT NULL,
msg_key TEXT NOT NULL,
payload JSONB NOT NULL,
available_at TIMESTAMPTZ NOT NULL DEFAULT now(),
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
published_at TIMESTAMPTZ
);
CREATE INDEX IF NOT EXISTS outbox_pending_idx ON outbox(available_at, id) WHERE published_at IS NULL;
+8
View File
@@ -0,0 +1,8 @@
fastapi==0.116.1
uvicorn[standard]==0.35.0
psycopg[pool,binary]==3.2.9
argon2-cffi==25.1.0
redis==6.4.0
boto3==1.40.20
Pillow==11.3.0
confluent-kafka==2.10.1
+37
View File
@@ -0,0 +1,37 @@
#!/usr/bin/env python3
"""Apply SQL migrations once, before any API replica is started."""
from pathlib import Path
import os
import time
import psycopg
ROOT = Path(__file__).resolve().parents[1]
CONNINFO = os.getenv("DATABASE_URL", "")
def main():
last_error = None
for _ in range(30):
try:
with psycopg.connect(CONNINFO, connect_timeout=3) as conn:
conn.execute("CREATE TABLE IF NOT EXISTS schema_migrations (version TEXT PRIMARY KEY, applied_at TIMESTAMPTZ NOT NULL DEFAULT now())")
conn.commit()
for path in sorted((ROOT / "migrations").glob("*.sql")):
version = path.name
if conn.execute("SELECT 1 FROM schema_migrations WHERE version=%s", (version,)).fetchone():
continue
sql = path.read_text(encoding="utf-8")
with conn.transaction():
for statement in sql.split(";"):
if statement.strip():
conn.execute(statement)
conn.execute("INSERT INTO schema_migrations(version) VALUES (%s)", (version,))
print(f"APPLIED {version}", flush=True)
print("PASS: migrations complete", flush=True)
return
except Exception as exc:
last_error = exc
time.sleep(1)
raise SystemExit(f"Migration failed after retries: {type(last_error).__name__}")
if __name__ == "__main__":
main()
+53
View File
@@ -0,0 +1,53 @@
#!/usr/bin/env python3
"""Operator tool: failed-job registry and SAFE explicit replay (lab5, level 4).
docker compose exec worker python scripts/replay_failed.py --list
docker compose exec worker python scripts/replay_failed.py --job-id <uuid> --reason "ticket-123"
Replay never executes a duplicate of a successful job: it requires status=failed, that the staging source still exists,
and that the effect (all variants of the pipeline version) is NOT already present. Everything happens in ONE transaction
(state change + audit event + outbox row), so the new attempt is delivered by the normal dispatcher path.
"""
import argparse
import os
import sys
import psycopg
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from src import jobs # noqa: E402
def main() -> int:
p = argparse.ArgumentParser()
p.add_argument("--list", action="store_true")
p.add_argument("--job-id")
p.add_argument("--reason", default="operator replay")
a = p.parse_args()
with psycopg.connect(connect_timeout=5) as conn:
if a.list or not a.job_id:
for r in conn.execute("SELECT job_id, attachment_id, error_code, attempt, updated_at FROM attachment_jobs WHERE status='failed' ORDER BY updated_at"):
print(*r, sep="\t")
return 0
row = conn.execute("SELECT j.status, j.attachment_id, j.pipeline_version, a.staging_key FROM attachment_jobs j "
"JOIN attachments a ON a.id=j.attachment_id WHERE j.job_id=%s FOR UPDATE OF j", (a.job_id,)).fetchone()
if row is None:
print("REFUSED: unknown job"); return 1
status, att_id, pv, staging_key = row
if status != "failed":
print(f"REFUSED: job is {status}, only failed jobs can be replayed"); return 1
done = conn.execute("SELECT count(*) FROM attachment_variants WHERE attachment_id=%s AND pipeline_version=%s", (att_id, pv)).fetchone()[0]
if done >= len(jobs.VARIANT_ORDER):
print("REFUSED: the effect already exists (all variants present)"); return 1
if not staging_key:
print("REFUSED: staging source expired (retention), upload the file again"); return 1
conn.execute("UPDATE attachment_jobs SET status='queued', attempt=0, error_code=NULL, lease_until=NULL, next_attempt_at=now(), updated_at=now() WHERE job_id=%s", (a.job_id,))
conn.execute("UPDATE attachments SET status='queued', error_code=NULL, updated_at=now() WHERE id=%s", (att_id,))
jobs.add_event(conn, a.job_id, "replay", a.reason[:200])
jobs.enqueue(conn, a.job_id, att_id, pv, 1)
print(f"REPLAYED job={a.job_id} attachment={att_id}")
return 0
if __name__ == "__main__":
sys.exit(main())
+5
View File
@@ -0,0 +1,5 @@
#!/bin/sh
set -eu
# Restart ALL state stores (PostgreSQL, Redis, S3, Kafka), then API, dispatcher and worker; named volumes are preserved.
docker compose restart postgres redis s3 kafka
docker compose restart api dispatcher worker
+24
View File
@@ -0,0 +1,24 @@
#!/bin/sh
set -eu
# Sequential replacement-equivalent exercise for Compose (which has no native rolling-update primitive).
# Run a load generator in another terminal while this script executes.
replicas="$(docker compose ps -q --all api)"
[ -n "$replicas" ] || { echo "No API containers found; start the stack first" >&2; exit 1; }
for id in $replicas; do
# Remove one replica; the LB excludes it after health-check failure.
docker stop --time 10 "$id" >/dev/null || true
docker rm -f "$id" >/dev/null
docker compose up -d --no-deps --scale api=2 api
# Wait for at least two healthy API containers before moving to the next one.
tries=0
while [ "$tries" -lt 30 ]; do
healthy=0
for candidate in $(docker compose ps -q api); do
status=$(docker inspect --format '{{.State.Health.Status}}' "$candidate" 2>/dev/null || echo starting)
[ "$status" = healthy ] && healthy=$((healthy+1))
done
[ "$healthy" -ge 2 ] && break
tries=$((tries+1)); sleep 1
done
[ "$healthy" -ge 2 ] || { echo "Fewer than two healthy replicas after replacement" >&2; exit 1; }
done
+74
View File
@@ -0,0 +1,74 @@
#!/usr/bin/env python3
"""Orphan sweeper (lab5): staging/ and attachments/ cleanup, retention of failed staging (see run_once).
Safety: only objects older than --grace-seconds are candidates (default 3600, far above the few seconds between
PUT and COMMIT), and the DB is re-checked immediately before each delete. Default mode is --dry-run.
"""
import argparse, os, sys, time
from datetime import datetime, timezone
import boto3, psycopg
from botocore.config import Config
def run_once(s3, conn, bucket, grace, apply, retention):
"""Lab5 cleanup rules (documented in REPORT.md):
1) staging/<k> without any attachments.staging_key row older than grace = orphan of a crash between PUT and COMMIT -> delete
(staging of ready attachments is deleted by the worker and its staging_key set to NULL, so a leftover is also an orphan);
2) attachments/<id>/... whose attachment row no longer exists -> delete;
3) staging of FAILED attachments older than retention -> delete object and NULL staging_key (replay is then impossible).
Current work (queued/processing, or failed within retention) is never touched; the DB is re-checked right before each delete."""
now = datetime.now(timezone.utc)
seen = old = orphans = deleted = 0
for prefix in ("staging/", "attachments/"):
for page in s3.get_paginator("list_objects_v2").paginate(Bucket=bucket, Prefix=prefix):
for obj in page.get("Contents", []):
seen += 1
key = obj["Key"]
if key == "attachments/.ready" or (now - obj["LastModified"]).total_seconds() < grace:
continue # fresh: may still be between PUT and COMMIT
old += 1
if prefix == "staging/":
referenced = lambda: conn.execute("SELECT 1 FROM attachments WHERE staging_key=%s", (key,)).fetchone()
else:
att = key.split("/")[1] if key.count("/") >= 2 else ""
referenced = lambda: conn.execute("SELECT 1 FROM attachments WHERE id::text=%s", (att,)).fetchone()
if referenced():
continue
orphans += 1
print(("DELETE " if apply else "WOULD-DELETE ") + key, flush=True)
if apply and not referenced():
s3.delete_object(Bucket=bucket, Key=key); deleted += 1
expired = conn.execute("SELECT id, staging_key FROM attachments WHERE status='failed' AND staging_key IS NOT NULL "
"AND updated_at < now() - make_interval(secs=>%s)", (retention,)).fetchall()
for att_id, skey in expired:
print(("EXPIRE-STAGING " if apply else "WOULD-EXPIRE-STAGING ") + skey, flush=True)
if apply:
s3.delete_object(Bucket=bucket, Key=skey)
conn.execute("UPDATE attachments SET staging_key=NULL WHERE id=%s", (att_id,))
deleted += 1
print(f"sweep: seen={seen} older_than_grace={old} orphans={orphans} failed_expired={len(expired)} deleted={deleted} apply={apply}", flush=True)
return orphans, deleted
def main():
p = argparse.ArgumentParser()
p.add_argument("--apply", action="store_true", help="really delete (default: dry-run)")
p.add_argument("--grace-seconds", type=int, default=int(os.getenv("SWEEP_GRACE_SECONDS", "3600")))
p.add_argument("--loop", action="store_true"); p.add_argument("--interval", type=int, default=int(os.getenv("SWEEP_INTERVAL_SECONDS", "600")))
p.add_argument("--retention-seconds", type=int, default=int(os.getenv("STAGING_RETENTION_SECONDS", "86400")))
a = p.parse_args()
if a.grace_seconds < 0: sys.exit("grace must be >= 0")
s3 = boto3.client("s3", endpoint_url=os.environ["S3_ENDPOINT"], region_name=os.getenv("S3_REGION", "us-east-1"),
aws_access_key_id=os.environ["S3_ACCESS_KEY_ID"], aws_secret_access_key=os.environ["S3_SECRET_ACCESS_KEY"],
config=Config(signature_version="s3v4", s3={"addressing_style": "path"}, connect_timeout=3, read_timeout=15, retries={"max_attempts": 2}))
bucket = os.environ["S3_BUCKET"]
while True:
try:
with psycopg.connect(connect_timeout=5, autocommit=True) as conn:
run_once(s3, conn, bucket, a.grace_seconds, a.apply, a.retention_seconds)
except Exception as exc: # keep looping through outages
print("sweep failed:", type(exc).__name__, flush=True)
if not a.loop: raise
if not a.loop: return
time.sleep(a.interval)
if __name__ == "__main__":
main()
+17
View File
@@ -0,0 +1,17 @@
#!/bin/sh
set -eu
: "${TARGET_INSTANCE_ID:?TARGET_INSTANCE_ID must be supplied by tests/resilience.py}"
found=""
for id in $(docker compose ps -q api); do
host=$(docker inspect --format '{{.Config.Hostname}}' "$id")
if [ "$host" = "$TARGET_INSTANCE_ID" ]; then
found="$id"
break
fi
done
if [ -z "$found" ]; then
echo "No running API container matches TARGET_INSTANCE_ID" >&2
exit 1
fi
# Stop exactly one API replica and intentionally leave it stopped for the failover check.
docker stop --time 10 "$found" >/dev/null
View File
+88
View File
@@ -0,0 +1,88 @@
"""Lab 5 dispatcher: transactional-outbox relay + lease reaper.
* relay : outbox rows (committed together with the state change) -> Kafka (acks=all, idempotent producer, delivery report)
* reaper : processing job with an expired lease (worker died) -> queued again (or failed when attempts are exhausted)
"""
from __future__ import annotations
import logging
import os
import time
from pathlib import Path
import psycopg
from confluent_kafka import Producer
from . import jobs
logging.basicConfig(level=os.getenv("LOG_LEVEL", "INFO"), format="%(asctime)s %(levelname)s dispatcher %(message)s")
log = logging.getLogger("dispatcher")
POLL_SECONDS = float(os.getenv("DISPATCH_POLL_SECONDS", "1"))
BATCH = int(os.getenv("OUTBOX_BATCH", "100"))
FAULT_FLAG = Path("/tmp/dispatcher_fault_done")
FAULT_EXIT_BEFORE_PUBLISH = os.getenv("FAULT_DISPATCHER_EXIT_BEFORE_PUBLISH", "0") == "1" # test hook: crash between COMMIT and publish
def relay(conn, producer: Producer) -> int:
with conn.transaction():
rows = conn.execute("SELECT id, topic, msg_key, payload::text FROM outbox WHERE published_at IS NULL AND available_at <= now() "
"ORDER BY id LIMIT %s FOR UPDATE SKIP LOCKED", (BATCH,)).fetchall()
if not rows:
return 0
if FAULT_EXIT_BEFORE_PUBLISH and not FAULT_FLAG.exists():
FAULT_FLAG.touch()
log.error("FAULT: exiting after reading the outbox, before publishing (COMMIT happened, publish did not)")
os._exit(137)
delivered, errors = set(), {}
for oid, topic, key, payload in rows:
def cb(err, msg, oid=oid):
(errors.__setitem__(oid, str(err)) if err else delivered.add(oid)) # publisher confirm arrives here
producer.produce(topic, key=key.encode(), value=payload.encode(), on_delivery=cb)
producer.flush(15)
if delivered:
conn.execute("UPDATE outbox SET published_at=now() WHERE id = ANY(%s)", (list(delivered),))
if errors:
log.warning("%d outbox rows not confirmed by the broker, will retry", len(errors))
return len(delivered)
def reap(conn) -> int:
n = 0
with conn.transaction():
for job_id, att_id, attempt, max_attempts, pv in conn.execute(
"SELECT job_id, attachment_id, attempt, max_attempts, pipeline_version FROM attachment_jobs "
"WHERE status='processing' AND lease_until < now() FOR UPDATE SKIP LOCKED").fetchall():
if attempt >= max_attempts:
jobs.fail_job(conn, job_id, att_id, "retry_exhausted", attempt)
else:
conn.execute("UPDATE attachment_jobs SET status='queued', lease_until=NULL, next_attempt_at=now(), updated_at=now() WHERE job_id=%s", (job_id,))
conn.execute("UPDATE attachments SET status='queued', updated_at=now() WHERE id=%s", (att_id,))
jobs.add_event(conn, job_id, "lease_expired", f"attempt={attempt}")
jobs.enqueue(conn, job_id, att_id, pv, attempt + 1)
n += 1
return n
def main() -> None:
producer = Producer({"bootstrap.servers": jobs.KAFKA_BOOTSTRAP, "acks": "all", "enable.idempotence": True,
"message.timeout.ms": 15000, "linger.ms": 5})
log.info("dispatcher started (poll=%.1fs)", POLL_SECONDS)
while True:
try:
with psycopg.connect(connect_timeout=5, autocommit=True, options="-c statement_timeout=20000") as conn:
while True:
done = reap(conn)
sent = relay(conn, producer)
if done or sent:
log.info("reaped=%d published=%d", done, sent)
if not sent:
time.sleep(POLL_SECONDS)
except psycopg.Error as exc:
log.warning("database unavailable: %s", type(exc).__name__); time.sleep(2)
except Exception as exc:
log.warning("dispatcher iteration failed: %s", type(exc).__name__); time.sleep(2)
if __name__ == "__main__":
main()
+97
View File
@@ -0,0 +1,97 @@
"""Image pipeline v1. Runs in a CHILD process with hard CPU/address-space limits (called by worker.py).
python -m src.imgproc <input-file> <output-dir>
Writes thumbnail.jpg, crop.jpg, optimized.webp and prints JSON metadata on stdout.
Exit codes: 0 ok, 2 invalid image, 3 too many pixels, 4 out of memory, other = unexpected (treated as transient).
"""
from __future__ import annotations
import hashlib
import json
import os
import resource
import sys
from pathlib import Path
from PIL import Image, ImageOps
MAX_IMAGE_PIXELS = int(os.getenv("MAX_IMAGE_PIXELS", "40000000"))
THUMB_BOX, CROP_SIZE, OPT_BOX = 256, 128, 1280
JPEG_QUALITY, WEBP_QUALITY = 85, 80 # documented in REPORT.md
EXIT_INVALID, EXIT_TOO_LARGE, EXIT_OOM = 2, 3, 4
def fit_box(w: int, h: int, box: int) -> tuple[int, int]:
"""Fit (w,h) into box x box, keep aspect ratio, never upscale, floor, min 1 px. Integer math: 320x200 -> 256x160."""
if w <= box and h <= box:
return w, h
if w >= h:
return box, max(1, h * box // w)
return max(1, w * box // h), box
def flatten_on_white(img: Image.Image) -> Image.Image:
"""RGB image; any alpha is composited on white (JPEG has no alpha)."""
if img.mode in ("RGBA", "LA") or (img.mode == "P" and "transparency" in img.info):
rgba = img.convert("RGBA")
bg = Image.new("RGB", rgba.size, (255, 255, 255))
bg.paste(rgba, mask=rgba.getchannel("A"))
return bg
return img.convert("RGB")
def save(img: Image.Image, path: Path, fmt: str, **opts) -> dict:
img.save(path, fmt, **opts) # no exif=/icc_profile= passed => metadata is stripped
data = path.read_bytes()
return {"size_bytes": len(data), "width": img.width, "height": img.height, "sha256": hashlib.sha256(data).hexdigest()}
def process(src: Path, out: Path) -> dict:
with Image.open(src) as img:
if img.format not in ("PNG", "JPEG"):
raise ValueError("unsupported format")
if img.width < 1 or img.height < 1:
raise ValueError("empty image")
if img.width * img.height > MAX_IMAGE_PIXELS: # decided from the header, before any pixel buffer exists
raise OverflowError("too many pixels")
img.load()
base = ImageOps.exif_transpose(img) # apply EXIF orientation once; result has no orientation tag
base.load()
result = {}
# thumbnail: JPEG, fit 256x256, no upscale, alpha -> white
flat = flatten_on_white(base)
t = flat.resize(fit_box(*flat.size, THUMB_BOX), Image.LANCZOS) if fit_box(*flat.size, THUMB_BOX) != flat.size else flat
result["thumbnail"] = {"content_type": "image/jpeg", **save(t, out / "thumbnail.jpg", "JPEG", quality=JPEG_QUALITY, optimize=True)}
# crop: JPEG, centered square 128x128 (small sources may be upscaled)
c = ImageOps.fit(flat, (CROP_SIZE, CROP_SIZE), Image.LANCZOS, centering=(0.5, 0.5))
result["crop"] = {"content_type": "image/jpeg", **save(c, out / "crop.jpg", "JPEG", quality=JPEG_QUALITY, optimize=True)}
# optimized: WebP, fit 1280x1280, no upscale, keeps alpha, lossy q=80, no EXIF/ICC
o = base if base.mode in ("RGB", "RGBA") else (base.convert("RGBA") if "A" in base.getbands() or base.mode == "P" else base.convert("RGB"))
size = fit_box(*o.size, OPT_BOX)
if size != o.size:
o = o.resize(size, Image.LANCZOS)
result["optimized"] = {"content_type": "image/webp", **save(o, out / "optimized.webp", "WEBP", quality=WEBP_QUALITY, method=4)}
return result
def main() -> int:
mem_mb, cpu_s = int(os.getenv("IMG_MEMORY_MB", "1024")), int(os.getenv("IMG_CPU_SECONDS", "60"))
resource.setrlimit(resource.RLIMIT_AS, (mem_mb * 1024 * 1024, mem_mb * 1024 * 1024))
resource.setrlimit(resource.RLIMIT_CPU, (cpu_s, cpu_s + 5))
Image.MAX_IMAGE_PIXELS = MAX_IMAGE_PIXELS
try:
print(json.dumps(process(Path(sys.argv[1]), Path(sys.argv[2]))))
return 0
except OverflowError:
return EXIT_TOO_LARGE
except Image.DecompressionBombError:
return EXIT_TOO_LARGE
except MemoryError:
return EXIT_OOM
except (ValueError, OSError, SyntaxError, Image.UnidentifiedImageError):
return EXIT_INVALID
if __name__ == "__main__":
sys.exit(main())
+81
View File
@@ -0,0 +1,81 @@
"""Shared constants and helpers for the API, the worker and the dispatcher (lab 5)."""
from __future__ import annotations
import json
import os
import random
import uuid
from datetime import datetime, timezone
import boto3
from botocore.config import Config as BotoConfig
KAFKA_BOOTSTRAP = os.getenv("KAFKA_BOOTSTRAP_SERVERS", "kafka:9092")
JOBS_TOPIC = os.getenv("JOBS_TOPIC", "attachment-jobs")
PIPELINE_VERSION = 1
JOB_TYPE = "attachment.process.v1"
JOB_MAX_ATTEMPTS = int(os.getenv("JOB_MAX_ATTEMPTS", "5"))
JOB_TIMEOUT_SECONDS = int(os.getenv("JOB_TIMEOUT_SECONDS", "60"))
LEASE_GRACE_SECONDS = int(os.getenv("LEASE_GRACE_SECONDS", "15"))
LEASE_SECONDS = JOB_TIMEOUT_SECONDS + LEASE_GRACE_SECONDS # lease > hard job timeout: a live worker never loses it
BACKOFF_BASE_SECONDS = float(os.getenv("JOB_BACKOFF_BASE_SECONDS", "2"))
BACKOFF_CAP_SECONDS = float(os.getenv("JOB_BACKOFF_CAP_SECONDS", "60"))
S3_ENDPOINT = os.getenv("S3_ENDPOINT", "http://s3:9000")
S3_REGION = os.getenv("S3_REGION", "us-east-1")
S3_BUCKET = os.getenv("S3_BUCKET", "messenger-attachments")
STAGING_PREFIX = "staging/"
FINAL_PREFIX = "attachments/"
VARIANT_ORDER = ("thumbnail", "crop", "optimized")
def final_key(attachment_id, pipeline_version: int, name: str) -> str:
"""Deterministic key: the effect key (attachment_id, pipeline_version, variant) maps 1:1 to an S3 object."""
return f"{FINAL_PREFIX}{attachment_id}/v{pipeline_version}/{name}"
def make_s3():
return boto3.client(
"s3", endpoint_url=S3_ENDPOINT, region_name=S3_REGION,
aws_access_key_id=os.getenv("S3_ACCESS_KEY_ID", ""), aws_secret_access_key=os.getenv("S3_SECRET_ACCESS_KEY", ""),
config=BotoConfig(signature_version="s3v4", s3={"addressing_style": "path"}, connect_timeout=2, read_timeout=20,
retries={"max_attempts": 2, "mode": "standard"},
request_checksum_calculation="when_required", response_checksum_validation="when_required"))
def backoff_seconds(attempt: int) -> float:
"""Exponential backoff with 'equal jitter': delay in [d/2, d], d = min(cap, base * 2^(attempt-1))."""
d = min(BACKOFF_CAP_SECONDS, BACKOFF_BASE_SECONDS * (2 ** max(0, attempt - 1)))
return random.uniform(d / 2, d)
def iso_now() -> str:
return datetime.now(timezone.utc).isoformat(timespec="seconds").replace("+00:00", "Z")
def add_event(conn, job_id, event: str, detail: str | None = None) -> None:
conn.execute("INSERT INTO job_events(job_id,event,detail) VALUES (%s,%s,%s)", (job_id, event, detail))
def enqueue(conn, job_id, attachment_id, pipeline_version: int, attempt: int, delay_seconds: float = 0.0) -> None:
"""Record the intent to publish a job message in the SAME transaction as the state change (transactional outbox).
The message carries only IDs (jobs.schema.json): no bytes, no credentials, no URLs.
"""
payload = {"schema_version": 1, "job_id": str(job_id), "type": JOB_TYPE, "attachment_id": str(attachment_id),
"pipeline_version": pipeline_version, "attempt": attempt, "created_at": iso_now()}
conn.execute(
"INSERT INTO outbox(job_id,topic,msg_key,payload,available_at) VALUES (%s,%s,%s,%s::jsonb, now() + make_interval(secs => %s))",
(job_id, JOBS_TOPIC, str(attachment_id), json.dumps(payload), delay_seconds))
def new_job_id() -> uuid.UUID:
return uuid.uuid4()
def fail_job(conn, job_id, att_id, code: str, attempt: int) -> None:
conn.execute("UPDATE attachment_jobs SET status='failed', error_code=%s, lease_until=NULL, updated_at=now() WHERE job_id=%s", (code, job_id))
conn.execute("UPDATE attachments SET status='failed', error_code=%s, updated_at=now() WHERE id=%s", (code, att_id))
add_event(conn, job_id, "failed", code)
+772
View File
@@ -0,0 +1,772 @@
"""Lab 5 REST API: FastAPI + PostgreSQL + Redis sessions + idempotency + S3 attachments with ASYNC processing (Kafka + outbox)."""
from __future__ import annotations
import asyncio
import anyio
import base64
import hashlib
import json
import logging
import os
import re
import secrets
import socket
import tempfile
import unicodedata
import uuid
from datetime import datetime, timedelta, timezone
from pathlib import Path
from typing import Any
from urllib.parse import quote
import psycopg
from psycopg_pool import ConnectionPool, PoolTimeout
import anyio
import boto3
import redis
from botocore.config import Config as BotoConfig
from botocore.exceptions import BotoCoreError, ClientError
from PIL import Image
from redis.exceptions import RedisError
from argon2 import PasswordHasher
from argon2.exceptions import InvalidHashError, VerificationError, VerifyMismatchError
from fastapi import FastAPI, Request
from fastapi.exceptions import RequestValidationError
from fastapi.responses import JSONResponse, Response, StreamingResponse
from starlette.concurrency import iterate_in_threadpool, run_in_threadpool
from starlette.requests import ClientDisconnect
from pydantic import BaseModel, ConfigDict, Field, ValidationError
from starlette.exceptions import HTTPException as StarletteHTTPException
from . import jobs
logging.basicConfig(level=os.getenv("LOG_LEVEL", "INFO"))
log = logging.getLogger("messenger")
DATABASE_URL = os.getenv("DATABASE_URL", "")
INSTANCE_ID = os.getenv("INSTANCE_ID") or socket.gethostname()
IDEMPOTENCY_KEY_RE = re.compile(r"^[A-Za-z0-9._:-]{1,128}$")
REDIS_URL = os.getenv("REDIS_URL", "redis://localhost:6379/0")
SESSION_TTL_SECONDS = int(os.getenv("SESSION_TTL_SECONDS", "3600"))
if SESSION_TTL_SECONDS < 1:
raise RuntimeError("SESSION_TTL_SECONDS must be positive")
# Redis uses its connection pool; session keys contain only a hash of the opaque token.
session_store = redis.Redis.from_url(REDIS_URL, decode_responses=True, socket_connect_timeout=3, socket_timeout=3, health_check_interval=30)
DB_POOL_MIN_SIZE = int(os.getenv("DB_POOL_MIN_SIZE", "1"))
DB_POOL_MAX_SIZE = int(os.getenv("DB_POOL_MAX_SIZE", "10"))
DB_POOL_TIMEOUT_SECONDS = float(os.getenv("DB_POOL_TIMEOUT_SECONDS", "5"))
if DB_POOL_MIN_SIZE < 1 or DB_POOL_MAX_SIZE < DB_POOL_MIN_SIZE:
raise RuntimeError("Invalid PostgreSQL pool size configuration")
# Every pooled connection gets bounded connect/query/idle-transaction timeouts.
db_pool = ConnectionPool(
conninfo=DATABASE_URL,
min_size=DB_POOL_MIN_SIZE,
max_size=DB_POOL_MAX_SIZE,
timeout=DB_POOL_TIMEOUT_SECONDS,
kwargs={
"connect_timeout": 5,
"options": "-c statement_timeout=5000 -c idle_in_transaction_session_timeout=10000",
},
open=False,
name="messenger-postgres",
)
# ---- Lab 4: S3 / attachment limits (names follow .env.example of the lab) ----
S3_ENDPOINT = os.getenv("S3_ENDPOINT", "http://s3:9000")
S3_REGION = os.getenv("S3_REGION", "us-east-1")
S3_BUCKET = os.getenv("S3_BUCKET", "messenger-attachments")
S3_ACCESS_KEY_ID = os.getenv("S3_ACCESS_KEY_ID", "")
S3_SECRET_ACCESS_KEY = os.getenv("S3_SECRET_ACCESS_KEY", "")
S3_KEY_PREFIX = "attachments/"
STAGING_PREFIX = jobs.STAGING_PREFIX # lab5: durable staging (private S3 prefix) for bytes accepted but not yet processed
MAX_UPLOAD_BYTES = int(os.getenv("MAX_UPLOAD_BYTES", "10485760"))
MAX_IMAGE_PIXELS = int(os.getenv("MAX_IMAGE_PIXELS", "40000000"))
UPLOAD_IDLE_TIMEOUT_SECONDS = float(os.getenv("UPLOAD_IDLE_TIMEOUT_SECONDS", "10"))
UPLOAD_TOTAL_TIMEOUT_SECONDS = float(os.getenv("UPLOAD_TOTAL_TIMEOUT_SECONDS", "60"))
IMAGE_DECODE_CONCURRENCY = int(os.getenv("IMAGE_DECODE_CONCURRENCY", "1"))
SPOOL_MEMORY_BYTES = 1024 * 1024 # larger bodies roll over to an anonymous temp file, never to RAM
CHUNK_BYTES = 64 * 1024
# Pillow's own bomb guard is a second line of defence behind the explicit check below.
Image.MAX_IMAGE_PIXELS = MAX_IMAGE_PIXELS
# Credentials are the restricted application user created by s3_init, never the MinIO root user.
_s3_common = dict(endpoint_url=S3_ENDPOINT, region_name=S3_REGION, aws_access_key_id=S3_ACCESS_KEY_ID, aws_secret_access_key=S3_SECRET_ACCESS_KEY)
s3 = boto3.client("s3", **_s3_common, config=BotoConfig(
signature_version="s3v4", s3={"addressing_style": "path"}, connect_timeout=2, read_timeout=20,
retries={"max_attempts": 2, "mode": "standard"}, max_pool_connections=32,
request_checksum_calculation="when_required", response_checksum_validation="when_required"))
# Readiness probe client: one attempt and tight timeouts so HAProxy's 2 s check never hangs on it.
s3_probe = boto3.client("s3", **_s3_common, config=BotoConfig(
signature_version="s3v4", s3={"addressing_style": "path"}, connect_timeout=1, read_timeout=1,
retries={"max_attempts": 1}, request_checksum_calculation="when_required", response_checksum_validation="when_required"))
decode_limiter = anyio.CapacityLimiter(IMAGE_DECODE_CONCURRENCY)
IMAGE_TYPES = {"PNG": "image/png", "JPEG": "image/jpeg"}
USERNAME_RE = re.compile(r"^[a-z0-9_]{3,32}$")
PASSWORDS = PasswordHasher()
class ApiError(Exception):
def __init__(self, status: int, code: str, message: str, headers: dict[str, str] | None = None):
self.status, self.code, self.message, self.headers = status, code, message, headers or {}
def fail(status: int, code: str, message: str, headers: dict[str, str] | None = None):
raise ApiError(status, code, message, headers)
class StrictModel(BaseModel):
model_config = ConfigDict(extra="forbid", strict=True)
class RegisterBody(StrictModel):
username: str
display_name: str
password: str
class ChatBody(StrictModel):
title: str
member_ids: list[str] = Field(min_length=1, max_length=100)
class MessageBody(StrictModel):
text: str
attachment_ids: list[str] = Field(default_factory=list, max_length=10)
app = FastAPI(docs_url=None, redoc_url=None, openapi_url=None)
@app.middleware("http")
async def request_id_middleware(request: Request, call_next):
request.state.request_id = uuid.uuid4().hex
try:
response = await call_next(request)
except Exception:
log.exception("Unhandled API exception request_id=%s", request.state.request_id)
response = JSONResponse(status_code=500, content={"error": {"code": "internal_error", "message": "Internal server error", "request_id": request.state.request_id}})
response.headers["X-Request-Id"] = request.state.request_id
response.headers["X-Instance-Id"] = INSTANCE_ID
return response
def error_response(status: int, code: str, message: str, request_id: str, headers: dict[str, str] | None = None):
return JSONResponse(status_code=status, content={"error": {"code": code, "message": message, "request_id": request_id}}, headers=headers or {})
@app.exception_handler(ApiError)
async def api_error_handler(request: Request, exc: ApiError):
return error_response(exc.status, exc.code, exc.message, request.state.request_id, exc.headers)
@app.exception_handler(PoolTimeout)
async def database_pool_timeout_handler(request: Request, exc: PoolTimeout):
log.warning("Database connection pool timed out request_id=%s", request.state.request_id)
return error_response(503, "unavailable", "Required storage is unavailable", request.state.request_id)
@app.exception_handler(psycopg.Error)
async def database_error_handler(request: Request, exc: psycopg.Error):
log.warning("Database operation failed request_id=%s", request.state.request_id)
return error_response(503, "unavailable", "Required storage is unavailable", request.state.request_id)
@app.exception_handler(RedisError)
async def redis_error_handler(request: Request, exc: RedisError):
log.warning("Session store operation failed request_id=%s", request.state.request_id)
return error_response(503, "unavailable", "Required storage is unavailable", request.state.request_id)
@app.exception_handler(BotoCoreError)
async def s3_core_error_handler(request: Request, exc: BotoCoreError):
log.warning("Object storage failure request_id=%s type=%s", request.state.request_id, type(exc).__name__)
return error_response(503, "unavailable", "Required storage is unavailable", request.state.request_id)
@app.exception_handler(ClientError)
async def s3_client_error_handler(request: Request, exc: ClientError):
log.warning("Object storage error request_id=%s code=%s", request.state.request_id, exc.response.get("Error", {}).get("Code"))
return error_response(503, "unavailable", "Required storage is unavailable", request.state.request_id)
@app.exception_handler(RequestValidationError)
async def validation_error_handler(request: Request, exc: RequestValidationError):
return error_response(400, "invalid_request", "Invalid request", request.state.request_id)
@app.exception_handler(StarletteHTTPException)
async def http_error_handler(request: Request, exc: StarletteHTTPException):
status = exc.status_code
code = "not_found" if status == 404 else "invalid_request"
message = "Object not found" if status == 404 else "Invalid request"
headers = dict(exc.headers or {})
return error_response(status, code, message, request.state.request_id, headers)
def db():
"""Borrow a bounded pooled connection; context exit commits/rolls back and returns it."""
return db_pool.connection(timeout=DB_POOL_TIMEOUT_SECONDS)
@app.on_event("startup")
def startup():
# Migrations run once in the dedicated `migrate` service before API replicas.
import time
db_pool.open(wait=False)
last_error = None
for _ in range(30):
try:
with db() as conn:
conn.execute("SELECT 1")
conn.execute("SELECT 1 FROM idempotency_records LIMIT 0")
conn.execute("SELECT 1 FROM outbox LIMIT 0")
session_store.ping()
s3.head_object(Bucket=S3_BUCKET, Key=S3_KEY_PREFIX + ".ready")
log.info("PostgreSQL, Redis and S3 bucket are ready; instance=%s", INSTANCE_ID)
return
except Exception as exc:
last_error = exc
time.sleep(1)
raise RuntimeError("PostgreSQL schema, Redis or S3 bucket unavailable during startup") from last_error
@app.on_event("shutdown")
def shutdown():
session_store.close()
db_pool.close()
def now_utc() -> datetime:
return datetime.now(timezone.utc)
def iso(value: datetime) -> str:
return value.astimezone(timezone.utc).isoformat(timespec="microseconds").replace("+00:00", "Z")
def user_dict(row: dict[str, Any]) -> dict[str, Any]:
return {"id": str(row["id"]), "username": row["username"], "display_name": row["display_name"], "created_at": iso(row["created_at"])}
def chat_dict(row: dict[str, Any], member_ids: list[str]) -> dict[str, Any]:
return {"id": str(row["id"]), "title": row["title"], "member_ids": member_ids, "created_at": iso(row["created_at"])}
def message_dict(row: dict[str, Any], attachment_ids: list[str] | None = None) -> dict[str, Any]:
return {"id": str(row["id"]), "chat_id": str(row["chat_id"]), "sender_id": str(row["sender_id"]), "text": row["text"], "created_at": iso(row["created_at"]), "attachment_ids": attachment_ids or []}
def validate_unicode(value: str, label: str) -> str:
"""Reject unpaired UTF-16 surrogates before sending text to PostgreSQL/KDF."""
try:
value.encode("utf-8")
except UnicodeEncodeError:
fail(400, "invalid_request", "Invalid Unicode in " + label)
return value
def parse_uuid(value: str, label: str = "UUID") -> uuid.UUID:
try:
# Keep the canonical UUID representation used by the Lab 1 API.
if not isinstance(value, str) or len(value) != 36:
raise ValueError("non-canonical UUID")
parsed = uuid.UUID(value)
if str(parsed) != value:
raise ValueError("non-canonical UUID")
return parsed
except (ValueError, TypeError, AttributeError):
fail(400, "invalid_request", "Invalid " + label)
def validate_body_type(request: Request):
content_type = request.headers.get("content-type", "").split(";", 1)[0].strip().lower()
if content_type != "application/json":
fail(415, "unsupported_media_type", "Content-Type must be application/json")
async def parse_body(request: Request, model):
validate_body_type(request)
try:
raw = await request.body()
data = json.loads(raw.decode("utf-8"))
return model.model_validate(data)
except (UnicodeDecodeError, json.JSONDecodeError, ValidationError, TypeError):
fail(400, "invalid_request", "Invalid JSON request body")
def challenge(scheme: str) -> dict[str, str]:
if scheme == "Basic":
return {"WWW-Authenticate": 'Basic realm="messenger", charset="UTF-8"'}
return {"WWW-Authenticate": 'Bearer realm="messenger"'}
def current_user(request: Request) -> dict[str, Any]:
auth = request.headers.get("authorization", "")
if not auth.startswith("Bearer ") or not auth[7:].strip():
fail(401, "unauthorized", "Valid Bearer session required", challenge("Bearer"))
token_hash = hashlib.sha256(auth[7:].strip().encode("utf-8")).hexdigest()
session_json = session_store.get("session:" + token_hash)
if session_json is None:
fail(401, "unauthorized", "Valid Bearer session required", challenge("Bearer"))
try:
session = json.loads(session_json)
user_id = uuid.UUID(session["user_id"])
expires_at = datetime.fromisoformat(session["expires_at"].replace("Z", "+00:00"))
except (ValueError, KeyError, TypeError, json.JSONDecodeError):
# Corrupt session data is invalidated instead of trusted.
session_store.delete("session:" + token_hash)
fail(401, "unauthorized", "Valid Bearer session required", challenge("Bearer"))
if expires_at <= now_utc():
session_store.delete("session:" + token_hash)
fail(401, "unauthorized", "Valid Bearer session required", challenge("Bearer"))
with db() as conn:
row = conn.execute("SELECT id,username,display_name,created_at FROM users WHERE id=%s", (user_id,)).fetchone()
if row is None:
session_store.delete("session:" + token_hash)
fail(401, "unauthorized", "Valid Bearer session required", challenge("Bearer"))
return {"id": row[0], "username": row[1], "display_name": row[2], "created_at": row[3], "expires_at": expires_at, "user": {"id": str(row[0]), "username": row[1], "display_name": row[2], "created_at": iso(row[3])}, "token_hash": token_hash}
def ensure_member(actor_id: uuid.UUID, chat_id: uuid.UUID):
with db() as conn:
row = conn.execute("SELECT 1 FROM chats c JOIN chat_members m ON m.chat_id=c.id WHERE c.id=%s AND m.user_id=%s", (chat_id, actor_id)).fetchone()
if row is None:
fail(404, "not_found", "Object not found")
def parse_page_args(limit: int | None, cursor: str | None, resource: str):
actual_limit = 50 if limit is None else limit
if isinstance(actual_limit, bool) or actual_limit < 1 or actual_limit > 100:
fail(400, "invalid_request", "limit must be an integer from 1 to 100")
after = None
if cursor is not None:
if not cursor:
fail(400, "invalid_request", "Invalid cursor")
try:
decoded = base64.urlsafe_b64decode(cursor + "=" * (-len(cursor) % 4)).decode("utf-8")
obj = json.loads(decoded)
if obj.get("resource") != resource:
fail(400, "invalid_request", "Cursor belongs to another resource")
after = (datetime.fromisoformat(obj["created_at"].replace("Z", "+00:00")), uuid.UUID(obj["id"]))
except ApiError:
raise
except Exception:
fail(400, "invalid_request", "Invalid cursor")
return actual_limit, after
def make_cursor(resource: str, created_at: datetime, item_id: uuid.UUID) -> str:
raw = json.dumps({"resource": resource, "created_at": iso(created_at), "id": str(item_id)}, separators=(",", ":")).encode()
return base64.urlsafe_b64encode(raw).decode().rstrip("=")
def json_rows(cursor, query: str, params=()):
with db() as conn:
cur = conn.execute(query, params)
names = [d.name for d in cur.description]
return [dict(zip(names, row)) for row in cur.fetchall()]
@app.get("/health/live")
def live():
return {"status": "ok"}
@app.get("/health/ready")
def ready():
try:
with db() as conn:
conn.execute("SELECT 1")
session_store.ping()
s3_probe.head_object(Bucket=S3_BUCKET, Key=S3_KEY_PREFIX + ".ready") # S3 is critical: uploads/downloads are part of the mandatory path
return {"status": "ok"}
except (psycopg.Error, PoolTimeout, RedisError, BotoCoreError, ClientError):
fail(503, "unavailable", "Required storage is unavailable")
@app.post("/api/v1/users", status_code=201)
async def register(request: Request):
body = await parse_body(request, RegisterBody)
if not USERNAME_RE.fullmatch(body.username):
fail(400, "invalid_request", "Invalid username")
validate_unicode(body.display_name, "display_name")
validate_unicode(body.password, "password")
if not body.display_name.strip() or len(body.display_name) > 100:
fail(400, "invalid_request", "Invalid display_name")
if not 12 <= len(body.password) <= 128:
fail(400, "invalid_request", "Password must contain 12 to 128 Unicode code points")
user_id = uuid.uuid4()
password_hash = PASSWORDS.hash(body.password)
try:
with db() as conn:
row = conn.execute("INSERT INTO users(id,username,display_name,password_hash) VALUES (%s,%s,%s,%s) RETURNING id,username,display_name,created_at", (user_id, body.username, body.display_name, password_hash)).fetchone()
except psycopg.errors.UniqueViolation:
fail(409, "conflict", "Username already exists")
return user_dict(dict(zip(("id", "username", "display_name", "created_at"), row)))
@app.get("/api/v1/users/me")
def me(request: Request):
return current_user(request)["user"]
@app.post("/api/v1/auth/sessions", status_code=201)
async def create_session(request: Request):
if await request.body():
fail(400, "invalid_request", "Session creation does not accept a request body")
auth = request.headers.get("authorization", "")
if not auth[:6].lower() == "basic ":
fail(401, "unauthorized", "Valid Basic credentials required", challenge("Basic"))
try:
decoded = base64.b64decode(auth[6:], validate=True).decode("utf-8")
username, password = decoded.split(":", 1)
except Exception:
fail(401, "unauthorized", "Invalid credentials", challenge("Basic"))
try:
validate_unicode(password, "password")
except ApiError:
fail(401, "unauthorized", "Invalid credentials", challenge("Basic"))
if not password or len(password) < 12 or len(password) > 128:
fail(401, "unauthorized", "Invalid credentials", challenge("Basic"))
with db() as conn:
row = conn.execute("SELECT id,username,display_name,password_hash,created_at FROM users WHERE username=%s", (username,)).fetchone()
if row is None:
fail(401, "unauthorized", "Invalid credentials", challenge("Basic"))
try:
PASSWORDS.verify(row[3], password)
except (VerifyMismatchError, VerificationError, InvalidHashError):
fail(401, "unauthorized", "Invalid credentials", challenge("Basic"))
token = secrets.token_urlsafe(32)
token_hash = hashlib.sha256(token.encode()).hexdigest()
expires_at = now_utc() + timedelta(seconds=SESSION_TTL_SECONDS)
session_json = json.dumps({"user_id": str(row[0]), "expires_at": iso(expires_at)}, separators=(",", ":"))
# Redis TTL enforces expiry; AOF + named volume make active sessions survive stack restarts.
session_store.setex("session:" + token_hash, SESSION_TTL_SECONDS, session_json)
user = user_dict({"id": row[0], "username": row[1], "display_name": row[2], "created_at": row[4]})
return {"token": token, "token_type": "Bearer", "expires_at": iso(expires_at), "user": user}
@app.delete("/api/v1/auth/sessions/current", status_code=204)
def revoke_session(request: Request):
actor = current_user(request)
session_store.delete("session:" + actor["token_hash"])
return Response(status_code=204)
@app.post("/api/v1/chats", status_code=201)
async def create_chat(request: Request):
actor = current_user(request)
body = await parse_body(request, ChatBody)
validate_unicode(body.title, "title")
if not body.title.strip() or len(body.title) > 100:
fail(400, "invalid_request", "Invalid title")
ids = [parse_uuid(value, "member UUID") for value in body.member_ids]
if len(ids) != len(set(ids)) or actor["id"] not in ids:
fail(400, "invalid_request", "Members must be unique and include the caller")
chat_id = uuid.uuid4()
try:
with db() as conn:
existing = conn.execute("SELECT id FROM users WHERE id = ANY(%s)", (ids,)).fetchall()
if len(existing) != len(ids):
fail(400, "invalid_request", "Unknown chat member")
row = conn.execute("INSERT INTO chats(id,title) VALUES (%s,%s) RETURNING id,title,created_at", (chat_id, body.title)).fetchone()
# psycopg 3: executemany exists on cursors, not on Connection (fix of a latent lab3 bug)
with conn.cursor() as cur:
cur.executemany("INSERT INTO chat_members(chat_id,user_id,position) VALUES (%s,%s,%s)", [(chat_id, uid, position) for position, uid in enumerate(ids)])
except psycopg.errors.ForeignKeyViolation:
fail(400, "invalid_request", "Unknown chat member")
return chat_dict({"id": row[0], "title": row[1], "created_at": row[2]}, [str(x) for x in ids])
@app.get("/api/v1/chats")
def list_chats(request: Request, limit: int | None = None, cursor: str | None = None):
actor = current_user(request)
chat_list_resource = "chats:" + str(actor["id"])
actual_limit, after = parse_page_args(limit, cursor, chat_list_resource)
params: list[Any] = [actor["id"]]
where = "m.user_id=%s"
if after:
where += " AND (c.created_at,c.id)>(%s,%s)"
params.extend(after)
params.append(actual_limit + 1)
rows = json_rows(None, f"SELECT c.id,c.title,c.created_at FROM chats c JOIN chat_members m ON m.chat_id=c.id WHERE {where} ORDER BY c.created_at,c.id LIMIT %s", tuple(params))
more = len(rows) > actual_limit
rows = rows[:actual_limit]
items = []
for row in rows:
members = json_rows(None, "SELECT user_id FROM chat_members WHERE chat_id=%s ORDER BY position,user_id", (row["id"],))
items.append(chat_dict(row, [str(m["user_id"]) for m in members]))
next_cursor = make_cursor(chat_list_resource, rows[-1]["created_at"], rows[-1]["id"]) if more and rows else None
return {"items": items, "next_cursor": next_cursor}
@app.get("/api/v1/chats/{chat_id}")
def get_chat(chat_id: str, request: Request):
actor = current_user(request)
parsed = parse_uuid(chat_id, "chat UUID")
ensure_member(actor["id"], parsed)
rows = json_rows(None, "SELECT id,title,created_at FROM chats WHERE id=%s", (parsed,))
members = json_rows(None, "SELECT user_id FROM chat_members WHERE chat_id=%s ORDER BY position,user_id", (parsed,))
return chat_dict(rows[0], [str(m["user_id"]) for m in members])
@app.post("/api/v1/chats/{chat_id}/messages", status_code=201)
async def create_message(chat_id: str, request: Request):
actor = current_user(request)
parsed_chat = parse_uuid(chat_id, "chat UUID")
ensure_member(actor["id"], parsed_chat)
body = await parse_body(request, MessageBody)
validate_unicode(body.text, "text")
attachment_ids = [parse_uuid(value, "attachment UUID") for value in body.attachment_ids]
if len(attachment_ids) != len(set(attachment_ids)):
fail(400, "invalid_request", "attachment_ids must be unique")
if len(body.text) > 4000 or (not body.text.strip() and not attachment_ids):
fail(400, "invalid_request", "Message needs 1 to 4000 non-whitespace code points of text or at least one attachment")
key = request.headers.get("Idempotency-Key", "")
if not IDEMPOTENCY_KEY_RE.fullmatch(key):
fail(400, "invalid_request", "Idempotency-Key must contain 1 to 128 ASCII letters, digits, '.', '_', ':', or '-'")
# Hash semantic JSON rather than raw bytes, so insignificant JSON formatting does not matter.
# attachment_ids is part of the hash only when non-empty: absent == [] and lab3 records stay valid.
canonical = {"text": body.text}
if attachment_ids:
canonical["attachment_ids"] = [str(x) for x in attachment_ids]
canonical_body = json.dumps(canonical, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
request_hash = hashlib.sha256(canonical_body.encode("utf-8")).hexdigest()
scope = f"{actor['id']}|POST|/api/v1/chats/{parsed_chat}/messages|{key}"
# The advisory transaction lock serializes the same key across every API replica.
# Message and idempotency result are committed atomically in the same PostgreSQL transaction.
with db() as conn:
conn.execute("SELECT pg_advisory_xact_lock(hashtextextended(%s, 0))", (scope,))
existing = conn.execute(
"SELECT request_hash, response_body FROM idempotency_records "
"WHERE user_id=%s AND chat_id=%s AND idempotency_key=%s AND created_at >= now() - interval '24 hours'",
(actor["id"], parsed_chat, key),
).fetchone()
if existing is not None:
if existing[0] != request_hash:
fail(409, "conflict", "Idempotency-Key was already used with a different request body")
replay = dict(existing[1])
replay.setdefault("attachment_ids", [])
return JSONResponse(status_code=201, content=replay)
# Keys are guaranteed for 24 hours; expired records may be replaced by a new attempt.
conn.execute(
"DELETE FROM idempotency_records WHERE user_id=%s AND chat_id=%s AND idempotency_key=%s AND created_at < now() - interval '24 hours'",
(actor["id"], parsed_chat, key),
)
if attachment_ids:
# Foreign, unknown and other-chat attachments are indistinguishable: all -> 404.
found = conn.execute("SELECT count(*) FROM attachments WHERE id = ANY(%s) AND chat_id=%s", (attachment_ids, parsed_chat)).fetchone()[0]
if found != len(attachment_ids):
fail(404, "not_found", "Object not found")
# lab5: queued/processing attachments may be attached, failed ones may not
if conn.execute("SELECT 1 FROM attachments WHERE id = ANY(%s) AND status='failed' LIMIT 1", (attachment_ids,)).fetchone():
fail(409, "conflict", "Attachment processing failed")
row = conn.execute(
"INSERT INTO messages(id,chat_id,sender_id,text) VALUES (%s,%s,%s,%s) "
"RETURNING id,chat_id,sender_id,text,created_at",
(uuid.uuid4(), parsed_chat, actor["id"], body.text),
).fetchone()
result = message_dict(dict(zip(("id", "chat_id", "sender_id", "text", "created_at"), row)), [str(x) for x in attachment_ids])
if attachment_ids:
with conn.cursor() as cur:
cur.executemany("INSERT INTO message_attachments(message_id,attachment_id,position) VALUES (%s,%s,%s)", [(row[0], aid, i) for i, aid in enumerate(attachment_ids)])
conn.execute(
"INSERT INTO idempotency_records(user_id,chat_id,idempotency_key,request_hash,response_body) "
"VALUES (%s,%s,%s,%s,%s::jsonb)",
(actor["id"], parsed_chat, key, request_hash, json.dumps(result, ensure_ascii=False)),
)
return result
@app.get("/api/v1/chats/{chat_id}/messages")
def list_messages(chat_id: str, request: Request, limit: int | None = None, cursor: str | None = None):
actor = current_user(request)
parsed_chat = parse_uuid(chat_id, "chat UUID")
ensure_member(actor["id"], parsed_chat)
actual_limit, after = parse_page_args(limit, cursor, "messages:" + str(parsed_chat))
params: list[Any] = [parsed_chat]
where = "chat_id=%s"
if after:
where += " AND (created_at,id)>(%s,%s)"
params.extend(after)
params.append(actual_limit + 1)
rows = json_rows(None, f"SELECT id,chat_id,sender_id,text,created_at FROM messages WHERE {where} ORDER BY created_at,id LIMIT %s", tuple(params))
more = len(rows) > actual_limit
rows = rows[:actual_limit]
links: dict[Any, list[str]] = {}
if rows:
link_rows = json_rows(None, "SELECT message_id, attachment_id FROM message_attachments WHERE message_id = ANY(%s) ORDER BY message_id, position", ([r["id"] for r in rows],))
for link in link_rows:
links.setdefault(link["message_id"], []).append(str(link["attachment_id"]))
items = [message_dict(row, links.get(row["id"])) for row in rows]
next_cursor = make_cursor("messages:" + str(parsed_chat), rows[-1]["created_at"], rows[-1]["id"]) if more and rows else None
return {"items": items, "next_cursor": next_cursor}
# ======================= Lab 4: attachments =======================
def attachment_dict(row: dict[str, Any], variants: list[dict[str, Any]] | None = None) -> dict[str, Any]:
return {"id": str(row["id"]), "chat_id": str(row["chat_id"]), "owner_id": str(row["owner_id"]), "filename": row["filename"],
"content_type": row["content_type"], "size_bytes": row["size_bytes"], "sha256": row["sha256"], "status": row["status"],
"variants": variants or [], "error_code": row["error_code"] if row["status"] == "failed" else None,
"created_at": iso(row["created_at"]), "updated_at": iso(row["updated_at"])}
def validate_filename(values: list[str]) -> str:
"""filename is display-only: it never reaches a path or an S3 key, but dangerous values are still rejected."""
if len(values) != 1:
fail(400, "invalid_request", "Exactly one filename query parameter is required")
name = values[0]
validate_unicode(name, "filename")
if not 1 <= len(name) <= 255 or not name.strip() or name in (".", ".."):
fail(400, "invalid_request", "Invalid filename")
if "/" in name or "\\" in name or any(unicodedata.category(ch) == "Cc" for ch in name):
fail(400, "invalid_request", "Invalid filename")
return name
def sniff_image_type(head: bytes) -> str | None:
if head.startswith(b"\x89PNG\r\n\x1a\n"):
return "image/png"
if head.startswith(b"\xff\xd8\xff"):
return "image/jpeg"
return None
async def read_body_to_spool(request: Request, declared: str):
"""Stream the request body into a bounded spool while hashing; abort as soon as a limit is broken.
RAM per upload stays ~1 MiB: bodies above SPOOL_MEMORY_BYTES roll over to an anonymous (already unlinked) temp file.
"""
spool = tempfile.SpooledTemporaryFile(max_size=SPOOL_MEMORY_BYTES)
digest, total, head = hashlib.sha256(), 0, b""
try:
iterator = request.stream().__aiter__()
with anyio.fail_after(UPLOAD_TOTAL_TIMEOUT_SECONDS):
while True:
try:
chunk = await asyncio.wait_for(iterator.__anext__(), UPLOAD_IDLE_TIMEOUT_SECONDS)
except StopAsyncIteration:
break
if not chunk:
continue
total += len(chunk)
if total > MAX_UPLOAD_BYTES:
fail(413, "payload_too_large", "Attachment exceeds the size limit")
if len(head) < 8:
head += chunk[: 8 - len(head)]
if len(head) >= 8 and sniff_image_type(head) != declared:
fail(415, "unsupported_media_type", "Content does not match the declared image type")
digest.update(chunk)
spool.write(chunk)
except ApiError:
spool.close()
raise
except (TimeoutError, asyncio.TimeoutError):
spool.close()
fail(400, "invalid_request", "Upload timed out")
except BaseException: # ClientDisconnect, cancellation: nothing was written to S3 yet
spool.close()
raise
if total == 0 or sniff_image_type(head) != declared:
spool.close()
fail(415, "unsupported_media_type", "Content is not a supported image")
return spool, total, digest.hexdigest()
def decode_image_checked(spool, declared: str) -> None:
"""Runs in a worker thread: header -> pixel limit -> full decode. Raises ApiError for every user mistake."""
try:
spool.seek(0)
with Image.open(spool) as img:
if IMAGE_TYPES.get(img.format or "") != declared:
fail(415, "unsupported_media_type", "Content does not match the declared image type")
width, height = img.size
if width < 1 or height < 1:
fail(415, "unsupported_media_type", "Corrupt image")
if width * height > MAX_IMAGE_PIXELS: # decided from the header, before any pixel buffer exists
fail(413, "payload_too_large", "Image has too many pixels")
img.load() # full decode detects truncated/corrupt data
except ApiError:
raise
except Image.DecompressionBombError:
fail(413, "payload_too_large", "Image has too many pixels")
except MemoryError:
fail(503, "unavailable", "Temporarily unable to process the image")
except Exception: # Pillow raises OSError/SyntaxError/ValueError/struct.error/... for corrupt input
fail(415, "unsupported_media_type", "Corrupt or unsupported image")
finally:
spool.seek(0)
def s3_put(key: str, spool, size: int, content_type: str) -> None:
spool.seek(0)
s3.put_object(Bucket=S3_BUCKET, Key=key, Body=spool, ContentLength=size, ContentType=content_type)
def s3_delete_quietly(key: str) -> None:
try:
s3.delete_object(Bucket=S3_BUCKET, Key=key)
except Exception:
# The periodic sweeper (scripts/s3_sweep.py) removes it after the grace period.
log.warning("Compensation delete failed; orphan left for sweeper key=%s", key)
async def compensate(key: str) -> None:
with anyio.CancelScope(shield=True):
await run_in_threadpool(s3_delete_quietly, key)
@app.post("/api/v1/chats/{chat_id}/attachments", status_code=202)
async def upload_attachment(chat_id: str, request: Request):
actor = current_user(request)
parsed_chat = parse_uuid(chat_id, "chat UUID")
ensure_member(actor["id"], parsed_chat)
filename = validate_filename(request.query_params.getlist("filename"))
declared = request.headers.get("content-type", "").split(";", 1)[0].strip().lower()
if declared not in ("image/png", "image/jpeg"):
fail(415, "unsupported_media_type", "Content-Type must be image/png or image/jpeg")
length = request.headers.get("content-length")
if length is not None:
if not length.isascii() or not length.isdigit():
fail(400, "invalid_request", "Invalid Content-Length")
if int(length) > MAX_UPLOAD_BYTES: # reject before reading a single body byte
fail(413, "payload_too_large", "Attachment exceeds the size limit")
spool, size, sha256 = await read_body_to_spool(request, declared)
key = STAGING_PREFIX + secrets.token_hex(16) # server-chosen random STAGING key, unrelated to filename or attachment id
try:
await anyio.to_thread.run_sync(decode_image_checked, spool, declared, limiter=decode_limiter)
await run_in_threadpool(s3_put, key, spool, size, declared)
except ApiError: # user error found by validation, before any S3 write
spool.close()
raise
except BaseException:
spool.close()
await compensate(key) # a PUT whose reply was lost may still have stored the object
raise
spool.close()
attachment_id, job_id = uuid.uuid4(), uuid.uuid4()
final_original = jobs.final_key(attachment_id, jobs.PIPELINE_VERSION, "original")
try:
# ONE transaction: metadata + job + outbox row. 202 is returned only after this COMMIT (durable handoff).
with db() as conn:
row = conn.execute(
"INSERT INTO attachments(id,chat_id,owner_id,filename,content_type,size_bytes,sha256,s3_key,staging_key,status,pipeline_version) "
"VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,'queued',%s) "
"RETURNING id,chat_id,owner_id,filename,content_type,size_bytes,sha256,status,error_code,created_at,updated_at",
(attachment_id, parsed_chat, actor["id"], filename, declared, size, sha256, final_original, key, jobs.PIPELINE_VERSION)).fetchone()
conn.execute("INSERT INTO attachment_jobs(job_id,attachment_id,pipeline_version,status,max_attempts) VALUES (%s,%s,%s,'queued',%s)",
(job_id, attachment_id, jobs.PIPELINE_VERSION, jobs.JOB_MAX_ATTEMPTS))
jobs.add_event(conn, job_id, "accepted")
jobs.enqueue(conn, job_id, attachment_id, jobs.PIPELINE_VERSION, 1)
except BaseException:
await compensate(key) # staging PUT succeeded but the COMMIT did not: remove the unaccounted object
raise
cols = ("id", "chat_id", "owner_id", "filename", "content_type", "size_bytes", "sha256", "status", "error_code", "created_at", "updated_at")
return JSONResponse(status_code=202, content=attachment_dict(dict(zip(cols, row))))
ATT_COLS = ("id", "chat_id", "owner_id", "filename", "content_type", "size_bytes", "sha256", "s3_key", "status", "error_code", "created_at", "updated_at")
def load_attachment(attachment_id: str, request: Request) -> dict[str, Any]:
actor = current_user(request)
parsed = parse_uuid(attachment_id, "attachment UUID")
rows = json_rows(None, "SELECT a.id,a.chat_id,a.owner_id,a.filename,a.content_type,a.size_bytes,a.sha256,a.s3_key,a.status,a.error_code,a.created_at,a.updated_at "
"FROM attachments a JOIN chat_members m ON m.chat_id=a.chat_id WHERE a.id=%s AND m.user_id=%s", (parsed, actor["id"]))
if not rows:
fail(404, "not_found", "Object not found")
return rows[0]
def variant_rows(attachment_id, pipeline_version: int = jobs.PIPELINE_VERSION) -> list[dict[str, Any]]:
rows = json_rows(None, "SELECT name,s3_key,content_type,size_bytes,width,height,sha256 FROM attachment_variants WHERE attachment_id=%s AND pipeline_version=%s", (attachment_id, pipeline_version))
return sorted(rows, key=lambda r: jobs.VARIANT_ORDER.index(r["name"]))
@app.get("/api/v1/attachments/{attachment_id}")
def get_attachment(attachment_id: str, request: Request):
row = load_attachment(attachment_id, request)
variants = []
if row["status"] == "ready": # variants are published only together with ready (one COMMIT in the worker)
variants = [{k: v[k] for k in ("name", "content_type", "size_bytes", "width", "height", "sha256")} for v in variant_rows(row["id"])]
return attachment_dict(row, variants)
@app.get("/api/v1/attachments/{attachment_id}/content")
def download_attachment(attachment_id: str, request: Request, variant: str = "original"):
row = load_attachment(attachment_id, request)
if variant != "original" and variant not in jobs.VARIANT_ORDER:
fail(400, "invalid_request", "Unknown variant")
if row["status"] != "ready":
fail(409, "conflict", "Attachment is not ready") # queued, processing and failed: no content
key, content_type, size = row["s3_key"], row["content_type"], row["size_bytes"]
if variant != "original":
found = [v for v in variant_rows(row["id"]) if v["name"] == variant]
if not found:
fail(404, "not_found", "Object not found")
key, content_type, size = found[0]["s3_key"], found[0]["content_type"], found[0]["size_bytes"]
try:
obj = s3.get_object(Bucket=S3_BUCKET, Key=key)
except ClientError as exc:
if exc.response.get("Error", {}).get("Code") in ("NoSuchKey", "404"):
log.error("INTEGRITY: metadata without object attachment=%s variant=%s", row["id"], variant)
fail(500, "internal_error", "Internal server error")
raise
body = obj["Body"]
async def stream():
try:
async for chunk in iterate_in_threadpool(body.iter_chunks(CHUNK_BYTES)):
yield chunk
finally:
body.close()
headers = {"Content-Length": str(size), "X-Content-Type-Options": "nosniff", "Cache-Control": "private, no-store",
"Content-Disposition": "inline; filename*=UTF-8''" + quote(row["filename"], safe="")}
return StreamingResponse(stream(), media_type=content_type, headers=headers)
if __name__ == "__main__":
import uvicorn
uvicorn.run("main:app", host=os.getenv("APP_HOST", "0.0.0.0"), port=int(os.getenv("APP_PORT", "8080")), timeout_graceful_shutdown=5)
+230
View File
@@ -0,0 +1,230 @@
"""Lab 5 worker: Kafka consumer -> claim (lease) in PostgreSQL -> staging -> pipeline -> final S3 + COMMIT -> offset commit.
PostgreSQL is the source of truth for job state; Kafka is the at-least-once transport. The Kafka offset is committed
only AFTER the durable effect (final objects + COMMIT of the result) or after a durable decision (retry scheduled / failed).
"""
from __future__ import annotations
import hashlib
import json
import logging
import os
import signal
import socket
import subprocess
import sys
import tempfile
import time
import uuid
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
import psycopg
from botocore.exceptions import BotoCoreError, ClientError
from confluent_kafka import Consumer, TopicPartition
from . import jobs
logging.basicConfig(level=os.getenv("LOG_LEVEL", "INFO"), format="%(asctime)s %(levelname)s worker %(message)s")
log = logging.getLogger("worker")
WORKER_ID = os.getenv("INSTANCE_ID") or socket.gethostname()
CONCURRENCY = int(os.getenv("WORKER_CONCURRENCY", "2"))
FAULT_EXIT_AFTER_S3 = os.getenv("FAULT_WORKER_EXIT_AFTER_S3", "0") == "1" # test hook: crash after the S3 write, before COMMIT/ack
IMG_MEMORY_MB = os.getenv("IMG_MEMORY_MB", "1024")
CODE_BY_EXIT = {2: "invalid_image", 3: "image_too_large", 4: "image_too_complex"} # permanent errors of imgproc
STOP = False
class Permanent(Exception):
def __init__(self, code: str):
super().__init__(code)
self.code = code
class Transient(Exception):
pass
def connect():
return psycopg.connect(connect_timeout=5, options="-c statement_timeout=10000")
def parse_job(raw: bytes) -> dict | None:
"""Validate against jobs.schema.json (the subset that matters). Poison messages are logged and skipped."""
try:
m = json.loads(raw)
assert m["schema_version"] == 1 and m["type"] == jobs.JOB_TYPE
uuid.UUID(m["job_id"]); uuid.UUID(m["attachment_id"])
assert isinstance(m["pipeline_version"], int) and m["pipeline_version"] >= 1
return m
except Exception:
log.error("poison message skipped (schema violation), size=%d", len(raw))
return None
def claim(msg: dict) -> dict | None:
"""queued -> processing with a lease. Returns None when this delivery must be a no-op (ack)."""
with connect() as conn:
row = conn.execute(
"SELECT status, attempt, max_attempts, attachment_id, pipeline_version, "
"(next_attempt_at <= now()) AS due, (lease_until IS NOT NULL AND lease_until < now()) AS lease_expired "
"FROM attachment_jobs WHERE job_id=%s FOR UPDATE", (msg["job_id"],)).fetchone()
if row is None:
log.warning("job %s unknown, skipped", msg["job_id"]); return None
status, attempt, max_attempts, att_id, pv, due, expired = row
if pv != msg["pipeline_version"]:
return None # stale job of another pipeline version never overwrites the newer one
if status in ("ready", "failed"):
log.info("job %s already %s: duplicate delivery is a no-op", msg["job_id"], status); return None
if status == "queued" and not due:
return None # early duplicate: the delayed outbox row will deliver it later
if status == "processing" and not expired:
log.info("job %s is leased by another worker: skip (dispatcher recovers it if that worker dies)", msg["job_id"]); return None
if attempt >= max_attempts: # lease expired and no attempts left
jobs.fail_job(conn, msg["job_id"], att_id, "retry_exhausted", attempt)
return None
attempt += 1
conn.execute("UPDATE attachment_jobs SET status='processing', attempt=%s, worker_id=%s, "
"lease_until=now()+make_interval(secs=>%s), updated_at=now() WHERE job_id=%s",
(attempt, WORKER_ID, jobs.LEASE_SECONDS, msg["job_id"]))
conn.execute("UPDATE attachments SET status='processing', updated_at=now() WHERE id=%s", (att_id,))
jobs.add_event(conn, msg["job_id"], "claimed", f"attempt={attempt} worker={WORKER_ID}")
a = conn.execute("SELECT sha256, staging_key, s3_key, content_type, filename FROM attachments WHERE id=%s", (att_id,)).fetchone()
return {"job_id": msg["job_id"], "attachment_id": att_id, "pv": pv, "attempt": attempt, "max_attempts": max_attempts,
"sha256": a[0].strip(), "staging_key": a[1], "original_key": a[2], "content_type": a[3]}
def run_pipeline(s3, c: dict) -> dict:
"""Download staging -> verify -> child process with limits -> idempotent PUTs of deterministic keys."""
if not c["staging_key"]:
raise Permanent("staging_missing")
with tempfile.TemporaryDirectory(prefix="job-") as tmp:
src, out = Path(tmp) / "in.bin", Path(tmp) / "out"
out.mkdir()
try:
s3.download_file(jobs.S3_BUCKET, c["staging_key"], str(src))
except ClientError as e:
if e.response.get("Error", {}).get("Code") in ("NoSuchKey", "404"):
raise Permanent("staging_missing")
raise
if hashlib.sha256(src.read_bytes()).hexdigest() != c["sha256"]:
raise Permanent("integrity_error") # bytes differ from what the API accepted
try:
r = subprocess.run([sys.executable, "-m", "src.imgproc", str(src), str(out)], capture_output=True, text=True,
timeout=jobs.JOB_TIMEOUT_SECONDS, env={"PATH": os.environ.get("PATH", ""), "IMG_MEMORY_MB": IMG_MEMORY_MB,
"IMG_CPU_SECONDS": str(jobs.JOB_TIMEOUT_SECONDS),
"MAX_IMAGE_PIXELS": os.getenv("MAX_IMAGE_PIXELS", "40000000")}) # no secrets in the child
except subprocess.TimeoutExpired:
raise Transient("job timeout") # child is killed by subprocess.run
if r.returncode in CODE_BY_EXIT:
raise Permanent(CODE_BY_EXIT[r.returncode])
if r.returncode != 0:
raise Transient(f"imgproc exit {r.returncode}")
meta = json.loads(r.stdout)
files = {"thumbnail": "thumbnail.jpg", "crop": "crop.jpg", "optimized": "optimized.webp"}
# 1) original into the final prefix (same key every time => repeated PUT is harmless)
s3.upload_file(str(src), jobs.S3_BUCKET, c["original_key"], ExtraArgs={"ContentType": c["content_type"]})
# 2) variants
for name in jobs.VARIANT_ORDER:
key = jobs.final_key(c["attachment_id"], c["pv"], name)
s3.upload_file(str(out / files[name]), jobs.S3_BUCKET, key, ExtraArgs={"ContentType": meta[name]["content_type"]})
meta[name]["s3_key"] = key
return meta
def commit_ready(c: dict, meta: dict) -> bool:
"""Atomically publish the result. False if this attempt no longer owns the job (lease lost to a newer attempt)."""
with connect() as conn:
row = conn.execute("SELECT status, attempt FROM attachment_jobs WHERE job_id=%s FOR UPDATE", (c["job_id"],)).fetchone()
if row is None or row[0] != "processing" or row[1] != c["attempt"]:
log.warning("job %s: attempt %s lost ownership, result not committed", c["job_id"], c["attempt"]); return False
for name in jobs.VARIANT_ORDER:
v = meta[name]
conn.execute("INSERT INTO attachment_variants(attachment_id,pipeline_version,name,s3_key,content_type,size_bytes,width,height,sha256) "
"VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s) ON CONFLICT (attachment_id,pipeline_version,name) DO NOTHING",
(c["attachment_id"], c["pv"], name, v["s3_key"], v["content_type"], v["size_bytes"], v["width"], v["height"], v["sha256"]))
conn.execute("UPDATE attachment_jobs SET status='ready', error_code=NULL, lease_until=NULL, updated_at=now() WHERE job_id=%s", (c["job_id"],))
conn.execute("UPDATE attachments SET status='ready', error_code=NULL, staging_key=NULL, updated_at=now() WHERE id=%s", (c["attachment_id"],))
jobs.add_event(conn, c["job_id"], "ready", f"attempt={c['attempt']}")
return True
def record_failure(c: dict, exc: Exception) -> None:
"""Permanent -> failed in the PostgreSQL registry. Transient -> queued with backoff+jitter (delayed outbox row) or failed when attempts are exhausted."""
with connect() as conn:
row = conn.execute("SELECT status, attempt FROM attachment_jobs WHERE job_id=%s FOR UPDATE", (c["job_id"],)).fetchone()
if row is None or row[0] != "processing" or row[1] != c["attempt"]:
return
if isinstance(exc, Permanent):
jobs.fail_job(conn, c["job_id"], c["attachment_id"], exc.code, c["attempt"])
elif c["attempt"] >= c["max_attempts"]:
jobs.fail_job(conn, c["job_id"], c["attachment_id"], "retry_exhausted", c["attempt"])
else:
delay = jobs.backoff_seconds(c["attempt"])
conn.execute("UPDATE attachment_jobs SET status='queued', lease_until=NULL, next_attempt_at=now()+make_interval(secs=>%s), "
"error_code=NULL, updated_at=now() WHERE job_id=%s", (delay, c["job_id"]))
conn.execute("UPDATE attachments SET status='queued', updated_at=now() WHERE id=%s", (c["attachment_id"],))
jobs.add_event(conn, c["job_id"], "retry_scheduled", f"attempt={c['attempt']} delay={delay:.1f}s reason={type(exc).__name__}")
jobs.enqueue(conn, c["job_id"], c["attachment_id"], c["pv"], c["attempt"] + 1, delay_seconds=delay)
def handle(s3, raw: bytes) -> None:
msg = parse_job(raw)
if msg is None:
return
c = claim(msg)
if c is None:
return
try:
meta = run_pipeline(s3, c)
if FAULT_EXIT_AFTER_S3 and c["attempt"] == 1:
log.error("FAULT: exiting after S3 write, before COMMIT and offset commit")
os._exit(137)
if commit_ready(c, meta):
try:
s3.delete_object(Bucket=jobs.S3_BUCKET, Key=c["staging_key"]) # staging cleanup after the safe COMMIT
except Exception:
log.warning("staging cleanup failed; sweeper will remove it")
except (Permanent, Transient, ClientError, BotoCoreError, OSError, json.JSONDecodeError) as exc:
log.warning("job %s attempt %s failed: %s", c["job_id"], c["attempt"], type(exc).__name__)
record_failure(c, exc)
def main() -> None:
def request_stop(*_):
global STOP
STOP = True
signal.signal(signal.SIGTERM, request_stop)
signal.signal(signal.SIGINT, request_stop)
s3 = jobs.make_s3()
consumer = Consumer({
"bootstrap.servers": jobs.KAFKA_BOOTSTRAP, "group.id": "attachment-workers",
"enable.auto.commit": False, "auto.offset.reset": "earliest", "isolation.level": "read_committed",
"max.poll.interval.ms": 600000, "session.timeout.ms": 30000,
"queued.max.messages.kbytes": 1024, "fetch.max.bytes": 1048576, # bounded client-side prefetch
})
consumer.subscribe([jobs.JOBS_TOPIC])
log.info("worker %s started, concurrency=%d, max_attempts=%d, lease=%ds", WORKER_ID, CONCURRENCY, jobs.JOB_MAX_ATTEMPTS, jobs.LEASE_SECONDS)
with ThreadPoolExecutor(max_workers=CONCURRENCY) as pool:
while not STOP:
msgs = consumer.consume(num_messages=CONCURRENCY, timeout=1.0) # at most CONCURRENCY jobs in flight
batch = []
for m in msgs:
if m.error():
log.warning("kafka error: %s", m.error()); continue
batch.append(m)
if not batch:
continue
futures = [pool.submit(handle, s3, m.value()) for m in batch]
for f in futures:
f.result() # unexpected exception => process exits, offsets stay uncommitted, Kafka redelivers
# ACK boundary: commit the highest processed offset+1 per partition, only after every job of the batch is durable
top = {}
for m in batch:
top[(m.topic(), m.partition())] = max(top.get((m.topic(), m.partition()), -1), m.offset() + 1)
consumer.commit(offsets=[TopicPartition(t, p, o) for (t, p), o in top.items()], asynchronous=False)
consumer.close()
if __name__ == "__main__":
main()
+176
View File
@@ -0,0 +1,176 @@
#!/usr/bin/env python3
"""Lab5 level-4 failure scenarios against a RUNNING compose stack (stdlib only; run from the repo root).
python3 student-tests/level4_failures.py all # or: worker_down | commit_publish_gap | crash_after_s3 | replay | s3_outage
Each scenario drives `docker compose` (never `down -v`: volumes stay) and prints PASS/FAIL with the observed facts.
"""
import base64, json, os, subprocess, sys, time, urllib.error, urllib.request, uuid
from pathlib import Path
BASE = os.getenv("BASE_URL", "http://localhost:8080")
PNG = (Path(__file__).resolve().parents[1] / "tests/fixtures/sample.png").read_bytes()
DC = ["docker", "compose", "--env-file", ".env"]
def sh(*args, env=None, check=True):
r = subprocess.run(list(args), capture_output=True, text=True, env={**os.environ, **(env or {})})
if check and r.returncode:
raise SystemExit(f"command failed: {' '.join(args)}\n{r.stderr[-500:]}")
return r.stdout.strip()
def sql(query):
return sh(*DC, "exec", "-T", "postgres", "sh", "-c", f'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -At -c "{query}"')
def http(method, path, token=None, body=None, headers=None, raw=None):
h = dict(headers or {})
if token:
h["Authorization"] = "Bearer " + token
data = raw if raw is not None else (json.dumps(body).encode() if body is not None else None)
if body is not None:
h["Content-Type"] = "application/json"
req = urllib.request.Request(BASE + path, data=data, method=method, headers=h)
try:
with urllib.request.urlopen(req, timeout=30) as r:
payload = r.read()
return r.status, (json.loads(payload) if payload and "json" in r.headers.get("Content-Type", "") else payload)
except urllib.error.HTTPError as e:
payload = e.read()
try:
return e.code, json.loads(payload)
except ValueError:
return e.code, payload
def setup():
name = "u" + uuid.uuid4().hex[:10]
pw = "correct-horse-battery"
st, user = http("POST", "/api/v1/users", body={"username": name, "display_name": "Tester", "password": pw})
assert st == 201, (st, user)
cred = base64.b64encode(f"{name}:{pw}".encode()).decode()
st, sess = http("POST", "/api/v1/auth/sessions", headers={"Authorization": "Basic " + cred})
token = sess["token"]
st, chat = http("POST", "/api/v1/chats", token, {"title": "lab5", "member_ids": [user["id"]]})
return token, chat["id"]
def upload(token, chat):
st, att = http("POST", f"/api/v1/chats/{chat}/attachments?filename=sample.png", token, raw=PNG, headers={"Content-Type": "image/png"})
assert st == 202 and att["status"] == "queued", (st, att)
return att["id"]
def wait(token, aid, want, timeout=90):
end, last = time.time() + timeout, None
while time.time() < end:
st, att = http("GET", f"/api/v1/attachments/{aid}", token)
last = att
if st == 200 and att["status"] == want:
return att
time.sleep(1)
raise AssertionError(f"attachment {aid} did not reach {want}; last={last}")
def wait_ready_api(timeout=90):
end = time.time() + timeout
while time.time() < end:
if http("GET", "/health/ready")[0] == 200:
return
time.sleep(1)
raise AssertionError("API not ready")
def check_variants(att):
v = {x["name"]: x for x in att["variants"]}
assert (v["thumbnail"]["width"], v["thumbnail"]["height"], v["thumbnail"]["content_type"]) == (256, 160, "image/jpeg"), v["thumbnail"]
assert (v["crop"]["width"], v["crop"]["height"]) == (128, 128), v["crop"]
assert v["optimized"]["content_type"] == "image/webp"
def events(aid):
return sql(f"SELECT string_agg(event, ',' ORDER BY id) FROM job_events e JOIN attachment_jobs j USING (job_id) WHERE j.attachment_id='{aid}'")
def worker_down():
token, chat = setup()
sh(*DC, "stop", "worker")
aid = upload(token, chat)
st, _ = http("GET", f"/api/v1/attachments/{aid}/content", token)
assert st == 409, st # content of a queued attachment
st, msg = http("POST", f"/api/v1/chats/{chat}/messages", token, {"text": "hi", "attachment_ids": [aid]}, {"Idempotency-Key": "k1"})
assert st == 201, (st, msg) # other features keep working; queued attachment can be attached
sh(*DC, "restart", "api"); wait_ready_api()
assert http("GET", f"/api/v1/attachments/{aid}", token)[1]["status"] == "queued" # accepted work survived the API restart
sh(*DC, "up", "-d", "worker")
att = wait(token, aid, "ready"); check_variants(att)
print("PASS worker_down:", aid, "events:", events(aid))
def commit_publish_gap():
token, chat = setup()
sh(*DC, "up", "-d", "dispatcher", env={"FAULT_DISPATCHER_EXIT_BEFORE_PUBLISH": "1"}, check=True)
sh(*DC, "exec", "-T", "dispatcher", "rm", "-f", "/tmp/dispatcher_fault_done", check=False)
sh(*DC, "restart", "dispatcher") # fresh container process: the fault fires once, on the first outbox batch
time.sleep(3)
aid = upload(token, chat) # COMMIT happens; the dispatcher dies before publishing
att = wait(token, aid, "ready", 90); check_variants(att)
sh(*DC, "up", "-d", "dispatcher") # back to the normal configuration (.env: fault off)
print("PASS commit_publish_gap:", aid, "events:", events(aid))
def crash_after_s3():
token, chat = setup()
env = {"FAULT_WORKER_EXIT_AFTER_S3": "1", "JOB_TIMEOUT_SECONDS": "10", "LEASE_GRACE_SECONDS": "5"}
sh(*DC, "up", "-d", "worker", env=env)
time.sleep(5)
aid = upload(token, chat)
att = wait(token, aid, "ready", 150); check_variants(att)
n = sql(f"SELECT count(*) FROM attachment_variants WHERE attachment_id='{aid}'")
assert n == "3", n # the retry completed the SAME result, no second set of variants
sh(*DC, "up", "-d", "worker") # restore normal worker config
print("PASS crash_after_s3:", aid, "events:", events(aid))
def replay():
token, chat = setup()
sh(*DC, "stop", "worker")
aid = upload(token, chat)
good = sql(f"SELECT sha256 FROM attachments WHERE id='{aid}'")
sql(f"UPDATE attachments SET sha256=repeat('0',64) WHERE id='{aid}'") # simulate corrupted/tampered staging bytes
sh(*DC, "up", "-d", "worker")
att = wait(token, aid, "failed", 60)
assert att["error_code"] == "integrity_error", att # permanent error: failed at once, no retries
st, _ = http("POST", f"/api/v1/chats/{chat}/messages", token, {"text": "", "attachment_ids": [aid]}, {"Idempotency-Key": "k2"})
assert st == 409, st
sql(f"UPDATE attachments SET sha256='{good}' WHERE id='{aid}'") # operator fixes the cause
job = sql(f"SELECT job_id FROM attachment_jobs WHERE attachment_id='{aid}'")
print(sh(*DC, "exec", "-T", "worker", "python", "scripts/replay_failed.py", "--job-id", job, "--reason", "student-test"))
att = wait(token, aid, "ready", 60); check_variants(att)
again = sh(*DC, "exec", "-T", "worker", "python", "scripts/replay_failed.py", "--job-id", job, check=False)
assert "REFUSED" in again, again # replay of a successful job is refused
print("PASS replay:", aid, "events:", events(aid))
def s3_outage():
token, chat = setup()
sh(*DC, "stop", "worker")
aid = upload(token, chat)
sh(*DC, "stop", "s3")
sh(*DC, "up", "-d", "--no-deps", "worker")
time.sleep(15)
ev = events(aid)
assert "retry_scheduled" in ev, ev # temporary error => backoff with jitter, not failed
sh(*DC, "start", "s3"); wait_ready_api()
att = wait(token, aid, "ready", 120); check_variants(att)
print("PASS s3_outage:", aid, "events:", events(aid))
SCENARIOS = {"worker_down": worker_down, "commit_publish_gap": commit_publish_gap, "crash_after_s3": crash_after_s3, "replay": replay, "s3_outage": s3_outage}
if __name__ == "__main__":
which = sys.argv[1] if len(sys.argv) > 1 else "all"
for name, fn in SCENARIOS.items():
if which in ("all", name):
fn()