forked from mtusi2026/mtusi2026lab5
Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
610732b49b |
@@ -18,3 +18,10 @@ __pycache__
|
||||
node_modules
|
||||
evidence
|
||||
.DS_Store
|
||||
.git
|
||||
.env
|
||||
.venv
|
||||
__pycache__
|
||||
.pytest_cache
|
||||
*.pyc
|
||||
evidence
|
||||
@@ -23,3 +23,7 @@ WORKER_CONCURRENCY=2
|
||||
JOB_MAX_ATTEMPTS=5
|
||||
JOB_TIMEOUT_SECONDS=60
|
||||
STAGING_RETENTION_SECONDS=86400
|
||||
RABBITMQ_USER=messenger
|
||||
RABBITMQ_PASSWORD=change-me
|
||||
RABBITMQ_QUEUE=image_jobs
|
||||
IMAGE_MAX_ATTEMPTS=5
|
||||
@@ -30,3 +30,8 @@ coverage/
|
||||
.vscode/
|
||||
# Keep small, sanitized evidence summaries in evidence/ under version control.
|
||||
# Inspect git diff --cached: ignore rules are not a secret scanner.
|
||||
.env
|
||||
.venv/
|
||||
__pycache__/
|
||||
*.pyc
|
||||
evidence/*.tmp
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
FROM python:3.12-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY requirements.txt .
|
||||
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
COPY src/ ./src/
|
||||
COPY migrations/ ./migrations/
|
||||
|
||||
ENV HOST=0.0.0.0
|
||||
ENV PORT=8080
|
||||
|
||||
EXPOSE 8080
|
||||
|
||||
CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8080", "--workers", "1"]
|
||||
@@ -1,37 +1,131 @@
|
||||
# Отчёт — лабораторная 5
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
- **Автор, группа:** Матвиенко, БПИ2304
|
||||
- **Целевая оценка:** 3
|
||||
- **Commit/PR:** добавить после публикации
|
||||
- **Использованные источники и библиотеки:** материалы лабораторной, OpenAPI-контракт, Python 3.12, FastAPI, PostgreSQL 16, RustFS 1.0.3, RabbitMQ 4.1.8, Pika 1.3.2, boto3, Pillow, Nginx 1.29, Docker Compose. При разработке и отладке использовалась помощь ИИ.
|
||||
|
||||
## Запуск из чистого clone
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
Необходимы Git, Docker Desktop с Docker Compose, Python 3 и Make.
|
||||
|
||||
Создать `.env` по образцу `.env.example`, указав собственные локальные секреты:
|
||||
|
||||
```dotenv
|
||||
POSTGRES_PASSWORD=<локальный_пароль>
|
||||
SESSION_TTL_SECONDS=3600
|
||||
S3_ACCESS_KEY=<локальный_ключ>
|
||||
S3_SECRET_KEY=<локальный_секрет>
|
||||
S3_BUCKET=messenger-attachments
|
||||
RABBITMQ_USER=messenger
|
||||
RABBITMQ_PASSWORD=<локальный_пароль_брокера>
|
||||
RABBITMQ_QUEUE=image_jobs
|
||||
IMAGE_MAX_ATTEMPTS=5
|
||||
```
|
||||
|
||||
Сборка, применение миграций, создание приватного bucket и запуск:
|
||||
|
||||
```bash
|
||||
docker compose up -d --build
|
||||
docker compose ps -a
|
||||
curl http://127.0.0.1:8080/health/ready
|
||||
```
|
||||
|
||||
Проверки:
|
||||
|
||||
```bash
|
||||
make test
|
||||
make test-persistence ACTION_SCRIPT=scripts/restart.sh
|
||||
make test-failover ACTION_SCRIPT=scripts/stop-one.sh
|
||||
python3 scripts/test-worker-recovery.py
|
||||
bash scripts/test-redelivery.sh
|
||||
```
|
||||
|
||||
Остановка с сохранением данных:
|
||||
|
||||
```bash
|
||||
docker compose down
|
||||
```
|
||||
|
||||
Намеренное удаление учебных данных и volumes:
|
||||
|
||||
```bash
|
||||
docker compose down -v
|
||||
```
|
||||
|
||||
## Архитектура и решения
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab4, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
**Схема:** клиент → Nginx → две API-реплики → PostgreSQL + RustFS → dispatcher → RabbitMQ → worker → RustFS + PostgreSQL.
|
||||
|
||||
Из lab4 сохранены авторизация, чаты, сообщения, балансировка API, идемпотентность и приватное S3-хранилище.
|
||||
|
||||
В lab5 реализована асинхронная обработка изображений:
|
||||
|
||||
- API сохраняет исходный файл в устойчивом S3 staging и создаёт задание в PostgreSQL.
|
||||
- Upload возвращает HTTP 202 со статусом `queued`.
|
||||
- Dispatcher публикует идентификаторы заданий в устойчивую очередь RabbitMQ с подтверждением публикации.
|
||||
- Отдельный worker обрабатывает файл и создаёт JPEG thumbnail в пределах 256×256 с сохранением пропорций.
|
||||
- Оригинал сохраняется без изменений.
|
||||
- Статусы `queued`, `processing`, `ready`, `failed` доступны через API.
|
||||
- Реализованы ограниченные повторы обработки — максимум 5 попыток.
|
||||
- Подтверждение сообщения RabbitMQ выполняется после завершения транзакции PostgreSQL.
|
||||
|
||||
Байты изображений и учётные данные в сообщениях брокера не передаются. PostgreSQL хранит метаданные и задания, RustFS — staging и конечные файлы. Стабильные object keys и проверка состояния задания защищают от повторной обработки завершённой работы.
|
||||
|
||||
Между S3 и PostgreSQL нет общей распределённой транзакции; при сбоях используются повторные попытки и идемпотентная обработка.
|
||||
|
||||
## Доказательства критериев
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
| Критерий | Команда или сценарий | Результат | Доказательство |
|
||||
|---|---|---|---|
|
||||
| Публичный smoke | `make test` | PASS | `evidence/smoke.txt` |
|
||||
| Сохранность данных | `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | PASS | `evidence/persistence.txt` |
|
||||
| Отказ API-реплики | `make test-failover ACTION_SCRIPT=scripts/stop-one.sh` | PASS, 121 запись, 0 ошибок | `evidence/failover.txt` |
|
||||
| Восстановление после отказа | Failover-сценарий | 0,783 секунды | `evidence/failover.txt` |
|
||||
| Остановленный worker и рестарт | `python3 scripts/test-worker-recovery.py` | PASS | `evidence/worker-recovery.txt` |
|
||||
| Повторная доставка задания | `bash scripts/test-redelivery.sh` | PASS, результат не изменился | `evidence/redelivery.txt` |
|
||||
| Подтверждение работы worker | Журнал worker | Два повторных задания распознаны как завершённые | `evidence/redelivery-worker.txt` |
|
||||
| Состояние инфраструктуры | `docker compose ps -a` | API и PostgreSQL healthy, остальные рабочие сервисы запущены | `evidence/containers.txt` |
|
||||
| Миграции | `docker compose run --rm --no-deps migrate` | Все 5 миграций уже применены | `evidence/migrations.txt` |
|
||||
| Перезапуск инфраструктуры | Сравнение времени запуска контейнеров | Сохранены данные до и после действия | `evidence/restart-before.txt`, `evidence/restart-after.txt` |
|
||||
|
||||
## Эксперименты и отказы
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
**Гипотеза:** принятое изображение не потеряется при остановке worker и перезапуске API или брокера. Повторная доставка завершённого задания не изменит результат.
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
Испытания выполнены на локальном стенде Windows / WSL Ubuntu / Docker Desktop, с RabbitMQ 4.1.8, PostgreSQL 16, RustFS 1.0.3 и двумя API-репликами.
|
||||
|
||||
**Проверка восстановления задания:**
|
||||
|
||||
1. Worker остановлен.
|
||||
2. Изображение загружено с результатом HTTP 202, статус `queued`.
|
||||
3. Создано сообщение с ожидающим вложением.
|
||||
4. Перезапущены RabbitMQ и API.
|
||||
5. Состояние `queued` сохранилось.
|
||||
6. После запуска worker вложение перешло в `ready`.
|
||||
7. Оригинал проверен по SHA-256, thumbnail создан как JPEG 256×160.
|
||||
|
||||
Результат: `PASS: durable worker recovery`.
|
||||
|
||||
**Проверка повторной доставки:**
|
||||
|
||||
Одно завершённое задание повторно опубликовано дважды. До и после проверки состояние осталось `ready`, число попыток — `1`, число вариантов — `1`, SHA-256 thumbnail не изменился. Worker дважды зарегистрировал получение уже завершённого задания.
|
||||
|
||||
Результат: `PASS: duplicate delivery did not change completed job`.
|
||||
|
||||
| Воздействие | Ожидание | Наблюдение | Результат |
|
||||
|---|---|---|---|
|
||||
| Остановка API-реплики | Запись через оставшуюся реплику | 121 успешная запись, 0 ошибок | Восстановление 0,783 с |
|
||||
| Перезапуск хранилищ и брокера | Сохранность данных и сессии | Persistence PASS | Потерь не выявлено |
|
||||
| Остановка worker и рестарт | Задание сохранится | `queued → ready`, оригинал и thumbnail доступны | PASS |
|
||||
| Две повторные доставки | Отсутствие повторной обработки | Статус, checksum и количество вариантов неизменны | PASS |
|
||||
|
||||
Отдельные измерения RPS, p50/p95/p99, CPU/RAM и длительные нагрузочные испытания не выполнялись. Проверка повторной доставки проведена для завершённого задания; аварийное завершение worker непосредственно перед ACK отдельно не моделировалось.
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
PostgreSQL, RustFS и RabbitMQ работают в одиночных экземплярах и остаются точками отказа. При намеренном удалении volumes данные теряются.
|
||||
|
||||
В перспективе можно добавить резервирование хранилищ, мониторинг очереди, очистку неиспользуемых staging-объектов и испытания с несколькими worker.
|
||||
|
||||
**Вывод:** асинхронная обработка изображений реализована. Публичный smoke, persistence, failover, восстановление заданий и повторная доставка успешно проверены. Принятое задание пережило остановку worker и перезапуск RabbitMQ и API, а повторная доставка не изменила готовый результат.
|
||||
+175
@@ -0,0 +1,175 @@
|
||||
|
||||
services:
|
||||
rabbitmq:
|
||||
image: rabbitmq:4.1.8-management
|
||||
hostname: rabbitmq
|
||||
environment:
|
||||
RABBITMQ_DEFAULT_USER: ${RABBITMQ_USER:?Set RABBITMQ_USER in .env}
|
||||
RABBITMQ_DEFAULT_PASS: ${RABBITMQ_PASSWORD:?Set RABBITMQ_PASSWORD in .env}
|
||||
volumes:
|
||||
- rabbitmq_data:/var/lib/rabbitmq
|
||||
healthcheck:
|
||||
test: ["CMD", "rabbitmq-diagnostics", "-q", "ping"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 15
|
||||
start_period: 15s
|
||||
restart: unless-stopped
|
||||
|
||||
db:
|
||||
image: postgres:16-alpine
|
||||
environment:
|
||||
POSTGRES_DB: messenger
|
||||
POSTGRES_USER: messenger
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U messenger -d messenger"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 20
|
||||
restart: unless-stopped
|
||||
|
||||
minio:
|
||||
image: rustfs/rustfs:1.0.3
|
||||
command: ["/data"]
|
||||
environment:
|
||||
RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY:?Set S3_ACCESS_KEY in .env}
|
||||
RUSTFS_SECRET_KEY: ${S3_SECRET_KEY:?Set S3_SECRET_KEY in .env}
|
||||
RUSTFS_ADDRESS: ":9000"
|
||||
volumes:
|
||||
- minio_data:/data
|
||||
restart: unless-stopped
|
||||
|
||||
s3-init:
|
||||
build: .
|
||||
command: ["python", "-m", "src.s3_setup"]
|
||||
environment: &s3_env
|
||||
S3_ENDPOINT: http://minio:9000
|
||||
S3_ACCESS_KEY: ${S3_ACCESS_KEY}
|
||||
S3_SECRET_KEY: ${S3_SECRET_KEY}
|
||||
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
|
||||
depends_on:
|
||||
minio:
|
||||
condition: service_started
|
||||
restart: "no"
|
||||
|
||||
migrate:
|
||||
build: .
|
||||
command: ["python", "-m", "src.migrate"]
|
||||
environment:
|
||||
DATABASE_URL: postgresql://messenger:${POSTGRES_PASSWORD}@db:5432/messenger
|
||||
depends_on:
|
||||
db:
|
||||
condition: service_healthy
|
||||
restart: "no"
|
||||
|
||||
api1:
|
||||
build: .
|
||||
environment:
|
||||
RABBITMQ_HOST: rabbitmq
|
||||
RABBITMQ_USER: ${RABBITMQ_USER}
|
||||
RABBITMQ_PASSWORD: ${RABBITMQ_PASSWORD}
|
||||
RABBITMQ_QUEUE: ${RABBITMQ_QUEUE:-image_jobs}
|
||||
IMAGE_MAX_ATTEMPTS: ${IMAGE_MAX_ATTEMPTS:-5}
|
||||
DATABASE_URL: postgresql://messenger:${POSTGRES_PASSWORD}@db:5432/messenger
|
||||
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
|
||||
<<: *s3_env
|
||||
depends_on:
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
s3-init:
|
||||
condition: service_completed_successfully
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=3)"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 20
|
||||
stop_grace_period: 15s
|
||||
restart: unless-stopped
|
||||
|
||||
api2:
|
||||
build: .
|
||||
environment:
|
||||
RABBITMQ_HOST: rabbitmq
|
||||
RABBITMQ_USER: ${RABBITMQ_USER}
|
||||
RABBITMQ_PASSWORD: ${RABBITMQ_PASSWORD}
|
||||
RABBITMQ_QUEUE: ${RABBITMQ_QUEUE:-image_jobs}
|
||||
IMAGE_MAX_ATTEMPTS: ${IMAGE_MAX_ATTEMPTS:-5}
|
||||
DATABASE_URL: postgresql://messenger:${POSTGRES_PASSWORD}@db:5432/messenger
|
||||
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
|
||||
<<: *s3_env
|
||||
depends_on:
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
s3-init:
|
||||
condition: service_completed_successfully
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=3)"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 20
|
||||
stop_grace_period: 15s
|
||||
restart: unless-stopped
|
||||
|
||||
|
||||
dispatcher:
|
||||
build: .
|
||||
command: ["python", "-m", "src.dispatcher"]
|
||||
environment:
|
||||
DATABASE_URL: postgresql://messenger:${POSTGRES_PASSWORD}@db:5432/messenger
|
||||
RABBITMQ_HOST: rabbitmq
|
||||
RABBITMQ_USER: ${RABBITMQ_USER}
|
||||
RABBITMQ_PASSWORD: ${RABBITMQ_PASSWORD}
|
||||
RABBITMQ_QUEUE: ${RABBITMQ_QUEUE:-image_jobs}
|
||||
depends_on:
|
||||
rabbitmq:
|
||||
condition: service_healthy
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
restart: unless-stopped
|
||||
stop_grace_period: 15s
|
||||
|
||||
|
||||
worker:
|
||||
build: .
|
||||
command: ["python", "-m", "src.worker"]
|
||||
environment:
|
||||
DATABASE_URL: postgresql://messenger:${POSTGRES_PASSWORD}@db:5432/messenger
|
||||
S3_ENDPOINT: http://minio:9000
|
||||
S3_ACCESS_KEY: ${S3_ACCESS_KEY}
|
||||
S3_SECRET_KEY: ${S3_SECRET_KEY}
|
||||
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
|
||||
RABBITMQ_HOST: rabbitmq
|
||||
RABBITMQ_USER: ${RABBITMQ_USER}
|
||||
RABBITMQ_PASSWORD: ${RABBITMQ_PASSWORD}
|
||||
RABBITMQ_QUEUE: ${RABBITMQ_QUEUE:-image_jobs}
|
||||
IMAGE_MAX_ATTEMPTS: ${IMAGE_MAX_ATTEMPTS:-5}
|
||||
depends_on:
|
||||
rabbitmq:
|
||||
condition: service_healthy
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
s3-init:
|
||||
condition: service_completed_successfully
|
||||
restart: unless-stopped
|
||||
stop_grace_period: 20s
|
||||
|
||||
nginx:
|
||||
image: nginx:1.29-alpine
|
||||
ports:
|
||||
- "127.0.0.1:8080:80"
|
||||
volumes:
|
||||
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
depends_on:
|
||||
api1:
|
||||
condition: service_healthy
|
||||
api2:
|
||||
condition: service_healthy
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
minio_data:
|
||||
rabbitmq_data:
|
||||
@@ -0,0 +1,11 @@
|
||||
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
|
||||
mtusi2026lab5-api1-1 mtusi2026lab5-api1 "uvicorn src.main:ap…" api1 10 minutes ago Up 2 minutes (healthy) 8080/tcp
|
||||
mtusi2026lab5-api2-1 mtusi2026lab5-api2 "uvicorn src.main:ap…" api2 10 minutes ago Up 2 minutes (healthy) 8080/tcp
|
||||
mtusi2026lab5-db-1 postgres:16-alpine "docker-entrypoint.s…" db 42 minutes ago Up 6 minutes (healthy) 5432/tcp
|
||||
mtusi2026lab5-dispatcher-1 mtusi2026lab5-dispatcher "python -m src.dispa…" dispatcher 10 minutes ago Up 6 minutes 8080/tcp
|
||||
mtusi2026lab5-migrate-1 mtusi2026lab5-migrate "python -m src.migra…" migrate 10 minutes ago Exited (0) 2 minutes ago
|
||||
mtusi2026lab5-minio-1 rustfs/rustfs:1.0.3 "/entrypoint.sh /data" minio 42 minutes ago Up 6 minutes 9000-9001/tcp
|
||||
mtusi2026lab5-nginx-1 nginx:1.29-alpine "/docker-entrypoint.…" nginx 10 minutes ago Up 10 minutes 127.0.0.1:8080->80/tcp
|
||||
mtusi2026lab5-rabbitmq-1 rabbitmq:4.1.8-management "docker-entrypoint.s…" rabbitmq 45 minutes ago Up 3 minutes (healthy) 4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 15691-15692/tcp, 25672/tcp
|
||||
mtusi2026lab5-s3-init-1 mtusi2026lab5-s3-init "python -m src.s3_se…" s3-init 10 minutes ago Exited (0) 2 minutes ago
|
||||
mtusi2026lab5-worker-1 mtusi2026lab5-worker "python -m src.worker" worker 10 minutes ago Up 2 minutes 8080/tcp
|
||||
@@ -0,0 +1,4 @@
|
||||
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://localhost:8080"
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||
{"observed_instances_before": 2, "successful_writes": 121, "failed_attempts": 0, "first_recovery_seconds": 0.783}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
@@ -0,0 +1,6 @@
|
||||
Already applied: 001_initial.sql
|
||||
Already applied: 002_idempotency.sql
|
||||
Already applied: 003_attachments.sql
|
||||
Already applied: 004_allow_empty_message_text.sql
|
||||
Already applied: 005_async_jobs.sql
|
||||
Migrations completed
|
||||
@@ -0,0 +1,3 @@
|
||||
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:8080"
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
@@ -0,0 +1,31 @@
|
||||
worker-1 | 2026-11-17 16:12:19,058 INFO Pika version 1.3.2 connecting to ('172.19.0.2', 5672)
|
||||
worker-1 | 2026-11-17 16:12:19,058 INFO Socket connected: <socket.socket fd=6, family=2, type=1, proto=6, laddr=('172.19.0.7', 39424), raddr=('172.19.0.2', 5672)>
|
||||
worker-1 | 2026-11-17 16:12:19,059 INFO Streaming transport linked up: (<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7731aaa89ee0>, _StreamingProtocolShim: <SelectConnection PROTOCOL transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7731aaa89ee0> params=<ConnectionParameters host=rabbitmq port=5672 virtual_host=/ ssl=False>>).
|
||||
worker-1 | 2026-11-17 16:12:19,060 INFO AMQPConnector - reporting success: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7731aaa89ee0> params=<ConnectionParameters host=rabbitmq port=5672 virtual_host=/ ssl=False>>
|
||||
worker-1 | 2026-11-17 16:12:19,061 INFO AMQPConnectionWorkflow - reporting success: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7731aaa89ee0> params=<ConnectionParameters host=rabbitmq port=5672 virtual_host=/ ssl=False>>
|
||||
worker-1 | 2026-11-17 16:12:19,061 INFO Connection workflow succeeded: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7731aaa89ee0> params=<ConnectionParameters host=rabbitmq port=5672 virtual_host=/ ssl=False>>
|
||||
worker-1 | 2026-11-17 16:12:19,061 INFO Created channel=1
|
||||
worker-1 | 2026-11-17 16:12:19,065 INFO Worker listening on image_jobs
|
||||
worker-1 | 2026-11-17 16:13:54,771 INFO Job da3f295c-1991-4fe3-96a1-4bed377479f5 ready, thumbnail 256x160
|
||||
worker-1 | 2026-11-17 16:14:27,063 INFO Job 37c1126c-dc6c-49d7-91bd-fd88f50883b4 ready, thumbnail 256x160
|
||||
worker-1 | 2026-11-17 16:15:06,446 INFO Job 7f37da8b-1f55-4b81-8cd3-aa5627756ecd ready, thumbnail 256x160
|
||||
worker-1 | 2026-11-17 16:15:36,525 INFO Pika version 1.3.2 connecting to ('172.19.0.2', 5672)
|
||||
worker-1 | 2026-11-17 16:15:36,525 INFO Socket connected: <socket.socket fd=6, family=2, type=1, proto=6, laddr=('172.19.0.6', 36710), raddr=('172.19.0.2', 5672)>
|
||||
worker-1 | 2026-11-17 16:15:36,525 INFO Streaming transport linked up: (<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x75c6b31cfc50>, _StreamingProtocolShim: <SelectConnection PROTOCOL transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x75c6b31cfc50> params=<ConnectionParameters host=rabbitmq port=5672 virtual_host=/ ssl=False>>).
|
||||
worker-1 | 2026-11-17 16:15:36,532 INFO AMQPConnector - reporting success: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x75c6b31cfc50> params=<ConnectionParameters host=rabbitmq port=5672 virtual_host=/ ssl=False>>
|
||||
worker-1 | 2026-11-17 16:15:36,532 INFO AMQPConnectionWorkflow - reporting success: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x75c6b31cfc50> params=<ConnectionParameters host=rabbitmq port=5672 virtual_host=/ ssl=False>>
|
||||
worker-1 | 2026-11-17 16:15:36,532 INFO Connection workflow succeeded: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x75c6b31cfc50> params=<ConnectionParameters host=rabbitmq port=5672 virtual_host=/ ssl=False>>
|
||||
worker-1 | 2026-11-17 16:15:36,532 INFO Created channel=1
|
||||
worker-1 | 2026-11-17 16:15:36,537 INFO Worker listening on image_jobs
|
||||
worker-1 | 2026-11-17 16:16:17,793 INFO Job cd52e633-e071-4966-93ff-6d9712c43c41 ready, thumbnail 256x160
|
||||
worker-1 | 2026-11-17 16:19:32,768 INFO Pika version 1.3.2 connecting to ('172.19.0.2', 5672)
|
||||
worker-1 | 2026-11-17 16:19:32,768 INFO Socket connected: <socket.socket fd=6, family=2, type=1, proto=6, laddr=('172.19.0.8', 48004), raddr=('172.19.0.2', 5672)>
|
||||
worker-1 | 2026-11-17 16:19:32,768 INFO Streaming transport linked up: (<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7a45b28b9340>, _StreamingProtocolShim: <SelectConnection PROTOCOL transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7a45b28b9340> params=<ConnectionParameters host=rabbitmq port=5672 virtual_host=/ ssl=False>>).
|
||||
worker-1 | 2026-11-17 16:19:32,770 INFO AMQPConnector - reporting success: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7a45b28b9340> params=<ConnectionParameters host=rabbitmq port=5672 virtual_host=/ ssl=False>>
|
||||
worker-1 | 2026-11-17 16:19:32,770 INFO AMQPConnectionWorkflow - reporting success: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7a45b28b9340> params=<ConnectionParameters host=rabbitmq port=5672 virtual_host=/ ssl=False>>
|
||||
worker-1 | 2026-11-17 16:19:32,770 INFO Connection workflow succeeded: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x7a45b28b9340> params=<ConnectionParameters host=rabbitmq port=5672 virtual_host=/ ssl=False>>
|
||||
worker-1 | 2026-11-17 16:19:32,770 INFO Created channel=1
|
||||
worker-1 | 2026-11-17 16:19:32,774 INFO Worker listening on image_jobs
|
||||
worker-1 | 2026-11-17 16:19:32,819 INFO Job 2d2befcd-5472-44c0-aaf8-4272a173fde6 ready, thumbnail 256x160
|
||||
worker-1 | 2026-11-17 16:21:50,498 INFO Job 2d2befcd-5472-44c0-aaf8-4272a173fde6 already completed
|
||||
worker-1 | 2026-11-17 16:21:50,517 INFO Job 2d2befcd-5472-44c0-aaf8-4272a173fde6 already completed
|
||||
@@ -0,0 +1,8 @@
|
||||
Finding completed image job...
|
||||
Job: 2d2befcd-5472-44c0-aaf8-4272a173fde6
|
||||
Before: ready|1|f9e3cf1ed71334dfb7824260b8777b7cc9dddcbc1a9c1d1059d22ab81932fc34|1
|
||||
Publishing the same job twice...
|
||||
Two duplicate messages published
|
||||
After: ready|1|f9e3cf1ed71334dfb7824260b8777b7cc9dddcbc1a9c1d1059d22ab81932fc34|1
|
||||
PASS: duplicate delivery did not change completed job
|
||||
PASS: thumbnail checksum and variant count preserved
|
||||
@@ -0,0 +1,5 @@
|
||||
/mtusi2026lab5-api1-1 2026-11-17T16:15:37.499975362Z
|
||||
/mtusi2026lab5-api2-1 2026-11-17T16:15:37.498279295Z
|
||||
/mtusi2026lab5-db-1 2026-11-17T16:15:27.473397186Z
|
||||
/mtusi2026lab5-minio-1 2026-11-17T16:15:27.347848806Z
|
||||
/mtusi2026lab5-rabbitmq-1 2026-11-17T16:15:28.371689705Z
|
||||
@@ -0,0 +1,5 @@
|
||||
/mtusi2026lab5-api1-1 2026-11-17T16:12:18.462950541Z
|
||||
/mtusi2026lab5-api2-1 2026-11-17T16:12:18.455718969Z
|
||||
/mtusi2026lab5-db-1 2026-11-17T15:39:33.580121321Z
|
||||
/mtusi2026lab5-minio-1 2026-11-17T15:39:33.581952593Z
|
||||
/mtusi2026lab5-rabbitmq-1 2026-11-17T15:36:42.420152689Z
|
||||
@@ -0,0 +1,2 @@
|
||||
python3 tests/smoke.py --base-url "http://localhost:8080"
|
||||
PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.
|
||||
@@ -0,0 +1,59 @@
|
||||
Stopping worker...
|
||||
Container mtusi2026lab5-worker-1 Stopping
|
||||
Container mtusi2026lab5-worker-1 Stopped
|
||||
Upload: 202 queued
|
||||
Message created while worker is stopped
|
||||
Restarting RabbitMQ and API...
|
||||
Container mtusi2026lab5-rabbitmq-1 Restarting
|
||||
Container mtusi2026lab5-rabbitmq-1 Started
|
||||
Container mtusi2026lab5-rabbitmq-1 Running
|
||||
Container mtusi2026lab5-rabbitmq-1 Waiting
|
||||
Container mtusi2026lab5-rabbitmq-1 Healthy
|
||||
Container mtusi2026lab5-api1-1 Restarting
|
||||
Container mtusi2026lab5-api2-1 Restarting
|
||||
Container mtusi2026lab5-api2-1 Started
|
||||
Container mtusi2026lab5-api1-1 Started
|
||||
Queued attachment survived restart
|
||||
Starting worker...
|
||||
Container mtusi2026lab5-db-1 Running
|
||||
Container mtusi2026lab5-rabbitmq-1 Running
|
||||
Container mtusi2026lab5-migrate-1 Created
|
||||
Container mtusi2026lab5-minio-1 Running
|
||||
Container mtusi2026lab5-s3-init-1 Created
|
||||
Container mtusi2026lab5-worker-1 Created
|
||||
Container mtusi2026lab5-s3-init-1 Starting
|
||||
Container mtusi2026lab5-db-1 Waiting
|
||||
Container mtusi2026lab5-s3-init-1 Started
|
||||
Container mtusi2026lab5-db-1 Healthy
|
||||
Container mtusi2026lab5-migrate-1 Starting
|
||||
Container mtusi2026lab5-migrate-1 Started
|
||||
Container mtusi2026lab5-rabbitmq-1 Waiting
|
||||
Container mtusi2026lab5-migrate-1 Waiting
|
||||
Container mtusi2026lab5-s3-init-1 Waiting
|
||||
Container mtusi2026lab5-s3-init-1 Exited
|
||||
Container mtusi2026lab5-migrate-1 Exited
|
||||
Container mtusi2026lab5-rabbitmq-1 Healthy
|
||||
Container mtusi2026lab5-worker-1 Starting
|
||||
Container mtusi2026lab5-worker-1 Started
|
||||
Attachment: ready
|
||||
Original: SHA-256 verified
|
||||
Thumbnail: JPEG 256x160 verified
|
||||
PASS: durable worker recovery
|
||||
Container mtusi2026lab5-minio-1 Running
|
||||
Container mtusi2026lab5-db-1 Running
|
||||
Container mtusi2026lab5-rabbitmq-1 Running
|
||||
Container mtusi2026lab5-s3-init-1 Created
|
||||
Container mtusi2026lab5-migrate-1 Created
|
||||
Container mtusi2026lab5-worker-1 Running
|
||||
Container mtusi2026lab5-s3-init-1 Starting
|
||||
Container mtusi2026lab5-db-1 Waiting
|
||||
Container mtusi2026lab5-s3-init-1 Started
|
||||
Container mtusi2026lab5-db-1 Healthy
|
||||
Container mtusi2026lab5-migrate-1 Starting
|
||||
Container mtusi2026lab5-migrate-1 Started
|
||||
Container mtusi2026lab5-s3-init-1 Waiting
|
||||
Container mtusi2026lab5-rabbitmq-1 Waiting
|
||||
Container mtusi2026lab5-migrate-1 Waiting
|
||||
Container mtusi2026lab5-rabbitmq-1 Healthy
|
||||
Container mtusi2026lab5-migrate-1 Exited
|
||||
Container mtusi2026lab5-s3-init-1 Exited
|
||||
@@ -0,0 +1,74 @@
|
||||
|
||||
CREATE TABLE IF NOT EXISTS users (
|
||||
id UUID PRIMARY KEY,
|
||||
username VARCHAR(32) NOT NULL UNIQUE,
|
||||
display_name VARCHAR(100) NOT NULL,
|
||||
password_hash TEXT NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
|
||||
|
||||
CONSTRAINT username_format
|
||||
CHECK (username ~ '^[a-z0-9_]{3,32}$'),
|
||||
CONSTRAINT display_name_length
|
||||
CHECK (char_length(display_name) BETWEEN 1 AND 100),
|
||||
CONSTRAINT display_name_not_blank
|
||||
CHECK (length(btrim(display_name)) > 0)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chats (
|
||||
id UUID PRIMARY KEY,
|
||||
title VARCHAR(100) NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
|
||||
|
||||
CONSTRAINT title_not_blank
|
||||
CHECK (length(btrim(title)) > 0)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chat_members (
|
||||
chat_id UUID NOT NULL
|
||||
REFERENCES chats(id) ON DELETE CASCADE,
|
||||
user_id UUID NOT NULL
|
||||
REFERENCES users(id),
|
||||
position INTEGER NOT NULL,
|
||||
|
||||
PRIMARY KEY (chat_id, user_id),
|
||||
UNIQUE (chat_id, position)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS messages (
|
||||
id UUID PRIMARY KEY,
|
||||
chat_id UUID NOT NULL
|
||||
REFERENCES chats(id) ON DELETE CASCADE,
|
||||
sender_id UUID NOT NULL
|
||||
REFERENCES users(id),
|
||||
text TEXT NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
|
||||
|
||||
CONSTRAINT message_length
|
||||
CHECK (char_length(text) BETWEEN 1 AND 4000),
|
||||
CONSTRAINT message_not_blank
|
||||
CHECK (length(btrim(text)) > 0)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS sessions (
|
||||
token_hash CHAR(64) PRIMARY KEY,
|
||||
user_id UUID NOT NULL
|
||||
REFERENCES users(id) ON DELETE CASCADE,
|
||||
expires_at TIMESTAMPTZ NOT NULL,
|
||||
revoked BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_chat_members_user
|
||||
ON chat_members(user_id, chat_id);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_chats_order
|
||||
ON chats(created_at, id);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_messages_history
|
||||
ON messages(chat_id, created_at, id);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_sessions_user
|
||||
ON sessions(user_id);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_sessions_expiry
|
||||
ON sessions(expires_at);
|
||||
@@ -0,0 +1,14 @@
|
||||
|
||||
CREATE TABLE IF NOT EXISTS message_idempotency (
|
||||
user_id UUID NOT NULL REFERENCES users(id),
|
||||
chat_id UUID NOT NULL REFERENCES chats(id),
|
||||
idempotency_key VARCHAR(128) NOT NULL,
|
||||
request_hash CHAR(64) NOT NULL,
|
||||
message_id UUID NOT NULL REFERENCES messages(id),
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
|
||||
|
||||
PRIMARY KEY (user_id, chat_id, idempotency_key)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_idempotency_created
|
||||
ON message_idempotency(created_at);
|
||||
@@ -0,0 +1,34 @@
|
||||
|
||||
CREATE TABLE IF NOT EXISTS attachments (
|
||||
id UUID PRIMARY KEY,
|
||||
chat_id UUID NOT NULL REFERENCES chats(id),
|
||||
uploader_id UUID NOT NULL REFERENCES users(id),
|
||||
filename TEXT NOT NULL,
|
||||
content_type TEXT NOT NULL,
|
||||
size_bytes BIGINT NOT NULL,
|
||||
sha256 CHAR(64) NOT NULL,
|
||||
object_key TEXT NOT NULL UNIQUE,
|
||||
width INTEGER NOT NULL,
|
||||
height INTEGER NOT NULL,
|
||||
status TEXT NOT NULL DEFAULT 'ready',
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
|
||||
CONSTRAINT attachment_size_valid
|
||||
CHECK (size_bytes BETWEEN 1 AND 10485760),
|
||||
|
||||
CONSTRAINT attachment_dimensions_valid
|
||||
CHECK (width > 0 AND height > 0),
|
||||
|
||||
CONSTRAINT attachment_status_valid
|
||||
CHECK (status IN ('ready', 'failed'))
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_attachments_chat
|
||||
ON attachments(chat_id);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS message_attachments (
|
||||
message_id UUID NOT NULL REFERENCES messages(id),
|
||||
attachment_id UUID NOT NULL REFERENCES attachments(id),
|
||||
|
||||
PRIMARY KEY (message_id, attachment_id)
|
||||
);
|
||||
@@ -0,0 +1,13 @@
|
||||
|
||||
-- Lab4: messages with attachments may have empty text.
|
||||
-- The API checks that a message contains text or attachments.
|
||||
|
||||
ALTER TABLE messages
|
||||
DROP CONSTRAINT IF EXISTS message_length;
|
||||
|
||||
ALTER TABLE messages
|
||||
DROP CONSTRAINT IF EXISTS message_not_blank;
|
||||
|
||||
ALTER TABLE messages
|
||||
ADD CONSTRAINT message_length
|
||||
CHECK (char_length(text) <= 4000);
|
||||
@@ -0,0 +1,95 @@
|
||||
-- Allow asynchronous attachment states.
|
||||
ALTER TABLE attachments
|
||||
DROP CONSTRAINT IF EXISTS attachment_status_valid;
|
||||
|
||||
ALTER TABLE attachments
|
||||
ADD CONSTRAINT attachment_status_valid
|
||||
CHECK (
|
||||
status IN (
|
||||
'queued',
|
||||
'processing',
|
||||
'ready',
|
||||
'failed'
|
||||
)
|
||||
);
|
||||
|
||||
-- Staging object stored in the private S3 bucket.
|
||||
ALTER TABLE attachments
|
||||
ADD COLUMN IF NOT EXISTS staging_key TEXT;
|
||||
|
||||
-- Machine-readable final error.
|
||||
ALTER TABLE attachments
|
||||
ADD COLUMN IF NOT EXISTS error_code TEXT;
|
||||
|
||||
-- Last metadata update timestamp.
|
||||
ALTER TABLE attachments
|
||||
ADD COLUMN IF NOT EXISTS updated_at
|
||||
TIMESTAMPTZ NOT NULL DEFAULT NOW();
|
||||
|
||||
-- Worker jobs and durable publication intent.
|
||||
CREATE TABLE IF NOT EXISTS attachment_jobs (
|
||||
id UUID PRIMARY KEY,
|
||||
attachment_id UUID NOT NULL UNIQUE
|
||||
REFERENCES attachments(id),
|
||||
|
||||
status TEXT NOT NULL DEFAULT 'queued',
|
||||
attempts INTEGER NOT NULL DEFAULT 0,
|
||||
max_attempts INTEGER NOT NULL DEFAULT 5,
|
||||
|
||||
published_at TIMESTAMPTZ,
|
||||
started_at TIMESTAMPTZ,
|
||||
completed_at TIMESTAMPTZ,
|
||||
next_attempt_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
|
||||
CONSTRAINT attachment_job_status_valid
|
||||
CHECK (
|
||||
status IN (
|
||||
'queued',
|
||||
'processing',
|
||||
'ready',
|
||||
'failed'
|
||||
)
|
||||
),
|
||||
|
||||
CONSTRAINT attachment_job_attempts_valid
|
||||
CHECK (
|
||||
attempts >= 0
|
||||
AND max_attempts BETWEEN 1 AND 20
|
||||
)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_attachment_jobs_pending
|
||||
ON attachment_jobs(status, next_attempt_at);
|
||||
|
||||
-- Image variants created by the worker.
|
||||
CREATE TABLE IF NOT EXISTS attachment_variants (
|
||||
attachment_id UUID NOT NULL
|
||||
REFERENCES attachments(id),
|
||||
|
||||
variant_name TEXT NOT NULL,
|
||||
object_key TEXT NOT NULL,
|
||||
content_type TEXT NOT NULL,
|
||||
size_bytes BIGINT NOT NULL,
|
||||
sha256 CHAR(64) NOT NULL,
|
||||
width INTEGER NOT NULL,
|
||||
height INTEGER NOT NULL,
|
||||
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
|
||||
PRIMARY KEY (attachment_id, variant_name),
|
||||
|
||||
CONSTRAINT variant_name_valid
|
||||
CHECK (variant_name IN ('thumbnail')),
|
||||
|
||||
CONSTRAINT variant_size_valid
|
||||
CHECK (size_bytes > 0),
|
||||
|
||||
CONSTRAINT variant_dimensions_valid
|
||||
CHECK (width > 0 AND height > 0)
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_attachment_variants_key
|
||||
ON attachment_variants(object_key);
|
||||
@@ -0,0 +1,37 @@
|
||||
events {
|
||||
worker_connections 1024;
|
||||
}
|
||||
|
||||
http {
|
||||
resolver 127.0.0.11 valid=2s ipv6=off;
|
||||
resolver_timeout 2s;
|
||||
|
||||
upstream messenger_api {
|
||||
zone messenger_api 64k;
|
||||
|
||||
server api1:8080 resolve max_fails=1 fail_timeout=2s;
|
||||
server api2:8080 resolve max_fails=1 fail_timeout=2s;
|
||||
}
|
||||
|
||||
server {
|
||||
client_max_body_size 11m;
|
||||
listen 80;
|
||||
|
||||
location / {
|
||||
proxy_pass http://messenger_api;
|
||||
|
||||
proxy_http_version 1.1;
|
||||
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
|
||||
proxy_connect_timeout 2s;
|
||||
proxy_read_timeout 30s;
|
||||
proxy_send_timeout 30s;
|
||||
|
||||
proxy_next_upstream error timeout http_502 http_503 http_504;
|
||||
proxy_next_upstream_tries 2;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
fastapi==0.115.12
|
||||
uvicorn==0.34.2
|
||||
psycopg[binary]==3.2.6
|
||||
argon2-cffi==23.1.0
|
||||
boto3==1.40.70
|
||||
Pillow==11.2.1
|
||||
pika==1.3.2
|
||||
@@ -0,0 +1,24 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
echo "Stopping API and image processing..."
|
||||
docker compose stop api1 api2 worker dispatcher
|
||||
|
||||
echo "Restarting persistent storage and broker..."
|
||||
docker compose restart db minio rabbitmq
|
||||
|
||||
echo "Waiting for PostgreSQL and RabbitMQ..."
|
||||
docker compose up -d --wait db rabbitmq
|
||||
|
||||
echo "Checking S3 bucket..."
|
||||
docker compose run --rm --no-deps s3-init
|
||||
|
||||
echo "Starting dispatcher and worker..."
|
||||
docker compose up -d dispatcher worker
|
||||
|
||||
echo "Starting API replicas..."
|
||||
docker compose up -d --wait api1 api2
|
||||
|
||||
echo "Restart complete. Persistent volumes preserved."
|
||||
@@ -0,0 +1,33 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
if [[ -z "${TARGET_INSTANCE_ID:-}" ]]; then
|
||||
echo "TARGET_INSTANCE_ID is not set" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
for service in api1 api2; do
|
||||
instance_id=$(
|
||||
docker compose exec -T "$service" python -c '
|
||||
import urllib.request
|
||||
|
||||
response = urllib.request.urlopen(
|
||||
"http://127.0.0.1:8080/health/live",
|
||||
timeout=2
|
||||
)
|
||||
|
||||
print(response.headers.get("X-Instance-Id", ""))
|
||||
' 2>/dev/null
|
||||
) || continue
|
||||
|
||||
if [[ "$instance_id" == "$TARGET_INSTANCE_ID" ]]; then
|
||||
echo "Stopping $service"
|
||||
docker compose stop -t 2 "$service"
|
||||
exit 0
|
||||
fi
|
||||
done
|
||||
|
||||
echo "Target API instance not found" >&2
|
||||
exit 1
|
||||
@@ -0,0 +1,77 @@
|
||||
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
sql() {
|
||||
docker compose exec -T db \
|
||||
psql -U messenger -d messenger -At -c "$1"
|
||||
}
|
||||
|
||||
echo "Finding completed image job..."
|
||||
|
||||
JOB_ID=$(sql "
|
||||
SELECT j.id
|
||||
FROM attachment_jobs j
|
||||
JOIN attachment_variants v
|
||||
ON v.attachment_id = j.attachment_id
|
||||
WHERE j.status = 'ready'
|
||||
AND v.variant_name = 'thumbnail'
|
||||
ORDER BY j.created_at DESC
|
||||
LIMIT 1;
|
||||
")
|
||||
|
||||
if [[ -z "$JOB_ID" ]]; then
|
||||
echo "FAIL: no completed image job found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Job: $JOB_ID"
|
||||
|
||||
STATE_QUERY="
|
||||
SELECT
|
||||
j.status || '|' ||
|
||||
j.attempts || '|' ||
|
||||
v.sha256 || '|' ||
|
||||
(
|
||||
SELECT COUNT(*)
|
||||
FROM attachment_variants
|
||||
WHERE attachment_id = j.attachment_id
|
||||
)
|
||||
FROM attachment_jobs j
|
||||
JOIN attachment_variants v
|
||||
ON v.attachment_id = j.attachment_id
|
||||
WHERE j.id = '$JOB_ID'
|
||||
AND v.variant_name = 'thumbnail';
|
||||
"
|
||||
|
||||
BEFORE=$(sql "$STATE_QUERY")
|
||||
|
||||
echo "Before: $BEFORE"
|
||||
echo "Publishing the same job twice..."
|
||||
|
||||
docker compose run --rm --no-deps api1 \
|
||||
python -c '
|
||||
import sys
|
||||
from src.queue import publish_job
|
||||
|
||||
job_id = sys.argv[1]
|
||||
publish_job(job_id)
|
||||
publish_job(job_id)
|
||||
print("Two duplicate messages published")
|
||||
' "$JOB_ID"
|
||||
|
||||
sleep 5
|
||||
|
||||
AFTER=$(sql "$STATE_QUERY")
|
||||
|
||||
echo "After: $AFTER"
|
||||
|
||||
if [[ "$BEFORE" != "$AFTER" ]]; then
|
||||
echo "FAIL: completed job changed after redelivery"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "PASS: duplicate delivery did not change completed job"
|
||||
echo "PASS: thumbnail checksum and variant count preserved"
|
||||
@@ -0,0 +1,246 @@
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
import json
|
||||
import subprocess
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
import uuid
|
||||
import zlib
|
||||
import struct
|
||||
|
||||
|
||||
BASE = "http://127.0.0.1:8080"
|
||||
|
||||
|
||||
def docker(*args):
|
||||
subprocess.run(
|
||||
["docker", "compose", *args],
|
||||
check=True,
|
||||
)
|
||||
|
||||
|
||||
def api(method, path, token=None, body=None, headers=None):
|
||||
request_headers = dict(headers or {})
|
||||
|
||||
if token:
|
||||
request_headers["Authorization"] = f"Bearer {token}"
|
||||
|
||||
if isinstance(body, dict):
|
||||
body = json.dumps(body).encode("utf-8")
|
||||
request_headers["Content-Type"] = "application/json"
|
||||
|
||||
request = urllib.request.Request(
|
||||
BASE + path,
|
||||
data=body,
|
||||
headers=request_headers,
|
||||
method=method,
|
||||
)
|
||||
|
||||
with urllib.request.urlopen(request, timeout=10) as response:
|
||||
data = response.read()
|
||||
return (
|
||||
response.status,
|
||||
json.loads(data) if data and
|
||||
"application/json" in response.headers.get(
|
||||
"Content-Type", ""
|
||||
) else data,
|
||||
)
|
||||
|
||||
|
||||
def png_chunk(name, data):
|
||||
value = name + data
|
||||
return (
|
||||
struct.pack(">I", len(data))
|
||||
+ value
|
||||
+ struct.pack(">I", zlib.crc32(value) & 0xFFFFFFFF)
|
||||
)
|
||||
|
||||
|
||||
def make_png():
|
||||
width, height = 320, 200
|
||||
scanline = b"\x00" + bytes([220, 80, 40]) * width
|
||||
|
||||
return (
|
||||
b"\x89PNG\r\n\x1a\n"
|
||||
+ png_chunk(
|
||||
b"IHDR",
|
||||
struct.pack(">IIBBBBB", width, height, 8, 2, 0, 0, 0),
|
||||
)
|
||||
+ png_chunk(
|
||||
b"IDAT", zlib.compress(scanline * height)
|
||||
)
|
||||
+ png_chunk(b"IEND", b"")
|
||||
)
|
||||
|
||||
|
||||
def wait_ready():
|
||||
for _ in range(90):
|
||||
try:
|
||||
status, data = api("GET", "/health/ready")
|
||||
if status == 200 and data.get("status") == "ok":
|
||||
return
|
||||
except Exception:
|
||||
pass
|
||||
time.sleep(1)
|
||||
|
||||
raise RuntimeError("API did not become ready")
|
||||
|
||||
|
||||
def main():
|
||||
suffix = uuid.uuid4().hex[:12]
|
||||
username = "lab5_" + suffix
|
||||
password = "Lab5TestPassword_" + suffix
|
||||
|
||||
print("Stopping worker...", flush=True)
|
||||
docker("stop", "worker")
|
||||
wait_ready()
|
||||
|
||||
status, user = api(
|
||||
"POST",
|
||||
"/api/v1/users",
|
||||
body={
|
||||
"username": username,
|
||||
"display_name": "Lab5 recovery test",
|
||||
"password": password,
|
||||
},
|
||||
)
|
||||
assert status == 201
|
||||
|
||||
credentials = base64.b64encode(
|
||||
f"{username}:{password}".encode()
|
||||
).decode()
|
||||
|
||||
status, session = api(
|
||||
"POST",
|
||||
"/api/v1/auth/sessions",
|
||||
headers={"Authorization": f"Basic {credentials}"},
|
||||
body=b"",
|
||||
)
|
||||
assert status == 201
|
||||
token = session["token"]
|
||||
|
||||
status, chat = api(
|
||||
"POST",
|
||||
"/api/v1/chats",
|
||||
token,
|
||||
{
|
||||
"title": "Worker recovery",
|
||||
"member_ids": [user["id"]],
|
||||
},
|
||||
)
|
||||
assert status == 201
|
||||
|
||||
original = make_png()
|
||||
path = (
|
||||
f"/api/v1/chats/{chat['id']}/attachments"
|
||||
"?filename=recovery.png"
|
||||
)
|
||||
|
||||
status, attachment = api(
|
||||
"POST",
|
||||
path,
|
||||
token,
|
||||
original,
|
||||
{"Content-Type": "image/png"},
|
||||
)
|
||||
|
||||
assert status == 202, f"Expected 202, got {status}"
|
||||
assert attachment["status"] == "queued"
|
||||
attachment_id = attachment["id"]
|
||||
print("Upload: 202 queued", flush=True)
|
||||
|
||||
status, message = api(
|
||||
"POST",
|
||||
f"/api/v1/chats/{chat['id']}/messages",
|
||||
token,
|
||||
{"text": "", "attachment_ids": [attachment_id]},
|
||||
{"Idempotency-Key": "recovery-" + suffix},
|
||||
)
|
||||
assert status == 201
|
||||
assert attachment_id in message["attachment_ids"]
|
||||
print("Message created while worker is stopped", flush=True)
|
||||
|
||||
status, metadata = api(
|
||||
"GET", f"/api/v1/attachments/{attachment_id}", token
|
||||
)
|
||||
assert status == 200 and metadata["status"] == "queued"
|
||||
|
||||
print("Restarting RabbitMQ and API...", flush=True)
|
||||
docker("restart", "rabbitmq")
|
||||
docker("up", "-d", "--wait", "rabbitmq")
|
||||
docker("restart", "api1", "api2")
|
||||
wait_ready()
|
||||
|
||||
status, metadata = api(
|
||||
"GET", f"/api/v1/attachments/{attachment_id}", token
|
||||
)
|
||||
assert metadata["status"] == "queued"
|
||||
print("Queued attachment survived restart", flush=True)
|
||||
|
||||
print("Starting worker...", flush=True)
|
||||
docker("up", "-d", "worker")
|
||||
|
||||
deadline = time.monotonic() + 90
|
||||
|
||||
while time.monotonic() < deadline:
|
||||
status, metadata = api(
|
||||
"GET", f"/api/v1/attachments/{attachment_id}", token
|
||||
)
|
||||
|
||||
if metadata["status"] == "ready":
|
||||
break
|
||||
|
||||
if metadata["status"] == "failed":
|
||||
raise RuntimeError(
|
||||
f"Processing failed: {metadata.get('error_code')}"
|
||||
)
|
||||
|
||||
time.sleep(1)
|
||||
else:
|
||||
raise RuntimeError("Attachment did not become ready")
|
||||
|
||||
status, downloaded = api(
|
||||
"GET",
|
||||
f"/api/v1/attachments/{attachment_id}/content",
|
||||
token,
|
||||
)
|
||||
assert status == 200
|
||||
assert downloaded == original
|
||||
assert hashlib.sha256(downloaded).hexdigest() == (
|
||||
hashlib.sha256(original).hexdigest()
|
||||
)
|
||||
|
||||
status, thumbnail = api(
|
||||
"GET",
|
||||
f"/api/v1/attachments/{attachment_id}/content"
|
||||
"?variant=thumbnail",
|
||||
token,
|
||||
)
|
||||
|
||||
assert status == 200
|
||||
assert thumbnail.startswith(b"\xff\xd8")
|
||||
assert thumbnail.endswith(b"\xff\xd9")
|
||||
|
||||
variants = metadata["variants"]
|
||||
assert any(
|
||||
v["name"] == "thumbnail"
|
||||
and v["width"] == 256
|
||||
and v["height"] == 160
|
||||
for v in variants
|
||||
)
|
||||
|
||||
print("Attachment: ready", flush=True)
|
||||
print("Original: SHA-256 verified", flush=True)
|
||||
print("Thumbnail: JPEG 256x160 verified", flush=True)
|
||||
print("PASS: durable worker recovery", flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
finally:
|
||||
# Restore the worker even when the test fails.
|
||||
docker("up", "-d", "worker")
|
||||
@@ -0,0 +1,108 @@
|
||||
|
||||
import io
|
||||
import re
|
||||
import uuid
|
||||
import hashlib
|
||||
|
||||
from PIL import Image, UnidentifiedImageError
|
||||
|
||||
|
||||
MAX_BYTES = 10 * 1024 * 1024
|
||||
MAX_PIXELS = 40_000_000
|
||||
|
||||
|
||||
def valid_filename(filename):
|
||||
if not isinstance(filename, str):
|
||||
return False
|
||||
|
||||
if not 1 <= len(filename) <= 255:
|
||||
return False
|
||||
|
||||
if filename.strip() == "":
|
||||
return False
|
||||
|
||||
if "/" in filename or "\\" in filename:
|
||||
return False
|
||||
|
||||
if any(ord(ch) < 32 or ord(ch) == 127 for ch in filename):
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
|
||||
def inspect_image(data, declared_mime):
|
||||
if len(data) > MAX_BYTES:
|
||||
raise ValueError("too_large")
|
||||
|
||||
if not data:
|
||||
raise ValueError("invalid_image")
|
||||
|
||||
if declared_mime not in ("image/png", "image/jpeg"):
|
||||
raise ValueError("invalid_image")
|
||||
|
||||
try:
|
||||
with Image.open(io.BytesIO(data)) as image:
|
||||
image_format = image.format
|
||||
|
||||
width, height = image.size
|
||||
|
||||
if width * height > MAX_PIXELS:
|
||||
raise ValueError("too_many_pixels")
|
||||
|
||||
if image_format not in ("PNG", "JPEG"):
|
||||
raise ValueError("invalid_image")
|
||||
|
||||
actual_mime = (
|
||||
"image/png"
|
||||
if image_format == "PNG"
|
||||
else "image/jpeg"
|
||||
)
|
||||
|
||||
if declared_mime != actual_mime:
|
||||
raise ValueError("invalid_image")
|
||||
|
||||
image.verify()
|
||||
|
||||
except Image.DecompressionBombError:
|
||||
raise ValueError("too_many_pixels")
|
||||
except (
|
||||
UnidentifiedImageError,
|
||||
OSError,
|
||||
SyntaxError,
|
||||
EOFError,
|
||||
):
|
||||
raise ValueError("invalid_image")
|
||||
|
||||
return {
|
||||
"content_type": actual_mime,
|
||||
"width": width,
|
||||
"height": height,
|
||||
"size_bytes": len(data),
|
||||
"sha256": hashlib.sha256(data).hexdigest(),
|
||||
}
|
||||
|
||||
def attachment_json(row, variants=None):
|
||||
from datetime import timezone
|
||||
|
||||
def timestamp(value):
|
||||
return value.astimezone(timezone.utc).isoformat(
|
||||
timespec="microseconds"
|
||||
).replace("+00:00", "Z")
|
||||
|
||||
return {
|
||||
"id": str(row["id"]),
|
||||
"chat_id": str(row["chat_id"]),
|
||||
"owner_id": str(row["uploader_id"]),
|
||||
"filename": row["filename"],
|
||||
"content_type": row["content_type"],
|
||||
"size_bytes": row["size_bytes"],
|
||||
"sha256": row["sha256"],
|
||||
"status": row["status"],
|
||||
"variants": variants if variants is not None else [],
|
||||
"error_code": row["error_code"],
|
||||
"created_at": timestamp(row["created_at"]),
|
||||
"updated_at": timestamp(row["updated_at"]),
|
||||
}
|
||||
|
||||
def object_key():
|
||||
return f"originals/{uuid.uuid4()}"
|
||||
@@ -0,0 +1,84 @@
|
||||
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
|
||||
import psycopg
|
||||
from psycopg.rows import dict_row
|
||||
|
||||
from src.queue import publish_job
|
||||
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s %(levelname)s %(message)s",
|
||||
)
|
||||
|
||||
logger = logging.getLogger("image-dispatcher")
|
||||
|
||||
DATABASE_URL = os.environ["DATABASE_URL"]
|
||||
|
||||
|
||||
def dispatch_one():
|
||||
with psycopg.connect(
|
||||
DATABASE_URL,
|
||||
row_factory=dict_row,
|
||||
) as conn:
|
||||
|
||||
# Only publish due, unpublished jobs.
|
||||
# The row lock also prevents two dispatchers
|
||||
# from selecting the same job simultaneously.
|
||||
job = conn.execute(
|
||||
"""
|
||||
SELECT id
|
||||
FROM attachment_jobs
|
||||
WHERE status = 'queued'
|
||||
AND published_at IS NULL
|
||||
AND next_attempt_at <= NOW()
|
||||
ORDER BY created_at, id
|
||||
LIMIT 1
|
||||
FOR UPDATE SKIP LOCKED
|
||||
""",
|
||||
).fetchone()
|
||||
|
||||
if job is None:
|
||||
return False
|
||||
|
||||
job_id = str(job["id"])
|
||||
|
||||
# RabbitMQ publisher confirms are enabled.
|
||||
# An exception rolls back this DB transaction.
|
||||
publish_job(job_id)
|
||||
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachment_jobs
|
||||
SET published_at = NOW(),
|
||||
updated_at = NOW()
|
||||
WHERE id = %s
|
||||
""",
|
||||
(job_id,),
|
||||
)
|
||||
|
||||
logger.info("Published job %s", job_id)
|
||||
|
||||
return True
|
||||
|
||||
|
||||
def main():
|
||||
logger.info("Dispatcher started")
|
||||
|
||||
while True:
|
||||
try:
|
||||
if not dispatch_one():
|
||||
time.sleep(1)
|
||||
|
||||
except Exception:
|
||||
logger.exception(
|
||||
"Dispatch failed; retrying"
|
||||
)
|
||||
time.sleep(3)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+1272
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,53 @@
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import psycopg
|
||||
|
||||
|
||||
DATABASE_URL = os.environ["DATABASE_URL"]
|
||||
MIGRATIONS_DIR = Path(__file__).resolve().parent.parent / "migrations"
|
||||
|
||||
|
||||
def run_migrations():
|
||||
with psycopg.connect(DATABASE_URL) as conn:
|
||||
with conn.cursor() as cur:
|
||||
# Prevent simultaneous migration runs.
|
||||
cur.execute("SELECT pg_advisory_xact_lock(2026002)")
|
||||
|
||||
cur.execute("""
|
||||
CREATE TABLE IF NOT EXISTS schema_migrations (
|
||||
version TEXT PRIMARY KEY,
|
||||
applied_at TIMESTAMPTZ NOT NULL
|
||||
DEFAULT clock_timestamp()
|
||||
)
|
||||
""")
|
||||
|
||||
for file in sorted(MIGRATIONS_DIR.glob("*.sql")):
|
||||
cur.execute(
|
||||
"SELECT 1 FROM schema_migrations WHERE version = %s",
|
||||
(file.name,)
|
||||
)
|
||||
|
||||
if cur.fetchone():
|
||||
print(f"Already applied: {file.name}")
|
||||
continue
|
||||
|
||||
sql = file.read_text(encoding="utf-8")
|
||||
|
||||
for statement in sql.split(";"):
|
||||
if statement.strip():
|
||||
cur.execute(statement)
|
||||
|
||||
cur.execute(
|
||||
"INSERT INTO schema_migrations (version) VALUES (%s)",
|
||||
(file.name,)
|
||||
)
|
||||
|
||||
print(f"Applied migration: {file.name}")
|
||||
|
||||
print("Migrations completed")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
run_migrations()
|
||||
@@ -0,0 +1,89 @@
|
||||
|
||||
import json
|
||||
import os
|
||||
import uuid
|
||||
|
||||
import pika
|
||||
|
||||
|
||||
RABBITMQ_HOST = os.getenv("RABBITMQ_HOST", "rabbitmq")
|
||||
RABBITMQ_USER = os.environ["RABBITMQ_USER"]
|
||||
RABBITMQ_PASSWORD = os.environ["RABBITMQ_PASSWORD"]
|
||||
RABBITMQ_QUEUE = os.getenv(
|
||||
"RABBITMQ_QUEUE",
|
||||
"image_jobs",
|
||||
)
|
||||
|
||||
|
||||
def connect():
|
||||
credentials = pika.PlainCredentials(
|
||||
RABBITMQ_USER,
|
||||
RABBITMQ_PASSWORD,
|
||||
)
|
||||
|
||||
parameters = pika.ConnectionParameters(
|
||||
host=RABBITMQ_HOST,
|
||||
port=5672,
|
||||
credentials=credentials,
|
||||
heartbeat=30,
|
||||
blocked_connection_timeout=15,
|
||||
connection_attempts=3,
|
||||
retry_delay=2,
|
||||
)
|
||||
|
||||
return pika.BlockingConnection(parameters)
|
||||
|
||||
|
||||
def declare_queue(channel):
|
||||
channel.queue_declare(
|
||||
queue=RABBITMQ_QUEUE,
|
||||
durable=True,
|
||||
arguments={
|
||||
"x-queue-type": "quorum",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def publish_job(job_id):
|
||||
connection = connect()
|
||||
|
||||
try:
|
||||
channel = connection.channel()
|
||||
declare_queue(channel)
|
||||
|
||||
# Wait for confirmation from RabbitMQ.
|
||||
channel.confirm_delivery()
|
||||
|
||||
payload = json.dumps({
|
||||
"job_id": str(uuid.UUID(str(job_id))),
|
||||
}).encode("utf-8")
|
||||
|
||||
channel.basic_publish(
|
||||
exchange="",
|
||||
routing_key=RABBITMQ_QUEUE,
|
||||
body=payload,
|
||||
properties=pika.BasicProperties(
|
||||
delivery_mode=2,
|
||||
content_type="application/json",
|
||||
message_id=str(job_id),
|
||||
),
|
||||
mandatory=True,
|
||||
)
|
||||
|
||||
finally:
|
||||
if connection.is_open:
|
||||
connection.close()
|
||||
|
||||
|
||||
def parse_job(body):
|
||||
payload = json.loads(body)
|
||||
|
||||
if not isinstance(payload, dict):
|
||||
raise ValueError("Invalid job payload")
|
||||
|
||||
job_id = payload.get("job_id")
|
||||
|
||||
if not isinstance(job_id, str):
|
||||
raise ValueError("Missing job_id")
|
||||
|
||||
return str(uuid.UUID(job_id))
|
||||
@@ -0,0 +1,60 @@
|
||||
|
||||
import os
|
||||
import time
|
||||
|
||||
import boto3
|
||||
from botocore.config import Config
|
||||
from botocore.exceptions import ClientError
|
||||
|
||||
|
||||
def main():
|
||||
bucket = os.environ["S3_BUCKET"]
|
||||
|
||||
client = boto3.client(
|
||||
"s3",
|
||||
endpoint_url=os.environ["S3_ENDPOINT"],
|
||||
aws_access_key_id=os.environ["S3_ACCESS_KEY"],
|
||||
aws_secret_access_key=os.environ["S3_SECRET_KEY"],
|
||||
region_name="us-east-1",
|
||||
config=Config(
|
||||
signature_version="s3v4",
|
||||
s3={"addressing_style": "path"},
|
||||
),
|
||||
)
|
||||
|
||||
for attempt in range(30):
|
||||
try:
|
||||
client.head_bucket(Bucket=bucket)
|
||||
print(f"Bucket already exists: {bucket}", flush=True)
|
||||
break
|
||||
|
||||
except ClientError as exc:
|
||||
code = exc.response["Error"]["Code"]
|
||||
|
||||
if code in ("404", "NoSuchBucket"):
|
||||
try:
|
||||
client.create_bucket(Bucket=bucket)
|
||||
print(f"Bucket created: {bucket}", flush=True)
|
||||
break
|
||||
except Exception as create_error:
|
||||
print(
|
||||
f"Waiting for S3: {create_error}",
|
||||
flush=True,
|
||||
)
|
||||
else:
|
||||
print(f"Waiting for S3: {exc}", flush=True)
|
||||
|
||||
except Exception as exc:
|
||||
print(f"Waiting for S3: {exc}", flush=True)
|
||||
|
||||
if attempt == 29:
|
||||
raise RuntimeError("Cannot initialize S3 bucket")
|
||||
|
||||
time.sleep(2)
|
||||
|
||||
# No public bucket policy is configured.
|
||||
print(f"Private S3 bucket is ready: {bucket}", flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+456
@@ -0,0 +1,456 @@
|
||||
|
||||
import hashlib
|
||||
import io
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
|
||||
import boto3
|
||||
import pika
|
||||
import psycopg
|
||||
from botocore.config import Config
|
||||
from PIL import Image, ImageOps
|
||||
from psycopg.rows import dict_row
|
||||
|
||||
from src.queue import (
|
||||
RABBITMQ_QUEUE,
|
||||
connect,
|
||||
declare_queue,
|
||||
parse_job,
|
||||
)
|
||||
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s %(levelname)s %(message)s",
|
||||
)
|
||||
|
||||
logger = logging.getLogger("image-worker")
|
||||
|
||||
DATABASE_URL = os.environ["DATABASE_URL"]
|
||||
S3_BUCKET = os.environ["S3_BUCKET"]
|
||||
MAX_BYTES = 10 * 1024 * 1024
|
||||
|
||||
s3 = boto3.client(
|
||||
"s3",
|
||||
endpoint_url=os.environ["S3_ENDPOINT"],
|
||||
aws_access_key_id=os.environ["S3_ACCESS_KEY"],
|
||||
aws_secret_access_key=os.environ["S3_SECRET_KEY"],
|
||||
region_name="us-east-1",
|
||||
config=Config(
|
||||
signature_version="s3v4",
|
||||
s3={"addressing_style": "path"},
|
||||
connect_timeout=3,
|
||||
read_timeout=15,
|
||||
retries={"max_attempts": 1},
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class InvalidImage(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def make_thumbnail(original):
|
||||
try:
|
||||
with Image.open(io.BytesIO(original)) as image:
|
||||
if image.format not in ("PNG", "JPEG"):
|
||||
raise InvalidImage("Unsupported image")
|
||||
|
||||
if image.width * image.height > 40_000_000:
|
||||
raise InvalidImage("Too many pixels")
|
||||
|
||||
image = ImageOps.exif_transpose(image)
|
||||
image.thumbnail(
|
||||
(256, 256),
|
||||
Image.Resampling.LANCZOS,
|
||||
)
|
||||
|
||||
# JPEG does not support transparency.
|
||||
rgba = image.convert("RGBA")
|
||||
|
||||
background = Image.new(
|
||||
"RGB",
|
||||
rgba.size,
|
||||
(255, 255, 255),
|
||||
)
|
||||
|
||||
background.paste(
|
||||
rgba,
|
||||
mask=rgba.getchannel("A"),
|
||||
)
|
||||
|
||||
output = io.BytesIO()
|
||||
|
||||
background.save(
|
||||
output,
|
||||
format="JPEG",
|
||||
quality=85,
|
||||
optimize=True,
|
||||
)
|
||||
|
||||
return (
|
||||
output.getvalue(),
|
||||
background.width,
|
||||
background.height,
|
||||
)
|
||||
|
||||
except (OSError, SyntaxError, EOFError) as exc:
|
||||
raise InvalidImage("Invalid image") from exc
|
||||
|
||||
|
||||
def get_staging_object(key):
|
||||
response = s3.get_object(
|
||||
Bucket=S3_BUCKET,
|
||||
Key=key,
|
||||
)
|
||||
|
||||
stream = response["Body"]
|
||||
|
||||
try:
|
||||
data = stream.read(MAX_BYTES + 1)
|
||||
finally:
|
||||
stream.close()
|
||||
|
||||
if not data or len(data) > MAX_BYTES:
|
||||
raise InvalidImage("Invalid staging size")
|
||||
|
||||
return data
|
||||
|
||||
|
||||
def process_job(job_id):
|
||||
"""
|
||||
Use a PostgreSQL row lock to prevent concurrent
|
||||
processing of the same logical job.
|
||||
|
||||
The transaction commits before RabbitMQ ACK.
|
||||
"""
|
||||
|
||||
with psycopg.connect(
|
||||
DATABASE_URL,
|
||||
row_factory=dict_row,
|
||||
) as conn:
|
||||
|
||||
job = conn.execute(
|
||||
"""
|
||||
SELECT j.*, a.staging_key,
|
||||
a.object_key, a.sha256,
|
||||
a.size_bytes, a.status AS attachment_status
|
||||
FROM attachment_jobs j
|
||||
JOIN attachments a
|
||||
ON a.id = j.attachment_id
|
||||
WHERE j.id = %s
|
||||
FOR UPDATE OF j, a
|
||||
""",
|
||||
(job_id,),
|
||||
).fetchone()
|
||||
|
||||
if job is None:
|
||||
logger.info("Unknown job %s, ignoring", job_id)
|
||||
return
|
||||
|
||||
if job["status"] == "ready":
|
||||
logger.info("Job %s already completed", job_id)
|
||||
return
|
||||
|
||||
if job["status"] == "failed":
|
||||
logger.info("Job %s already failed", job_id)
|
||||
return
|
||||
|
||||
if job["next_attempt_at"] is not None:
|
||||
from datetime import datetime, timezone
|
||||
|
||||
if job["next_attempt_at"] > datetime.now(timezone.utc):
|
||||
logger.info("Job %s is not due yet", job_id)
|
||||
return
|
||||
|
||||
attempts = job["attempts"] + 1
|
||||
|
||||
if attempts > job["max_attempts"]:
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'failed',
|
||||
completed_at = NOW(),
|
||||
updated_at = NOW()
|
||||
WHERE id = %s
|
||||
""",
|
||||
(job_id,),
|
||||
)
|
||||
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachments
|
||||
SET status = 'failed',
|
||||
error_code = 'processing_failed',
|
||||
updated_at = NOW()
|
||||
WHERE id = %s
|
||||
""",
|
||||
(job["attachment_id"],),
|
||||
)
|
||||
|
||||
return
|
||||
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'processing',
|
||||
attempts = %s,
|
||||
started_at = NOW(),
|
||||
updated_at = NOW()
|
||||
WHERE id = %s
|
||||
""",
|
||||
(attempts, job_id),
|
||||
)
|
||||
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachments
|
||||
SET status = 'processing',
|
||||
updated_at = NOW()
|
||||
WHERE id = %s
|
||||
""",
|
||||
(job["attachment_id"],),
|
||||
)
|
||||
|
||||
try:
|
||||
original = get_staging_object(
|
||||
job["staging_key"]
|
||||
)
|
||||
|
||||
if (
|
||||
len(original) != job["size_bytes"]
|
||||
or hashlib.sha256(original).hexdigest()
|
||||
!= job["sha256"]
|
||||
):
|
||||
raise InvalidImage("Staging checksum mismatch")
|
||||
|
||||
thumbnail, width, height = make_thumbnail(
|
||||
original
|
||||
)
|
||||
|
||||
attachment_id = str(job["attachment_id"])
|
||||
|
||||
# Stable keys make duplicate delivery safe.
|
||||
original_key = job["object_key"]
|
||||
thumb_key = (
|
||||
f"thumbnails/{attachment_id}.jpg"
|
||||
)
|
||||
|
||||
s3.put_object(
|
||||
Bucket=S3_BUCKET,
|
||||
Key=original_key,
|
||||
Body=original,
|
||||
ContentType="image/png"
|
||||
if original[:8] == b"\x89PNG\r\n\x1a\n"
|
||||
else "image/jpeg",
|
||||
)
|
||||
|
||||
s3.put_object(
|
||||
Bucket=S3_BUCKET,
|
||||
Key=thumb_key,
|
||||
Body=thumbnail,
|
||||
ContentType="image/jpeg",
|
||||
)
|
||||
|
||||
thumb_sha256 = hashlib.sha256(
|
||||
thumbnail
|
||||
).hexdigest()
|
||||
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO attachment_variants
|
||||
(
|
||||
attachment_id,
|
||||
variant_name,
|
||||
object_key,
|
||||
content_type,
|
||||
size_bytes,
|
||||
sha256,
|
||||
width,
|
||||
height
|
||||
)
|
||||
VALUES (
|
||||
%s, 'thumbnail', %s, 'image/jpeg',
|
||||
%s, %s, %s, %s
|
||||
)
|
||||
ON CONFLICT (
|
||||
attachment_id, variant_name
|
||||
)
|
||||
DO UPDATE SET
|
||||
object_key = EXCLUDED.object_key,
|
||||
size_bytes = EXCLUDED.size_bytes,
|
||||
sha256 = EXCLUDED.sha256,
|
||||
width = EXCLUDED.width,
|
||||
height = EXCLUDED.height
|
||||
""",
|
||||
(
|
||||
job["attachment_id"],
|
||||
thumb_key,
|
||||
len(thumbnail),
|
||||
thumb_sha256,
|
||||
width,
|
||||
height,
|
||||
),
|
||||
)
|
||||
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachments
|
||||
SET status = 'ready',
|
||||
error_code = NULL,
|
||||
updated_at = NOW()
|
||||
WHERE id = %s
|
||||
""",
|
||||
(job["attachment_id"],),
|
||||
)
|
||||
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'ready',
|
||||
completed_at = NOW(),
|
||||
updated_at = NOW()
|
||||
WHERE id = %s
|
||||
""",
|
||||
(job_id,),
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"Job %s ready, thumbnail %sx%s",
|
||||
job_id,
|
||||
width,
|
||||
height,
|
||||
)
|
||||
|
||||
except Exception as exc:
|
||||
logger.exception(
|
||||
"Processing failed for job %s: %s",
|
||||
job_id,
|
||||
type(exc).__name__,
|
||||
)
|
||||
|
||||
if (
|
||||
attempts >= job["max_attempts"]
|
||||
or isinstance(exc, InvalidImage)
|
||||
):
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'failed',
|
||||
completed_at = NOW(),
|
||||
updated_at = NOW()
|
||||
WHERE id = %s
|
||||
""",
|
||||
(job_id,),
|
||||
)
|
||||
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachments
|
||||
SET status = 'failed',
|
||||
error_code = 'processing_failed',
|
||||
updated_at = NOW()
|
||||
WHERE id = %s
|
||||
""",
|
||||
(job["attachment_id"],),
|
||||
)
|
||||
|
||||
else:
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'queued',
|
||||
next_attempt_at =
|
||||
NOW() + INTERVAL '5 seconds',
|
||||
published_at = NULL,
|
||||
updated_at = NOW()
|
||||
WHERE id = %s
|
||||
""",
|
||||
(job_id,),
|
||||
)
|
||||
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE attachments
|
||||
SET status = 'queued',
|
||||
updated_at = NOW()
|
||||
WHERE id = %s
|
||||
""",
|
||||
(job["attachment_id"],),
|
||||
)
|
||||
|
||||
|
||||
def on_message(channel, method, properties, body):
|
||||
try:
|
||||
job_id = parse_job(body)
|
||||
except (ValueError, TypeError, KeyError):
|
||||
logger.warning("Invalid queue message")
|
||||
channel.basic_ack(
|
||||
delivery_tag=method.delivery_tag
|
||||
)
|
||||
return
|
||||
|
||||
try:
|
||||
process_job(job_id)
|
||||
|
||||
# ACK only after PostgreSQL transaction commits.
|
||||
channel.basic_ack(
|
||||
delivery_tag=method.delivery_tag
|
||||
)
|
||||
|
||||
except Exception:
|
||||
logger.exception(
|
||||
"Unexpected failure for job %s",
|
||||
job_id,
|
||||
)
|
||||
|
||||
channel.basic_nack(
|
||||
delivery_tag=method.delivery_tag,
|
||||
requeue=True,
|
||||
)
|
||||
|
||||
time.sleep(2)
|
||||
|
||||
|
||||
def main():
|
||||
while True:
|
||||
try:
|
||||
connection = connect()
|
||||
channel = connection.channel()
|
||||
|
||||
declare_queue(channel)
|
||||
|
||||
channel.basic_qos(
|
||||
prefetch_count=1
|
||||
)
|
||||
|
||||
channel.basic_consume(
|
||||
queue=RABBITMQ_QUEUE,
|
||||
on_message_callback=on_message,
|
||||
auto_ack=False,
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"Worker listening on %s",
|
||||
RABBITMQ_QUEUE,
|
||||
)
|
||||
|
||||
channel.start_consuming()
|
||||
|
||||
except Exception:
|
||||
logger.exception(
|
||||
"Worker disconnected, reconnecting"
|
||||
)
|
||||
time.sleep(3)
|
||||
|
||||
finally:
|
||||
if (
|
||||
"connection" in locals()
|
||||
and connection.is_open
|
||||
):
|
||||
connection.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user