ЛР5 — БПИ2302 — Бондаренко — на 4 #78
+22
@@ -0,0 +1,22 @@
|
||||
FROM python:3.11.7-slim-bookworm AS builder
|
||||
|
||||
WORKDIR /app
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt
|
||||
|
||||
FROM python:3.11.7-slim-bookworm AS runner
|
||||
|
||||
RUN useradd -u 10001 -m appuser
|
||||
WORKDIR /home/appuser
|
||||
|
||||
COPY --from=builder /install /usr/local
|
||||
COPY app.py worker.py ./
|
||||
|
||||
USER appuser
|
||||
EXPOSE 8080
|
||||
|
||||
HEALTHCHECK --interval=5s --timeout=3s --retries=3 \
|
||||
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8080/health/ready')"
|
||||
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8080", "--timeout-graceful-shutdown", "10"]
|
||||
@@ -1,37 +1,73 @@
|
||||
# Отчёт — лабораторная 5
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
|
||||
## Запуск из чистого clone
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
- Автор, группа: Бондаренко Мария, БПИ2302
|
||||
- Целевая оценка: 4
|
||||
|
||||
## Архитектура и решения
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab4, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
Система построена по микросервисной/событийно-ориентированной схеме с разделением синхронного HTTP API и асинхронного фонового обработчика медиафайлов:
|
||||
|
||||
## Доказательства критериев
|
||||
1. API-сервисы (api-1, api-2):
|
||||
- Реализованы на базе FastAPI, балансируются через Nginx
|
||||
- При загрузке изображения выполняют первичную валидацию сигнатур (PNG/JPEG), проверку размера и разрешения
|
||||
- Сохраняют исходный файл во временную зону объектного хранилища S3
|
||||
- В рамках единой транзакции PostgreSQL фиксируют метаданные вложения в таблице attachments и записывают событие в таблицу outbox
|
||||
2. Транзакционный Outbox:
|
||||
- Фоновый поток в API-инстансах вычитывает неопубликованные записи, отправляет задачи в стрим брокера attachment_jobs и проставляет published_at. Это исключает потерю задач при сбое между COMMIT в БД и публикацией в брокер
|
||||
3. Брокер сообщений:
|
||||
- Запущен с включенной персистентностью
|
||||
- Используются Consumer Groups для балансировки нагрузки между воркерами и подтверждения обработки
|
||||
4. Объектное хранилище:
|
||||
- Хранит временные файлы в staging/ и финальные артефакты в attachments/{chat_id}/{attachment_id}/{variant}
|
||||
5. Фоновый обработчик:
|
||||
- Многопоточный воркер
|
||||
- Вычитывает задачи пачками и обрабатывает их идемпотентно с блокировкой строки в БД
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
## Пайплайн обработки изображений и варианты (Variants)
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
Перед генерацией производных форматов исходное изображение проходит нормализацию:
|
||||
- EXIF-ориентация: применяется ImageOps.exif_transpose(img) для корректного поворота фотографий с мобильных устройств перед ресайзом и кропом
|
||||
- Цветовое пространство: изображения с альфа-каналом (RGBA, LA, P с прозрачностью) накладываются на белый фон (255, 255, 255) и приводятся к RGB
|
||||
- Защита от Decompression Bomb: установлено ограничение Image.MAX_IMAGE_PIXELS = 40000000
|
||||
|
||||
## Эксперименты и отказы
|
||||
## Спецификация генерируемых вариантов (Grade 4):
|
||||
1. thumbnail (превью для списка/чата):
|
||||
- Формат и качество: image/jpeg, quality=85, optimize=True.
|
||||
- Геометрия: пропорциональное масштабирование с вписыванием в квадрат 256256 пикселей (без апскейла, если оригинал меньше)
|
||||
2. optimized (оптимизированный просмотр):
|
||||
- Формат и качество: image/webp, quality=85.
|
||||
- Геометрия: пропорциональное масштабирование с вписыванием в рамку 12801280 пикселей (без апскейла)
|
||||
3. crop (квадратная миниатюра/аватар):
|
||||
- Формат и качество: image/jpeg, quality=85, optimize=True.
|
||||
- Геометрия: центральное квадратное кадрирование с последующим приведением к точному размеру 128128 пикселей
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
## Отказоустойчивость, ретраи и восстановление
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
Для обеспечения надежности уровня на оценку 4 реализованы следующие механизмы:
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
- Разделение ошибок (Permanent / Transient):
|
||||
- Постоянные ошибки сразу переводят вложение в терминальный статус failed с соответствующим error_code и подтверждаются в брокере без бесполезных повторов
|
||||
- Временные ошибки обрабатываются с повторами до 5 раз
|
||||
- Exponential Backoff + Jitter:
|
||||
- Перед постановкой следующей попытки ыполняется задержка для предотвращения лавинообразной нагрузки на S3/БД
|
||||
- Dead Letter Queue (DLQ) и ручной Replay:
|
||||
- При исчерпании попыток вложение получает статус failed, а сообщение перекладывается в отдельный стрим attachment_jobs_dlq
|
||||
- Реализован эндпоинт POST /api/v1/attachments/{attachment_id}/replay, который сбрасывает счетчик попыток, переводит статус обратно в queued и транзакционно создает новую задачу в outbox
|
||||
- Защита от зависаний:
|
||||
- Каждая задача ограничена по времени выполнения (30 секунд).
|
||||
- Фоновый цикл воркера каждые 15 секунд вызывает XAUTOCLAIM, перехватывая зависшие сообщения от упавших экземпляров воркеров
|
||||
- Очистка временного хранилища:
|
||||
- Воркер периодически (раз в 10 минут) сканирует префикс staging/ в S3 и удаляет осиротевшие объекты старше 1 часа.
|
||||
|
||||
## Результаты проверки и артефакты
|
||||
|
||||
1. Контрактные и функциональные тесты:
|
||||
- scripts/check_contract.py — подтверждено соответствие спецификации OpenAPI, наличие заголовков X-Request-Id и X-Instance-Id
|
||||
- tests/smoke.py (evidence/smoke.txt) — пройден полный публичный сценарий
|
||||
2. Демонстрация асинхронности и переживания рестартов:
|
||||
- При остановленном контейнере worker (demo-01c-worker-stopped.txt) загружен файл sample.png (9becbd32-e413-417a-8762-79bcc195e296), получен ответ 202 Accepted со статусом queued (demo-02-uppload-202.json).
|
||||
- Выполнены рестарты API-реплик (demo-04-api-restarted.txt) и брокера Redis (demo-08-after-broker-restart.json).
|
||||
- После запуска воркера (demo-05-worker-started.txt) задача успешно обработана из персистентной очереди, статус перешел в ready (demo-06-ready.json), а вариант thumbnail корректно отдан клиенту (demo-07-thumbnail.jpg).
|
||||
3. Артефакты расширенного пайплайна:
|
||||
- Сформированы и верифицированы все три варианта (g4-variants.txt, g4-thunbnail.jpg, g4-crop.jpg) с корректными MIME-типами (image/jpeg, image/webp) и размерами
|
||||
+218
@@ -0,0 +1,218 @@
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:15.5-alpine3.19
|
||||
environment:
|
||||
POSTGRES_DB: messenger
|
||||
POSTGRES_USER: postgres
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}
|
||||
volumes:
|
||||
- pgdata:/var/lib/postgresql/data
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '0.50'
|
||||
memory: 512M
|
||||
reservations:
|
||||
cpus: '0.10'
|
||||
memory: 128M
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U postgres -d messenger"]
|
||||
interval: 3s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
|
||||
redis:
|
||||
image: redis:7.2.4-alpine
|
||||
command: redis-server --appendonly yes --appendfsync everysec --loglevel warning
|
||||
volumes:
|
||||
- redisdata:/data
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '0.25'
|
||||
memory: 256M
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 3s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
|
||||
migrate:
|
||||
image: postgres:15.5-alpine3.19
|
||||
environment:
|
||||
PGPASSWORD: ${POSTGRES_PASSWORD}
|
||||
volumes:
|
||||
- ./migrations:/migrations
|
||||
command: >
|
||||
sh -c "until pg_isready -h postgres -U postgres; do sleep 1; done;
|
||||
psql -h postgres -U postgres -d messenger -f /migrations/init.sql"
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
restart: "no"
|
||||
|
||||
s3:
|
||||
image: chrislusf/seaweedfs:latest
|
||||
command: ["server", "-s3", "-dir=/data", "-s3.port=8333"]
|
||||
environment:
|
||||
- AWS_ACCESS_KEY_ID=appuser
|
||||
- AWS_SECRET_ACCESS_KEY=appsecretpassword
|
||||
volumes:
|
||||
- s3data:/data
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '0.50'
|
||||
memory: 512M
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "wget -q -O /dev/null http://127.0.0.1:8333/healthz || exit 1"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
start_period: 10s
|
||||
|
||||
s3-init:
|
||||
image: python:3.11.7-slim-bookworm
|
||||
volumes:
|
||||
- ./scripts:/scripts
|
||||
- ./requirements.txt:/requirements.txt
|
||||
environment:
|
||||
- S3_ENDPOINT=http://s3:8333
|
||||
- S3_BUCKET=messenger-attachments
|
||||
- S3_ROOT_ACCESS_KEY=appuser
|
||||
- S3_ROOT_SECRET_KEY=appsecretpassword
|
||||
command: >
|
||||
sh -c "pip install --no-cache-dir -r /requirements.txt &&
|
||||
python /scripts/bootstrap_s3.py"
|
||||
depends_on:
|
||||
s3:
|
||||
condition: service_healthy
|
||||
restart: "no"
|
||||
|
||||
api-1:
|
||||
build: .
|
||||
stop_grace_period: 15s
|
||||
environment:
|
||||
- DATABASE_URL=postgresql://postgres:${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}@postgres:5432/messenger
|
||||
- REDIS_URL=redis://redis:6379/0
|
||||
- SESSION_TTL_SECONDS=3600
|
||||
- INSTANCE_ID=api-replica-1
|
||||
- SECRET_HMAC_KEY=${SECRET_HMAC_KEY}
|
||||
- S3_ENDPOINT=http://s3:8333
|
||||
- S3_BUCKET=messenger-attachments
|
||||
- S3_ACCESS_KEY_ID=appuser
|
||||
- S3_SECRET_ACCESS_KEY=appsecretpassword
|
||||
- BROKER_STREAM=attachment_jobs
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '0.50'
|
||||
memory: 256M
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
s3-init:
|
||||
condition: service_completed_successfully
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8080/health/ready')"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 3
|
||||
|
||||
api-2:
|
||||
build: .
|
||||
stop_grace_period: 15s
|
||||
environment:
|
||||
- DATABASE_URL=postgresql://postgres:${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}@postgres:5432/messenger
|
||||
- REDIS_URL=redis://redis:6379/0
|
||||
- SESSION_TTL_SECONDS=3600
|
||||
- INSTANCE_ID=api-replica-2
|
||||
- SECRET_HMAC_KEY=${SECRET_HMAC_KEY}
|
||||
- S3_ENDPOINT=http://s3:8333
|
||||
- S3_BUCKET=messenger-attachments
|
||||
- S3_ACCESS_KEY_ID=appuser
|
||||
- S3_SECRET_ACCESS_KEY=appsecretpassword
|
||||
- BROKER_STREAM=attachment_jobs
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '0.50'
|
||||
memory: 256M
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
s3-init:
|
||||
condition: service_completed_successfully
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8080/health/ready')"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 3
|
||||
|
||||
worker:
|
||||
build: .
|
||||
command: ["python", "worker.py"]
|
||||
stop_grace_period: 15s
|
||||
environment:
|
||||
- DATABASE_URL=postgresql://postgres:${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}@postgres:5432/messenger
|
||||
- REDIS_URL=redis://redis:6379/0
|
||||
- S3_ENDPOINT=http://s3:8333
|
||||
- S3_BUCKET=messenger-attachments
|
||||
- S3_ACCESS_KEY_ID=appuser
|
||||
- S3_SECRET_ACCESS_KEY=appsecretpassword
|
||||
- BROKER_STREAM=attachment_jobs
|
||||
- BROKER_GROUP=attachment_workers
|
||||
- JOB_MAX_ATTEMPTS=5
|
||||
- MAX_IMAGE_PIXELS=40000000
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '0.50'
|
||||
memory: 256M
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
s3-init:
|
||||
condition: service_completed_successfully
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import os, redis; redis.Redis.from_url(os.environ['REDIS_URL']).ping()"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 3
|
||||
|
||||
nginx:
|
||||
image: nginx:1.25.4-alpine
|
||||
ports:
|
||||
- "8080:8080"
|
||||
volumes:
|
||||
- ./nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '0.25'
|
||||
memory: 128M
|
||||
depends_on:
|
||||
- api-1
|
||||
- api-2
|
||||
healthcheck:
|
||||
test: ["CMD", "nginx", "-t"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 3
|
||||
|
||||
volumes:
|
||||
pgdata:
|
||||
redisdata:
|
||||
s3data:
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"id": "fdeb71d4-ee4f-47a6-8968-ffc7740e5242",
|
||||
"username": "demo_75866",
|
||||
"display_name": "Demo User",
|
||||
"created_at": "2026-11-16T13:43:38.059472Z"
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"token": "grSsHp1gJpFLckF14UT44obEonSGQRvP878JhLyCaNzM6PGBH1LwrdgZIFF8Zu8U",
|
||||
"token_type": "Bearer",
|
||||
"expires_at": "2026-11-16T14:43:38.850596Z",
|
||||
"user": {
|
||||
"id": "fdeb71d4-ee4f-47a6-8968-ffc7740e5242",
|
||||
"username": "demo_75866",
|
||||
"display_name": "Demo User",
|
||||
"created_at": "2026-11-16T13:43:38.059472Z"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"id": "5db90ff5-9604-4853-b002-db4144868bcb",
|
||||
"title": "Demo chat",
|
||||
"member_ids": [
|
||||
"fdeb71d4-ee4f-47a6-8968-ffc7740e5242"
|
||||
],
|
||||
"created_at": "2026-11-16T13:43:39.240968Z"
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
|
||||
mtusi2026lab5-api-1-1 mtusi2026lab5-api-1 "uvicorn app:app --hтАж" api-1 30 minutes ago Up 6 minutes (healthy) 8080/tcp
|
||||
mtusi2026lab5-api-2-1 mtusi2026lab5-api-2 "uvicorn app:app --hтАж" api-2 30 minutes ago Up 6 minutes (healthy) 8080/tcp
|
||||
mtusi2026lab5-migrate-1 postgres:15.5-alpine3.19 "docker-entrypoint.sтАж" migrate 30 minutes ago Exited (0) 5 minutes ago
|
||||
mtusi2026lab5-nginx-1 nginx:1.25.4-alpine "/docker-entrypoint.тАж" nginx 30 minutes ago Up 25 minutes (healthy) 0.0.0.0:8080->8080/tcp, [::]:8080->8080/tcp
|
||||
mtusi2026lab5-postgres-1 postgres:15.5-alpine3.19 "docker-entrypoint.sтАж" postgres 30 minutes ago Up 30 minutes (healthy) 5432/tcp
|
||||
mtusi2026lab5-redis-1 redis:7.2.4-alpine "docker-entrypoint.sтАж" redis 30 minutes ago Up 4 minutes (healthy) 6379/tcp
|
||||
mtusi2026lab5-s3-1 chrislusf/seaweedfs:latest "/entrypoint.sh servтАж" s3 30 minutes ago Up 30 minutes (healthy) 7333/tcp, 8080/tcp, 8333/tcp, 8888/tcp, 9333/tcp, 18080/tcp, 18888/tcp, 19333/tcp
|
||||
mtusi2026lab5-s3-init-1 python:3.11.7-slim-bookworm "sh -c 'pip install тАж" s3-init 30 minutes ago Exited (0) 5 minutes ago
|
||||
mtusi2026lab5-worker-1 mtusi2026lab5-worker "python worker.py" worker 30 minutes ago Exited (0) Less than a second ago
|
||||
@@ -0,0 +1 @@
|
||||
{"id":"9becbd32-e413-417a-8762-79bcc195e296","chat_id":"5db90ff5-9604-4853-b002-db4144868bcb","owner_id":"fdeb71d4-ee4f-47a6-8968-ffc7740e5242","filename":"sample.png","content_type":"image/png","size_bytes":111817,"sha256":"99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75","status":"queued","variants":[],"error_code":null,"created_at":"2026-11-16T13:43:41.734766Z","updated_at":"2026-11-16T13:43:41.734766Z"}
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"id": "74dc2e91-a901-4067-8993-da9d2cc4ca55",
|
||||
"chat_id": "5db90ff5-9604-4853-b002-db4144868bcb",
|
||||
"sender_id": "fdeb71d4-ee4f-47a6-8968-ffc7740e5242",
|
||||
"text": "",
|
||||
"created_at": "2026-11-16T13:43:42.100558Z",
|
||||
"attachment_ids": [
|
||||
"9becbd32-e413-417a-8762-79bcc195e296"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
|
||||
mtusi2026lab5-api-1-1 mtusi2026lab5-api-1 "uvicorn app:app --hтАж" api-1 30 minutes ago Up Less than a second (health: starting) 8080/tcp
|
||||
mtusi2026lab5-api-2-1 mtusi2026lab5-api-2 "uvicorn app:app --hтАж" api-2 30 minutes ago Up Less than a second (health: starting) 8080/tcp
|
||||
mtusi2026lab5-nginx-1 nginx:1.25.4-alpine "/docker-entrypoint.тАж" nginx 30 minutes ago Up 25 minutes (healthy) 0.0.0.0:8080->8080/tcp, [::]:8080->8080/tcp
|
||||
mtusi2026lab5-postgres-1 postgres:15.5-alpine3.19 "docker-entrypoint.sтАж" postgres 30 minutes ago Up 30 minutes (healthy) 5432/tcp
|
||||
mtusi2026lab5-redis-1 redis:7.2.4-alpine "docker-entrypoint.sтАж" redis 30 minutes ago Up 4 minutes (healthy) 6379/tcp
|
||||
mtusi2026lab5-s3-1 chrislusf/seaweedfs:latest "/entrypoint.sh servтАж" s3 30 minutes ago Up 30 minutes (healthy) 7333/tcp, 8080/tcp, 8333/tcp, 8888/tcp, 9333/tcp, 18080/tcp, 18888/tcp, 19333/tcp
|
||||
@@ -0,0 +1,10 @@
|
||||
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
|
||||
mtusi2026lab5-api-1-1 mtusi2026lab5-api-1 "uvicorn app:app --hтАж" api-1 30 minutes ago Up 10 seconds (healthy) 8080/tcp
|
||||
mtusi2026lab5-api-2-1 mtusi2026lab5-api-2 "uvicorn app:app --hтАж" api-2 30 minutes ago Up 9 seconds (healthy) 8080/tcp
|
||||
mtusi2026lab5-migrate-1 postgres:15.5-alpine3.19 "docker-entrypoint.sтАж" migrate 30 minutes ago Exited (0) 7 seconds ago
|
||||
mtusi2026lab5-nginx-1 nginx:1.25.4-alpine "/docker-entrypoint.тАж" nginx 30 minutes ago Up 26 minutes (healthy) 0.0.0.0:8080->8080/tcp, [::]:8080->8080/tcp
|
||||
mtusi2026lab5-postgres-1 postgres:15.5-alpine3.19 "docker-entrypoint.sтАж" postgres 30 minutes ago Up 30 minutes (healthy) 5432/tcp
|
||||
mtusi2026lab5-redis-1 redis:7.2.4-alpine "docker-entrypoint.sтАж" redis 30 minutes ago Up 4 minutes (healthy) 6379/tcp
|
||||
mtusi2026lab5-s3-1 chrislusf/seaweedfs:latest "/entrypoint.sh servтАж" s3 30 minutes ago Up 30 minutes (healthy) 7333/tcp, 8080/tcp, 8333/tcp, 8888/tcp, 9333/tcp, 18080/tcp, 18888/tcp, 19333/tcp
|
||||
mtusi2026lab5-s3-init-1 python:3.11.7-slim-bookworm "sh -c 'pip install тАж" s3-init 30 minutes ago Exited (0) Less than a second ago
|
||||
mtusi2026lab5-worker-1 mtusi2026lab5-worker "python worker.py" worker 30 minutes ago Up Less than a second (health: starting) 8080/tcp
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"id": "9becbd32-e413-417a-8762-79bcc195e296",
|
||||
"chat_id": "5db90ff5-9604-4853-b002-db4144868bcb",
|
||||
"owner_id": "fdeb71d4-ee4f-47a6-8968-ffc7740e5242",
|
||||
"filename": "sample.png",
|
||||
"content_type": "image/png",
|
||||
"size_bytes": 111817,
|
||||
"sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
|
||||
"status": "ready",
|
||||
"variants": [
|
||||
{
|
||||
"name": "thumbnail",
|
||||
"width": 256,
|
||||
"height": 160,
|
||||
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
|
||||
"size_bytes": 4679,
|
||||
"content_type": "image/jpeg"
|
||||
}
|
||||
],
|
||||
"error_code": null,
|
||||
"created_at": "2026-11-16T13:43:41.734766Z",
|
||||
"updated_at": "2026-11-16T13:43:55.375481Z"
|
||||
}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 4.6 KiB |
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"id": "9becbd32-e413-417a-8762-79bcc195e296",
|
||||
"chat_id": "5db90ff5-9604-4853-b002-db4144868bcb",
|
||||
"owner_id": "fdeb71d4-ee4f-47a6-8968-ffc7740e5242",
|
||||
"filename": "sample.png",
|
||||
"content_type": "image/png",
|
||||
"size_bytes": 111817,
|
||||
"sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
|
||||
"status": "ready",
|
||||
"variants": [
|
||||
{
|
||||
"name": "thumbnail",
|
||||
"width": 256,
|
||||
"height": 160,
|
||||
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
|
||||
"size_bytes": 4679,
|
||||
"content_type": "image/jpeg"
|
||||
}
|
||||
],
|
||||
"error_code": null,
|
||||
"created_at": "2026-11-16T13:43:41.734766Z",
|
||||
"updated_at": "2026-11-16T13:43:55.375481Z"
|
||||
}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 1.8 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 4.6 KiB |
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,74 @@
|
||||
CREATE TABLE IF NOT EXISTS users (
|
||||
id UUID PRIMARY KEY,
|
||||
username VARCHAR(32) UNIQUE NOT NULL,
|
||||
display_name VARCHAR(100) NOT NULL,
|
||||
password_hash VARCHAR(255) NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chats (
|
||||
id UUID PRIMARY KEY,
|
||||
title VARCHAR(100) NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chat_members (
|
||||
chat_id UUID REFERENCES chats(id) ON DELETE CASCADE,
|
||||
user_id UUID REFERENCES users(id) ON DELETE CASCADE,
|
||||
PRIMARY KEY (chat_id, user_id)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS messages (
|
||||
id UUID PRIMARY KEY,
|
||||
chat_id UUID REFERENCES chats(id) ON DELETE CASCADE,
|
||||
sender_id UUID REFERENCES users(id) ON DELETE CASCADE,
|
||||
text VARCHAR(4000) NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL,
|
||||
attachment_ids UUID[] DEFAULT '{}'
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_messages_chat_pagination ON messages(chat_id, created_at, id);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS idempotency_keys (
|
||||
user_id UUID NOT NULL,
|
||||
method VARCHAR(16) NOT NULL,
|
||||
path TEXT NOT NULL,
|
||||
key VARCHAR(128) NOT NULL,
|
||||
request_hash VARCHAR(64) NOT NULL,
|
||||
response_status INT NOT NULL,
|
||||
response_body JSONB NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
PRIMARY KEY (user_id, method, path, key)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS attachments (
|
||||
id UUID PRIMARY KEY,
|
||||
chat_id UUID REFERENCES chats(id) ON DELETE CASCADE,
|
||||
owner_id UUID REFERENCES users(id) ON DELETE CASCADE,
|
||||
filename VARCHAR(255) NOT NULL,
|
||||
content_type VARCHAR(50) NOT NULL,
|
||||
size_bytes BIGINT NOT NULL,
|
||||
sha256 VARCHAR(64) NOT NULL,
|
||||
status VARCHAR(32) NOT NULL DEFAULT 'queued',
|
||||
variants JSONB NOT NULL DEFAULT '[]',
|
||||
error_code VARCHAR(64),
|
||||
pipeline_version INT NOT NULL DEFAULT 1,
|
||||
attempt INT NOT NULL DEFAULT 0,
|
||||
created_at TIMESTAMPTZ NOT NULL,
|
||||
updated_at TIMESTAMPTZ NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS outbox (
|
||||
id UUID PRIMARY KEY,
|
||||
stream VARCHAR(64) NOT NULL,
|
||||
payload JSONB NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
published_at TIMESTAMPTZ
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_attachments_chat ON attachments(chat_id);
|
||||
|
||||
ALTER TABLE messages ADD COLUMN IF NOT EXISTS attachment_ids UUID[] DEFAULT '{}';
|
||||
ALTER TABLE attachments ADD COLUMN IF NOT EXISTS pipeline_version INT NOT NULL DEFAULT 1;
|
||||
ALTER TABLE attachments ADD COLUMN IF NOT EXISTS attempt INT NOT NULL DEFAULT 0;
|
||||
CREATE INDEX IF NOT EXISTS idx_outbox_unpublished ON outbox(created_at) WHERE published_at IS NULL;
|
||||
+28
@@ -0,0 +1,28 @@
|
||||
events { worker_connections 1024; }
|
||||
|
||||
http {
|
||||
upstream messenger_api {
|
||||
server api-1:8080 max_fails=1 fail_timeout=5s;
|
||||
server api-2:8080 max_fails=1 fail_timeout=5s;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 8080;
|
||||
|
||||
location / {
|
||||
proxy_pass http://messenger_api;
|
||||
|
||||
proxy_connect_timeout 2s;
|
||||
proxy_send_timeout 5s;
|
||||
proxy_read_timeout 5s;
|
||||
|
||||
proxy_next_upstream error timeout http_502 http_503 http_504;
|
||||
proxy_next_upstream_tries 2;
|
||||
proxy_next_upstream_timeout 5s;
|
||||
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
fastapi==0.110.0
|
||||
uvicorn==0.28.0
|
||||
pydantic==2.6.2
|
||||
starlette==0.36.3
|
||||
idna==3.6
|
||||
anyio==4.3.0
|
||||
sniffio==1.3.1
|
||||
click==8.1.7
|
||||
h11==0.14.0
|
||||
annotated-types==0.6.0
|
||||
psycopg2-binary==2.9.9
|
||||
bcrypt==4.1.2
|
||||
redis==5.0.1
|
||||
boto3==1.34.0
|
||||
Pillow==10.2.0
|
||||
@@ -0,0 +1,29 @@
|
||||
import os
|
||||
import boto3
|
||||
from botocore.client import Config
|
||||
|
||||
S3_ENDPOINT = os.getenv("S3_ENDPOINT", "http://s3:8333")
|
||||
S3_BUCKET = os.getenv("S3_BUCKET", "messenger-attachments")
|
||||
S3_ROOT_ACCESS_KEY = os.getenv("S3_ROOT_ACCESS_KEY", "minioadmin")
|
||||
S3_ROOT_SECRET_KEY = os.getenv("S3_ROOT_SECRET_KEY", "minioadminpassword")
|
||||
|
||||
s3_client = boto3.client(
|
||||
"s3",
|
||||
endpoint_url=S3_ENDPOINT,
|
||||
aws_access_key_id=S3_ROOT_ACCESS_KEY,
|
||||
aws_secret_access_key=S3_ROOT_SECRET_KEY,
|
||||
config=Config(signature_version="s3v4"),
|
||||
region_name="us-east-1"
|
||||
)
|
||||
|
||||
def init_bucket():
|
||||
print(f"Checking/Creating bucket: {S3_BUCKET}")
|
||||
try:
|
||||
s3_client.head_bucket(Bucket=S3_BUCKET)
|
||||
print("Bucket already exists.")
|
||||
except Exception:
|
||||
s3_client.create_bucket(Bucket=S3_BUCKET)
|
||||
print("Bucket successfully created.")
|
||||
|
||||
if __name__ == "__main__":
|
||||
init_bucket()
|
||||
@@ -0,0 +1,11 @@
|
||||
import os, sys, urllib.request, urllib.error
|
||||
BUCKET = os.getenv("S3_BUCKET", "messenger-attachments")
|
||||
KEY = os.getenv("S3_KEY")
|
||||
URL = f"http://s3:8333/{BUCKET}/{KEY}"
|
||||
try:
|
||||
urllib.request.urlopen(URL)
|
||||
print("FAIL: object accessible without auth"); sys.exit(1)
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code in (401, 403):
|
||||
print(f"PASS: anonymous GET -> {e.code}"); sys.exit(0)
|
||||
print(f"FAIL: unexpected {e.code}"); sys.exit(1)
|
||||
@@ -8,8 +8,8 @@ import re
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
def main():
|
||||
lab = json.loads((ROOT/'lab.json').read_text())['number']
|
||||
spec = json.loads((ROOT/'contracts/openapi.json').read_text())
|
||||
lab = json.loads((ROOT/'lab.json').read_text(encoding='utf-8'))['number']
|
||||
spec = json.loads((ROOT/'contracts/openapi.json').read_text(encoding='utf-8'))
|
||||
assert spec['openapi']=='3.1.0'
|
||||
assert spec['info']['version']=='1.%d.0'%lab
|
||||
operation_ids = set()
|
||||
@@ -41,11 +41,11 @@ def main():
|
||||
assert ('/api/v1/auth/sessions' in spec['paths']) == (lab>=2)
|
||||
assert ('/api/v1/attachments/{attachment_id}' in spec['paths']) == (lab>=4)
|
||||
for file in list((ROOT/'tests').glob('*.py'))+list((ROOT/'scripts').glob('*.py')):
|
||||
ast.parse(file.read_text(),filename=str(file))
|
||||
ast.parse(file.read_text(encoding='utf-8'),filename=str(file))
|
||||
for file in ROOT.rglob('*.json'):
|
||||
if any(part in {'.git','.venv','node_modules','vendor','output'} for part in file.parts):
|
||||
continue
|
||||
json.loads(file.read_text())
|
||||
json.loads(file.read_text(encoding='utf-8-sig'))
|
||||
for required in ('README.md','COURSE.md','CONTRIBUTING.md','REPORT.md','contracts/README.md','tests/smoke.py','tests/README.md'):
|
||||
assert (ROOT/required).is_file(), 'missing '+required
|
||||
print('PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.')
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
import os
|
||||
import sys
|
||||
import boto3
|
||||
from botocore.client import Config
|
||||
import psycopg2
|
||||
import psycopg2.extras
|
||||
from datetime import datetime, timezone, timedelta
|
||||
|
||||
DATABASE_URL = os.getenv("DATABASE_URL", "postgresql://postgres:postgres@postgres:5432/messenger")
|
||||
S3_ENDPOINT = os.getenv("S3_ENDPOINT", "http://s3:8333")
|
||||
S3_BUCKET = os.getenv("S3_BUCKET", "messenger-attachments")
|
||||
S3_ACCESS_KEY = os.getenv("S3_ACCESS_KEY_ID", "appuser")
|
||||
S3_SECRET_KEY = os.getenv("S3_SECRET_ACCESS_KEY", "appsecretpassword")
|
||||
|
||||
s3_client = boto3.client(
|
||||
"s3",
|
||||
endpoint_url=S3_ENDPOINT,
|
||||
aws_access_key_id=S3_ACCESS_KEY,
|
||||
aws_secret_access_key=S3_SECRET_KEY,
|
||||
config=Config(signature_version="s3v4"),
|
||||
region_name="us-east-1"
|
||||
)
|
||||
|
||||
def run_reconciliation(dry_run: bool = False):
|
||||
print(f"Starting reconciliation (dry_run={dry_run})...")
|
||||
conn = psycopg2.connect(DATABASE_URL, cursor_factory=psycopg2.extras.RealDictCursor)
|
||||
|
||||
threshold = datetime.now(timezone.utc) - timedelta(hours=1)
|
||||
|
||||
with conn.cursor() as cur:
|
||||
cur.execute("""
|
||||
SELECT id, chat_id, filename, created_at FROM attachments
|
||||
WHERE status = 'pending' AND created_at < %s
|
||||
""", (threshold,))
|
||||
stale_pending = cur.fetchall()
|
||||
|
||||
for row in stale_pending:
|
||||
att_id = str(row['id'])
|
||||
chat_id = str(row['chat_id'])
|
||||
filename = row['filename']
|
||||
s3_key = f"attachments/{chat_id}/{att_id}/{filename}"
|
||||
print(f"[DB Stale Pending] Found stale attachment {att_id}, marking as failed and cleaning S3.")
|
||||
|
||||
if not dry_run:
|
||||
try:
|
||||
s3_client.delete_object(Bucket=S3_BUCKET, Key=s3_key)
|
||||
except Exception as e:
|
||||
print(f"Failed to delete S3 key {s3_key}: {e}")
|
||||
with conn.cursor() as cur:
|
||||
cur.execute("UPDATE attachments SET status = 'failed', error_code = 'timeout_stale' WHERE id = %s", (att_id,))
|
||||
conn.commit()
|
||||
|
||||
with conn.cursor() as cur:
|
||||
cur.execute("SELECT id, chat_id, filename FROM attachments")
|
||||
all_db_attachments = cur.fetchall()
|
||||
|
||||
valid_s3_keys = set()
|
||||
for row in all_db_attachments:
|
||||
valid_s3_keys.add(f"attachments/{str(row['chat_id'])}/{str(row['id'])}/{row['filename']}")
|
||||
|
||||
paginator = s3_client.get_paginator('list_objects_v2')
|
||||
for page in paginator.paginate(Bucket=S3_BUCKET, Prefix="attachments/"):
|
||||
for obj in page.get('Contents', []):
|
||||
key = obj['Key']
|
||||
last_modified = obj['LastModified']
|
||||
|
||||
if key not in valid_s3_keys and last_modified < threshold:
|
||||
print(f"[S3 Orphan] Found orphan S3 object {key} (Modified: {last_modified})")
|
||||
if not dry_run:
|
||||
try:
|
||||
s3_client.delete_object(Bucket=S3_BUCKET, Key=key)
|
||||
print(f"Deleted orphan object {key} from S3.")
|
||||
except Exception as e:
|
||||
print(f"Failed to delete orphan {key}: {e}")
|
||||
|
||||
conn.close()
|
||||
print("Reconciliation completed successfully.")
|
||||
|
||||
if __name__ == "__main__":
|
||||
dry_run_mode = "--dry-run" in sys.argv
|
||||
run_reconciliation(dry_run=dry_run_mode)
|
||||
@@ -0,0 +1,3 @@
|
||||
@echo off
|
||||
docker compose restart
|
||||
exit /b %ERRORLEVEL%
|
||||
@@ -0,0 +1,3 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
docker compose restart
|
||||
@@ -0,0 +1,35 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
|
||||
wait_ready() {
|
||||
svc=$1
|
||||
timeout=60
|
||||
echo "Waiting for $svc readiness (up to ${timeout}s)..."
|
||||
while [ "$timeout" -gt 0 ]; do
|
||||
if docker compose exec -T "$svc" python -c \
|
||||
"import urllib.request; urllib.request.urlopen('http://localhost:8080/health/ready')" 2>/dev/null; then
|
||||
echo "$svc is ready!"
|
||||
return 0
|
||||
fi
|
||||
sleep 1
|
||||
timeout=$((timeout - 1))
|
||||
done
|
||||
echo "$svc failed to become ready" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
echo "Starting Rolling Update"
|
||||
|
||||
echo "1. Rebuilding and replacing api-1..."
|
||||
docker compose up -d --no-deps --build --force-recreate api-1
|
||||
wait_ready api-1
|
||||
echo "Reloading nginx to pick up new api-1 IP..."
|
||||
docker compose exec -T nginx nginx -s reload
|
||||
|
||||
echo "2. Rebuilding and replacing api-2..."
|
||||
docker compose up -d --no-deps --build --force-recreate api-2
|
||||
wait_ready api-2
|
||||
echo "Reloading nginx to pick up new api-2 IP..."
|
||||
docker compose exec -T nginx nginx -s reload
|
||||
|
||||
echo "Rolling Update Completed Successfully"
|
||||
@@ -0,0 +1,12 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
: "${TARGET_INSTANCE_ID:?test provides target API instance alias}"
|
||||
|
||||
if [ "$TARGET_INSTANCE_ID" = "api-replica-1" ]; then
|
||||
docker compose stop -t 3 api-1
|
||||
elif [ "$TARGET_INSTANCE_ID" = "api-replica-2" ]; then
|
||||
docker compose stop -t 3 api-2
|
||||
else
|
||||
echo "Unknown target instance: $TARGET_INSTANCE_ID" >&2
|
||||
exit 1
|
||||
fi
|
||||
@@ -0,0 +1,54 @@
|
||||
import base64, concurrent.futures, json, pathlib, time, urllib.request, uuid
|
||||
|
||||
BASE = "http://localhost:8080"
|
||||
|
||||
def _req(method, path, body=None, headers=None, token=None):
|
||||
h = {"Accept": "application/json"}
|
||||
if token: h["Authorization"] = f"Bearer {token}"
|
||||
h.update(headers or {})
|
||||
req = urllib.request.Request(BASE + path, data=body, headers=h, method=method)
|
||||
with urllib.request.urlopen(req) as r:
|
||||
return r.status, r.read()
|
||||
|
||||
def register_and_login():
|
||||
u = f"u_{uuid.uuid4().hex[:10]}"
|
||||
p = "T3st-" + uuid.uuid4().hex
|
||||
_req("POST", "/api/v1/users",
|
||||
json.dumps({"username": u, "display_name": "Load", "password": p}).encode(),
|
||||
{"Content-Type": "application/json"})
|
||||
basic = base64.b64encode(f"{u}:{p}".encode()).decode()
|
||||
_, body = _req("POST", "/api/v1/auth/sessions",
|
||||
headers={"Authorization": f"Basic {basic}"})
|
||||
return json.loads(body)["token"]
|
||||
|
||||
def make_chat(token):
|
||||
_, me = _req("GET", "/api/v1/users/me", token=token)
|
||||
uid = json.loads(me)["id"]
|
||||
_, chat = _req("POST", "/api/v1/chats",
|
||||
json.dumps({"title": "load", "member_ids": [uid]}).encode(),
|
||||
{"Content-Type": "application/json", "Idempotency-Key": uuid.uuid4().hex},
|
||||
token=token)
|
||||
return json.loads(chat)["id"]
|
||||
|
||||
def main():
|
||||
token = register_and_login()
|
||||
chat = make_chat(token)
|
||||
png = pathlib.Path("tests/fixtures/sample.png").read_bytes()
|
||||
|
||||
def one(_):
|
||||
try:
|
||||
status, _ = _req("POST",
|
||||
f"/api/v1/chats/{chat}/attachments?filename=f.png",
|
||||
png, {"Content-Type": "image/png"}, token)
|
||||
return status
|
||||
except Exception as e:
|
||||
return repr(e)
|
||||
|
||||
t0 = time.monotonic()
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as ex:
|
||||
results = list(ex.map(one, range(10)))
|
||||
print(json.dumps({"results": results,
|
||||
"elapsed_s": round(time.monotonic() - t0, 3)}))
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+1
-1
@@ -14,7 +14,7 @@ import uuid
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
LAB = json.loads((ROOT / 'lab.json').read_text())['number']
|
||||
SPEC = json.loads((ROOT / 'contracts/openapi.json').read_text())
|
||||
SPEC = json.loads((ROOT / 'contracts/openapi.json').read_text(encoding='utf-8'))
|
||||
|
||||
class Failure(AssertionError):
|
||||
pass
|
||||
|
||||
@@ -0,0 +1,109 @@
|
||||
import concurrent.futures
|
||||
import urllib.request
|
||||
import urllib.error
|
||||
import json
|
||||
import unittest
|
||||
from client import Client
|
||||
|
||||
class TestMessengerLab4(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.client = Client()
|
||||
self.alice = self.client.user("alice")
|
||||
self.bob = self.client.user("bob")
|
||||
self.eve = self.client.user("eve")
|
||||
|
||||
res = self.client.call(
|
||||
'POST', '/api/v1/chats', 201,
|
||||
actor=self.alice,
|
||||
body={"title": "Test Chat", "member_ids": [self.alice["id"], self.bob["id"]]},
|
||||
schema='Chat'
|
||||
)
|
||||
self.chat = res['json']
|
||||
|
||||
def test_unauthorized_access_to_chat(self):
|
||||
self.client.call('GET', f"/api/v1/chats/{self.chat['id']}", 404, actor=self.eve)
|
||||
|
||||
def test_empty_message(self):
|
||||
self.client.call('POST', f"/api/v1/chats/{self.chat['id']}/messages", 400, actor=self.alice, body={"text": ""})
|
||||
|
||||
def test_malformed_json(self):
|
||||
url = self.client.base + f"/api/v1/chats/{self.chat['id']}/messages"
|
||||
req = urllib.request.Request(
|
||||
url,
|
||||
data=b"{broken json content without quotes",
|
||||
headers={
|
||||
"Content-Type": "application/json",
|
||||
"X-User-Id": self.alice["id"]
|
||||
},
|
||||
method="POST"
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req) as resp:
|
||||
self.fail("Expected HTTP 400 for malformed JSON")
|
||||
except urllib.error.HTTPError as e:
|
||||
self.assertEqual(e.code, 400)
|
||||
body = json.loads(e.read().decode())
|
||||
self.assertEqual(body["error"]["code"], "invalid_request")
|
||||
|
||||
def test_duplicate_username(self):
|
||||
self.client.call('POST', '/api/v1/users', 409, body=self.alice["registration"])
|
||||
|
||||
def test_pagination_multiple_pages(self):
|
||||
import time
|
||||
msg_texts = ["msg-page-1", "msg-page-2", "msg-page-3"]
|
||||
for txt in msg_texts:
|
||||
self.client.message(self.alice, self.chat, txt)
|
||||
time.sleep(0.01)
|
||||
|
||||
path = f"/api/v1/chats/{self.chat['id']}/messages"
|
||||
collected_items = []
|
||||
cursor = None
|
||||
|
||||
for _ in range(5):
|
||||
url_suffix = "?limit=1"
|
||||
if cursor:
|
||||
url_suffix += f"&cursor={cursor}"
|
||||
|
||||
page_res = self.client.call('GET', path + url_suffix, actor=self.alice, schema='MessagePage')
|
||||
page = page_res['json']
|
||||
|
||||
for item in page['items']:
|
||||
self.assertNotIn(item['id'], [x['id'] for x in collected_items])
|
||||
collected_items.append(item)
|
||||
|
||||
cursor = page.get('next_keyword') or page.get('next_cursor')
|
||||
if not cursor:
|
||||
break
|
||||
|
||||
self.assertEqual(len(collected_items), 3)
|
||||
self.assertEqual(sorted([x['text'] for x in collected_items]), sorted(msg_texts))
|
||||
|
||||
def test_concurrent_messages_strict(self):
|
||||
def worker_send(worker_idx):
|
||||
local_client = Client()
|
||||
ids = []
|
||||
for i in range(10):
|
||||
res = local_client.message(self.alice, self.chat, f"w-{worker_idx}-msg-{i}")
|
||||
ids.append(res['json']['id'])
|
||||
return ids
|
||||
|
||||
all_ids = []
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
|
||||
futures = [executor.submit(worker_send, w) for w in range(10)]
|
||||
for f in concurrent.futures.as_completed(futures):
|
||||
all_ids.extend(f.result())
|
||||
|
||||
self.assertEqual(len(all_ids), 100)
|
||||
self.assertEqual(len(set(all_ids)), 100)
|
||||
|
||||
page = self.client.call(
|
||||
'GET', f"/api/v1/chats/{self.chat['id']}/messages?limit=100",
|
||||
actor=self.alice,
|
||||
schema='MessagePage'
|
||||
)
|
||||
server_ids = [m['id'] for m in page['json']['items']]
|
||||
self.assertEqual(len(server_ids), 100)
|
||||
self.assertEqual(len(set(server_ids)), 100)
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -0,0 +1,139 @@
|
||||
import urllib.request
|
||||
import urllib.error
|
||||
import json
|
||||
import unittest
|
||||
from client import Client
|
||||
|
||||
class TestMessengerLevel4Complete(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.client = Client()
|
||||
self.alice = self.client.user("alice")
|
||||
self.bob = self.client.user("bob")
|
||||
self.eve = self.client.user("eve")
|
||||
|
||||
res_chat1 = self.client.call(
|
||||
'POST', '/api/v1/chats', 201,
|
||||
actor=self.alice,
|
||||
body={"title": "Team Chat", "member_ids": [self.alice["id"], self.bob["id"]]},
|
||||
schema='Chat'
|
||||
)
|
||||
self.chat1 = res_chat1['json']
|
||||
|
||||
res_chat2 = self.client.call(
|
||||
'POST', '/api/v1/chats', 201,
|
||||
actor=self.eve,
|
||||
body={"title": "Private Chat", "member_ids": [self.eve["id"], self.bob["id"]]},
|
||||
schema='Chat'
|
||||
)
|
||||
self.chat2 = res_chat2['json']
|
||||
|
||||
def test_unsupported_media_type(self):
|
||||
"""Проверка 415: передача текстового файла под видом картинки."""
|
||||
fake_bytes = b"Just a simple plain text file content."
|
||||
url = self.client.base + f"/api/v1/chats/{self.chat1['id']}/attachments?filename=notes.txt"
|
||||
req = urllib.request.Request(
|
||||
url, data=fake_bytes,
|
||||
headers={"Authorization": f"Bearer {self.alice['token']}", "Content-Type": "image/png"},
|
||||
method="POST"
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req) as resp:
|
||||
self.fail("Expected HTTP 415")
|
||||
except urllib.error.HTTPError as e:
|
||||
self.assertEqual(e.code, 415)
|
||||
|
||||
def test_payload_too_large(self):
|
||||
"""Проверка 413: файл размером больше 10 MiB."""
|
||||
large_data = b'\x89PNG\r\n\x1a\n' + b'\x00' * (10 * 1024 * 1024 + 100)
|
||||
url = self.client.base + f"/api/v1/chats/{self.chat1['id']}/attachments?filename=huge.png"
|
||||
req = urllib.request.Request(
|
||||
url, data=large_data,
|
||||
headers={"Authorization": f"Bearer {self.alice['token']}", "Content-Type": "image/png"},
|
||||
method="POST"
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req) as resp:
|
||||
self.fail("Expected HTTP 413")
|
||||
except urllib.error.HTTPError as e:
|
||||
self.assertEqual(e.code, 413)
|
||||
|
||||
def test_corrupted_image_signature(self):
|
||||
"""Проверка 415: битая структура / неверные размеры PNG."""
|
||||
corrupted_png = b'\x89PNG\r\n\x1a\n' + b'\xFF' * 50
|
||||
url = self.client.base + f"/api/v1/chats/{self.chat1['id']}/attachments?filename=corrupt.png"
|
||||
req = urllib.request.Request(
|
||||
url, data=corrupted_png,
|
||||
headers={"Authorization": f"Bearer {self.alice['token']}", "Content-Type": "image/png"},
|
||||
method="POST"
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req) as resp:
|
||||
self.fail("Expected HTTP 415")
|
||||
except urllib.error.HTTPError as e:
|
||||
self.assertEqual(e.code, 415)
|
||||
|
||||
def test_invalid_filename_characters(self):
|
||||
"""Проверка 400: недопустимые символы (path traversal) в имени файла."""
|
||||
valid_png = (
|
||||
b'\x89PNG\r\n\x1a\n'
|
||||
b'\x00\x00\x00\rIHDR'
|
||||
b'\x00\x00\x00\n\x00\x00\x00\n'
|
||||
b'\x08\x02\x00\x00\x00'
|
||||
b'\x02p\xdd\xb5'
|
||||
b'\x00\x00\x00\x0fcIDATx\x9cc`\x00\x00\x00\x02\x00\x01H\xaf\xa4q'
|
||||
b'\x00\x00\x00\x00IEND\xaeB`\x82'
|
||||
)
|
||||
url = self.client.base + f"/api/v1/chats/{self.chat1['id']}/attachments?filename=../../etc/passwd"
|
||||
req = urllib.request.Request(
|
||||
url, data=valid_png,
|
||||
headers={"Authorization": f"Bearer {self.alice['token']}", "Content-Type": "image/png"},
|
||||
method="POST"
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req) as resp:
|
||||
self.fail("Expected HTTP 400")
|
||||
except urllib.error.HTTPError as e:
|
||||
self.assertEqual(e.code, 400)
|
||||
|
||||
def test_nonexistent_attachment_id(self):
|
||||
"""Проверка 404: передача несуществующего ID вложения при отправке сообщения."""
|
||||
fake_uuid = "12345678-1234-4123-8123-123456789abc"
|
||||
res = self.client.call(
|
||||
'POST', f"/api/v1/chats/{self.chat1['id']}/messages", 404,
|
||||
actor=self.alice,
|
||||
body={"text": "Check this out", "attachment_ids": [fake_uuid]},
|
||||
headers={"Idempotency-Key": "test-key-lvl4-1"}
|
||||
)
|
||||
self.assertEqual(res['status'], 404)
|
||||
|
||||
def test_cross_chat_attachment_isolation(self):
|
||||
"""Проверка 404: попытка прикрепить вложение из чужого чата."""
|
||||
valid_png = (
|
||||
b'\x89PNG\r\n\x1a\n'
|
||||
b'\x00\x00\x00\rIHDR'
|
||||
b'\x00\x00\x00\n\x00\x00\x00\n'
|
||||
b'\x08\x02\x00\x00\x00'
|
||||
b'\x02p\xdd\xb5'
|
||||
b'\x00\x00\x00\x0fcID1Tx\x9cc`\x00\x00\x00\x02\x00\x01H\xaf\xa4q'
|
||||
b'\x00\x00\x00\x00IEND\xaeB`\x82'
|
||||
)
|
||||
url = self.client.base + f"/api/v1/chats/{self.chat2['id']}/attachments?filename=secret.png"
|
||||
req = urllib.request.Request(
|
||||
url, data=valid_png,
|
||||
headers={"Authorization": f"Bearer {self.eve['token']}", "Content-Type": "image/png"},
|
||||
method="POST"
|
||||
)
|
||||
with urllib.request.urlopen(req) as resp:
|
||||
data = json.loads(resp.read().decode())
|
||||
eve_att_id = data["id"]
|
||||
|
||||
res = self.client.call(
|
||||
'POST', f"/api/v1/chats/{self.chat1['id']}/messages", 404,
|
||||
actor=self.alice,
|
||||
body={"text": "Stealing file", "attachment_ids": [eve_att_id]},
|
||||
headers={"Idempotency-Key": "test-key-lvl4-2"}
|
||||
)
|
||||
self.assertEqual(res['status'], 404)
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
+1
-1
@@ -55,7 +55,7 @@ def main():
|
||||
check(len(observed)>=2,'less than two API instances observed before failure; verify LB/session affinity')
|
||||
print('Executing explicit '+args.mode+' hook; hook output suppressed to avoid leaking secrets.')
|
||||
# No shell interpolation and no Docker commands in this test.
|
||||
proc = subprocess.Popen([str(script)],env=env,stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL)
|
||||
proc = subprocess.Popen(['sh', str(script)], env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
started = time.monotonic()
|
||||
successes,failures,after_instances = [],0,[]
|
||||
try:
|
||||
|
||||
@@ -0,0 +1,405 @@
|
||||
import os
|
||||
import io
|
||||
import json
|
||||
import time
|
||||
import signal
|
||||
import random
|
||||
import hashlib
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
|
||||
import psycopg2
|
||||
import psycopg2.extras
|
||||
import redis
|
||||
import boto3
|
||||
import botocore.exceptions
|
||||
from botocore.client import Config
|
||||
from PIL import Image, ImageOps, UnidentifiedImageError
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
|
||||
logger = logging.getLogger("attachment-worker")
|
||||
|
||||
DATABASE_URL = os.getenv("DATABASE_URL", "postgresql://postgres:postgres@postgres:5432/messenger")
|
||||
REDIS_URL = os.getenv("REDIS_URL", "redis://redis:6379/0")
|
||||
S3_ENDPOINT = os.getenv("S3_ENDPOINT", "http://s3:8333")
|
||||
S3_BUCKET = os.getenv("S3_BUCKET", "messenger-attachments")
|
||||
S3_ACCESS_KEY = os.getenv("S3_ACCESS_KEY_ID", "appuser")
|
||||
S3_SECRET_KEY = os.getenv("S3_SECRET_ACCESS_KEY", "appsecretpassword")
|
||||
|
||||
STREAM_NAME = os.getenv("BROKER_STREAM", "attachment_jobs")
|
||||
DLQ_STREAM_NAME = f"{STREAM_NAME}_dlq"
|
||||
GROUP_NAME = os.getenv("BROKER_GROUP", "attachment_workers")
|
||||
CONSUMER_NAME = os.getenv("HOSTNAME", "worker-1")
|
||||
JOB_MAX_ATTEMPTS = int(os.getenv("JOB_MAX_ATTEMPTS", "5"))
|
||||
MAX_IMAGE_PIXELS = int(os.getenv("MAX_IMAGE_PIXELS", "40000000"))
|
||||
JOB_TIMEOUT_SECONDS = int(os.getenv("JOB_TIMEOUT_SECONDS", "30"))
|
||||
WORKER_CONCURRENCY = int(os.getenv("WORKER_CONCURRENCY", "4"))
|
||||
|
||||
CLAIM_MIN_IDLE_MS = (JOB_TIMEOUT_SECONDS + 20) * 1000
|
||||
|
||||
Image.MAX_IMAGE_PIXELS = MAX_IMAGE_PIXELS
|
||||
|
||||
redis_client = redis.Redis.from_url(REDIS_URL, decode_responses=True)
|
||||
s3_client = boto3.client(
|
||||
"s3", endpoint_url=S3_ENDPOINT,
|
||||
aws_access_key_id=S3_ACCESS_KEY, aws_secret_access_key=S3_SECRET_KEY,
|
||||
config=Config(signature_version="s3v4"), region_name="us-east-1"
|
||||
)
|
||||
|
||||
running = True
|
||||
|
||||
def handle_sigterm(signum, frame):
|
||||
global running
|
||||
logger.info("Received shutdown signal, finishing in-flight jobs...")
|
||||
running = False
|
||||
|
||||
signal.signal(signal.SIGTERM, handle_sigterm)
|
||||
signal.signal(signal.SIGINT, handle_sigterm)
|
||||
|
||||
|
||||
def get_db_connection():
|
||||
return psycopg2.connect(DATABASE_URL, cursor_factory=psycopg2.extras.RealDictCursor)
|
||||
|
||||
|
||||
def ensure_consumer_groups():
|
||||
while running:
|
||||
try:
|
||||
redis_client.xgroup_create(STREAM_NAME, GROUP_NAME, id="0", mkstream=True)
|
||||
redis_client.xgroup_create(DLQ_STREAM_NAME, GROUP_NAME, id="0", mkstream=True)
|
||||
logger.info("Consumer groups ensured on '%s' and '%s'", STREAM_NAME, DLQ_STREAM_NAME)
|
||||
return
|
||||
except redis.exceptions.ResponseError as e:
|
||||
if "BUSYGROUP" in str(e):
|
||||
return
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.warning("Waiting for Redis: %s", type(e).__name__)
|
||||
time.sleep(1)
|
||||
|
||||
|
||||
def prepare_base_image(raw_bytes):
|
||||
img = Image.open(io.BytesIO(raw_bytes))
|
||||
img = ImageOps.exif_transpose(img)
|
||||
if img.mode in ("RGBA", "LA") or (img.mode == "P" and "transparency" in img.info):
|
||||
rgba = img.convert("RGBA")
|
||||
bg = Image.new("RGB", rgba.size, (255, 255, 255))
|
||||
bg.paste(rgba, mask=rgba.split()[3])
|
||||
img = bg
|
||||
elif img.mode != "RGB":
|
||||
img = img.convert("RGB")
|
||||
return img
|
||||
|
||||
|
||||
def generate_grade4_variants(raw_bytes):
|
||||
base_img = prepare_base_image(raw_bytes)
|
||||
orig_w, orig_h = base_img.size
|
||||
meta = []
|
||||
blobs = {}
|
||||
|
||||
thumb = base_img.copy()
|
||||
if orig_w <= 256 and orig_h <= 256:
|
||||
t_w, t_h = orig_w, orig_h
|
||||
else:
|
||||
s = min(256.0 / orig_w, 256.0 / orig_h)
|
||||
t_w, t_h = max(1, int(orig_w * s)), max(1, int(orig_h * s))
|
||||
if (t_w, t_h) != (orig_w, orig_h):
|
||||
thumb = thumb.resize((t_w, t_h), Image.Resampling.LANCZOS)
|
||||
buf = io.BytesIO(); thumb.save(buf, format="JPEG", quality=85, optimize=True)
|
||||
tb = buf.getvalue(); blobs["thumbnail"] = tb
|
||||
meta.append({"name": "thumbnail", "content_type": "image/jpeg",
|
||||
"size_bytes": len(tb), "width": t_w, "height": t_h,
|
||||
"sha256": hashlib.sha256(tb).hexdigest()})
|
||||
|
||||
opt = base_img.copy()
|
||||
if orig_w <= 1280 and orig_h <= 1280:
|
||||
o_w, o_h = orig_w, orig_h
|
||||
else:
|
||||
s = min(1280.0 / orig_w, 1280.0 / orig_h)
|
||||
o_w, o_h = max(1, int(orig_w * s)), max(1, int(orig_h * s))
|
||||
if (o_w, o_h) != (orig_w, orig_h):
|
||||
opt = opt.resize((o_w, o_h), Image.Resampling.LANCZOS)
|
||||
buf = io.BytesIO(); opt.save(buf, format="WEBP", quality=85)
|
||||
ob = buf.getvalue(); blobs["optimized"] = ob
|
||||
meta.append({"name": "optimized", "content_type": "image/webp",
|
||||
"size_bytes": len(ob), "width": o_w, "height": o_h,
|
||||
"sha256": hashlib.sha256(ob).hexdigest()})
|
||||
|
||||
c = min(orig_w, orig_h)
|
||||
left, top = (orig_w - c) // 2, (orig_h - c) // 2
|
||||
crop = base_img.crop((left, top, left + c, top + c))
|
||||
if c != 128:
|
||||
crop = crop.resize((128, 128), Image.Resampling.LANCZOS)
|
||||
buf = io.BytesIO(); crop.save(buf, format="JPEG", quality=85, optimize=True)
|
||||
cb = buf.getvalue(); blobs["crop"] = cb
|
||||
meta.append({"name": "crop", "content_type": "image/jpeg",
|
||||
"size_bytes": len(cb), "width": 128, "height": 128,
|
||||
"sha256": hashlib.sha256(cb).hexdigest()})
|
||||
|
||||
return meta, blobs
|
||||
|
||||
|
||||
def mark_failed(attachment_id, error_code):
|
||||
now = datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
|
||||
conn = get_db_connection()
|
||||
try:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute("UPDATE attachments SET status='failed', error_code=%s, updated_at=%s "
|
||||
"WHERE id=%s AND status != 'ready'",
|
||||
(error_code, now, attachment_id))
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
PERMANENT_ERRORS = (
|
||||
UnidentifiedImageError, Image.DecompressionBombError, ValueError,
|
||||
psycopg2.errors.DataError,
|
||||
)
|
||||
TRANSIENT_ERRORS = (
|
||||
TimeoutError,
|
||||
botocore.exceptions.EndpointConnectionError,
|
||||
botocore.exceptions.ReadTimeoutError,
|
||||
psycopg2.OperationalError,
|
||||
redis.exceptions.ConnectionError,
|
||||
)
|
||||
|
||||
|
||||
def process_job(message_id, fields):
|
||||
try:
|
||||
job = json.loads(fields.get("payload", "{}"))
|
||||
except Exception:
|
||||
logger.error("Malformed job %s, ack to drop", message_id)
|
||||
redis_client.xack(STREAM_NAME, GROUP_NAME, message_id)
|
||||
return
|
||||
|
||||
attachment_id = job.get("attachment_id")
|
||||
job_id = job.get("job_id")
|
||||
pipeline_version = int(job.get("pipeline_version", 1))
|
||||
attempt = int(job.get("attempt", 1))
|
||||
|
||||
if not attachment_id:
|
||||
redis_client.xack(STREAM_NAME, GROUP_NAME, message_id)
|
||||
return
|
||||
|
||||
conn = get_db_connection()
|
||||
try:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute("SELECT * FROM attachments WHERE id=%s FOR UPDATE", (attachment_id,))
|
||||
att = cur.fetchone()
|
||||
if not att:
|
||||
conn.commit()
|
||||
redis_client.xack(STREAM_NAME, GROUP_NAME, message_id)
|
||||
return
|
||||
if att["status"] == "ready" or att["pipeline_version"] > pipeline_version:
|
||||
conn.commit()
|
||||
redis_client.xack(STREAM_NAME, GROUP_NAME, message_id)
|
||||
return
|
||||
if att["status"] == "failed":
|
||||
conn.commit()
|
||||
redis_client.xack(STREAM_NAME, GROUP_NAME, message_id)
|
||||
return
|
||||
|
||||
if att["status"] == "processing":
|
||||
updated = att["updated_at"]
|
||||
if isinstance(updated, str):
|
||||
updated = datetime.fromisoformat(updated.replace("Z", "+00:00"))
|
||||
age_seconds = (datetime.now(timezone.utc) - updated).total_seconds()
|
||||
if age_seconds < (JOB_TIMEOUT_SECONDS + 10):
|
||||
conn.commit()
|
||||
logger.info("Attachment %s already processing (age=%.1fs), ack duplicate",
|
||||
attachment_id, age_seconds)
|
||||
redis_client.xack(STREAM_NAME, GROUP_NAME, message_id)
|
||||
return
|
||||
|
||||
now = datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
|
||||
cur.execute(
|
||||
"UPDATE attachments SET status='processing', attempt=%s, updated_at=%s WHERE id=%s",
|
||||
(attempt, now, attachment_id)
|
||||
)
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
chat_id = str(att["chat_id"])
|
||||
staging_key = f"staging/{chat_id}/{attachment_id}/original"
|
||||
final_original_key = f"attachments/{chat_id}/{attachment_id}/original"
|
||||
|
||||
start_time = time.monotonic()
|
||||
|
||||
def check_timeout():
|
||||
if time.monotonic() - start_time > JOB_TIMEOUT_SECONDS:
|
||||
raise TimeoutError("Job exceeded time limit")
|
||||
|
||||
try:
|
||||
try:
|
||||
obj = s3_client.get_object(Bucket=S3_BUCKET, Key=staging_key)
|
||||
except Exception:
|
||||
obj = s3_client.get_object(Bucket=S3_BUCKET, Key=final_original_key)
|
||||
raw_bytes = obj["Body"].read()
|
||||
check_timeout()
|
||||
|
||||
if hashlib.sha256(raw_bytes).hexdigest() != att["sha256"] or len(raw_bytes) != att["size_bytes"]:
|
||||
raise ValueError("Staging checksum mismatch")
|
||||
|
||||
variants_meta, variants_bytes = generate_grade4_variants(raw_bytes)
|
||||
check_timeout()
|
||||
|
||||
s3_client.put_object(
|
||||
Bucket=S3_BUCKET, Key=final_original_key,
|
||||
Body=raw_bytes, ContentType=att["content_type"]
|
||||
)
|
||||
for var_name, var_bytes in variants_bytes.items():
|
||||
ct = "image/webp" if var_name == "optimized" else "image/jpeg"
|
||||
s3_client.put_object(
|
||||
Bucket=S3_BUCKET,
|
||||
Key=f"attachments/{chat_id}/{attachment_id}/{var_name}",
|
||||
Body=var_bytes, ContentType=ct
|
||||
)
|
||||
check_timeout()
|
||||
|
||||
ready_at = datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
|
||||
conn = get_db_connection()
|
||||
try:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute("""
|
||||
UPDATE attachments SET status='ready', variants=%s::jsonb,
|
||||
error_code=NULL, updated_at=%s
|
||||
WHERE id=%s AND pipeline_version <= %s
|
||||
""", (json.dumps(variants_meta), ready_at, attachment_id, pipeline_version))
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
redis_client.xack(STREAM_NAME, GROUP_NAME, message_id)
|
||||
logger.info("Job %s completed: attachment %s ready", job_id, attachment_id)
|
||||
|
||||
try:
|
||||
s3_client.delete_object(Bucket=S3_BUCKET, Key=staging_key)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
except (UnidentifiedImageError, Image.DecompressionBombError) as e:
|
||||
logger.error("Permanent image error for %s: %s", attachment_id, type(e).__name__)
|
||||
mark_failed(attachment_id, "invalid_image")
|
||||
redis_client.xack(STREAM_NAME, GROUP_NAME, message_id)
|
||||
|
||||
except ValueError as e:
|
||||
logger.error("Checksum error for %s: %s", attachment_id, e)
|
||||
mark_failed(attachment_id, "invalid_checksum")
|
||||
redis_client.xack(STREAM_NAME, GROUP_NAME, message_id)
|
||||
|
||||
except psycopg2.errors.DataError as e:
|
||||
logger.error("DB data error for %s: %s", attachment_id, type(e).__name__)
|
||||
mark_failed(attachment_id, "data_error")
|
||||
redis_client.xack(STREAM_NAME, GROUP_NAME, message_id)
|
||||
|
||||
except TRANSIENT_ERRORS as e:
|
||||
logger.warning(
|
||||
"Transient error for %s (attempt %d/%d): %s",
|
||||
attachment_id, attempt, JOB_MAX_ATTEMPTS, type(e).__name__
|
||||
)
|
||||
if attempt >= JOB_MAX_ATTEMPTS:
|
||||
mark_failed(attachment_id, "retry_exhausted")
|
||||
try:
|
||||
dlq_job = dict(job)
|
||||
dlq_job["attempt"] = 1
|
||||
redis_client.xadd(
|
||||
DLQ_STREAM_NAME,
|
||||
{"payload": json.dumps(dlq_job), "error": str(e)}
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
redis_client.xack(STREAM_NAME, GROUP_NAME, message_id)
|
||||
else:
|
||||
delay = min(2 ** attempt, 30) + random.uniform(0, 1)
|
||||
time.sleep(delay)
|
||||
next_job = dict(job)
|
||||
next_job["attempt"] = attempt + 1
|
||||
redis_client.xadd(STREAM_NAME, {"payload": json.dumps(next_job)})
|
||||
redis_client.xack(STREAM_NAME, GROUP_NAME, message_id)
|
||||
|
||||
except Exception as e:
|
||||
logger.error("Unknown error for %s: %s, marking failed",
|
||||
attachment_id, type(e).__name__)
|
||||
mark_failed(attachment_id, "processing_failed")
|
||||
redis_client.xack(STREAM_NAME, GROUP_NAME, message_id)
|
||||
|
||||
def run_staging_cleanup():
|
||||
try:
|
||||
cutoff = datetime.now(timezone.utc).timestamp() - 3600
|
||||
paginator = s3_client.get_paginator("list_objects_v2")
|
||||
for page in paginator.paginate(Bucket=S3_BUCKET, Prefix="staging/"):
|
||||
for obj in page.get("Contents", []):
|
||||
if obj["LastModified"].timestamp() < cutoff:
|
||||
try:
|
||||
s3_client.delete_object(Bucket=S3_BUCKET, Key=obj["Key"])
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as e:
|
||||
logger.debug("Staging cleanup: %s", e)
|
||||
|
||||
|
||||
def safe_process(msg_id, fields):
|
||||
"""Wrapper - exceptions в threads иначе исчезают молча."""
|
||||
try:
|
||||
process_job(msg_id, fields)
|
||||
except Exception as e:
|
||||
logger.error("Unhandled in process_job %s: %s", msg_id, type(e).__name__)
|
||||
|
||||
|
||||
def main():
|
||||
ensure_consumer_groups()
|
||||
logger.info("Worker '%s' started, concurrency=%d", CONSUMER_NAME, WORKER_CONCURRENCY)
|
||||
|
||||
executor = ThreadPoolExecutor(max_workers=WORKER_CONCURRENCY)
|
||||
last_claim = 0
|
||||
last_cleanup = 0
|
||||
|
||||
try:
|
||||
while running:
|
||||
try:
|
||||
now = time.time()
|
||||
|
||||
if now - last_claim > 15:
|
||||
last_claim = now
|
||||
try:
|
||||
claimed = redis_client.xautoclaim(
|
||||
STREAM_NAME, GROUP_NAME, CONSUMER_NAME,
|
||||
min_idle_time=CLAIM_MIN_IDLE_MS,
|
||||
start_id="0-0", count=WORKER_CONCURRENCY
|
||||
)
|
||||
if claimed and len(claimed) > 1 and claimed[1]:
|
||||
for msg_id, fields in claimed[1]:
|
||||
if not running:
|
||||
break
|
||||
executor.submit(safe_process, msg_id, fields)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if now - last_cleanup > 600:
|
||||
last_cleanup = now
|
||||
executor.submit(run_staging_cleanup)
|
||||
|
||||
resp = redis_client.xreadgroup(
|
||||
GROUP_NAME, CONSUMER_NAME, {STREAM_NAME: ">"},
|
||||
count=WORKER_CONCURRENCY, block=2000
|
||||
)
|
||||
if not resp:
|
||||
continue
|
||||
for _, messages in resp:
|
||||
for msg_id, fields in messages:
|
||||
if not running:
|
||||
break
|
||||
executor.submit(safe_process, msg_id, fields)
|
||||
except Exception as e:
|
||||
logger.warning("Worker loop: %s", type(e).__name__)
|
||||
time.sleep(1)
|
||||
finally:
|
||||
logger.info("Shutting down, waiting for in-flight jobs...")
|
||||
executor.shutdown(wait=True)
|
||||
logger.info("Worker stopped cleanly")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user