Compare commits

4 Commits
Author SHA1 Message Date
Getlles 9c5c004625 report 2026-11-16 20:54:49 +03:00
Getlles 749109aaf4 grade4 2026-11-16 20:34:32 +03:00
Getlles d475bf29ed grade3 2026-11-16 17:38:53 +03:00
Getlles 3571422836 lab4 2026-11-16 14:44:03 +03:00
36 changed files with 2423 additions and 30 deletions
+22
View File
@@ -0,0 +1,22 @@
FROM python:3.11.7-slim-bookworm AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt
FROM python:3.11.7-slim-bookworm AS runner
RUN useradd -u 10001 -m appuser
WORKDIR /home/appuser
COPY --from=builder /install /usr/local
COPY app.py worker.py ./
USER appuser
EXPOSE 8080
HEALTHCHECK --interval=5s --timeout=3s --retries=3 \
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8080/health/ready')"
ENV PYTHONUNBUFFERED=1
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8080", "--timeout-graceful-shutdown", "10"]
+60 -24
View File
@@ -1,37 +1,73 @@
# Отчёт — лабораторная 5
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
## Запуск из чистого clone
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
- Автор, группа: Бондаренко Мария, БПИ2302
- Целевая оценка: 4
## Архитектура и решения
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab4, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
Система построена по микросервисной/событийно-ориентированной схеме с разделением синхронного HTTP API и асинхронного фонового обработчика медиафайлов:
## Доказательства критериев
1. API-сервисы (api-1, api-2):
- Реализованы на базе FastAPI, балансируются через Nginx
- При загрузке изображения выполняют первичную валидацию сигнатур (PNG/JPEG), проверку размера и разрешения
- Сохраняют исходный файл во временную зону объектного хранилища S3
- В рамках единой транзакции PostgreSQL фиксируют метаданные вложения в таблице attachments и записывают событие в таблицу outbox
2. Транзакционный Outbox:
- Фоновый поток в API-инстансах вычитывает неопубликованные записи, отправляет задачи в стрим брокера attachment_jobs и проставляет published_at. Это исключает потерю задач при сбое между COMMIT в БД и публикацией в брокер
3. Брокер сообщений:
- Запущен с включенной персистентностью
- Используются Consumer Groups для балансировки нагрузки между воркерами и подтверждения обработки
4. Объектное хранилище:
- Хранит временные файлы в staging/ и финальные артефакты в attachments/{chat_id}/{attachment_id}/{variant}
5. Фоновый обработчик:
- Многопоточный воркер
- Вычитывает задачи пачками и обрабатывает их идемпотентно с блокировкой строки в БД
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
## Пайплайн обработки изображений и варианты (Variants)
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
Перед генерацией производных форматов исходное изображение проходит нормализацию:
- EXIF-ориентация: применяется ImageOps.exif_transpose(img) для корректного поворота фотографий с мобильных устройств перед ресайзом и кропом
- Цветовое пространство: изображения с альфа-каналом (RGBA, LA, P с прозрачностью) накладываются на белый фон (255, 255, 255) и приводятся к RGB
- Защита от Decompression Bomb: установлено ограничение Image.MAX_IMAGE_PIXELS = 40000000
## Эксперименты и отказы
## Спецификация генерируемых вариантов (Grade 4):
1. thumbnail (превью для списка/чата):
- Формат и качество: image/jpeg, quality=85, optimize=True.
- Геометрия: пропорциональное масштабирование с вписыванием в квадрат 256256 пикселей (без апскейла, если оригинал меньше)
2. optimized (оптимизированный просмотр):
- Формат и качество: image/webp, quality=85.
- Геометрия: пропорциональное масштабирование с вписыванием в рамку 12801280 пикселей (без апскейла)
3. crop (квадратная миниатюра/аватар):
- Формат и качество: image/jpeg, quality=85, optimize=True.
- Геометрия: центральное квадратное кадрирование с последующим приведением к точному размеру 128128 пикселей
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
## Отказоустойчивость, ретраи и восстановление
## Ограничения и следующие шаги
Для обеспечения надежности уровня на оценку 4 реализованы следующие механизмы:
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
- Разделение ошибок (Permanent / Transient):
- Постоянные ошибки сразу переводят вложение в терминальный статус failed с соответствующим error_code и подтверждаются в брокере без бесполезных повторов
- Временные ошибки обрабатываются с повторами до 5 раз
- Exponential Backoff + Jitter:
- Перед постановкой следующей попытки ыполняется задержка для предотвращения лавинообразной нагрузки на S3/БД
- Dead Letter Queue (DLQ) и ручной Replay:
- При исчерпании попыток вложение получает статус failed, а сообщение перекладывается в отдельный стрим attachment_jobs_dlq
- Реализован эндпоинт POST /api/v1/attachments/{attachment_id}/replay, который сбрасывает счетчик попыток, переводит статус обратно в queued и транзакционно создает новую задачу в outbox
- Защита от зависаний:
- Каждая задача ограничена по времени выполнения (30 секунд).
- Фоновый цикл воркера каждые 15 секунд вызывает XAUTOCLAIM, перехватывая зависшие сообщения от упавших экземпляров воркеров
- Очистка временного хранилища:
- Воркер периодически (раз в 10 минут) сканирует префикс staging/ в S3 и удаляет осиротевшие объекты старше 1 часа.
## Результаты проверки и артефакты
1. Контрактные и функциональные тесты:
- scripts/check_contract.py — подтверждено соответствие спецификации OpenAPI, наличие заголовков X-Request-Id и X-Instance-Id
- tests/smoke.py (evidence/smoke.txt) — пройден полный публичный сценарий
2. Демонстрация асинхронности и переживания рестартов:
- При остановленном контейнере worker (demo-01c-worker-stopped.txt) загружен файл sample.png (9becbd32-e413-417a-8762-79bcc195e296), получен ответ 202 Accepted со статусом queued (demo-02-uppload-202.json).
- Выполнены рестарты API-реплик (demo-04-api-restarted.txt) и брокера Redis (demo-08-after-broker-restart.json).
- После запуска воркера (demo-05-worker-started.txt) задача успешно обработана из персистентной очереди, статус перешел в ready (demo-06-ready.json), а вариант thumbnail корректно отдан клиенту (demo-07-thumbnail.jpg).
3. Артефакты расширенного пайплайна:
- Сформированы и верифицированы все три варианта (g4-variants.txt, g4-thunbnail.jpg, g4-crop.jpg) с корректными MIME-типами (image/jpeg, image/webp) и размерами
+1010
View File
File diff suppressed because it is too large Load Diff
+218
View File
@@ -0,0 +1,218 @@
services:
postgres:
image: postgres:15.5-alpine3.19
environment:
POSTGRES_DB: messenger
POSTGRES_USER: postgres
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}
volumes:
- pgdata:/var/lib/postgresql/data
deploy:
resources:
limits:
cpus: '0.50'
memory: 512M
reservations:
cpus: '0.10'
memory: 128M
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres -d messenger"]
interval: 3s
timeout: 3s
retries: 5
redis:
image: redis:7.2.4-alpine
command: redis-server --appendonly yes --appendfsync everysec --loglevel warning
volumes:
- redisdata:/data
deploy:
resources:
limits:
cpus: '0.25'
memory: 256M
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 3s
timeout: 3s
retries: 5
migrate:
image: postgres:15.5-alpine3.19
environment:
PGPASSWORD: ${POSTGRES_PASSWORD}
volumes:
- ./migrations:/migrations
command: >
sh -c "until pg_isready -h postgres -U postgres; do sleep 1; done;
psql -h postgres -U postgres -d messenger -f /migrations/init.sql"
depends_on:
postgres:
condition: service_healthy
restart: "no"
s3:
image: chrislusf/seaweedfs:latest
command: ["server", "-s3", "-dir=/data", "-s3.port=8333"]
environment:
- AWS_ACCESS_KEY_ID=appuser
- AWS_SECRET_ACCESS_KEY=appsecretpassword
volumes:
- s3data:/data
deploy:
resources:
limits:
cpus: '0.50'
memory: 512M
healthcheck:
test: ["CMD-SHELL", "wget -q -O /dev/null http://127.0.0.1:8333/healthz || exit 1"]
interval: 5s
timeout: 3s
retries: 5
start_period: 10s
s3-init:
image: python:3.11.7-slim-bookworm
volumes:
- ./scripts:/scripts
- ./requirements.txt:/requirements.txt
environment:
- S3_ENDPOINT=http://s3:8333
- S3_BUCKET=messenger-attachments
- S3_ROOT_ACCESS_KEY=appuser
- S3_ROOT_SECRET_KEY=appsecretpassword
command: >
sh -c "pip install --no-cache-dir -r /requirements.txt &&
python /scripts/bootstrap_s3.py"
depends_on:
s3:
condition: service_healthy
restart: "no"
api-1:
build: .
stop_grace_period: 15s
environment:
- DATABASE_URL=postgresql://postgres:${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}@postgres:5432/messenger
- REDIS_URL=redis://redis:6379/0
- SESSION_TTL_SECONDS=3600
- INSTANCE_ID=api-replica-1
- SECRET_HMAC_KEY=${SECRET_HMAC_KEY}
- S3_ENDPOINT=http://s3:8333
- S3_BUCKET=messenger-attachments
- S3_ACCESS_KEY_ID=appuser
- S3_SECRET_ACCESS_KEY=appsecretpassword
- BROKER_STREAM=attachment_jobs
deploy:
resources:
limits:
cpus: '0.50'
memory: 256M
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_healthy
migrate:
condition: service_completed_successfully
s3-init:
condition: service_completed_successfully
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8080/health/ready')"]
interval: 5s
timeout: 3s
retries: 3
api-2:
build: .
stop_grace_period: 15s
environment:
- DATABASE_URL=postgresql://postgres:${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}@postgres:5432/messenger
- REDIS_URL=redis://redis:6379/0
- SESSION_TTL_SECONDS=3600
- INSTANCE_ID=api-replica-2
- SECRET_HMAC_KEY=${SECRET_HMAC_KEY}
- S3_ENDPOINT=http://s3:8333
- S3_BUCKET=messenger-attachments
- S3_ACCESS_KEY_ID=appuser
- S3_SECRET_ACCESS_KEY=appsecretpassword
- BROKER_STREAM=attachment_jobs
deploy:
resources:
limits:
cpus: '0.50'
memory: 256M
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_healthy
migrate:
condition: service_completed_successfully
s3-init:
condition: service_completed_successfully
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8080/health/ready')"]
interval: 5s
timeout: 3s
retries: 3
worker:
build: .
command: ["python", "worker.py"]
stop_grace_period: 15s
environment:
- DATABASE_URL=postgresql://postgres:${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}@postgres:5432/messenger
- REDIS_URL=redis://redis:6379/0
- S3_ENDPOINT=http://s3:8333
- S3_BUCKET=messenger-attachments
- S3_ACCESS_KEY_ID=appuser
- S3_SECRET_ACCESS_KEY=appsecretpassword
- BROKER_STREAM=attachment_jobs
- BROKER_GROUP=attachment_workers
- JOB_MAX_ATTEMPTS=5
- MAX_IMAGE_PIXELS=40000000
deploy:
resources:
limits:
cpus: '0.50'
memory: 256M
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_healthy
migrate:
condition: service_completed_successfully
s3-init:
condition: service_completed_successfully
healthcheck:
test: ["CMD", "python", "-c", "import os, redis; redis.Redis.from_url(os.environ['REDIS_URL']).ping()"]
interval: 5s
timeout: 3s
retries: 3
nginx:
image: nginx:1.25.4-alpine
ports:
- "8080:8080"
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
deploy:
resources:
limits:
cpus: '0.25'
memory: 128M
depends_on:
- api-1
- api-2
healthcheck:
test: ["CMD", "nginx", "-t"]
interval: 5s
timeout: 3s
retries: 3
volumes:
pgdata:
redisdata:
s3data:
+6
View File
@@ -0,0 +1,6 @@
{
"id": "fdeb71d4-ee4f-47a6-8968-ffc7740e5242",
"username": "demo_75866",
"display_name": "Demo User",
"created_at": "2026-11-16T13:43:38.059472Z"
}
+11
View File
@@ -0,0 +1,11 @@
{
"token": "grSsHp1gJpFLckF14UT44obEonSGQRvP878JhLyCaNzM6PGBH1LwrdgZIFF8Zu8U",
"token_type": "Bearer",
"expires_at": "2026-11-16T14:43:38.850596Z",
"user": {
"id": "fdeb71d4-ee4f-47a6-8968-ffc7740e5242",
"username": "demo_75866",
"display_name": "Demo User",
"created_at": "2026-11-16T13:43:38.059472Z"
}
}
+8
View File
@@ -0,0 +1,8 @@
{
"id": "5db90ff5-9604-4853-b002-db4144868bcb",
"title": "Demo chat",
"member_ids": [
"fdeb71d4-ee4f-47a6-8968-ffc7740e5242"
],
"created_at": "2026-11-16T13:43:39.240968Z"
}
+10
View File
@@ -0,0 +1,10 @@
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
mtusi2026lab5-api-1-1 mtusi2026lab5-api-1 "uvicorn app:app --hтАж" api-1 30 minutes ago Up 6 minutes (healthy) 8080/tcp
mtusi2026lab5-api-2-1 mtusi2026lab5-api-2 "uvicorn app:app --hтАж" api-2 30 minutes ago Up 6 minutes (healthy) 8080/tcp
mtusi2026lab5-migrate-1 postgres:15.5-alpine3.19 "docker-entrypoint.sтАж" migrate 30 minutes ago Exited (0) 5 minutes ago
mtusi2026lab5-nginx-1 nginx:1.25.4-alpine "/docker-entrypoint.тАж" nginx 30 minutes ago Up 25 minutes (healthy) 0.0.0.0:8080->8080/tcp, [::]:8080->8080/tcp
mtusi2026lab5-postgres-1 postgres:15.5-alpine3.19 "docker-entrypoint.sтАж" postgres 30 minutes ago Up 30 minutes (healthy) 5432/tcp
mtusi2026lab5-redis-1 redis:7.2.4-alpine "docker-entrypoint.sтАж" redis 30 minutes ago Up 4 minutes (healthy) 6379/tcp
mtusi2026lab5-s3-1 chrislusf/seaweedfs:latest "/entrypoint.sh servтАж" s3 30 minutes ago Up 30 minutes (healthy) 7333/tcp, 8080/tcp, 8333/tcp, 8888/tcp, 9333/tcp, 18080/tcp, 18888/tcp, 19333/tcp
mtusi2026lab5-s3-init-1 python:3.11.7-slim-bookworm "sh -c 'pip install тАж" s3-init 30 minutes ago Exited (0) 5 minutes ago
mtusi2026lab5-worker-1 mtusi2026lab5-worker "python worker.py" worker 30 minutes ago Exited (0) Less than a second ago
+1
View File
@@ -0,0 +1 @@
{"id":"9becbd32-e413-417a-8762-79bcc195e296","chat_id":"5db90ff5-9604-4853-b002-db4144868bcb","owner_id":"fdeb71d4-ee4f-47a6-8968-ffc7740e5242","filename":"sample.png","content_type":"image/png","size_bytes":111817,"sha256":"99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75","status":"queued","variants":[],"error_code":null,"created_at":"2026-11-16T13:43:41.734766Z","updated_at":"2026-11-16T13:43:41.734766Z"}
+10
View File
@@ -0,0 +1,10 @@
{
"id": "74dc2e91-a901-4067-8993-da9d2cc4ca55",
"chat_id": "5db90ff5-9604-4853-b002-db4144868bcb",
"sender_id": "fdeb71d4-ee4f-47a6-8968-ffc7740e5242",
"text": "",
"created_at": "2026-11-16T13:43:42.100558Z",
"attachment_ids": [
"9becbd32-e413-417a-8762-79bcc195e296"
]
}
+7
View File
@@ -0,0 +1,7 @@
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
mtusi2026lab5-api-1-1 mtusi2026lab5-api-1 "uvicorn app:app --hтАж" api-1 30 minutes ago Up Less than a second (health: starting) 8080/tcp
mtusi2026lab5-api-2-1 mtusi2026lab5-api-2 "uvicorn app:app --hтАж" api-2 30 minutes ago Up Less than a second (health: starting) 8080/tcp
mtusi2026lab5-nginx-1 nginx:1.25.4-alpine "/docker-entrypoint.тАж" nginx 30 minutes ago Up 25 minutes (healthy) 0.0.0.0:8080->8080/tcp, [::]:8080->8080/tcp
mtusi2026lab5-postgres-1 postgres:15.5-alpine3.19 "docker-entrypoint.sтАж" postgres 30 minutes ago Up 30 minutes (healthy) 5432/tcp
mtusi2026lab5-redis-1 redis:7.2.4-alpine "docker-entrypoint.sтАж" redis 30 minutes ago Up 4 minutes (healthy) 6379/tcp
mtusi2026lab5-s3-1 chrislusf/seaweedfs:latest "/entrypoint.sh servтАж" s3 30 minutes ago Up 30 minutes (healthy) 7333/tcp, 8080/tcp, 8333/tcp, 8888/tcp, 9333/tcp, 18080/tcp, 18888/tcp, 19333/tcp
+10
View File
@@ -0,0 +1,10 @@
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
mtusi2026lab5-api-1-1 mtusi2026lab5-api-1 "uvicorn app:app --hтАж" api-1 30 minutes ago Up 10 seconds (healthy) 8080/tcp
mtusi2026lab5-api-2-1 mtusi2026lab5-api-2 "uvicorn app:app --hтАж" api-2 30 minutes ago Up 9 seconds (healthy) 8080/tcp
mtusi2026lab5-migrate-1 postgres:15.5-alpine3.19 "docker-entrypoint.sтАж" migrate 30 minutes ago Exited (0) 7 seconds ago
mtusi2026lab5-nginx-1 nginx:1.25.4-alpine "/docker-entrypoint.тАж" nginx 30 minutes ago Up 26 minutes (healthy) 0.0.0.0:8080->8080/tcp, [::]:8080->8080/tcp
mtusi2026lab5-postgres-1 postgres:15.5-alpine3.19 "docker-entrypoint.sтАж" postgres 30 minutes ago Up 30 minutes (healthy) 5432/tcp
mtusi2026lab5-redis-1 redis:7.2.4-alpine "docker-entrypoint.sтАж" redis 30 minutes ago Up 4 minutes (healthy) 6379/tcp
mtusi2026lab5-s3-1 chrislusf/seaweedfs:latest "/entrypoint.sh servтАж" s3 30 minutes ago Up 30 minutes (healthy) 7333/tcp, 8080/tcp, 8333/tcp, 8888/tcp, 9333/tcp, 18080/tcp, 18888/tcp, 19333/tcp
mtusi2026lab5-s3-init-1 python:3.11.7-slim-bookworm "sh -c 'pip install тАж" s3-init 30 minutes ago Exited (0) Less than a second ago
mtusi2026lab5-worker-1 mtusi2026lab5-worker "python worker.py" worker 30 minutes ago Up Less than a second (health: starting) 8080/tcp
+23
View File
@@ -0,0 +1,23 @@
{
"id": "9becbd32-e413-417a-8762-79bcc195e296",
"chat_id": "5db90ff5-9604-4853-b002-db4144868bcb",
"owner_id": "fdeb71d4-ee4f-47a6-8968-ffc7740e5242",
"filename": "sample.png",
"content_type": "image/png",
"size_bytes": 111817,
"sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
"status": "ready",
"variants": [
{
"name": "thumbnail",
"width": 256,
"height": 160,
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"content_type": "image/jpeg"
}
],
"error_code": null,
"created_at": "2026-11-16T13:43:41.734766Z",
"updated_at": "2026-11-16T13:43:55.375481Z"
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 4.6 KiB

@@ -0,0 +1,23 @@
{
"id": "9becbd32-e413-417a-8762-79bcc195e296",
"chat_id": "5db90ff5-9604-4853-b002-db4144868bcb",
"owner_id": "fdeb71d4-ee4f-47a6-8968-ffc7740e5242",
"filename": "sample.png",
"content_type": "image/png",
"size_bytes": 111817,
"sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
"status": "ready",
"variants": [
{
"name": "thumbnail",
"width": 256,
"height": 160,
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"content_type": "image/jpeg"
}
],
"error_code": null,
"created_at": "2026-11-16T13:43:41.734766Z",
"updated_at": "2026-11-16T13:43:55.375481Z"
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 1.8 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 4.6 KiB

Binary file not shown.
Binary file not shown.
+74
View File
@@ -0,0 +1,74 @@
CREATE TABLE IF NOT EXISTS users (
id UUID PRIMARY KEY,
username VARCHAR(32) UNIQUE NOT NULL,
display_name VARCHAR(100) NOT NULL,
password_hash VARCHAR(255) NOT NULL,
created_at TIMESTAMPTZ NOT NULL
);
CREATE TABLE IF NOT EXISTS chats (
id UUID PRIMARY KEY,
title VARCHAR(100) NOT NULL,
created_at TIMESTAMPTZ NOT NULL
);
CREATE TABLE IF NOT EXISTS chat_members (
chat_id UUID REFERENCES chats(id) ON DELETE CASCADE,
user_id UUID REFERENCES users(id) ON DELETE CASCADE,
PRIMARY KEY (chat_id, user_id)
);
CREATE TABLE IF NOT EXISTS messages (
id UUID PRIMARY KEY,
chat_id UUID REFERENCES chats(id) ON DELETE CASCADE,
sender_id UUID REFERENCES users(id) ON DELETE CASCADE,
text VARCHAR(4000) NOT NULL,
created_at TIMESTAMPTZ NOT NULL,
attachment_ids UUID[] DEFAULT '{}'
);
CREATE INDEX IF NOT EXISTS idx_messages_chat_pagination ON messages(chat_id, created_at, id);
CREATE TABLE IF NOT EXISTS idempotency_keys (
user_id UUID NOT NULL,
method VARCHAR(16) NOT NULL,
path TEXT NOT NULL,
key VARCHAR(128) NOT NULL,
request_hash VARCHAR(64) NOT NULL,
response_status INT NOT NULL,
response_body JSONB NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
PRIMARY KEY (user_id, method, path, key)
);
CREATE TABLE IF NOT EXISTS attachments (
id UUID PRIMARY KEY,
chat_id UUID REFERENCES chats(id) ON DELETE CASCADE,
owner_id UUID REFERENCES users(id) ON DELETE CASCADE,
filename VARCHAR(255) NOT NULL,
content_type VARCHAR(50) NOT NULL,
size_bytes BIGINT NOT NULL,
sha256 VARCHAR(64) NOT NULL,
status VARCHAR(32) NOT NULL DEFAULT 'queued',
variants JSONB NOT NULL DEFAULT '[]',
error_code VARCHAR(64),
pipeline_version INT NOT NULL DEFAULT 1,
attempt INT NOT NULL DEFAULT 0,
created_at TIMESTAMPTZ NOT NULL,
updated_at TIMESTAMPTZ NOT NULL
);
CREATE TABLE IF NOT EXISTS outbox (
id UUID PRIMARY KEY,
stream VARCHAR(64) NOT NULL,
payload JSONB NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
published_at TIMESTAMPTZ
);
CREATE INDEX IF NOT EXISTS idx_attachments_chat ON attachments(chat_id);
ALTER TABLE messages ADD COLUMN IF NOT EXISTS attachment_ids UUID[] DEFAULT '{}';
ALTER TABLE attachments ADD COLUMN IF NOT EXISTS pipeline_version INT NOT NULL DEFAULT 1;
ALTER TABLE attachments ADD COLUMN IF NOT EXISTS attempt INT NOT NULL DEFAULT 0;
CREATE INDEX IF NOT EXISTS idx_outbox_unpublished ON outbox(created_at) WHERE published_at IS NULL;
+28
View File
@@ -0,0 +1,28 @@
events { worker_connections 1024; }
http {
upstream messenger_api {
server api-1:8080 max_fails=1 fail_timeout=5s;
server api-2:8080 max_fails=1 fail_timeout=5s;
}
server {
listen 8080;
location / {
proxy_pass http://messenger_api;
proxy_connect_timeout 2s;
proxy_send_timeout 5s;
proxy_read_timeout 5s;
proxy_next_upstream error timeout http_502 http_503 http_504;
proxy_next_upstream_tries 2;
proxy_next_upstream_timeout 5s;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
}
+15
View File
@@ -0,0 +1,15 @@
fastapi==0.110.0
uvicorn==0.28.0
pydantic==2.6.2
starlette==0.36.3
idna==3.6
anyio==4.3.0
sniffio==1.3.1
click==8.1.7
h11==0.14.0
annotated-types==0.6.0
psycopg2-binary==2.9.9
bcrypt==4.1.2
redis==5.0.1
boto3==1.34.0
Pillow==10.2.0
+29
View File
@@ -0,0 +1,29 @@
import os
import boto3
from botocore.client import Config
S3_ENDPOINT = os.getenv("S3_ENDPOINT", "http://s3:8333")
S3_BUCKET = os.getenv("S3_BUCKET", "messenger-attachments")
S3_ROOT_ACCESS_KEY = os.getenv("S3_ROOT_ACCESS_KEY", "minioadmin")
S3_ROOT_SECRET_KEY = os.getenv("S3_ROOT_SECRET_KEY", "minioadminpassword")
s3_client = boto3.client(
"s3",
endpoint_url=S3_ENDPOINT,
aws_access_key_id=S3_ROOT_ACCESS_KEY,
aws_secret_access_key=S3_ROOT_SECRET_KEY,
config=Config(signature_version="s3v4"),
region_name="us-east-1"
)
def init_bucket():
print(f"Checking/Creating bucket: {S3_BUCKET}")
try:
s3_client.head_bucket(Bucket=S3_BUCKET)
print("Bucket already exists.")
except Exception:
s3_client.create_bucket(Bucket=S3_BUCKET)
print("Bucket successfully created.")
if __name__ == "__main__":
init_bucket()
+11
View File
@@ -0,0 +1,11 @@
import os, sys, urllib.request, urllib.error
BUCKET = os.getenv("S3_BUCKET", "messenger-attachments")
KEY = os.getenv("S3_KEY")
URL = f"http://s3:8333/{BUCKET}/{KEY}"
try:
urllib.request.urlopen(URL)
print("FAIL: object accessible without auth"); sys.exit(1)
except urllib.error.HTTPError as e:
if e.code in (401, 403):
print(f"PASS: anonymous GET -> {e.code}"); sys.exit(0)
print(f"FAIL: unexpected {e.code}"); sys.exit(1)
+4 -4
View File
@@ -8,8 +8,8 @@ import re
ROOT = Path(__file__).resolve().parents[1]
def main():
lab = json.loads((ROOT/'lab.json').read_text())['number']
spec = json.loads((ROOT/'contracts/openapi.json').read_text())
lab = json.loads((ROOT/'lab.json').read_text(encoding='utf-8'))['number']
spec = json.loads((ROOT/'contracts/openapi.json').read_text(encoding='utf-8'))
assert spec['openapi']=='3.1.0'
assert spec['info']['version']=='1.%d.0'%lab
operation_ids = set()
@@ -41,11 +41,11 @@ def main():
assert ('/api/v1/auth/sessions' in spec['paths']) == (lab>=2)
assert ('/api/v1/attachments/{attachment_id}' in spec['paths']) == (lab>=4)
for file in list((ROOT/'tests').glob('*.py'))+list((ROOT/'scripts').glob('*.py')):
ast.parse(file.read_text(),filename=str(file))
ast.parse(file.read_text(encoding='utf-8'),filename=str(file))
for file in ROOT.rglob('*.json'):
if any(part in {'.git','.venv','node_modules','vendor','output'} for part in file.parts):
continue
json.loads(file.read_text())
json.loads(file.read_text(encoding='utf-8-sig'))
for required in ('README.md','COURSE.md','CONTRIBUTING.md','REPORT.md','contracts/README.md','tests/smoke.py','tests/README.md'):
assert (ROOT/required).is_file(), 'missing '+required
print('PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.')
+81
View File
@@ -0,0 +1,81 @@
import os
import sys
import boto3
from botocore.client import Config
import psycopg2
import psycopg2.extras
from datetime import datetime, timezone, timedelta
DATABASE_URL = os.getenv("DATABASE_URL", "postgresql://postgres:postgres@postgres:5432/messenger")
S3_ENDPOINT = os.getenv("S3_ENDPOINT", "http://s3:8333")
S3_BUCKET = os.getenv("S3_BUCKET", "messenger-attachments")
S3_ACCESS_KEY = os.getenv("S3_ACCESS_KEY_ID", "appuser")
S3_SECRET_KEY = os.getenv("S3_SECRET_ACCESS_KEY", "appsecretpassword")
s3_client = boto3.client(
"s3",
endpoint_url=S3_ENDPOINT,
aws_access_key_id=S3_ACCESS_KEY,
aws_secret_access_key=S3_SECRET_KEY,
config=Config(signature_version="s3v4"),
region_name="us-east-1"
)
def run_reconciliation(dry_run: bool = False):
print(f"Starting reconciliation (dry_run={dry_run})...")
conn = psycopg2.connect(DATABASE_URL, cursor_factory=psycopg2.extras.RealDictCursor)
threshold = datetime.now(timezone.utc) - timedelta(hours=1)
with conn.cursor() as cur:
cur.execute("""
SELECT id, chat_id, filename, created_at FROM attachments
WHERE status = 'pending' AND created_at < %s
""", (threshold,))
stale_pending = cur.fetchall()
for row in stale_pending:
att_id = str(row['id'])
chat_id = str(row['chat_id'])
filename = row['filename']
s3_key = f"attachments/{chat_id}/{att_id}/{filename}"
print(f"[DB Stale Pending] Found stale attachment {att_id}, marking as failed and cleaning S3.")
if not dry_run:
try:
s3_client.delete_object(Bucket=S3_BUCKET, Key=s3_key)
except Exception as e:
print(f"Failed to delete S3 key {s3_key}: {e}")
with conn.cursor() as cur:
cur.execute("UPDATE attachments SET status = 'failed', error_code = 'timeout_stale' WHERE id = %s", (att_id,))
conn.commit()
with conn.cursor() as cur:
cur.execute("SELECT id, chat_id, filename FROM attachments")
all_db_attachments = cur.fetchall()
valid_s3_keys = set()
for row in all_db_attachments:
valid_s3_keys.add(f"attachments/{str(row['chat_id'])}/{str(row['id'])}/{row['filename']}")
paginator = s3_client.get_paginator('list_objects_v2')
for page in paginator.paginate(Bucket=S3_BUCKET, Prefix="attachments/"):
for obj in page.get('Contents', []):
key = obj['Key']
last_modified = obj['LastModified']
if key not in valid_s3_keys and last_modified < threshold:
print(f"[S3 Orphan] Found orphan S3 object {key} (Modified: {last_modified})")
if not dry_run:
try:
s3_client.delete_object(Bucket=S3_BUCKET, Key=key)
print(f"Deleted orphan object {key} from S3.")
except Exception as e:
print(f"Failed to delete orphan {key}: {e}")
conn.close()
print("Reconciliation completed successfully.")
if __name__ == "__main__":
dry_run_mode = "--dry-run" in sys.argv
run_reconciliation(dry_run=dry_run_mode)
+3
View File
@@ -0,0 +1,3 @@
@echo off
docker compose restart
exit /b %ERRORLEVEL%
+3
View File
@@ -0,0 +1,3 @@
#!/bin/sh
set -eu
docker compose restart
+35
View File
@@ -0,0 +1,35 @@
#!/bin/sh
set -eu
wait_ready() {
svc=$1
timeout=60
echo "Waiting for $svc readiness (up to ${timeout}s)..."
while [ "$timeout" -gt 0 ]; do
if docker compose exec -T "$svc" python -c \
"import urllib.request; urllib.request.urlopen('http://localhost:8080/health/ready')" 2>/dev/null; then
echo "$svc is ready!"
return 0
fi
sleep 1
timeout=$((timeout - 1))
done
echo "$svc failed to become ready" >&2
return 1
}
echo "Starting Rolling Update"
echo "1. Rebuilding and replacing api-1..."
docker compose up -d --no-deps --build --force-recreate api-1
wait_ready api-1
echo "Reloading nginx to pick up new api-1 IP..."
docker compose exec -T nginx nginx -s reload
echo "2. Rebuilding and replacing api-2..."
docker compose up -d --no-deps --build --force-recreate api-2
wait_ready api-2
echo "Reloading nginx to pick up new api-2 IP..."
docker compose exec -T nginx nginx -s reload
echo "Rolling Update Completed Successfully"
+12
View File
@@ -0,0 +1,12 @@
#!/bin/sh
set -eu
: "${TARGET_INSTANCE_ID:?test provides target API instance alias}"
if [ "$TARGET_INSTANCE_ID" = "api-replica-1" ]; then
docker compose stop -t 3 api-1
elif [ "$TARGET_INSTANCE_ID" = "api-replica-2" ]; then
docker compose stop -t 3 api-2
else
echo "Unknown target instance: $TARGET_INSTANCE_ID" >&2
exit 1
fi
+54
View File
@@ -0,0 +1,54 @@
import base64, concurrent.futures, json, pathlib, time, urllib.request, uuid
BASE = "http://localhost:8080"
def _req(method, path, body=None, headers=None, token=None):
h = {"Accept": "application/json"}
if token: h["Authorization"] = f"Bearer {token}"
h.update(headers or {})
req = urllib.request.Request(BASE + path, data=body, headers=h, method=method)
with urllib.request.urlopen(req) as r:
return r.status, r.read()
def register_and_login():
u = f"u_{uuid.uuid4().hex[:10]}"
p = "T3st-" + uuid.uuid4().hex
_req("POST", "/api/v1/users",
json.dumps({"username": u, "display_name": "Load", "password": p}).encode(),
{"Content-Type": "application/json"})
basic = base64.b64encode(f"{u}:{p}".encode()).decode()
_, body = _req("POST", "/api/v1/auth/sessions",
headers={"Authorization": f"Basic {basic}"})
return json.loads(body)["token"]
def make_chat(token):
_, me = _req("GET", "/api/v1/users/me", token=token)
uid = json.loads(me)["id"]
_, chat = _req("POST", "/api/v1/chats",
json.dumps({"title": "load", "member_ids": [uid]}).encode(),
{"Content-Type": "application/json", "Idempotency-Key": uuid.uuid4().hex},
token=token)
return json.loads(chat)["id"]
def main():
token = register_and_login()
chat = make_chat(token)
png = pathlib.Path("tests/fixtures/sample.png").read_bytes()
def one(_):
try:
status, _ = _req("POST",
f"/api/v1/chats/{chat}/attachments?filename=f.png",
png, {"Content-Type": "image/png"}, token)
return status
except Exception as e:
return repr(e)
t0 = time.monotonic()
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as ex:
results = list(ex.map(one, range(10)))
print(json.dumps({"results": results,
"elapsed_s": round(time.monotonic() - t0, 3)}))
if __name__ == "__main__":
main()
+1 -1
View File
@@ -14,7 +14,7 @@ import uuid
ROOT = Path(__file__).resolve().parents[1]
LAB = json.loads((ROOT / 'lab.json').read_text())['number']
SPEC = json.loads((ROOT / 'contracts/openapi.json').read_text())
SPEC = json.loads((ROOT / 'contracts/openapi.json').read_text(encoding='utf-8'))
class Failure(AssertionError):
pass
+109
View File
@@ -0,0 +1,109 @@
import concurrent.futures
import urllib.request
import urllib.error
import json
import unittest
from client import Client
class TestMessengerLab4(unittest.TestCase):
def setUp(self):
self.client = Client()
self.alice = self.client.user("alice")
self.bob = self.client.user("bob")
self.eve = self.client.user("eve")
res = self.client.call(
'POST', '/api/v1/chats', 201,
actor=self.alice,
body={"title": "Test Chat", "member_ids": [self.alice["id"], self.bob["id"]]},
schema='Chat'
)
self.chat = res['json']
def test_unauthorized_access_to_chat(self):
self.client.call('GET', f"/api/v1/chats/{self.chat['id']}", 404, actor=self.eve)
def test_empty_message(self):
self.client.call('POST', f"/api/v1/chats/{self.chat['id']}/messages", 400, actor=self.alice, body={"text": ""})
def test_malformed_json(self):
url = self.client.base + f"/api/v1/chats/{self.chat['id']}/messages"
req = urllib.request.Request(
url,
data=b"{broken json content without quotes",
headers={
"Content-Type": "application/json",
"X-User-Id": self.alice["id"]
},
method="POST"
)
try:
with urllib.request.urlopen(req) as resp:
self.fail("Expected HTTP 400 for malformed JSON")
except urllib.error.HTTPError as e:
self.assertEqual(e.code, 400)
body = json.loads(e.read().decode())
self.assertEqual(body["error"]["code"], "invalid_request")
def test_duplicate_username(self):
self.client.call('POST', '/api/v1/users', 409, body=self.alice["registration"])
def test_pagination_multiple_pages(self):
import time
msg_texts = ["msg-page-1", "msg-page-2", "msg-page-3"]
for txt in msg_texts:
self.client.message(self.alice, self.chat, txt)
time.sleep(0.01)
path = f"/api/v1/chats/{self.chat['id']}/messages"
collected_items = []
cursor = None
for _ in range(5):
url_suffix = "?limit=1"
if cursor:
url_suffix += f"&cursor={cursor}"
page_res = self.client.call('GET', path + url_suffix, actor=self.alice, schema='MessagePage')
page = page_res['json']
for item in page['items']:
self.assertNotIn(item['id'], [x['id'] for x in collected_items])
collected_items.append(item)
cursor = page.get('next_keyword') or page.get('next_cursor')
if not cursor:
break
self.assertEqual(len(collected_items), 3)
self.assertEqual(sorted([x['text'] for x in collected_items]), sorted(msg_texts))
def test_concurrent_messages_strict(self):
def worker_send(worker_idx):
local_client = Client()
ids = []
for i in range(10):
res = local_client.message(self.alice, self.chat, f"w-{worker_idx}-msg-{i}")
ids.append(res['json']['id'])
return ids
all_ids = []
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
futures = [executor.submit(worker_send, w) for w in range(10)]
for f in concurrent.futures.as_completed(futures):
all_ids.extend(f.result())
self.assertEqual(len(all_ids), 100)
self.assertEqual(len(set(all_ids)), 100)
page = self.client.call(
'GET', f"/api/v1/chats/{self.chat['id']}/messages?limit=100",
actor=self.alice,
schema='MessagePage'
)
server_ids = [m['id'] for m in page['json']['items']]
self.assertEqual(len(server_ids), 100)
self.assertEqual(len(set(server_ids)), 100)
if __name__ == '__main__':
unittest.main()
+139
View File
@@ -0,0 +1,139 @@
import urllib.request
import urllib.error
import json
import unittest
from client import Client
class TestMessengerLevel4Complete(unittest.TestCase):
def setUp(self):
self.client = Client()
self.alice = self.client.user("alice")
self.bob = self.client.user("bob")
self.eve = self.client.user("eve")
res_chat1 = self.client.call(
'POST', '/api/v1/chats', 201,
actor=self.alice,
body={"title": "Team Chat", "member_ids": [self.alice["id"], self.bob["id"]]},
schema='Chat'
)
self.chat1 = res_chat1['json']
res_chat2 = self.client.call(
'POST', '/api/v1/chats', 201,
actor=self.eve,
body={"title": "Private Chat", "member_ids": [self.eve["id"], self.bob["id"]]},
schema='Chat'
)
self.chat2 = res_chat2['json']
def test_unsupported_media_type(self):
"""Проверка 415: передача текстового файла под видом картинки."""
fake_bytes = b"Just a simple plain text file content."
url = self.client.base + f"/api/v1/chats/{self.chat1['id']}/attachments?filename=notes.txt"
req = urllib.request.Request(
url, data=fake_bytes,
headers={"Authorization": f"Bearer {self.alice['token']}", "Content-Type": "image/png"},
method="POST"
)
try:
with urllib.request.urlopen(req) as resp:
self.fail("Expected HTTP 415")
except urllib.error.HTTPError as e:
self.assertEqual(e.code, 415)
def test_payload_too_large(self):
"""Проверка 413: файл размером больше 10 MiB."""
large_data = b'\x89PNG\r\n\x1a\n' + b'\x00' * (10 * 1024 * 1024 + 100)
url = self.client.base + f"/api/v1/chats/{self.chat1['id']}/attachments?filename=huge.png"
req = urllib.request.Request(
url, data=large_data,
headers={"Authorization": f"Bearer {self.alice['token']}", "Content-Type": "image/png"},
method="POST"
)
try:
with urllib.request.urlopen(req) as resp:
self.fail("Expected HTTP 413")
except urllib.error.HTTPError as e:
self.assertEqual(e.code, 413)
def test_corrupted_image_signature(self):
"""Проверка 415: битая структура / неверные размеры PNG."""
corrupted_png = b'\x89PNG\r\n\x1a\n' + b'\xFF' * 50
url = self.client.base + f"/api/v1/chats/{self.chat1['id']}/attachments?filename=corrupt.png"
req = urllib.request.Request(
url, data=corrupted_png,
headers={"Authorization": f"Bearer {self.alice['token']}", "Content-Type": "image/png"},
method="POST"
)
try:
with urllib.request.urlopen(req) as resp:
self.fail("Expected HTTP 415")
except urllib.error.HTTPError as e:
self.assertEqual(e.code, 415)
def test_invalid_filename_characters(self):
"""Проверка 400: недопустимые символы (path traversal) в имени файла."""
valid_png = (
b'\x89PNG\r\n\x1a\n'
b'\x00\x00\x00\rIHDR'
b'\x00\x00\x00\n\x00\x00\x00\n'
b'\x08\x02\x00\x00\x00'
b'\x02p\xdd\xb5'
b'\x00\x00\x00\x0fcIDATx\x9cc`\x00\x00\x00\x02\x00\x01H\xaf\xa4q'
b'\x00\x00\x00\x00IEND\xaeB`\x82'
)
url = self.client.base + f"/api/v1/chats/{self.chat1['id']}/attachments?filename=../../etc/passwd"
req = urllib.request.Request(
url, data=valid_png,
headers={"Authorization": f"Bearer {self.alice['token']}", "Content-Type": "image/png"},
method="POST"
)
try:
with urllib.request.urlopen(req) as resp:
self.fail("Expected HTTP 400")
except urllib.error.HTTPError as e:
self.assertEqual(e.code, 400)
def test_nonexistent_attachment_id(self):
"""Проверка 404: передача несуществующего ID вложения при отправке сообщения."""
fake_uuid = "12345678-1234-4123-8123-123456789abc"
res = self.client.call(
'POST', f"/api/v1/chats/{self.chat1['id']}/messages", 404,
actor=self.alice,
body={"text": "Check this out", "attachment_ids": [fake_uuid]},
headers={"Idempotency-Key": "test-key-lvl4-1"}
)
self.assertEqual(res['status'], 404)
def test_cross_chat_attachment_isolation(self):
"""Проверка 404: попытка прикрепить вложение из чужого чата."""
valid_png = (
b'\x89PNG\r\n\x1a\n'
b'\x00\x00\x00\rIHDR'
b'\x00\x00\x00\n\x00\x00\x00\n'
b'\x08\x02\x00\x00\x00'
b'\x02p\xdd\xb5'
b'\x00\x00\x00\x0fcID1Tx\x9cc`\x00\x00\x00\x02\x00\x01H\xaf\xa4q'
b'\x00\x00\x00\x00IEND\xaeB`\x82'
)
url = self.client.base + f"/api/v1/chats/{self.chat2['id']}/attachments?filename=secret.png"
req = urllib.request.Request(
url, data=valid_png,
headers={"Authorization": f"Bearer {self.eve['token']}", "Content-Type": "image/png"},
method="POST"
)
with urllib.request.urlopen(req) as resp:
data = json.loads(resp.read().decode())
eve_att_id = data["id"]
res = self.client.call(
'POST', f"/api/v1/chats/{self.chat1['id']}/messages", 404,
actor=self.alice,
body={"text": "Stealing file", "attachment_ids": [eve_att_id]},
headers={"Idempotency-Key": "test-key-lvl4-2"}
)
self.assertEqual(res['status'], 404)
if __name__ == '__main__':
unittest.main()
+1 -1
View File
@@ -55,7 +55,7 @@ def main():
check(len(observed)>=2,'less than two API instances observed before failure; verify LB/session affinity')
print('Executing explicit '+args.mode+' hook; hook output suppressed to avoid leaking secrets.')
# No shell interpolation and no Docker commands in this test.
proc = subprocess.Popen([str(script)],env=env,stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL)
proc = subprocess.Popen(['sh', str(script)], env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
started = time.monotonic()
successes,failures,after_instances = [],0,[]
try:
+405
View File
@@ -0,0 +1,405 @@
import os
import io
import json
import time
import signal
import random
import hashlib
import logging
from datetime import datetime, timezone
from concurrent.futures import ThreadPoolExecutor
import psycopg2
import psycopg2.extras
import redis
import boto3
import botocore.exceptions
from botocore.client import Config
from PIL import Image, ImageOps, UnidentifiedImageError
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("attachment-worker")
DATABASE_URL = os.getenv("DATABASE_URL", "postgresql://postgres:postgres@postgres:5432/messenger")
REDIS_URL = os.getenv("REDIS_URL", "redis://redis:6379/0")
S3_ENDPOINT = os.getenv("S3_ENDPOINT", "http://s3:8333")
S3_BUCKET = os.getenv("S3_BUCKET", "messenger-attachments")
S3_ACCESS_KEY = os.getenv("S3_ACCESS_KEY_ID", "appuser")
S3_SECRET_KEY = os.getenv("S3_SECRET_ACCESS_KEY", "appsecretpassword")
STREAM_NAME = os.getenv("BROKER_STREAM", "attachment_jobs")
DLQ_STREAM_NAME = f"{STREAM_NAME}_dlq"
GROUP_NAME = os.getenv("BROKER_GROUP", "attachment_workers")
CONSUMER_NAME = os.getenv("HOSTNAME", "worker-1")
JOB_MAX_ATTEMPTS = int(os.getenv("JOB_MAX_ATTEMPTS", "5"))
MAX_IMAGE_PIXELS = int(os.getenv("MAX_IMAGE_PIXELS", "40000000"))
JOB_TIMEOUT_SECONDS = int(os.getenv("JOB_TIMEOUT_SECONDS", "30"))
WORKER_CONCURRENCY = int(os.getenv("WORKER_CONCURRENCY", "4"))
CLAIM_MIN_IDLE_MS = (JOB_TIMEOUT_SECONDS + 20) * 1000
Image.MAX_IMAGE_PIXELS = MAX_IMAGE_PIXELS
redis_client = redis.Redis.from_url(REDIS_URL, decode_responses=True)
s3_client = boto3.client(
"s3", endpoint_url=S3_ENDPOINT,
aws_access_key_id=S3_ACCESS_KEY, aws_secret_access_key=S3_SECRET_KEY,
config=Config(signature_version="s3v4"), region_name="us-east-1"
)
running = True
def handle_sigterm(signum, frame):
global running
logger.info("Received shutdown signal, finishing in-flight jobs...")
running = False
signal.signal(signal.SIGTERM, handle_sigterm)
signal.signal(signal.SIGINT, handle_sigterm)
def get_db_connection():
return psycopg2.connect(DATABASE_URL, cursor_factory=psycopg2.extras.RealDictCursor)
def ensure_consumer_groups():
while running:
try:
redis_client.xgroup_create(STREAM_NAME, GROUP_NAME, id="0", mkstream=True)
redis_client.xgroup_create(DLQ_STREAM_NAME, GROUP_NAME, id="0", mkstream=True)
logger.info("Consumer groups ensured on '%s' and '%s'", STREAM_NAME, DLQ_STREAM_NAME)
return
except redis.exceptions.ResponseError as e:
if "BUSYGROUP" in str(e):
return
raise
except Exception as e:
logger.warning("Waiting for Redis: %s", type(e).__name__)
time.sleep(1)
def prepare_base_image(raw_bytes):
img = Image.open(io.BytesIO(raw_bytes))
img = ImageOps.exif_transpose(img)
if img.mode in ("RGBA", "LA") or (img.mode == "P" and "transparency" in img.info):
rgba = img.convert("RGBA")
bg = Image.new("RGB", rgba.size, (255, 255, 255))
bg.paste(rgba, mask=rgba.split()[3])
img = bg
elif img.mode != "RGB":
img = img.convert("RGB")
return img
def generate_grade4_variants(raw_bytes):
base_img = prepare_base_image(raw_bytes)
orig_w, orig_h = base_img.size
meta = []
blobs = {}
thumb = base_img.copy()
if orig_w <= 256 and orig_h <= 256:
t_w, t_h = orig_w, orig_h
else:
s = min(256.0 / orig_w, 256.0 / orig_h)
t_w, t_h = max(1, int(orig_w * s)), max(1, int(orig_h * s))
if (t_w, t_h) != (orig_w, orig_h):
thumb = thumb.resize((t_w, t_h), Image.Resampling.LANCZOS)
buf = io.BytesIO(); thumb.save(buf, format="JPEG", quality=85, optimize=True)
tb = buf.getvalue(); blobs["thumbnail"] = tb
meta.append({"name": "thumbnail", "content_type": "image/jpeg",
"size_bytes": len(tb), "width": t_w, "height": t_h,
"sha256": hashlib.sha256(tb).hexdigest()})
opt = base_img.copy()
if orig_w <= 1280 and orig_h <= 1280:
o_w, o_h = orig_w, orig_h
else:
s = min(1280.0 / orig_w, 1280.0 / orig_h)
o_w, o_h = max(1, int(orig_w * s)), max(1, int(orig_h * s))
if (o_w, o_h) != (orig_w, orig_h):
opt = opt.resize((o_w, o_h), Image.Resampling.LANCZOS)
buf = io.BytesIO(); opt.save(buf, format="WEBP", quality=85)
ob = buf.getvalue(); blobs["optimized"] = ob
meta.append({"name": "optimized", "content_type": "image/webp",
"size_bytes": len(ob), "width": o_w, "height": o_h,
"sha256": hashlib.sha256(ob).hexdigest()})
c = min(orig_w, orig_h)
left, top = (orig_w - c) // 2, (orig_h - c) // 2
crop = base_img.crop((left, top, left + c, top + c))
if c != 128:
crop = crop.resize((128, 128), Image.Resampling.LANCZOS)
buf = io.BytesIO(); crop.save(buf, format="JPEG", quality=85, optimize=True)
cb = buf.getvalue(); blobs["crop"] = cb
meta.append({"name": "crop", "content_type": "image/jpeg",
"size_bytes": len(cb), "width": 128, "height": 128,
"sha256": hashlib.sha256(cb).hexdigest()})
return meta, blobs
def mark_failed(attachment_id, error_code):
now = datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
conn = get_db_connection()
try:
with conn.cursor() as cur:
cur.execute("UPDATE attachments SET status='failed', error_code=%s, updated_at=%s "
"WHERE id=%s AND status != 'ready'",
(error_code, now, attachment_id))
conn.commit()
finally:
conn.close()
PERMANENT_ERRORS = (
UnidentifiedImageError, Image.DecompressionBombError, ValueError,
psycopg2.errors.DataError,
)
TRANSIENT_ERRORS = (
TimeoutError,
botocore.exceptions.EndpointConnectionError,
botocore.exceptions.ReadTimeoutError,
psycopg2.OperationalError,
redis.exceptions.ConnectionError,
)
def process_job(message_id, fields):
try:
job = json.loads(fields.get("payload", "{}"))
except Exception:
logger.error("Malformed job %s, ack to drop", message_id)
redis_client.xack(STREAM_NAME, GROUP_NAME, message_id)
return
attachment_id = job.get("attachment_id")
job_id = job.get("job_id")
pipeline_version = int(job.get("pipeline_version", 1))
attempt = int(job.get("attempt", 1))
if not attachment_id:
redis_client.xack(STREAM_NAME, GROUP_NAME, message_id)
return
conn = get_db_connection()
try:
with conn.cursor() as cur:
cur.execute("SELECT * FROM attachments WHERE id=%s FOR UPDATE", (attachment_id,))
att = cur.fetchone()
if not att:
conn.commit()
redis_client.xack(STREAM_NAME, GROUP_NAME, message_id)
return
if att["status"] == "ready" or att["pipeline_version"] > pipeline_version:
conn.commit()
redis_client.xack(STREAM_NAME, GROUP_NAME, message_id)
return
if att["status"] == "failed":
conn.commit()
redis_client.xack(STREAM_NAME, GROUP_NAME, message_id)
return
if att["status"] == "processing":
updated = att["updated_at"]
if isinstance(updated, str):
updated = datetime.fromisoformat(updated.replace("Z", "+00:00"))
age_seconds = (datetime.now(timezone.utc) - updated).total_seconds()
if age_seconds < (JOB_TIMEOUT_SECONDS + 10):
conn.commit()
logger.info("Attachment %s already processing (age=%.1fs), ack duplicate",
attachment_id, age_seconds)
redis_client.xack(STREAM_NAME, GROUP_NAME, message_id)
return
now = datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
cur.execute(
"UPDATE attachments SET status='processing', attempt=%s, updated_at=%s WHERE id=%s",
(attempt, now, attachment_id)
)
conn.commit()
finally:
conn.close()
chat_id = str(att["chat_id"])
staging_key = f"staging/{chat_id}/{attachment_id}/original"
final_original_key = f"attachments/{chat_id}/{attachment_id}/original"
start_time = time.monotonic()
def check_timeout():
if time.monotonic() - start_time > JOB_TIMEOUT_SECONDS:
raise TimeoutError("Job exceeded time limit")
try:
try:
obj = s3_client.get_object(Bucket=S3_BUCKET, Key=staging_key)
except Exception:
obj = s3_client.get_object(Bucket=S3_BUCKET, Key=final_original_key)
raw_bytes = obj["Body"].read()
check_timeout()
if hashlib.sha256(raw_bytes).hexdigest() != att["sha256"] or len(raw_bytes) != att["size_bytes"]:
raise ValueError("Staging checksum mismatch")
variants_meta, variants_bytes = generate_grade4_variants(raw_bytes)
check_timeout()
s3_client.put_object(
Bucket=S3_BUCKET, Key=final_original_key,
Body=raw_bytes, ContentType=att["content_type"]
)
for var_name, var_bytes in variants_bytes.items():
ct = "image/webp" if var_name == "optimized" else "image/jpeg"
s3_client.put_object(
Bucket=S3_BUCKET,
Key=f"attachments/{chat_id}/{attachment_id}/{var_name}",
Body=var_bytes, ContentType=ct
)
check_timeout()
ready_at = datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
conn = get_db_connection()
try:
with conn.cursor() as cur:
cur.execute("""
UPDATE attachments SET status='ready', variants=%s::jsonb,
error_code=NULL, updated_at=%s
WHERE id=%s AND pipeline_version <= %s
""", (json.dumps(variants_meta), ready_at, attachment_id, pipeline_version))
conn.commit()
finally:
conn.close()
redis_client.xack(STREAM_NAME, GROUP_NAME, message_id)
logger.info("Job %s completed: attachment %s ready", job_id, attachment_id)
try:
s3_client.delete_object(Bucket=S3_BUCKET, Key=staging_key)
except Exception:
pass
except (UnidentifiedImageError, Image.DecompressionBombError) as e:
logger.error("Permanent image error for %s: %s", attachment_id, type(e).__name__)
mark_failed(attachment_id, "invalid_image")
redis_client.xack(STREAM_NAME, GROUP_NAME, message_id)
except ValueError as e:
logger.error("Checksum error for %s: %s", attachment_id, e)
mark_failed(attachment_id, "invalid_checksum")
redis_client.xack(STREAM_NAME, GROUP_NAME, message_id)
except psycopg2.errors.DataError as e:
logger.error("DB data error for %s: %s", attachment_id, type(e).__name__)
mark_failed(attachment_id, "data_error")
redis_client.xack(STREAM_NAME, GROUP_NAME, message_id)
except TRANSIENT_ERRORS as e:
logger.warning(
"Transient error for %s (attempt %d/%d): %s",
attachment_id, attempt, JOB_MAX_ATTEMPTS, type(e).__name__
)
if attempt >= JOB_MAX_ATTEMPTS:
mark_failed(attachment_id, "retry_exhausted")
try:
dlq_job = dict(job)
dlq_job["attempt"] = 1
redis_client.xadd(
DLQ_STREAM_NAME,
{"payload": json.dumps(dlq_job), "error": str(e)}
)
except Exception:
pass
redis_client.xack(STREAM_NAME, GROUP_NAME, message_id)
else:
delay = min(2 ** attempt, 30) + random.uniform(0, 1)
time.sleep(delay)
next_job = dict(job)
next_job["attempt"] = attempt + 1
redis_client.xadd(STREAM_NAME, {"payload": json.dumps(next_job)})
redis_client.xack(STREAM_NAME, GROUP_NAME, message_id)
except Exception as e:
logger.error("Unknown error for %s: %s, marking failed",
attachment_id, type(e).__name__)
mark_failed(attachment_id, "processing_failed")
redis_client.xack(STREAM_NAME, GROUP_NAME, message_id)
def run_staging_cleanup():
try:
cutoff = datetime.now(timezone.utc).timestamp() - 3600
paginator = s3_client.get_paginator("list_objects_v2")
for page in paginator.paginate(Bucket=S3_BUCKET, Prefix="staging/"):
for obj in page.get("Contents", []):
if obj["LastModified"].timestamp() < cutoff:
try:
s3_client.delete_object(Bucket=S3_BUCKET, Key=obj["Key"])
except Exception:
pass
except Exception as e:
logger.debug("Staging cleanup: %s", e)
def safe_process(msg_id, fields):
"""Wrapper - exceptions в threads иначе исчезают молча."""
try:
process_job(msg_id, fields)
except Exception as e:
logger.error("Unhandled in process_job %s: %s", msg_id, type(e).__name__)
def main():
ensure_consumer_groups()
logger.info("Worker '%s' started, concurrency=%d", CONSUMER_NAME, WORKER_CONCURRENCY)
executor = ThreadPoolExecutor(max_workers=WORKER_CONCURRENCY)
last_claim = 0
last_cleanup = 0
try:
while running:
try:
now = time.time()
if now - last_claim > 15:
last_claim = now
try:
claimed = redis_client.xautoclaim(
STREAM_NAME, GROUP_NAME, CONSUMER_NAME,
min_idle_time=CLAIM_MIN_IDLE_MS,
start_id="0-0", count=WORKER_CONCURRENCY
)
if claimed and len(claimed) > 1 and claimed[1]:
for msg_id, fields in claimed[1]:
if not running:
break
executor.submit(safe_process, msg_id, fields)
except Exception:
pass
if now - last_cleanup > 600:
last_cleanup = now
executor.submit(run_staging_cleanup)
resp = redis_client.xreadgroup(
GROUP_NAME, CONSUMER_NAME, {STREAM_NAME: ">"},
count=WORKER_CONCURRENCY, block=2000
)
if not resp:
continue
for _, messages in resp:
for msg_id, fields in messages:
if not running:
break
executor.submit(safe_process, msg_id, fields)
except Exception as e:
logger.warning("Worker loop: %s", type(e).__name__)
time.sleep(1)
finally:
logger.info("Shutting down, waiting for in-flight jobs...")
executor.shutdown(wait=True)
logger.info("Worker stopped cleanly")
if __name__ == "__main__":
main()