Compare commits

8 Commits
Author SHA1 Message Date
Mikhail Verkovykh 8cadd96586 chore(report) added metadata 2026-11-20 20:50:48 +03:00
Mikhail Verkovykh 186fc95d6e docs(lab5): fill pull request template 2026-11-20 19:05:32 +03:00
Mikhail Verkovykh b6953ae6ee feat(lab5): add exactly_right scenario and refresh evidence 2026-11-20 19:05:29 +03:00
Mikhail Verkovykh d0f533b371 feat(lab5): time pipeline stages and centralize variant constants 2026-11-20 19:05:27 +03:00
Mikhail Verkovykh 1ce93185af chore(lab5): remove stale templates and refresh source comments 2026-11-20 19:05:23 +03:00
Mikhail Verkovykh fec2882c9a feat(lab5): track 1 queue experiment and evidence
- 3 profiles (serial 1/1, balanced 4/4, eager 8/32) x 2 image sizes
- measure throughput, DB-derived queue lag and p95 time-to-ready, worker RSS
- separate MAX_PENDING_JOBS=3 backpressure and worker recovery run
- self-contained HTML/SVG dashboard plus JSON/Markdown evidence
- document methodology, results and limitations in REPORT/README/CHANGELOG
2026-11-15 00:44:45 +03:00
Mikhail Verkovykh 9e43faa536 feat(lab5): durable async attachment pipeline through grade 4
- RabbitMQ quorum queue, persistent messages and publisher confirms
- S3 staging plus PostgreSQL attachments/jobs outbox transaction
- worker variants (thumbnail, crop, optimized WebP), DB lease and lease sweeper
- jittered retry, permanent failures, explicit replay and admission cap
- deterministic crash hooks, resilience script and evidence for grade 4
- async-aware custom tests, oversized pixel case, docs and changelog
2026-11-15 00:30:36 +03:00
Mikhail Verkovykh 28d68dc30b chore(migrate) moved codebase from lab4 2026-11-14 23:17:36 +03:00
121 changed files with 15038 additions and 120 deletions
+65 -11
View File
@@ -1,25 +1,79 @@
# Copy to .env for your LOCAL synthetic test environment; never commit .env. # Copy to .env for your LOCAL synthetic test environment; never commit .env.
# These are logical interface suggestions; map actual framework names in REPORT.md. # Fill every empty value. The compose file uses ${VAR:?} so a missing secret
# fails fast instead of silently starting with a guessable password.
APP_HOST=0.0.0.0 APP_HOST=0.0.0.0
APP_PORT=8080 APP_PORT=8080
APP_ENV=lab APP_ENV=lab
SESSION_TTL_SECONDS=3600 SESSION_TTL_SECONDS=3600
# Choose credentials locally. File paths contain no secret values.
DATABASE_URL_FILE=/run/secrets/database_url
# Optional on 3, required use on 4 in lab2; Redis OR Valkey:
SESSION_STORE=postgres
CACHE_URL_FILE=/run/secrets/cache_url
API_REPLICAS=2
IDEMPOTENCY_TTL_SECONDS=86400 IDEMPOTENCY_TTL_SECONDS=86400
S3_ENDPOINT=http://s3:9000 SESSION_STORE=redis
API_REPLICAS=2
POSTGRES_USER=messenger
POSTGRES_DB=messenger
POSTGRES_PASSWORD=
DB_POOL_MAX_CONNS=10
# Redis/Valkey session store.
CACHE_URL=redis://redis:6379/0
# Private S3-compatible bucket (RustFS in compose.yaml).
# RustFS identifies an IAM user by its access key, so S3_APP_USER must equal
# S3_ACCESS_KEY_ID. The bootstrap refuses to start when they diverge instead of
# leaving the API healthy-looking but unable to authenticate.
S3_ENDPOINT=http://rustfs:9000
S3_REGION=us-east-1 S3_REGION=us-east-1
S3_BUCKET=messenger-attachments S3_BUCKET=messenger-attachments
S3_ACCESS_KEY_ID_FILE=/run/secrets/s3_access_key_id S3_ROOT_ACCESS_KEY=
S3_SECRET_ACCESS_KEY_FILE=/run/secrets/s3_secret_access_key S3_ROOT_SECRET_KEY=
S3_ACCESS_KEY_ID=messenger-api
S3_SECRET_ACCESS_KEY=
S3_APP_USER=messenger-api
S3_APP_POLICY=messenger-attachments-rw
MAX_UPLOAD_BYTES=10485760 MAX_UPLOAD_BYTES=10485760
MAX_IMAGE_PIXELS=40000000 MAX_IMAGE_PIXELS=40000000
BROKER_URL_FILE=/run/secrets/broker_url # Shared decode memory budget per API process and per worker process.
IMAGE_DECODE_MEMORY_MB=256
# RabbitMQ broker (lab5). The management UI is bound to 127.0.0.1 only.
RABBITMQ_USER=messenger
RABBITMQ_PASSWORD=
RABBITMQ_MGMT_PORT=15672
JOB_QUEUE=attachment.process.v1
# Async worker and retry policy. MAX_PENDING_JOBS is an approximate admission
# cap across API replicas; 0 disables it.
WORKER_ONLY=false
WORKER_CONCURRENCY=2 WORKER_CONCURRENCY=2
WORKER_PREFETCH=2
WORKER_REPLICAS=1
MAX_PENDING_JOBS=1000
JOB_MAX_ATTEMPTS=5 JOB_MAX_ATTEMPTS=5
JOB_TIMEOUT_SECONDS=60 JOB_TIMEOUT_SECONDS=60
JOB_LEASE_SECONDS=90
JOB_RETRY_BASE_SECONDS=2
JOB_RETRY_MAX_SECONDS=30
LEASE_SWEEP_SECONDS=15
STAGING_RETENTION_SECONDS=86400 STAGING_RETENTION_SECONDS=86400
# Deterministic grade-4 fault hooks. Keep them off outside evidence runs.
WORKER_FAULT_MODE=none
WORKER_CRASH_AFTER_EFFECT=false
WORKER_CRASH_AFTER_CLAIM=false
# Container limits (used by the lab5 experiment).
API_CPU_LIMIT=0.50
API_MEM_LIMIT=512M
WORKER_CPU_LIMIT=1.0
WORKER_MEM_LIMIT=1G
POSTGRES_CPU_LIMIT=1.0
POSTGRES_MEM_LIMIT=512M
REDIS_CPU_LIMIT=0.50
REDIS_MEM_LIMIT=256M
RUSTFS_CPU_LIMIT=1.0
RUSTFS_MEM_LIMIT=512M
RABBITMQ_CPU_LIMIT=1.0
RABBITMQ_MEM_LIMIT=512M
RECONCILER_CPU_LIMIT=0.25
RECONCILER_MEM_LIMIT=128M
+64 -14
View File
@@ -1,18 +1,68 @@
## Лабораторная 5 ## Лабораторная 5
Автор / группа: TODO Автор / группа: Булла Максим, БПИ2301
Целевая оценка (3/4/5): TODO Целевая оценка (3/4/5): 5
Трек 5, если заявлен: TODO Трек 5, если заявлен: трек 1 — «Очередь под перегрузкой»
Что реализовано и что перенесено: TODO Что реализовано и что перенесено:
Команды запуска из чистого clone: TODO Реализован durable-асинхронный pipeline: upload пишет байты в S3 staging и в
Результаты проверок и ссылка на REPORT.md: TODO одной транзакции создаёт `attachments(queued)` + `attachment_jobs` (outbox);
Ограничения / известные ошибки: TODO publisher подтверждает доставку в RabbitMQ quorum queue, отдельный worker
Источники и использование ИИ: TODO перечитывает метаданные из PostgreSQL, под DB-lease проверяет staging, пишет
оригинал и три варианта (`thumbnail`, `crop`, `optimized`), затем атомарно
фиксирует `ready` и только после этого делает ack. Добавлены retry с
экспоненциальным backoff и jitter, различение постоянных/временных ошибок,
replay, lease recovery, reconciler для брошенного staging и admission cap с
`503`/`Retry-After`. Параметры вариантов вынесены в
`app/internal/config/config.go`. Обязательный минимум предыдущих работ сохранён;
устаревшие файлы `*.lab3`/`*.lab4` удалены, публичный контракт и тесты не
ослаблены. Для трека 1 добавлен сценарий `exactly_right` (1280×1280) и
измерение compression overhead отдельно от decode/scaling.
- [ ] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами. Команды запуска из чистого clone:
- [ ] Контракт и публичные тесты не ослаблены.
- [ ] make check и make test выполнены; приложен реальный вывод. ```sh
- [ ] Сценарии отказов/безопасности своего уровня воспроизведены. cp .env.example .env
- [ ] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens. # заполнить POSTGRES_PASSWORD, S3_ROOT_ACCESS_KEY, S3_ROOT_SECRET_KEY,
- [ ] README задания сохранён, REPORT.md содержит инструкцию именно моего решения. # S3_SECRET_ACCESS_KEY, RABBITMQ_PASSWORD; S3_ACCESS_KEY_ID == S3_APP_USER
docker compose --env-file .env up -d --build
curl -fsS http://localhost:8080/health/ready
make check
make test BASE_URL=http://localhost:8080
make track1 BASE_URL=http://localhost:8080 # эксперимент трека 1
```
Результаты проверок и ссылка на REPORT.md:
`make check` — PASS (offline structure, local refs, Python syntax); `make test` —
`PASS: lab5 public API scenario`; `go test ./...` в `app` — все пакеты зелёные.
Сценарии уровня 4 воспроизведены: broker outage/outbox, worker crash до и после
эффекта, retry, permanent failure + replay, lease recovery, перезапуск всех
контейнеров с сохранением volumes. Трек 1: 9 ячеек (3 профиля × 3 размера),
eager/small — 15.58 jobs/s против 6.84 у serial; `exactly_right`/`huge_few` —
~0.44–0.50 jobs/s, encode share 82–83% против 61–64%; backpressure — `503` +
`Retry-After: 5`, recovery 7.653 s. Подробности, таблица критериев и ограничения —
[REPORT.md](REPORT.md); сырые данные — `evidence/track1.json`,
`evidence/lab5-grade4.json`.
Ограничения / известные ошибки:
Стенд учебный: один RabbitMQ, один PostgreSQL и один worker, поэтому проверяется
durability на volume, а не HA-кластер. `MAX_PENDING_JOBS` считается по каждой
API-реплике приблизительно, без общей транзакции с вставкой. Время recovery
включает запуск контейнера и healthcheck. PNG-фикстуры синтетические; один
прогон на ячейку, p95 по 12–18 заданиям чувствителен к выбросу.
`exactly_right` убирает scaling, но не уравнивает объём WebP-encode с
`huge_few`.
Источники и использование ИИ:
RabbitMQ quorum queues и publisher confirms; `amqp091-go`, `gen2brain/webp`,
`golang.org/x/image/draw`, `golang.org/x/sync/semaphore`; стандартные
`image/png`, `image/jpeg`. ИИ: DeepSeek Flash V4.1 + OpenCode harness для генерации
решения и частичного ревью. Все приведённые результаты получены
запуском команд на этом стенде.
- [x] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
- [x] Контракт и публичные тесты не ослаблены.
- [x] make check и make test выполнены; приложен реальный вывод.
- [x] Сценарии отказов/безопасности своего уровня воспроизведены.
- [x] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
- [x] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
+75
View File
@@ -0,0 +1,75 @@
# Changelog
Все заметные изменения стенда. Формат — по мотивам Keep a Changelog; версии
учебные, даты в UTC.
## [Unreleased]
### Added
- Асинхронный pipeline lab5: RabbitMQ quorum queue `attachment.process.v1`,
persistent-сообщения и publisher confirms.
- Durable staging в RustFS `staging/{chat}/{attachment}` и отдельный worker,
который пишет оригинал и варианты `thumbnail` (JPEG 256 fit), `crop`
(JPEG 128×128), `optimized` (WebP 1280 fit).
- PostgreSQL-таблица `attachment_jobs` как transactional outbox
(`published_at IS NULL`) и источник состояний `queued/processing/ready/failed`.
- DB-lease на job, sweeper просроченных lease, retry с экспоненциальным backoff
±50% jitter, различение постоянных/временных ошибок и явный
`worker -replay <attachment_uuid>`.
- Admission cap `MAX_PENDING_JOBS` с `503` и `Retry-After: 5`.
- Детерминированные хуки `WORKER_FAULT_MODE`, `WORKER_CRASH_AFTER_EFFECT`,
`WORKER_CRASH_AFTER_CLAIM` и скрипт `scripts/lab5_resilience.py`.
- Миграции `0004_async_jobs` и `0005_retry_error_state`.
- Reconciler очищает брошенные staging/pending-объекты, не трогая `ready`.
- Проверки: async-aware custom tests, негативный случай 42 MP (413), evidence
для smoke, отказов, persistence и failover.
- `scripts/restart.sh` перезапускает RabbitMQ и worker с сохранением volumes.
- `scripts/track1_experiment.py` и `make track1` — трек 1: матрица
`serial/balanced/eager` × `small_many`/`exactly_right`/`huge_few`,
throughput, queue lag, p95 time-to-ready, RSS, декомпозиция
decode/scale/JPEG/WebP и отдельный backpressure/recovery прогон;
self-contained HTML-дашборд с inline SVG.
- `exactly_right` (1280×1280): optimized WebP кодируется без scaling, поэтому
encode share показывает нижнюю границу стоимости сжатия.
- Документация: README с запуском и архитектурой, отчёт с таблицей критериев,
CHANGELOG.
### Removed
- 34 устаревших файла `*.lab3`/`*.lab4` (Makefile, contracts, tests, scripts,
CI templates и fixtures) удалены из рабочего дерева.
### Changed
- Параметры вариантов (`ThumbnailMax`, `CropSize`, `OptimizedMax`, качества
JPEG/WebP и WebP method) перенесены в `app/internal/config/config.go`.
- Комментарии в Go/SQL/shell/Python больше не ссылаются на номера лабораторных;
поведение описывается по существу.
- `POST /chats/{id}/attachments` отвечает `202` после durable staging и COMMIT
задания; готовность определяется опросом метаданных, content до `ready` — 409.
- Загрузка потоковая (буфер 32 KiB), лимиты bytes/pixels проверяются до
декодирования; декодирование ограничено общим semaphore-budget.
- `attachment_jobs.error_code` разрешён для `queued/processing`, чтобы
сохранять причину временной ошибки между попытками.
- RustFS bootstrap требует `S3_APP_USER == S3_ACCESS_KEY_ID` и падает быстро при
расхождении; `.env.example` это документирует.
### Fixed
- Retry, потерявший ownership строки, больше не подтверждается молча: worker
возвращает сообщение в очередь.
- JSON-сообщения брокера корректно разбирают `created_at` через
`RFCFormTime.UnmarshalJSON`.
- Устранён разрыв между COMMIT задания и публикацией: outbox повторяет
публикацию до подтверждения брокера.
- Повторная доставка после crash не создаёт второй набор вариантов и не
перезаписывает более новую версию результата.
### Evidence
- `evidence/lab5-smoke.txt`, `evidence/lab5-negative-cases.txt`,
`evidence/lab5-grade4.{json,md}`, `evidence/lab5-persistence.{txt}`,
`evidence/lab5-persistence-containers.txt`, `evidence/lab5-failover.txt`,
`evidence/reconcile.txt`, `evidence/track1.{json,html,md}`.
+25
View File
@@ -0,0 +1,25 @@
FROM golang:1.27.1-alpine AS build
WORKDIR /src
COPY app/go.mod app/go.sum ./
RUN go mod download
COPY app/ ./
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/server ./cmd/server \
&& CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/worker ./cmd/worker \
&& CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/reconciler ./cmd/reconciler
FROM alpine:3.22
# The runtime image needs no package manager at build time; copying the
# CA bundle from the Go builder avoids a network dependency on Alpine mirrors.
RUN addgroup -S -g 10001 app \
&& adduser -S -u 10001 -G app app
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY --from=build /out/server /usr/local/bin/server
COPY --from=build /out/worker /usr/local/bin/worker
COPY --from=build /out/reconciler /usr/local/bin/reconciler
USER app
EXPOSE 8080
ENTRYPOINT ["/usr/local/bin/server"]
+4 -1
View File
@@ -4,7 +4,7 @@ CA_FILE ?=
ACTION_SCRIPT ?= ACTION_SCRIPT ?=
TLS_ARGS = $(if $(CA_FILE),--ca-file "$(CA_FILE)",) TLS_ARGS = $(if $(CA_FILE),--ca-file "$(CA_FILE)",)
.PHONY: help check validate test load test-persistence test-failover .PHONY: help check validate test load test-persistence test-failover track1
help: help:
@echo "check: offline structure; validate: full OpenAPI (requirements-dev.txt); test: running API; load: synthetic writes" @echo "check: offline structure; validate: full OpenAPI (requirements-dev.txt); test: running API; load: synthetic writes"
@@ -28,3 +28,6 @@ test-persistence:
test-failover: test-failover:
@test -n "$(ACTION_SCRIPT)" || (echo "Set ACTION_SCRIPT to your implemented executable stop-one hook"; exit 2) @test -n "$(ACTION_SCRIPT)" || (echo "Set ACTION_SCRIPT to your implemented executable stop-one hook"; exit 2)
$(PYTHON) tests/resilience.py --mode failover --action-script "$(ACTION_SCRIPT)" --base-url "$(BASE_URL)" $(TLS_ARGS) $(PYTHON) tests/resilience.py --mode failover --action-script "$(ACTION_SCRIPT)" --base-url "$(BASE_URL)" $(TLS_ARGS)
track1:
$(PYTHON) scripts/track1_experiment.py --base-url "$(BASE_URL)"
+162 -76
View File
@@ -1,105 +1,191 @@
# Лабораторная 5. Очереди и асинхронная обработка изображений # Лабораторная 5. Очереди и асинхронная обработка изображений
Отделить приём файла от тяжёлой обработки, корректно переживать повторную доставку и остановку worker. Реализация лабораторной 5 на Go: API принимает файл, сохраняет оригинал в
durable staging, фиксирует задание в PostgreSQL и отвечает **202**. Отдельный
worker обрабатывает задание через RabbitMQ и делает варианты изображения.
Целевая оценка — **5, трек 1** (очередь под перегрузкой).
Это **skeleton**, а не готовый сервер. Здесь есть задание, контракт и публичные проверки. Реализацию приложения, Dockerfile и требуемую инфраструктуру пишет студент. Язык и framework свободные; UI не обязателен. [Карта курса](COURSE.md) · [Процесс fork/PR](CONTRIBUTING.md) · [OpenAPI](contracts/openapi.json) · [Семантика API](contracts/README.md) · [Проверки](tests/README.md) · [Отчёт](REPORT.md) · [Примеры JSON](contracts/examples.json) · [Changelog](CHANGELOG.md)
[Карта курса](COURSE.md) · [Процесс fork/PR](CONTRIBUTING.md) · [OpenAPI](contracts/openapi.json) · [Семантика API](contracts/README.md) · [Проверки](tests/README.md) · [Отчёт](REPORT.md) · [Примеры JSON](contracts/examples.json) · [Памятка преподавателю](MAINTAINERS.md) ## Стек и состав стенда
## Откуда начинаем | Компонент | Версия | Роль |
| --- | --- | --- |
| Go + Fiber v3 | Go 1.27.1 | API, worker, reconciler |
| PostgreSQL | 16.10 | источник истины: attachments, jobs/outbox, idempotency |
| RabbitMQ | 4.2 | durable quorum queue `attachment.process.v1` |
| RustFS (S3) | 1.0.4 | staging и конечные объекты, private bucket |
| Redis | 7.4.7 | сессии |
| HAProxy | 3.2.6 | балансировка двух API-реплик |
Нужен минимум lab4. POST upload теперь возвращает 202 queued, а готовность выясняется через метаданные. В HTTP upload клиент по-прежнему передаёт все байты. Контейнеры: `postgres`, `redis`, `rustfs`, `s3-init`, `migrate`, два `api`,
`worker`, `reconciler`, `haproxy`. Секреты живут только в локальном `.env`.
**Архитектура:** API → устойчивый staging + запись задания → брокер → worker → конечный S3 + PostgreSQL. Клиент опрашивает статус. ## Запуск из чистого clone
## Что сделать Нужны Docker Engine с Compose v2 и Python 3.9+ для проверок.
1. Выберите RabbitMQ, Redis/Valkey Streams, NATS JetStream или другой брокер с persistence и подтверждениями. Обычный Pub/Sub без хранения не подходит.
2. Разделите приём, durable staging, публикацию задания, обработку и фиксацию результата. Прочитайте contracts/attachment-states.md.
3. Реализуйте worker отдельно от API: публикация оригинала в конечный S3-prefix и thumbnail 256×256 по правилам контракта.
4. Добавьте конечные состояния, ограниченные повторы и идемпотентную запись результатов. Не помещайте bytes или credentials в сообщения брокера.
5. Остановите worker, загрузите файл, перезапустите API/worker, убедитесь в завершении принятой задачи. Затем проверьте повторную доставку.
## Оценка «3»: работающий минимум
- Весь минимум lab4 сохранён с переходом upload 201 → 202. API возвращает 202 только после устойчивого приёма оригинала и сохранения задания; остановка API после ответа не теряет работу.
- В общем Compose/Swarm есть отдельные broker и worker. Сообщения durable, доставка at least once, подтверждение только после устойчивого эффекта. Недоступность worker не заставляет API выполнять resize синхронно.
- Состояния queued/processing/ready/failed доступны в API. Worker сохраняет оригинал и один thumbnail; original неизменен, thumbnail соответствует контракту. Публичный smoke проходит.
- Есть ограниченный retry (конкретное число попыток в конфигурации, например 5), финальное failed с безопасным error_code. Повтор одной job не создаёт второго набора вариантов.
- Демонстрация: worker остановлен → upload 202 → сообщение с вложением создано → API перезапущен → worker запущен → ready и доступный thumbnail. Принятое задание не исчезает при рестарте broker с volume.
## Оценка «4»: уровень стажёра/джуна
Весь уровень 3 **и все** пункты ниже:
- Добавлены crop 128×128 и optimized WebP по контракту, параметры качества/удаления метаданных описаны. Невалидные/огромные изображения ограничены по bytes, пикселям, времени и памяти worker.
- Разрыв между PostgreSQL и broker закрыт transactional outbox или другой доказанной схемой восстановления намерения. Внедрён crash между COMMIT и publish; после восстановления задача доходит до ready без ручного создания нового задания.
- Retry использует backoff с jitter, различает постоянные/временные ошибки; есть DLQ/реестр failed и безопасный явный replay. «Повторить» не означает безусловно исполнить дубль уже успешной job.
- Проверена остановка worker после записи объекта, но до ack; повтор завершает тот же результат. Есть timeout/lease для зависшего processing, ограниченная конкуренция/prefetch и очистка staging после безопасной фиксации.
## Оценка «5»: исследование и доказанный результат
Весь уровень 4 **и один законченный трек на выбор**. Приведите гипотезу, повторяемую методику, результаты и ограничения.
**Трек 1. Очередь под перегрузкой.** Сравните ≥3 настройки concurrency/prefetch на разных размерах изображений. Измерьте throughput, queue lag, p95 времени до ready и RSS; покажите backpressure и ограничение количества принятой работы. Докажите восстановление после перегрузки, объясните компромисс latency/ресурсы.
**Трек 2. Доказанная идемпотентность pipeline.** Постройте versioned pipeline и автоматический fault-injection для дубля, reorder, crash до/после side effect и устаревшей job после новой версии. Зафиксируйте инварианты, покажите отсутствие потери подтверждённых задач и перезаписи нового результата старым. Обсудите почему это не exactly-once доставка.
## Как начать и проверить
```sh ```sh
# Работает прямо в skeleton, Python 3.9+; приложение не запускается: cp .env.example .env
make check
# После реализации запустите свой сервер/стенд по REPORT.md. # Заполните только секреты; S3_ACCESS_KEY_ID и S3_APP_USER должны совпадать.
# По умолчанию тест использует http://localhost:8080. python3 - <<'PY'
make test from pathlib import Path
``` import secrets
p = Path(".env")
text = p.read_text()
for key in ("POSTGRES_PASSWORD", "S3_ROOT_ACCESS_KEY", "S3_ROOT_SECRET_KEY",
"S3_SECRET_ACCESS_KEY", "RABBITMQ_PASSWORD"):
text = "\n".join(
f"{key}={secrets.token_urlsafe(24)}" if line.startswith(key + "=") else line
for line in text.splitlines()
) + "\n"
p.write_text(text)
PY
`.env.example`, `compose.example.yaml` и `contracts/infrastructure.json` описывают настройки и роли компонентов. **compose.example.yaml — список ролей с пустым services, не запускаемый стенд.** Реализуйте свой `compose.yaml`/`stack.yaml`, закрепите версии образов, заполните локальный `.env` и опишите bootstrap/миграции. Все зависимости поднимаются из этого репозитория; ссылка «у меня БД уже установлена» не заменяет воспроизводимость.
```sh
# Compose-вариант после реализации:
docker compose --env-file .env config --quiet
docker compose --env-file .env up -d --build docker compose --env-file .env up -d --build
# Дождитесь /health/ready; затем make test. # Дождитесь healthy у api/worker и 200 от readiness.
# Для Swarm укажите эквивалентные build/push/deploy-команды в REPORT.md. curl -fsS http://localhost:8080/health/ready
make test BASE_URL=http://localhost:8080
``` ```
`migrate` применяет миграции 0001–0005, `s3-init` идемпотентно создаёт private
bucket и scoped-пользователя. Повторный `up -d` не удаляет volumes. Полная
остановка без потери данных — `docker compose down`; намеренное удаление
учебных данных — `docker compose down -v` (разрушительно).
Примеры запросов доступны в `examples/requests.http`, примеры JSON — в `contracts/examples.json`. Для полной проверки схемы (по желанию локально; CI делает её автоматически): Если Docker не может скачать базовые образы из-за DNS, соберите приложение на
хосте и запустите без сборки:
```sh ```sh
python3 -m venv .venv DOCKER_BUILDKIT=1 docker build --network=host -t mtusi2026lab-api:local .
.venv/bin/python -m pip install -r requirements-dev.txt docker compose --env-file .env up -d --no-build
make validate PYTHON=.venv/bin/python
``` ```
Smoke не выставляет оценку автоматически. Проверку сохранности/отказов запускайте явно по [tests/README.md](tests/README.md), остальные доказательства приведите в отчёте. ## Как устроен путь файла
## Сценарий защиты 1. **API / upload.** Проверяются членство в чате, имя и `Content-Type`, затем
тело потоково копируется в temp-файл (буфер 32 KiB) с SHA-256. Байтовый
лимит 10 MiB и пиксельный лимит 40 MP проверяются до выделения памяти под
декодирование. Валидный оригинал кладётся в `staging/{chat}/{attachment}`.
2. **Одна транзакция.** В PostgreSQL вставляются `attachments` (`queued`) и
`attachment_jobs` (`published_at IS NULL`). Только после COMMIT клиент
получает `202`; в этот момент durable уже и байты, и намерение.
3. **Outbox → RabbitMQ.** Каждая API-реплика (и worker) раз в 250 мс публикует
непубликованные строки в quorum queue с persistent delivery и ждёт publisher
confirm. Затем `published_at` выставляется compare-and-set. Падение между
COMMIT и publish восстанавливается следующим проходом.
4. **Worker.** Сообщение содержит только `job_id`, `attachment_id`, версию и
попытку — без байтов, ключей и credentials. Worker берёт DB-lease, заново
читает доверенные метаданные, сверяет размер и SHA-256 staging-объекта.
5. **Эффект.** Оригинал копируется без изменений в `attachments/...`; рядом
пишутся детерминированные варианты `thumbnail`, `crop`, `optimized`.
`CompleteJob` атомарно ставит `ready` только для текущей попытки/версии.
Лишь после этого staging удаляется, а сообщение подтверждается (ack).
6. **Клиент** опрашивает `GET /api/v1/attachments/{id}` до `ready`; content
отдаётся только для `ready`, иначе `409`.
1. Запустить make test: API возвращает 202, GET доходит до ready, оригинал и настоящий JPEG thumbnail проверяются. ### Варианты
2. Остановить worker своим скриптом, загрузить файл, убедиться, что content отвечает 409, а другие сообщения продолжают работать.
3. Перезапустить API/broker с сохранением volumes, запустить worker; та же запись должна стать ready. Зафиксировать порядок действий и ID без токенов.
4. Для 4: воспроизвести crash после side effect до ack, разрыв COMMIT/publish и replay failed; для 5 — выбранный эксперимент.
## Что должно быть в PR | Имя | Формат | Правило |
| --- | --- | --- |
| `thumbnail` | JPEG q=85 | fit в 256×256, белый фон, без upscale |
| `crop` | JPEG q=85 | центральный квадрат 128×128 |
| `optimized` | WebP q=82 | fit в 1280×1280, без upscale |
Worker, брокер в общем стенде, схема jobs, migrations/outbox по уровню, конфигурация retry, тесты сбоев и отчёт. EXIF orientation применяется до преобразований. Оригинал сохраняется
byte-for-byte и сохраняет исходный SHA-256. Метаданные вариантов не
копируются. Все операции детерминированы: повторная доставка перезаписывает
те же ключи теми же байтами.
Заполните `REPORT.md` и [шаблон PR](.github/pull_request_template.md), укажите целевую оценку/трек. Не меняйте обязательный контракт и публичные tests ради зелёного результата. Сохраняйте возможности предыдущих лабораторных в пределах [объявленных переходов](COURSE.md). ### Надёжность и повторы
## Вопросы на защите - Очередь — RabbitMQ quorum queue, сообщения persistent, publisher confirms.
- Ack только после durable effect (или осознанного no-op). `ErrRequeue`
используется, если не удалось записать состояние в PostgreSQL.
- Ошибки делятся на постоянные и временные. Временные повторяются с
экспоненциальным backoff ±50% jitter, до `JOB_MAX_ATTEMPTS`.
- Постоянная ошибка даёт `failed` с безопасным `error_code`; реестр failed
виден через API, явный replay — `worker -replay <attachment_uuid>`.
- Просроченный lease (`processing` без живого worker) возвращается в очередь
sweeper-ом. Повтор не создаёт второй набор вариантов.
- `MAX_PENDING_JOBS` — приблизительный admission cap: при переполнении upload
получает `503` и `Retry-After: 5`, не доходя до чтения тела.
- Reconciler удаляет только брошенные staging/pending-объекты после grace
period; готовые вложения не трогает.
- Чем publisher confirm отличается от consumer ack? ## Конфигурация
- Что именно уже сохранено в момент 202?
- Почему повторная доставка нормальна и где находится ключ идемпотентности job?
- Как отличить зависший worker от долго работающего и не запустить два неконтролируемых дубля?
## Первичные материалы Все переменные и значения по умолчанию перечислены в `.env.example`. Главные:
- [RabbitMQ reliability](https://www.rabbitmq.com/docs/reliability) | Переменная | Смысл |
- [Publisher confirms and acknowledgements](https://www.rabbitmq.com/docs/confirms) | --- | --- |
- [Redis Streams](https://redis.io/docs/latest/develop/data-types/streams/) | `MAX_UPLOAD_BYTES` | байтовый лимит, не больше 10 MiB |
- [NATS JetStream](https://docs.nats.io/nats-concepts/jetstream) | `MAX_IMAGE_PIXELS` | пиксельный лимит, не больше 40 MP |
| `IMAGE_DECODE_MEMORY_MB` | общий budget декодирования на процесс |
| `WORKER_CONCURRENCY` / `WORKER_PREFETCH` | параллелизм handler-ов / unacked сообщений |
| `MAX_PENDING_JOBS` | admission cap очереди (0 — выключен) |
| `JOB_MAX_ATTEMPTS`, `JOB_TIMEOUT_SECONDS`, `JOB_LEASE_SECONDS` | retry/lease policy |
| `JOB_RETRY_BASE_SECONDS`, `JOB_RETRY_MAX_SECONDS` | backoff |
| `WORKER_FAULT_MODE`, `WORKER_CRASH_AFTER_EFFECT`, `WORKER_CRASH_AFTER_CLAIM` | детерминированные хуки для доказательств |
## Проверки и доказательства
```sh
make check
make test BASE_URL=http://localhost:8080 # публичный smoke
python3 tests/custom/attachments.py --base-url ... # 413/415/404/400, без 5xx
python3 tests/custom/concurrent_uploads.py --base-url ... # конкурентные upload
# attachment_flow.py — двухшаговый helper для verify-attachment-failover.sh,
# а не самостоятельный тест: ему нужны --action upload|download и --state.
python3 scripts/lab5_resilience.py --base-url ... # 6 сценариев уровня 4
python3 scripts/track1_experiment.py # трек 1 уровня 5
```
`scripts/lab5_resilience.py` останавливает worker и RabbitMQ, внедряет crash
до/после эффекта, проверяет retry, replay и lease recovery. Результаты —
`evidence/lab5-grade4.{json,md}`. Persistence/failover проверяются публичными
`make test-persistence` / `make test-failover` с `scripts/restart.sh` и
`scripts/stop-one.sh`; см. `evidence/lab5-persistence*` и `evidence/lab5-failover.txt`.
Трек 1 запускает матрицу `serial/balanced/eager` на трёх размерах:
`small_many` (192×128), `exactly_right` (1280×1280, optimized WebP без scaling)
и `huge_few` (2400×1800). Метрики: throughput, queue lag, p95 time-to-ready,
RSS и разложение времени на decode / scale / JPEG / WebP. Результаты и графики —
`evidence/track1.{json,html,md}`, разбор — в [REPORT.md](REPORT.md).
```sh
# Полный прогон: 30 small + 18 exactly_right + 12 huge на профиль.
make track1 BASE_URL=http://localhost:8080
# То же без make.
python3 scripts/track1_experiment.py --base-url http://localhost:8080
# Быстрая проверка механики и/или свои лимиты worker-а.
python3 scripts/track1_experiment.py --small-jobs 5 --exactly-jobs 3 --huge-jobs 3 --skip-backpressure
TRACK1_WORKER_CPU_LIMIT=2.0 TRACK1_WORKER_MEM_LIMIT=3G \
python3 scripts/track1_experiment.py --base-url http://localhost:8080
```
Эксперимент пересоздаёт worker с `WORKER_CONCURRENCY`/`WORKER_PREFETCH`,
поднимает memory limit до 2 GiB (чтобы `eager` не падал по OOM), а в `finally`
возвращает конфигурацию из `.env`. Он управляет только локальным Compose-стендом.
Кратко: на 192×128 eager дал 15.58 jobs/s против 6.84 у serial (2.3×) при RSS
28→32 MiB. На 1280×1280 и 2400×1800 все профили дали ~0.44–0.50 jobs/s:
worker ограничен 1 CPU, а encode share составил 82–83% и 61–64% — сжатие, а не
только scaling, задаёт нижнюю границу. RSS вырос до 818–950 MiB, queue lag p95
упал, но ready p95 остался ~26–36 s. При `MAX_PENDING_JOBS=3` API вернул
`503` + `Retry-After: 5`; после старта worker принятые задания стали ready за
7.65 s.
## Ограничения
Стенд учебный и однокомпонентный по каждому сервису: RabbitMQ и PostgreSQL
запущены в одном экземпляре, поэтому отказ их контейнера проверяется
перезапуском с volume, а не HA-кластером. `MAX_PENDING_JOBS` считается по
каждой API-реплике отдельно, это приблизительный, а не глобальный лимит.
Измерения RSS на Docker Desktop/в CI могут отличаться от bare-metal. Секреты,
tokens и содержимое пользовательских файлов в evidence не сохраняются.
+215 -17
View File
@@ -1,37 +1,235 @@
# Отчёт — лабораторная 5 # Отчёт — лабораторная 5
- Автор, группа: TODO - Автор, группа: Булла Максим, БПИ2301
- Целевая оценка; выбранный трек 5, если нужен: TODO - Целевая оценка: **5**, трек 1 («Очередь под перегрузкой»)
- Commit/PR: TODO после публикации - Commit/PR: ``186fc95``
- Использованные источники, библиотеки и помощь ИИ: TODO - Использованные источники и библиотеки: RabbitMQ quorum queues и publisher
confirms; `amqp091-go`, `gen2brain/webp`, `golang.org/x/image/draw`,
`golang.org/x/sync/semaphore`; стандартные `image/png`, `image/jpeg`.
Генерация отчёта и графиков — Python standard library. DeepSeek V4.1 Flash
использовался для реализации, ревью кода и подготовки текста; все
приведённые результаты получены запуском команд на этом стенде.
## Запуск из чистого clone ## Запуск из чистого clone
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных. Требования: Docker Engine + Compose v2, Python 3.9+. Точные версии образов
закреплены в `compose.yaml` (PostgreSQL 16.10, RabbitMQ 4.2, RustFS 1.0.4,
Redis 7.4.7, HAProxy 3.2.6); приложение собирается из `golang:1.27.1-alpine`.
```sh
cp .env.example .env
# Заполнить POSTGRES_PASSWORD, S3_ROOT_ACCESS_KEY, S3_ROOT_SECRET_KEY,
# S3_SECRET_ACCESS_KEY, RABBITMQ_PASSWORD. S3_ACCESS_KEY_ID и S3_APP_USER
# обязаны совпадать: RustFS использует имя пользователя как access key.
docker compose --env-file .env up -d --build
curl -fsS http://localhost:8080/health/ready
make test BASE_URL=http://localhost:8080
```
`migrate` применяет миграции 0001–0005; `s3-init` создаёт приватный bucket и
scoped-пользователя. Оператору не нужно вручную создавать очередь: worker и
publisher объявляют durable quorum queue `attachment.process.v1`.
Безопасная остановка: `docker compose down` (volumes сохраняются).
Намеренное удаление учебных данных: `docker compose down -v`. Секреты в git не
попадают; `.env` находится в `.gitignore`.
## Архитектура и решения ## Архитектура и решения
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab4, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации. ```
client ──HTTP──> API ──stream──> staging/{chat}/{attachment} (S3)
│
└─ ONE TX: attachments(queued) + attachment_jobs(outbox)
│
outbox loop │ publisher confirm
v
RabbitMQ quorum queue
│ at-least-once
v
worker
DB lease ─> read metadata ─> verify staging
│
├─> attachments/{chat}/{attachment} (original, byte-equal)
├─> .../thumbnail (JPEG 256 fit)
├─> .../crop (JPEG 128 square)
└─> .../optimized (WebP 1280 fit)
│
└─ CompleteJob (attempt/version CAS) ─> ready
│
ack + delete staging
```
**Источник истины — PostgreSQL.** Сообщение RabbitMQ — только идентификатор и
снимок намерения; worker перечитывает attachment из БД. Поэтому подмена
сообщения не даёт доступа к чужим байтам, а повторная доставка безопасна.
**Граница 202.** До ответа выполнены: авторизация, валидация заголовка и
лимитов, потоковая запись во временный файл, SHA-256, PUT в durable staging и
одна транзакция, создающая attachment + job. Ответ `202` означает «байты и
намерение сохранены», а не «resize выполнен».
**Outbox.** `attachment_jobs.published_at IS NULL` — committed intent без
publisher confirm. Outbox-цикл работает в каждой API-реплике и в worker; дубли
публикации схлопываются DB-lease. Это закрывает окно «COMMIT прошёл, publish
не прошёл».
**Идемпотентность эффекта.** `ClaimJob` выдаёт lease только одной попытке.
`CompleteJob` обновляет строку при совпадении `job_id + attempt + processing` и
проверяет `pipeline_version`; устаревшая попытка не может перезаписать новый
результат. Ключи вариантов детерминированы, повтор пишет те же байты.
**Retry.** Постоянные ошибки (`invalid_image`, `staging_corrupt`,
`staging_missing`) сразу дают `failed`. Временные (`storage_unavailable`,
`processing_timeout`, `processing_failed`) повторяются с экспоненциальным
backoff ±50% jitter до `JOB_MAX_ATTEMPTS`. После исчерпания — `failed` с
`retry_exhausted`. Явный replay: `docker compose exec worker /usr/local/bin/worker -replay <uuid>`.
**Память и лимиты.** Upload не использует `io.ReadAll`: тело копируется
буфером 32 KiB. API и worker декодируют изображение только после проверки
`width*height`, под общим semaphore-budget `IMAGE_DECODE_MEMORY_MB`. Байтовый
лимит 10 MiB и пиксельный 40 MP — жёсткие сверху, конфигурация может только
ужесточить их.
**Reconciler.** Удаляет только брошенные `pending`-строки и объекты без
метаданных после grace period; `ready` не трогает. Это не часть happy path, а
страховка от падений между PUT и COMMIT.
## Доказательства критериев ## Доказательства критериев
Все команды выполнялись на локальном Compose-стенде; из sandbox-контейнера
использовался адрес Docker host `http://172.18.0.1:8080`, для обычного хоста —
`http://localhost:8080`. Токены и секреты не сохранены.
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства | | Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- | | --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO | | Публичный smoke (уровень 3) | `make test` | `PASS: lab5 public API scenario`; upload 202 → ready, оригинал и JPEG thumbnail проверены | `evidence/lab5-smoke.txt` |
| Остальные пункты уровня 3 | TODO | TODO | TODO | | Upload 202 только после durable приёма | `scripts/lab5_resilience.py` сценарий `worker_stopped` | worker остановлен: `202 queued`, content `409`, после старта worker — `ready`; queue 2.316 s | `evidence/lab5-grade4.json` |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO | | Отдельные broker и worker | `docker compose ps` | контейнеры `rabbitmq`, `worker`, два `api`, `postgres`, `rustfs`, `reconciler` | `evidence/lab5-persistence-containers.txt` |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO | | Остановка API/broker не теряет принятое | `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | PASS; все контейнеры перезапущены с новыми `StartedAt`, volumes сохранены | `evidence/lab5-persistence.txt`, `evidence/lab5-persistence-containers.txt` |
| Broker outage / outbox | сценарий `broker_outage` | 202 при выключенном RabbitMQ, `published_at IS NULL`, readiness 200; после старта broker job опубликован и ready, attempt 1 | `evidence/lab5-grade4.json` |
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены. | Retry с backoff и jitter | сценарий `transient_retry` | первая попытка failed, retry через 2.25 s, attempt 2 → ready | `evidence/lab5-grade4.md` |
| Permanent failure + replay | сценарий `permanent_failure_replay` | `invalid_image`, content/link 409, replay_count 1, после replay ready, 3 варианта | `evidence/lab5-grade4.json` |
| Crash после эффекта до ack | сценарий `crash_after_effect` | worker restart 0→1, повтор завершает ту же job, 3 варианта | `evidence/lab5-grade4.md` |
| Зависший processing / lease | сценарий `lease_recovery` | one-shot exit 10, статус processing, sweeper вернул job, attempt 2 → ready | `evidence/lab5-grade4.md` |
| Ограниченная конкуренция/prefetch | конфигурация worker | `WORKER_CONCURRENCY`, `WORKER_PREFETCH`, `ch.Qos(prefetch)`; матрица трека 1 | `REPORT.md`, `evidence/track1.json` |
| Очистка staging после фиксации | сценарии `worker_stopped`, `crash_after_effect` | `staging_cleared: true`; reconciler удаляет только брошенное | `evidence/lab5-grade4.json`, `evidence/reconcile.txt` |
| Негативные случаи и лимиты | `python3 tests/custom/attachments.py` | 413 по bytes и по 42 MP, 415 corrupt/spoof, 404/400, нет 5xx | `evidence/lab5-negative-cases.txt` |
| Failover API | `make test-failover ACTION_SCRIPT=scripts/stop-one.sh` | 118 успешных записей, 1 переходная ошибка, восстановление 2.659 s; остановлена именно `api-2` | `evidence/lab5-failover.txt` |
| Трек 1, уровень 5 | `python3 scripts/track1_experiment.py` | 3 профиля × 3 размера; eager/small 15.58 jobs/s против serial 6.84; exactly_right/huge_few ~0.44–0.50 jobs/s, encode share 83% против 61–64%; RSS до 950 MiB; 503 + Retry-After 5, recovery 7.653 s | `evidence/track1.json`, `evidence/track1.html` |
## Эксперименты и отказы ## Эксперименты и отказы
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения. | Воздействие | Что ожидали | Что наблюдали | Время восстановления / потеря данных |
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- | | --- | --- | --- | --- |
| TODO | TODO | TODO | TODO | | Worker остановлен после upload | 202, content 409, job ждёт | 202 queued; после старта ready, attempt 1 | 2.316 s до ready, потерь нет |
| RabbitMQ остановлен между COMMIT и publish | 202 и восстановление outbox | 202, `published_at IS NULL`, readiness 200; после старта job ready | 9.005 s до ready, потерь нет |
| Worker `exit 9` после записи вариантов до ack | redelivery завершает ту же job | restart 0→1, 3 объекта, ready | без потери и без второго набора |
| Worker `exit 10` после claim | lease истекает, sweeper возвращает job | processing → recovered → attempt 2 → ready | 8 s (lease sweep), потерь нет |
| Временная ошибка storage | retry с jitter | attempt 1 failed, attempt 2 ready | ~2.25 s backoff |
| Постоянная ошибка `invalid_image` | failed + безопасный replay | content/link 409, replay_count 1, ready | ручной replay, без дубля |
| Остановка одной API-реплики | сессия и подтверждённые данные живы | 118 успешных записей, 1 ошибка, recovery 2.659 s | потерь нет |
| Перезапуск всех контейнеров с volumes | данные и сессия сохраняются | persistence PASS, новые `StartedAt` | потерь нет |
### Ограничения измерения отказов
`restart.sh` перезапускает контейнеры, а не эмулирует падение ядра или потеру
диска. RabbitMQ и PostgreSQL — одиночные экземпляры, поэтому это проверка
durability на volume, а не HA-кластера. Failover измеряет один API-инстанс за
HAProxy; отказ БД или брокера в этом сценарии не имитируется. Времена включают
опрос готовности и запуск контейнера, поэтому это верхняя оценка, не SLA.
## Эксперимент трека 1 (уровень 5)
<!-- TRACK1:START -->
### Гипотеза
Стоимость задания = decode + scaling + encode. `exactly_right` (1280×1280) не
масштабирует optimized WebP и показывает нижнюю границу стоимости сжатия;
`huge_few` добавляет downscale. Ожидаем: на тяжёлых изображениях concurrency
снижает queue lag, но не throughput при 1 CPU, а RSS растёт.
### Методика
- Профили: `serial` 1/1, `balanced` 4/4, `eager` 8/32 (concurrency/prefetch).
- Сценарии: `small_many` 30×192×128; `exactly_right` 18×1280×1280 (optimized
WebP без scaling); `huge_few` 12×2400×1800 (optimized → 1280×960).
- 9 ячеек; worker пересоздаётся на профиль, изображения загружаются параллельно,
все задания доводятся до терминального состояния.
- Queue lag, time-to-ready и processing — из `attachment_jobs`; throughput =
ready / (последний `finished_at` − первый `created_at`).
- Decode/scale/JPEG/WebP — из worker-лога; encode share =
`encode / (decode + scale + encode)`. Это wall-clock под данным профилем.
- RSS — `VmRSS` из `/proc/1/status` примерно каждые 100 мс.
- Worker 1 CPU, memory limit 2 GiB (в `.env` 1 GiB), чтобы `eager` не падал по
OOM; зафиксировано в JSON.
- Backpressure: `MAX_PENDING_JOBS=3`, worker остановлен, upload-ы до `503`, затем
recovery. Cap выбран намеренно; важны `Retry-After` и отсутствие потерь.
Команда:
```sh
make track1 BASE_URL=http://localhost:8080
```
Сырые данные: `evidence/track1.json`; графики: `evidence/track1.html`;
краткая таблица: `evidence/track1.md`.
### Результаты
| Профиль | Сценарий | Jobs/s | Queue lag p95, ms | Ready p95, ms | Processing p95, ms | Encode p95, ms | Encode share, % | RSS peak, MiB | Max parallel |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| serial | small_many | 6.837 | 3 256 | 3 338 | 170 | 80 | 83.8 | 28.2 | 1 |
| serial | exactly_right | 0.475 | 33 591 | 35 714 | 2 180 | 1 722 | 82.0 | 174.3 | 1 |
| serial | huge_few | 0.436 | 23 668 | 25 896 | 2 359 | 1 508 | 63.8 | 212.9 | 1 |
| balanced | small_many | 13.694 | 1 147 | 1 274 | 396 | 90 | 83.8 | 28.9 | 4 |
| balanced | exactly_right | 0.495 | 31 510 | 35 764 | 8 174 | 6 562 | 83.1 | 433.8 | 4 |
| balanced | huge_few | 0.441 | 18 161 | 26 576 | 9 046 | 5 606 | 61.4 | 756.4 | 4 |
| eager | small_many | 15.576 | 924 | 1 166 | 410 | 95 | 79.9 | 32.1 | 8 |
| eager | exactly_right | 0.495 | 31 735 | 35 797 | 16 363 | 13 330 | 83.1 | 818.4 | 8 |
| eager | huge_few | 0.443 | 17 648 | 26 641 | 17 678 | 11 124 | 61.4 | 949.6 | 8 |
Backpressure: cap `3` задан намеренно; первый `503` пришёл с `Retry-After: 5`.
После старта worker принятые задания стали `ready` за **7.653 s** (включая
запуск контейнера и healthcheck), probe upload получил `202` и дошёл до `ready`.
### Выводы
1. **Compression, not scaling, is the floor.** `exactly_right` даёт 82–83%
encode share и ~0.48 jobs/s; `huge_few` — 61–64% и ~0.44 jobs/s. Убрать
downscale недостаточно, чтобы 1280×1280 стал дешёвым: доминирует WebP.
На `huge_few` scale p95 выше (balanced: 3.4 s против 1.5 s), но optimized
output меньше (1.23 MP против 1.64 MP), поэтому throughput почти совпадает.
2. **Маленькие изображения масштабируются concurrency.** 6.84 → 13.69 →
15.58 jobs/s, RSS 28 → 32 MiB.
3. **Тяжёлые изображения CPU-bound.** ~0.44–0.50 jobs/s у всех профилей;
queue lag падает, ready p95 держится ~26–36 s, RSS растёт 174 → 818 MiB
(`exactly_right`) и 213 → 950 MiB (`huge_few`). Concurrency покупает latency
ценой памяти, а не throughput.
4. **Backpressure.** `503` + `Retry-After: 5`; принятые задания восстановились
за 7.65 s.
5. **Практический выбор.** `balanced` для тяжёлых, `eager` для лёгких.
### Ограничения
- `exactly_right` кодирует 1.64 MP WebP, `huge_few` после downscale — ~1.23 MP:
сценарий убирает scaling, но не уравнивает объём encode.
- PNG-фикстуры синтетические; один прогон на ячейку, p95 по 12–18 заданиям
чувствителен к выбросу.
- Время recovery включает `docker compose start` и healthcheck.
<!-- TRACK1:END -->
## Ограничения и следующие шаги ## Ограничения и следующие шаги
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза. - Стенд учебный: один RabbitMQ, один PostgreSQL, один worker; горизонтальное
масштабирование worker проверяется конфигурацией, но не HA-кластером.
- `MAX_PENDING_JOBS` считается по репликам приблизительно; глобальный лимит
потребовал бы общего счётчика (например, Redis) и отдельной транзакционной
семантики.
- Replay намеренно ручной и явный: автоматический replay постоянных ошибок
маскировал бы дефекты входных данных.
- WebP-энкодер `gen2brain/webp` выбран без CGO; качество и скорость могут
отличаться от libwebp CLI, но результаты детерминированы на стенде.
- Измерения RSS зависят от cgroup/докер-обвязки; для production нужен
длительный профиль и p99, а не короткий учебный прогон.
- Следующий шаг: versioned pipeline и автоматический fault-injection для
трека 2, если потребуется доказывать более сильную идемпотентность.
Executable
+59
View File
@@ -0,0 +1,59 @@
#:schema https://json.schemastore.org/any.json
env_files = []
root = "."
testdata_dir = "testdata"
tmp_dir = "tmp"
[build]
args_bin = []
bin = "./tmp/main"
cmd = "go build -o ./tmp/main ./cmd/server/"
delay = 1000
entrypoint = ["./tmp/main"]
exclude_dir = ["assets", "tmp", "vendor", "testdata"]
exclude_file = []
exclude_regex = ["_test.go"]
exclude_unchanged = false
follow_symlink = false
full_bin = ""
ignore_dangerous_root_dir = false
include_dir = []
include_ext = ["go", "tpl", "tmpl", "html"]
include_file = []
kill_delay = "0s"
log = "build-errors.log"
poll = false
poll_interval = 0
post_cmd = []
pre_cmd = []
rerun = false
rerun_delay = 500
send_interrupt = false
stop_on_error = false
[color]
app = ""
build = "yellow"
main = "magenta"
mode = ""
runner = "green"
watcher = "cyan"
[log]
main_only = false
silent = false
time = false
[misc]
clean_on_exit = false
[proxy]
app_port = 0
app_start_timeout = 0
enabled = false
proxy_port = 0
[screen]
clear_on_rebuild = false
keep_scroll = true
+105
View File
@@ -0,0 +1,105 @@
// Command reconciler periodically removes abandoned attachment objects and
// pending metadata rows. It is safe to run concurrently with the API and safe
// to re-run; see internal/reconcile for the grace-period rules.
package main
import (
"context"
"flag"
"log"
"log/slog"
"os/signal"
"syscall"
"time"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/config"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/database"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/objectstore"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/reconcile"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/store"
)
func main() {
once := flag.Bool("once", false, "run one reconciliation pass and exit")
dryRun := flag.Bool("dry-run", false, "log what would be removed without changing either store")
grace := flag.Duration("grace", 0, "override RECONCILE_GRACE_SECONDS for this run")
interval := flag.Duration("interval", 0, "override RECONCILE_INTERVAL_SECONDS for this run")
flag.Parse()
cfg, err := config.Load()
if err != nil {
log.Fatalf("configuration: %v", err)
}
if *grace > 0 {
cfg.ReconcileGrace = *grace
}
if *interval > 0 {
cfg.ReconcileInterval = *interval
}
rootCtx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
pool, err := database.Connect(rootCtx, cfg.DatabaseURL, cfg.DBMaxConns, cfg.StartupWait)
if err != nil {
log.Fatalf("database: %v", err)
}
defer pool.Close()
objects, err := objectstore.New(objectstore.Config{
Endpoint: cfg.S3Endpoint,
Region: cfg.S3Region,
Bucket: cfg.S3Bucket,
AccessKey: cfg.S3AccessKey,
SecretKey: cfg.S3SecretKey,
})
if err != nil {
log.Fatalf("object storage: %v", err)
}
worker := reconcile.New(
store.New(pool, cfg.DBTimeout, cfg.IdempotencyTTL),
objects,
cfg.ReconcileGrace,
slog.Default(),
)
if *once {
stats, runErr := worker.RunOnce(rootCtx, *dryRun)
logStats(stats, *dryRun)
if runErr != nil {
log.Fatalf("reconcile: %v", runErr)
}
return
}
for {
stats, runErr := worker.RunOnce(rootCtx, *dryRun)
if runErr != nil {
slog.Error("reconcile pass failed", "err", runErr)
} else {
logStats(stats, *dryRun)
}
// A timer avoids the ticker behaviour where missed ticks accumulate:
// the next pass starts one full interval after this one finished.
timer := time.NewTimer(cfg.ReconcileInterval)
select {
case <-rootCtx.Done():
timer.Stop()
return
case <-timer.C:
}
}
}
func logStats(stats reconcile.Stats, dryRun bool) {
slog.Info("reconcile pass complete",
"dry_run", dryRun,
"stale_staging", stats.StaleStaging,
"orphan_objects", stats.OrphanObjects,
"deleted_staging", stats.DeletedStaging,
"deleted_objects", stats.DeletedObjects,
"skipped", stats.Skipped,
)
}
+91
View File
@@ -0,0 +1,91 @@
// Command server starts the messenger HTTP API, applies optional migrations,
// and shuts down gracefully on SIGINT/SIGTERM.
package main
import (
"context"
"log"
"net"
"os/signal"
"syscall"
"time"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/attachments"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/broker"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/config"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/database"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/health"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/outbox"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/session"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/store"
)
func main() {
cfg, err := config.Load()
if err != nil {
log.Fatalf("configuration: %v", err)
}
rootCtx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
pool, err := database.Connect(rootCtx, cfg.DatabaseURL, cfg.DBMaxConns, cfg.StartupWait)
if err != nil {
log.Fatalf("database: %v", err)
}
defer pool.Close()
if cfg.MigrateOnly || cfg.RunMigrations {
if err = database.Migrate(cfg.DatabaseURL); err != nil {
log.Fatalf("migrations: %v", err)
}
}
if cfg.MigrateOnly {
log.Printf("migrations applied; migrate-only process exiting")
return
}
sessions, err := session.New(cfg.CacheURL, cfg.SessionTTL, cfg.RedisTimeout)
if err != nil {
log.Fatalf("session store: %v", err)
}
defer sessions.Close()
attachments.CleanupStaleUploads(cfg.UploadTempDir, time.Hour)
st := store.New(pool, cfg.DBTimeout, cfg.IdempotencyTTL)
// AGENT: the outbox publisher runs in every API replica. Duplicate
// publication is harmless because workers claim by job_id; the important
// property is that a committed intent is retried until a confirm.
publisher := broker.NewPublisher(cfg.BrokerURL, cfg.JobQueue)
defer publisher.Close()
go outbox.New(st, publisher, nil).Run(rootCtx)
app, err := internal.New(cfg, st, sessions)
if err != nil {
log.Fatalf("application: %v", err)
}
addr := net.JoinHostPort(cfg.Host, cfg.Port)
listenErr := make(chan error, 1)
go func() {
listenErr <- app.Listen(addr)
}()
select {
case err = <-listenErr:
if err != nil {
log.Fatalf("server: %v", err)
}
case <-rootCtx.Done():
}
log.Printf("shutting down")
health.SignalShutdown()
if err = app.ShutdownWithTimeout(10 * time.Second); err != nil {
log.Printf("shutdown: %v", err)
}
}
+104
View File
@@ -0,0 +1,104 @@
// Command worker consumes durable attachment jobs, writes variants and acks
// only after the ready snapshot commits. WORKER_ONLY=true skips Redis setup.
package main
import (
"context"
"flag"
"log"
"log/slog"
"os"
"os/signal"
"syscall"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/broker"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/config"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/database"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/objectstore"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/outbox"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/store"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/worker"
"github.com/google/uuid"
)
func main() {
replay := flag.String("replay", "", "explicitly requeue one failed attachment UUID and exit")
flag.Parse()
cfg, err := config.Load()
if err != nil {
log.Fatalf("configuration: %v", err)
}
rootCtx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
pool, err := database.Connect(rootCtx, cfg.DatabaseURL, cfg.DBMaxConns, cfg.StartupWait)
if err != nil {
log.Fatalf("database: %v", err)
}
defer pool.Close()
st := store.New(pool, cfg.DBTimeout, cfg.IdempotencyTTL)
if *replay != "" {
attachmentID, parseErr := uuid.Parse(*replay)
if parseErr != nil {
log.Fatalf("replay: invalid attachment UUID: %v", parseErr)
}
changed, replayErr := st.ReplayFailed(rootCtx, attachmentID)
if replayErr != nil {
log.Fatalf("replay: %v", replayErr)
}
if !changed {
log.Printf("replay: attachment %s is not failed; nothing changed", attachmentID)
return
}
// Publish immediately so a replay works even when no API replica is
// currently running its outbox loop. Duplicate publication is safe.
publisher := broker.NewPublisher(cfg.BrokerURL, cfg.JobQueue)
defer publisher.Close()
if replayErr = outbox.New(st, publisher, nil).PublishBatch(rootCtx); replayErr != nil {
log.Fatalf("replay publish: %v", replayErr)
}
log.Printf("replay: attachment %s requeued", attachmentID)
return
}
objects, err := objectstore.New(objectstore.Config{
Endpoint: cfg.S3Endpoint,
Region: cfg.S3Region,
Bucket: cfg.S3Bucket,
AccessKey: cfg.S3AccessKey,
SecretKey: cfg.S3SecretKey,
})
if err != nil {
log.Fatalf("object storage: %v", err)
}
logger := slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelInfo}))
slog.SetDefault(logger)
processor := worker.New(cfg, st, objects, logger)
go processor.StartLeaseSweeper(rootCtx)
// The worker also drains the outbox. This keeps retries/replay moving even
// if every API replica is down. Two publishers only create duplicates that
// the job lease collapses.
publisher := broker.NewPublisher(cfg.BrokerURL, cfg.JobQueue)
defer publisher.Close()
go outbox.New(st, publisher, logger).Run(rootCtx)
logger.Info("worker starting",
"queue", cfg.JobQueue,
"concurrency", cfg.WorkerConcurrency,
"prefetch", cfg.WorkerPrefetch,
"max_attempts", cfg.JobMaxAttempts,
"job_timeout", cfg.JobTimeout,
"fault_mode", cfg.WorkerFaultMode)
consumer := broker.NewConsumer(cfg.BrokerURL, cfg.JobQueue, logger)
if err = consumer.Run(rootCtx, cfg.WorkerConcurrency, cfg.WorkerPrefetch, processor.Handle); err != nil {
log.Fatalf("worker: %v", err)
}
logger.Info("worker stopped")
}
Executable
+57
View File
@@ -0,0 +1,57 @@
module gitflic.ru/project/bulla-max/mtusi2026lab
go 1.27.1
require (
github.com/aws/aws-sdk-go-v2 v1.47.5
github.com/aws/aws-sdk-go-v2/config v1.33.13
github.com/aws/aws-sdk-go-v2/credentials v1.20.13
github.com/aws/aws-sdk-go-v2/service/s3 v1.114.6
github.com/aws/smithy-go v1.28.7
github.com/gen2brain/webp v0.6.4
github.com/gofiber/fiber/v3 v3.5.0
github.com/golang-migrate/migrate/v4 v4.20.1
github.com/google/uuid v1.6.0
github.com/jackc/pgx/v5 v5.11.0
github.com/rabbitmq/amqp091-go v1.16.0
github.com/redis/go-redis/v9 v9.23.0
golang.org/x/crypto v0.58.0
golang.org/x/image v0.48.0
golang.org/x/sync v0.25.0
)
require (
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.25 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.20.5 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.8 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.8 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.5.8 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.23 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.11.9 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.8 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.20.8 // indirect
github.com/aws/aws-sdk-go-v2/service/signin v1.12.1 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.39.1 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.44.1 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.51.6 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/ebitengine/purego v0.10.1 // indirect
github.com/gofiber/schema v1.8.8 // indirect
github.com/gofiber/utils/v2 v2.6.1 // indirect
github.com/jackc/pgerrcode v0.0.0-20220416144525-469b46aa5efa // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/klauspost/compress v1.20.1 // indirect
github.com/mattn/go-colorable v0.1.15 // indirect
github.com/mattn/go-isatty v0.0.24 // indirect
github.com/molecule-man/go-brrr v1.1.1 // indirect
github.com/philhofer/fwd v1.2.0 // indirect
github.com/tinylib/msgp v1.6.5 // indirect
github.com/valyala/bytebufferpool v1.0.0 // indirect
github.com/valyala/fasthttp v1.74.0 // indirect
go.uber.org/atomic v1.12.0 // indirect
golang.org/x/net v0.61.0 // indirect
golang.org/x/sys v0.50.0 // indirect
golang.org/x/text v0.44.0 // indirect
)
Executable
+177
View File
@@ -0,0 +1,177 @@
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c h1:udKWzYgxTojEKWjV8V+WSxDXJ4NFATAsZjh8iIbsQIg=
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E=
github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERoyfY=
github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU=
github.com/aws/aws-sdk-go-v2 v1.47.5 h1:5yA+A+BiXbDq1ZLbYwPViwhEGDMdR0MEhL8Jh2LnLS4=
github.com/aws/aws-sdk-go-v2 v1.47.5/go.mod h1:iNIkogrRvFwm+1OQIUOP2lPBtlXZboOIUWiCa+pHEMw=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.25 h1:MW0Iql1SOTgQPj8mCxrU/eH/K/AIDWi08ItOsXaWRLU=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.25/go.mod h1:+ucVYmHYB3JqVueY0E1RHahdXAnKQbIHqybSbvOUbOM=
github.com/aws/aws-sdk-go-v2/config v1.33.13 h1:u52x6ISQOx172tpwM0VRxhJzEsTJYpMcEYUGM2A1wrw=
github.com/aws/aws-sdk-go-v2/config v1.33.13/go.mod h1:/ixtz6XPHb42zekjQ0LqD9XjTpTysNWhIGwufY1OzI4=
github.com/aws/aws-sdk-go-v2/credentials v1.20.13 h1:j4z41OtspmQ3ftQycaDunJ26b69a9SHfaSVUPqgRv8k=
github.com/aws/aws-sdk-go-v2/credentials v1.20.13/go.mod h1:SgPA1oS3b+dlMxZb618jDmGNscpXSCaxz15igS52OlE=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.20.5 h1:iXjy2SFhGYNB1jWZBhQqq8HbhsbPF23w+/zbMvPXsTg=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.20.5/go.mod h1:k0RUiBnyXU0AXRJ9tJmY2jQke6XUIX2PNVnWtvgaPtc=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.8 h1:gzIWc3DTPfPbWKrQnmL6BV6WD7NPD1htxzgdk/bQ/Z4=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.8/go.mod h1:vvUA5cDAZWGo4R14FZPZfXfJCmgsafOWEq78Kg0D+ug=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.8 h1:PWv+vG8aCLS8qd+HPdYhK7Vees90aKHWFFQ1AGYD/8E=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.8/go.mod h1:xxPZEpflYHJA12366ZGyq66z9dHPCoeCIu99Jck+MPw=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.5.8 h1:WOp5De9iWdEHDPh2Kb0tBP3vGBm/dVAuRGh4a4tqduo=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.5.8/go.mod h1:z8W8YgfhbWl62XqcTtK+CxPrE+iHGRVUOtrrbIOp7ZY=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.23 h1:MCIR6iZaCKO4UE0hUvoWWLI6JVjppopV64JfAJaMjng=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.23/go.mod h1:pphWFS+z0MmIPivj8c/6bATd1Z+Av7Ust75DPdEFsPA=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.11.9 h1:GACgECI422Xt5n4Pojr3O0aocPv8lfH4PJe2Aon5Wv8=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.11.9/go.mod h1:lPfnRkHpn1qWZJLbVEw10rIqSuCee1pNQTyGkUjAYh8=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.8 h1:+PMpWlJYOn2tm7sm6OZ/ZLd4jStl17T7wjNapd1Rczs=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.8/go.mod h1:cnG3xZLGQL4c/bt8pjdzlFtZsW7epBBuNS04a5xMjdo=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.20.8 h1:RS1Wt+K0r4X0ci9Vf4JYTUfHzsQihYA2LgGJeEneKpk=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.20.8/go.mod h1:adMnTy5kyoUW86jhXnGSVc/mvMQI4GZd1qYWMeaS8M8=
github.com/aws/aws-sdk-go-v2/service/s3 v1.114.6 h1:to7+Vr3rLkpYRwVKJ5QRriI0eE306vrVN/6TnVHgxvY=
github.com/aws/aws-sdk-go-v2/service/s3 v1.114.6/go.mod h1:Jc89ASa2k7HMAskj3bSnt8bf+k+p2IApzkoXnEjkgUw=
github.com/aws/aws-sdk-go-v2/service/signin v1.12.1 h1:WZLC+k/CZx11kZGQDqu9mpA8yhiXGX2I//jhuJE1FDo=
github.com/aws/aws-sdk-go-v2/service/signin v1.12.1/go.mod h1:wxAk0D2oVBSjuXEIzvcKxTZfM90zticdAbh/VD42L0Y=
github.com/aws/aws-sdk-go-v2/service/sso v1.39.1 h1:4VB/dnDo7pmXSLQMszSoBzWndENuc7mzwTpD6lQn77A=
github.com/aws/aws-sdk-go-v2/service/sso v1.39.1/go.mod h1:UGsEYgWz4Gi+46vDQ5zkFLqUgJziyOE2SXjZjgmfmHc=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.44.1 h1:w8yui2ZqdHmGwVBDGvazoIvKw/DnLENBLkYnN8q8iqo=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.44.1/go.mod h1:gJSdIY6YUzDvqQLcMtPVZgBRHrFwzHryvpE8Igzbxps=
github.com/aws/aws-sdk-go-v2/service/sts v1.51.6 h1:BVjKrtRUtSYKztBCCfE1oAB+d3ViKcW2cKZfserOqMQ=
github.com/aws/aws-sdk-go-v2/service/sts v1.51.6/go.mod h1:eSukFLeLkPMm+1OxDR+F0joauZdyBkF47Ow7MqjZiJw=
github.com/aws/smithy-go v1.28.7 h1:h/n1SHrwVtaDYLz8mR1/Kq4Xh4RaFbFW7bOYckg9mfE=
github.com/aws/smithy-go v1.28.7/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/containerd/errdefs v1.0.0 h1:tg5yIfIlQIrxYtu9ajqY42W3lpS19XqdxRQeEwYG8PI=
github.com/containerd/errdefs v1.0.0/go.mod h1:+YBYIdtsnF4Iw6nWZhJcqGSg/dwvV7tyJ/kCkyJ2k+M=
github.com/containerd/errdefs/pkg v0.3.0 h1:9IKJ06FvyNlexW690DXuQNx2KA2cUJXx151Xdx3ZPPE=
github.com/containerd/errdefs/pkg v0.3.0/go.mod h1:NJw6s9HwNuRhnjJhM7pylWwMyAkmCQvQ4GpJHEqRLVk=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dhui/dktest v0.4.6 h1:+DPKyScKSEp3VLtbMDHcUq6V5Lm5zfZZVb0Sk7Ahom4=
github.com/dhui/dktest v0.4.6/go.mod h1:JHTSYDtKkvFNFHJKqCzVzqXecyv+tKt8EzceOmQOgbU=
github.com/distribution/reference v0.6.0 h1:0IXCQ5g4/QMHHkarYzh5l+u8T3t73zM5QvfrDyIgxBk=
github.com/distribution/reference v0.6.0/go.mod h1:BbU0aIcezP1/5jX/8MP0YiH4SdvB5Y4f/wlDRiLyi3E=
github.com/docker/docker v28.5.2+incompatible h1:DBX0Y0zAjZbSrm1uzOkdr1onVghKaftjlSWt4AFexzM=
github.com/docker/docker v28.5.2+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
github.com/docker/go-connections v0.7.0 h1:6SsRfJddP22WMrCkj19x9WKjEDTB+ahsdiGYf0mN39c=
github.com/docker/go-connections v0.7.0/go.mod h1:no1qkHdjq7kLMGUXYAduOhYPSJxxvgWBh7ogVvptn3Q=
github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4=
github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
github.com/ebitengine/purego v0.10.1 h1:dewVBCBT2GaMu1SrNTYxQhgQBethzfhiwvZiLGP/qyY=
github.com/ebitengine/purego v0.10.1/go.mod h1:iIjxzd6CiRiOG0UyXP+V1+jWqUXVjPKLAI0mRfJZTmQ=
github.com/felixge/httpsnoop v1.1.0 h1:3YtUj32ZZkqZtt3sZZsClsymw/QDuVfpNhoA31zeORc=
github.com/felixge/httpsnoop v1.1.0/go.mod h1:Zqxgdd+1Rkcz8euOqdr7lqgCRJztwr5hp9vDSi5UZCE=
github.com/fxamacker/cbor/v2 v2.9.4 h1:xwjVlxEMR3S605oUlgBjKLTTeGFciYPGYCtF/35LKGo=
github.com/fxamacker/cbor/v2 v2.9.4/go.mod h1:vM4b+DJCtHn+zz7h3FFp/hDAI9WNWCsZj23V5ytsSxQ=
github.com/gen2brain/webp v0.6.4 h1:SUDdmxADOAiPQ+5ylNmuHhuYf2dOi0KgKZHL5vpVCNU=
github.com/gen2brain/webp v0.6.4/go.mod h1:iGWMaCSw7t3I/Cv9llzEKmpnR36S8lS8VL/ZVjxU0JE=
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/gofiber/fiber/v3 v3.5.0 h1:dk7TOUH6DXJGtOLsN2XEG+0ZML7cznzHILTVozbNEK8=
github.com/gofiber/fiber/v3 v3.5.0/go.mod h1:GOVDTW+gjJvfe0iJyVujbQ1Lnx+JUjFySJRI/9/xX/w=
github.com/gofiber/schema v1.8.8 h1:md5t65+Z9LttOuj7FdMcQ63D2+6GniT2ganLXwvTJM4=
github.com/gofiber/schema v1.8.8/go.mod h1:Hzf0L/fTPP1IwOAwKipKflm9C2722aebnSKJCMOgb1I=
github.com/gofiber/utils/v2 v2.6.1 h1:x3zsS/oUQz0ia2Zx8XXtG/7naR5a+/95FRO8mwQ5QQ4=
github.com/gofiber/utils/v2 v2.6.1/go.mod h1:tc9KqdRAQZRD/u0mN2d1ZHROGheSZv+0FcC9cg5OEdw=
github.com/golang-migrate/migrate/v4 v4.20.1 h1:2N/ToVTKrKl58ynBpgeVJ4In7VcLCjWTZtm4eP1LxhU=
github.com/golang-migrate/migrate/v4 v4.20.1/go.mod h1:DDPgKVb4ovSWc4FwSPfV2Uz1160f4XBiTHTrAJtljmM=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/jackc/pgerrcode v0.0.0-20220416144525-469b46aa5efa h1:s+4MhCQ6YrzisK6hFJUX53drDT4UsSW3DEhKn0ifuHw=
github.com/jackc/pgerrcode v0.0.0-20220416144525-469b46aa5efa/go.mod h1:a/s9Lp5W7n/DD0VrVoyJ00FbP2ytTPDVOivvn2bMlds=
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
github.com/jackc/pgx/v5 v5.11.0 h1:IzBBtyK9AHqf98cctWFifYSci2hgQR/cd56wB4p+ogg=
github.com/jackc/pgx/v5 v5.11.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/klauspost/compress v1.20.1 h1:T7kKElXUMXrUJ2E9QhQhxFtcK5rPyLdsGZvdbLMPdiQ=
github.com/klauspost/compress v1.20.1/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/klauspost/cpuid/v2 v2.4.0 h1:S6Hrbc7+ywsr0r+RLapfGBHfyefhCTwEh3A0tV913Dw=
github.com/klauspost/cpuid/v2 v2.4.0/go.mod h1:19jmZ9mjzoF//ddRSUsv0zfBTJWh3QJh9FNxZTMrGxU=
github.com/lib/pq v1.10.9 h1:YXG7RB+JIjhP29X+OtkiDnYaXQwpS4JEWq7dtCCRUEw=
github.com/lib/pq v1.10.9/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o=
github.com/mattn/go-colorable v0.1.15 h1:+u9SLTRGnXv73cEsnsmoZBom+dMU88B2M0aDcWy0/jY=
github.com/mattn/go-colorable v0.1.15/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8=
github.com/mattn/go-isatty v0.0.24 h1:tGZZoVgT/KiqK1c8ocVLeDS8BSWMRd47J3Lbz7vsReI=
github.com/mattn/go-isatty v0.0.24/go.mod h1:nMCL3Zebbrt45jsMDgnfIwz6ydEQApk5oEI3HqDio6A=
github.com/moby/docker-image-spec v1.3.1 h1:jMKff3w6PgbfSa69GfNg+zN/XLhfXJGnEx3Nl2EsFP0=
github.com/moby/docker-image-spec v1.3.1/go.mod h1:eKmb5VW8vQEh/BAr2yvVNvuiJuY6UIocYsFu/DxxRpo=
github.com/moby/moby/api v1.54.2 h1:wiat9QAhnDQjA7wk1kh/TqHz2I1uUA7M7t9SAl/JNXg=
github.com/moby/moby/api v1.54.2/go.mod h1:+RQ6wluLwtYaTd1WnPLykIDPekkuyD/ROWQClE83pzs=
github.com/moby/moby/client v0.4.1 h1:DMQgisVoMkmMs7fp3ROSdiBnoAu8+vo3GggFl06M/wY=
github.com/moby/moby/client v0.4.1/go.mod h1:z52C9O2POPOsnxZAy//WtKcQ32P+jT/NGeXu/7nfjGQ=
github.com/moby/term v0.5.2 h1:6qk3FJAFDs6i/q3W/pQ97SX192qKfZgGjCQqfCJkgzQ=
github.com/moby/term v0.5.2/go.mod h1:d3djjFCrjnB+fl8NJux+EJzu0msscUP+f8it8hPkFLc=
github.com/molecule-man/go-brrr v1.1.1 h1:KYwusQhjtV3smALnhmafZf4OBrh7vP4p8O47DyUmmuY=
github.com/molecule-man/go-brrr v1.1.1/go.mod h1:7ybW6/7gA3oKY45jOfVNjSJDtrr6ea4tzbsTkjmQDC4=
github.com/morikuni/aec v1.0.0 h1:nP9CBfwrvYnBRgY6qfDQkygYDmYwOilePFkwzv4dU8A=
github.com/morikuni/aec v1.0.0/go.mod h1:BbKIizmSmc5MMPqRYbxO4ZU0S0+P200+tUnFx7PXmsc=
github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U=
github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM=
github.com/opencontainers/image-spec v1.1.1 h1:y0fUlFfIZhPF1W537XOLg0/fcx6zcHCJwooC2xJA040=
github.com/opencontainers/image-spec v1.1.1/go.mod h1:qpqAh3Dmcf36wStyyWU+kCeDgrGnAve2nCC8+7h8Q0M=
github.com/philhofer/fwd v1.2.0 h1:e6DnBTl7vGY+Gz322/ASL4Gyp1FspeMvx1RNDoToZuM=
github.com/philhofer/fwd v1.2.0/go.mod h1:RqIHx9QI14HlwKwm98g9Re5prTQ6LdeRQn+gXJFxsJM=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/rabbitmq/amqp091-go v1.16.0 h1:1O8oQqnj9hLACoo6CVMgn57z2pLWXvgemhh4GsquV+Y=
github.com/rabbitmq/amqp091-go v1.16.0/go.mod h1:Hy4jKW5kQART1u+JkDTF9YYOQUHXqMuhrgxOEeS7G4o=
github.com/redis/go-redis/v9 v9.23.0 h1:/TTVdJa8BRNsybwwGGnmiYktHNlUXtbOQxlWpHBPL3s=
github.com/redis/go-redis/v9 v9.23.0/go.mod h1:EWP2UUk+XDaI1s5nQB5mablZN6vccXlmBQTyfnF87D0=
github.com/shamaton/msgpack/v3 v3.2.3 h1:0r0aZtrXDFOPpwWJTEOpFq+Ih/JmMLm0pqLN5mSilrw=
github.com/shamaton/msgpack/v3 v3.2.3/go.mod h1:sgBYvEiyz8JR1NC3yGRoPVME9xXovpnh3l/plW1nfRo=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
github.com/tinylib/msgp v1.6.5 h1:iAH6XTP7BpzErjBZlujJQXxA+GmRi2YWs6M4KdLtNbE=
github.com/tinylib/msgp v1.6.5/go.mod h1:RSp0LW9oSxFut3KzESt5Voq4GVWyS+PSulT77roAqEA=
github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
github.com/valyala/fasthttp v1.74.0 h1:wMS9fnO2QTALozYx5pId2Vi7ZwU/epUkY8i/KPWCHoU=
github.com/valyala/fasthttp v1.74.0/go.mod h1:3ARmLamUcw7ElxVtC8PXaGzQ6VEuvnetlkrwIklQBSE=
github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM=
github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg=
github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs=
github.com/zeebo/xxh3 v1.1.0/go.mod h1:IisAie1LELR4xhVinxWS5+zf1lA4p0MW4T+w+W07F5s=
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.69.0 h1:8tvICD4vSTOOsNrsI4Ljf6C+6UKvpTEH5XY3JMoyPoo=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.69.0/go.mod h1:z9+yiacE0IHRqM4qFfkbt/JYlmYXgss8GY/jXoNuPJI=
go.opentelemetry.io/otel v1.44.0 h1:JjwHmHpA4iZ3wBxluu2fbbE7j4kqlE8jXyAyPXH7HqU=
go.opentelemetry.io/otel v1.44.0/go.mod h1:BMgjTHL9WPRlRjL2oZCBTL4whCGtXch2H4BhOPIAyYc=
go.opentelemetry.io/otel/metric v1.44.0 h1:1w0gILTcHdr3YI+ixLyjemwrVnsMURbTZFrSYCdDdmc=
go.opentelemetry.io/otel/metric v1.44.0/go.mod h1:8O7hanEPBNgEMmybD3s2VBKcgWOCsA6tzHBPODAiquo=
go.opentelemetry.io/otel/trace v1.44.0 h1:jxF5CsGYCe74MCRx2X4g7WsY/VBKRqqpNvXlX/6gtIk=
go.opentelemetry.io/otel/trace v1.44.0/go.mod h1:oLl1jrMQAVo6v3GAggN+1VH9VIz9iUSvW53sW1Q8PIE=
go.uber.org/atomic v1.12.0 h1:BvcXdFKuviU4fTL/f+SxdQ5qJX/Jix8pAkgdUcb3XOE=
go.uber.org/atomic v1.12.0/go.mod h1:I6c4cg+6HCxRjfjSsYtApoFILnpc0CGUdGkXVqbYVNk=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
golang.org/x/crypto v0.58.0 h1:COkYLr4k7nDI4r6QtMML2AfzeIswWHAJthLS6K/M/54=
golang.org/x/crypto v0.58.0/go.mod h1:Xh+kl5A+M0gMiWDytH2Hhr5Z7WbNCpv/eB/EFgfSk+w=
golang.org/x/image v0.48.0 h1:1AvuPcuoOTJpA7s4VZ5umrb90T3So0rkB/bSGmxkxik=
golang.org/x/image v0.48.0/go.mod h1:mmtg64iqJKGqEgy+r13dfybl85tmxUd2lMnmaiSdWYY=
golang.org/x/net v0.61.0 h1:RR+6j/BTBCrggOLugxa+66R8FvoN6MHlzDlZIbN9YFw=
golang.org/x/net v0.61.0/go.mod h1:WgDCOTH8iwtB66T4/GfI94AHo/hhdb7Hah9NDseQfiI=
golang.org/x/sync v0.25.0 h1:dJnifr0nMV8/SBnBvdo4eSGGnrc3O4FFx6IqvtLcmRs=
golang.org/x/sync v0.25.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
golang.org/x/sys v0.50.0 h1:dkKZEFTEKaLr1sjhs0WiKqoi8Jj5nHbS3Njmx1UoMJs=
golang.org/x/sys v0.50.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/text v0.44.0 h1:YItdp/mQNGnsmrgpcdA5yQ3bnRxCfo/Tj/iByGW9GGo=
golang.org/x/text v0.44.0/go.mod h1:S4FR1waQ9LbjQTls97SyRjDaUUVkaM+cXwwuG/G+MwY=
golang.org/x/tools v0.52.0 h1:ryxg4U63oaUYk7uOLTF37n43f4qgtGvqV4Ioit1BaEs=
golang.org/x/tools v0.52.0/go.mod h1:ItcP5UNaVGG+Yc4eIYmaFLNJ4soEopkx5Y2d4lcEurs=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+119
View File
@@ -0,0 +1,119 @@
// Package internal wires HTTP routes, middleware and storage into one Fiber
// application and owns the process-wide error handler.
package internal
import (
"errors"
"log/slog"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/attachments"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/auth"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/chats"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/config"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/health"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/messages"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/objectstore"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/session"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/store"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/users"
"github.com/gofiber/fiber/v3"
"github.com/gofiber/fiber/v3/middleware/recover"
"github.com/gofiber/fiber/v3/middleware/requestid"
)
func New(cfg config.Config, st *store.Store, sessions *session.Store) (*fiber.App, error) {
objects, err := objectstore.New(objectstore.Config{
Endpoint: cfg.S3Endpoint,
Region: cfg.S3Region,
Bucket: cfg.S3Bucket,
AccessKey: cfg.S3AccessKey,
SecretKey: cfg.S3SecretKey,
})
if err != nil {
return nil, err
}
app := fiber.New(fiber.Config{
ReadTimeout: cfg.ReadTimeout,
WriteTimeout: cfg.WriteTimeout,
IdleTimeout: cfg.IdleTimeout,
// BodyLimit is the coarse HTTP envelope. The attachment handler and
// BindJSON enforce their own smaller budgets.
BodyLimit: cfg.HTTPBodyLimit,
// Without StreamRequestBody fasthttp buffers the entire body
// before the handler runs. Streaming lets the upload path copy in
// 32 KiB chunks and stops a 10 MiB body from becoming a per-request
// allocation even when the request is later rejected.
StreamRequestBody: true,
ErrorHandler: handleError,
})
app.Use(recover.New())
app.Use(requestid.New())
app.Use(func(c fiber.Ctx) error {
c.Set("X-Instance-Id", cfg.InstanceID)
return c.Next()
})
healthHandler := health.NewHandler(st, sessions, objects)
healthRoutes := app.Group("/health")
healthRoutes.Get("/live", healthHandler.Live)
healthRoutes.Get("/ready", healthHandler.Ready)
authHandler := auth.NewHandler(st, sessions)
usersHandler := users.NewHandler(st, config.MaxJSONBodyBytes)
chatsHandler := chats.NewHandler(st, config.MaxJSONBodyBytes)
messagesHandler := messages.NewHandler(st, config.MaxJSONBodyBytes)
attachmentsHandler := attachments.NewHandler(
st,
objects,
attachments.NewValidator(cfg.MaxImagePixels, cfg.ImageDecodeBytes),
cfg.MaxUploadBytes,
cfg.UploadDrainLimit,
cfg.MaxPendingJobs,
cfg.JobMaxAttempts,
cfg.UploadTimeout,
cfg.DownloadTimeout,
cfg.UploadTempDir,
)
api := app.Group("/api/v1")
api.Post("/users", usersHandler.CreateUser)
api.Post("/auth/sessions", authHandler.CreateSession)
protected := api.Group("", auth.BearerAuth(sessions))
protected.Get("/users/me", usersHandler.GetCurrentUser)
protected.Delete("/auth/sessions/current", authHandler.RevokeSession)
protected.Get("/chats", chatsHandler.ListChats)
protected.Post("/chats", chatsHandler.CreateChat)
protected.Get("/chats/:chat_id", chatsHandler.GetChat)
protected.Get("/chats/:chat_id/messages", messagesHandler.ListMessages)
protected.Post("/chats/:chat_id/messages", messagesHandler.CreateMessage)
protected.Post("/chats/:chat_id/attachments", attachmentsHandler.Upload)
protected.Get("/attachments/:attachment_id", attachmentsHandler.Get)
protected.Get("/attachments/:attachment_id/content", attachmentsHandler.Download)
return app, nil
}
// handleError is the only writer of error responses: handlers and the store
// return *common.AppError and this function decides status, headers and body.
func handleError(c fiber.Ctx, err error) error {
if appErr, ok := errors.AsType[*common.AppError](err); ok {
if appErr.Challenge != "" {
c.Set(fiber.HeaderWWWAuthenticate, appErr.Challenge)
}
return common.SendError(c, appErr.Status, appErr.Code, appErr.Message)
}
if fiberErr, ok := errors.AsType[*fiber.Error](err); ok {
return common.SendError(c, fiberErr.Code,
common.ErrorCodeFromStatus(fiberErr.Code), fiberErr.Message)
}
slog.Error("unhandled request error",
"request_id", requestid.FromContext(c), "err", err)
return common.SendError(c, fiber.StatusInternalServerError,
common.ErrorCodeInternal, "Internal server error")
}
+359
View File
@@ -0,0 +1,359 @@
package attachments
import (
"bufio"
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"io"
"log/slog"
"mime"
"os"
"strconv"
"time"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/model"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/objectstore"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/store"
"github.com/gofiber/fiber/v3"
"github.com/google/uuid"
)
const copyBufferSize = 32 * 1024
type Handler struct {
store *store.Store
objects *objectstore.Client
validator *Validator
maxUploadBytes int64
drainLimit int64
maxPendingJobs int
jobMaxAttempts int
uploadTimeout time.Duration
downloadTimeout time.Duration
tempDir string
}
func NewHandler(st *store.Store, client *objectstore.Client, validator *Validator, maxUploadBytes, drainLimit int64, maxPendingJobs, jobMaxAttempts int, uploadTimeout, downloadTimeout time.Duration, tempDir string) *Handler {
return &Handler{
store: st,
objects: client,
validator: validator,
maxUploadBytes: maxUploadBytes,
drainLimit: drainLimit,
maxPendingJobs: maxPendingJobs,
jobMaxAttempts: jobMaxAttempts,
uploadTimeout: uploadTimeout,
downloadTimeout: downloadTimeout,
tempDir: tempDir,
}
}
// Upload stages the original, then commits the attachment snapshot and its
// outbox job together. The 202 is therefore a promise about durable bytes plus
// durable intent, not a promise that resizing already happened.
func (h *Handler) Upload(c fiber.Ctx) error {
userID := common.MustUserID(c)
chatID, err := common.PathUUID(c, "chat_id")
if err != nil {
return err
}
// Authorization precedes any body I/O: an outsider cannot make the server
// read a large upload before learning that the chat is not theirs.
isMember, err := h.store.IsMember(c.Context(), userID, chatID)
if err != nil {
return err
}
if !isMember {
return common.NotFound("chat not found")
}
filename := c.Query("filename")
if err := common.ValidateFilename(filename); err != nil {
return err
}
contentType, _, err := mime.ParseMediaType(c.Get(fiber.HeaderContentType))
if err != nil || (contentType != "image/png" && contentType != "image/jpeg") {
return common.UnsupportedMedia("Content-Type must be image/png or image/jpeg")
}
// Backpressure: refuse new work before reading its bytes. The check is
// approximate across API replicas but bounds staging growth under overload.
if h.maxPendingJobs > 0 {
pending, countErr := h.store.CountPendingJobs(c.Context())
if countErr != nil {
return countErr
}
if pending >= h.maxPendingJobs {
c.Set(fiber.HeaderRetryAfter, "5")
return common.Unavailable("processing queue is full")
}
}
// Reject an obviously oversized request before allocating a temp file or
// decoding anything. Chunked requests are caught by the streaming counter
// below.
//
// The declared body is drained first, but only within the bounded
// HTTP envelope (upload limit plus the JSON body slack). If the server
// closed immediately, the client would often see a connection reset instead
// of the required 413 while it was still sending. Draining is bounded and
// uses a 32 KiB buffer, so it does not weaken the memory guarantee.
if rawLength := c.Get(fiber.HeaderContentLength); rawLength != "" {
declared, parseErr := strconv.ParseInt(rawLength, 10, 64)
if parseErr != nil || declared < 0 {
return common.BadRequest("invalid Content-Length")
}
if declared > h.maxUploadBytes {
if declared <= h.drainLimit {
_, _ = io.CopyBuffer(io.Discard, io.LimitReader(requestBody(c), declared), make([]byte, copyBufferSize))
}
return common.PayloadTooLarge("upload exceeds the byte limit")
}
}
if err = os.MkdirAll(h.tempDir, 0o700); err != nil {
slog.Error("create upload temp dir", "err", err)
return common.Internal()
}
file, err := os.CreateTemp(h.tempDir, "upload-*.part")
if err != nil {
slog.Error("create upload temp file", "err", err)
return common.Internal()
}
filePath := file.Name()
defer func() {
_ = file.Close()
_ = os.Remove(filePath)
}()
// The 32 KiB copy buffer is the only request-sized allocation here.
// The body is never read with io.ReadAll, so ten simultaneous 10 MiB
// uploads do not require ten 10 MiB Go slices.
var reader io.Reader = requestBody(c)
hasher := sha256.New()
copyBuffer := make([]byte, copyBufferSize)
written, err := io.CopyBuffer(io.MultiWriter(file, hasher), io.LimitReader(reader, h.maxUploadBytes+1), copyBuffer)
if err != nil {
return common.BadRequest("incomplete upload body")
}
if written == 0 {
return common.UnsupportedMedia("upload body is empty")
}
if written > h.maxUploadBytes {
return common.PayloadTooLarge("upload exceeds the byte limit")
}
ctx, cancel := context.WithTimeout(c.Context(), h.uploadTimeout)
defer cancel()
info, err := h.validator.Validate(ctx, file, contentType)
if err != nil {
return err
}
attachmentID := uuid.New()
jobID := uuid.New()
attachment := model.Attachment{
ID: attachmentID,
ChatID: chatID,
OwnerID: userID,
Filename: filename,
ContentType: info.ContentType,
SizeBytes: written,
SHA256: hex.EncodeToString(hasher.Sum(nil)),
Status: "queued",
Variants: []model.Variant{},
ErrorCode: nil,
CreatedAt: common.NowUTC(),
UpdatedAt: common.NowUTC(),
// AGENT: keys are server-generated from UUIDs, never the client
// filename. staging/ is the durable handoff prefix; the worker copies
// the same bytes to the final attachments/ key unchanged.
ObjectKey: objectKey(chatID, attachmentID),
StagingKey: stagingKey(chatID, attachmentID),
PipelineVersion: 1,
}
if _, err = file.Seek(0, 0); err != nil {
return common.BadRequest("cannot read upload body")
}
if err = h.objects.Put(ctx, attachment.StagingKey, file, written, attachment.ContentType); err != nil {
slog.Error("stage attachment object", "err", err, "attachment_id", attachment.ID)
return common.Unavailable("object storage unavailable")
}
job := model.JobRecord{
Job: model.Job{
JobID: jobID,
AttachmentID: attachmentID,
PipelineVersion: attachment.PipelineVersion,
CreatedAt: attachment.CreatedAt,
},
MaxAttempts: h.jobMaxAttempts,
}
if err = h.store.CreateQueuedAttachment(ctx, attachment, job); err != nil {
h.compensateStaging(attachment.StagingKey)
return err
}
return c.Status(fiber.StatusAccepted).JSON(attachment)
}
// compensateStaging is best effort. If it fails, the reconciler removes an
// object with no metadata row after the grace period.
func (h *Handler) compensateStaging(key string) {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := h.objects.Delete(ctx, key); err != nil && !errors.Is(err, objectstore.ErrNotFound) {
slog.Error("compensate: delete staging object", "err", err)
}
}
func (h *Handler) Get(c fiber.Ctx) error {
userID := common.MustUserID(c)
attachmentID, err := common.PathUUID(c, "attachment_id")
if err != nil {
return err
}
attachment, err := h.store.GetAttachmentForUser(c.Context(), userID, attachmentID)
if err != nil {
return err
}
// Queued/processing snapshots; the client polls this
// endpoint until status becomes ready or failed.
return c.JSON(attachment)
}
func (h *Handler) Download(c fiber.Ctx) error {
userID := common.MustUserID(c)
attachmentID, err := common.PathUUID(c, "attachment_id")
if err != nil {
return err
}
variant := c.Query("variant", "original")
if variant == "" {
variant = "original"
}
switch variant {
case "original", "thumbnail", "crop", "optimized":
default:
return common.BadRequest("unsupported variant")
}
attachment, err := h.store.GetAttachmentForUser(c.Context(), userID, attachmentID)
if err != nil {
return err
}
if attachment.Status != "ready" {
return common.Conflict("attachment is not ready")
}
key := attachment.ObjectKey
contentType := attachment.ContentType
sizeBytes := attachment.SizeBytes
if variant != "original" {
found := false
for _, candidate := range attachment.Variants {
if candidate.Name == variant {
key = store.VariantObjectKey(attachment.ObjectKey, candidate.Name)
contentType = candidate.ContentType
sizeBytes = candidate.SizeBytes
found = true
break
}
}
if !found {
return common.NotFound("attachment variant not found")
}
}
ctx, cancel := context.WithTimeout(c.Context(), h.downloadTimeout)
body, size, err := h.objects.Get(ctx, key)
if err != nil {
cancel()
if errors.Is(err, objectstore.ErrNotFound) {
slog.Error("attachment object missing", "attachment_id", attachment.ID)
return common.NotFound("attachment content not found")
}
slog.Error("read attachment object", "err", err, "attachment_id", attachment.ID)
return common.Unavailable("object storage unavailable")
}
if size != sizeBytes {
_ = body.Close()
cancel()
slog.Error("attachment object size mismatch",
"attachment_id", attachment.ID, "metadata_size", sizeBytes, "object_size", size)
return common.Unavailable("attachment content is inconsistent")
}
c.Set(fiber.HeaderContentType, contentType)
c.Set(fiber.HeaderContentLength, strconv.FormatInt(sizeBytes, 10))
c.Set(fiber.HeaderContentDisposition, mime.FormatMediaType("inline", map[string]string{"filename": attachment.Filename}))
c.Set(fiber.HeaderCacheControl, "private, no-store")
c.Set("X-Content-Type-Options", "nosniff")
// AGENT: SendStreamWriter copies the S3 body straight to the response. The
// API never materializes the whole object in RAM and never redirects the
// client to a bucket URL, so authorization is enforced on every byte.
//
// The context and body are released inside the callback, not by a defer in
// this handler: the handler returns before fasthttp copies the stream, and
// canceling the S3 request context earlier truncates the response.
return c.SendStreamWriter(func(writer *bufio.Writer) {
defer cancel()
defer func() { _ = body.Close() }()
if _, copyErr := io.Copy(writer, body); copyErr != nil {
slog.Error("stream attachment to client", "err", copyErr, "attachment_id", attachment.ID)
}
})
}
func requestBody(c fiber.Ctx) io.Reader {
if reader := c.Request().BodyStream(); reader != nil {
return reader
}
return bytes.NewReader(c.Body())
}
func objectKey(chatID, attachmentID uuid.UUID) string {
return fmt.Sprintf("attachments/%s/%s", chatID, attachmentID)
}
func stagingKey(chatID, attachmentID uuid.UUID) string {
return fmt.Sprintf("staging/%s/%s", chatID, attachmentID)
}
// CleanupStaleUploads removes temp files abandoned by a previous crash. A
// running request removes its own file in a defer; this only handles the
// process-died-mid-upload case.
func CleanupStaleUploads(dir string, olderThan time.Duration) {
entries, err := os.ReadDir(dir)
if err != nil {
return
}
cutoff := time.Now().Add(-olderThan)
for _, entry := range entries {
if entry.IsDir() {
continue
}
info, infoErr := entry.Info()
if infoErr != nil || info.ModTime().After(cutoff) {
continue
}
_ = os.Remove(dir + string(os.PathSeparator) + entry.Name())
}
}
+129
View File
@@ -0,0 +1,129 @@
// Package attachments implements the synchronous upload, metadata lookup and
// authorized download path for chat attachments.
package attachments
import (
"context"
"errors"
"image"
_ "image/jpeg"
_ "image/png"
"io"
"mime"
"net/http"
"os"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
"golang.org/x/sync/semaphore"
)
type Validator struct {
maxPixels int64
decodeBudget *semaphore.Weighted
budgetBytes int64
}
type ImageInfo struct {
ContentType string
Width int
Height int
}
func NewValidator(maxPixels, decodeBudgetBytes int64) *Validator {
return &Validator{
maxPixels: maxPixels,
decodeBudget: semaphore.NewWeighted(decodeBudgetBytes),
budgetBytes: decodeBudgetBytes,
}
}
// Validate checks the declared media type against the real bytes and then
// decodes the image under a shared memory budget. The decode is what turns a
// structurally broken or truncated file into 415 instead of a saved object.
//
// The image.DecodeConfig call first reads only the header. This is the important
// ordering: a decompression bomb is rejected from width×height before the
// decoder is allowed to allocate pixel memory.
func (v *Validator) Validate(ctx context.Context, file *os.File, declared string) (ImageInfo, error) {
mediaType, _, err := mime.ParseMediaType(declared)
if err != nil {
return ImageInfo{}, common.UnsupportedMedia("Content-Type must be image/png or image/jpeg")
}
if mediaType != "image/png" && mediaType != "image/jpeg" {
return ImageInfo{}, common.UnsupportedMedia("Content-Type must be image/png or image/jpeg")
}
if _, err = file.Seek(0, 0); err != nil {
return ImageInfo{}, common.BadRequest("cannot read upload body")
}
head := make([]byte, 512)
n, readErr := file.Read(head)
if readErr != nil && !errors.Is(readErr, io.EOF) {
return ImageInfo{}, common.BadRequest("cannot read upload body")
}
detected := http.DetectContentType(head[:n])
if detected != mediaType {
return ImageInfo{}, common.UnsupportedMedia("file content does not match Content-Type")
}
if _, err = file.Seek(0, 0); err != nil {
return ImageInfo{}, common.BadRequest("cannot read upload body")
}
config, format, err := image.DecodeConfig(file)
if err != nil {
return ImageInfo{}, common.UnsupportedMedia("invalid or corrupted image")
}
if format != "png" && format != "jpeg" {
return ImageInfo{}, common.UnsupportedMedia("only PNG and JPEG images are accepted")
}
if (format == "png" && mediaType != "image/png") || (format == "jpeg" && mediaType != "image/jpeg") {
return ImageInfo{}, common.UnsupportedMedia("file content does not match Content-Type")
}
if config.Width <= 0 || config.Height <= 0 {
return ImageInfo{}, common.UnsupportedMedia("invalid or corrupted image")
}
pixels := int64(config.Width) * int64(config.Height)
if pixels <= 0 || pixels > v.maxPixels {
return ImageInfo{}, common.PayloadTooLarge("image has too many pixels")
}
// Approximate decoded RGBA size. The semaphore caps the sum across
// concurrent requests; a single 40 Mp image needs ~160 MiB, so a lower
// configured budget is a deliberate operator choice and becomes a 413.
weight := pixels * 4
if weight > v.budgetBytes {
return ImageInfo{}, common.PayloadTooLarge("image is too large to validate")
}
if err = v.decodeBudget.Acquire(ctx, weight); err != nil {
return ImageInfo{}, common.Unavailable("image validation is busy")
}
defer v.decodeBudget.Release(weight)
if _, err = file.Seek(0, 0); err != nil {
return ImageInfo{}, common.BadRequest("cannot read upload body")
}
info, decodeErr := safeDecode(file)
if decodeErr != nil {
return ImageInfo{}, common.UnsupportedMedia("invalid or corrupted image")
}
bounds := info.Bounds()
if bounds.Dx() != config.Width || bounds.Dy() != config.Height {
return ImageInfo{}, common.UnsupportedMedia("invalid or corrupted image")
}
return ImageInfo{ContentType: mediaType, Width: config.Width, Height: config.Height}, nil
}
// safeDecode converts a decoder panic into an ordinary error. A malformed
// client file must never crash the process or become a 500.
func safeDecode(file *os.File) (img image.Image, err error) {
defer func() {
if recovered := recover(); recovered != nil {
img = nil
err = errors.New("image decoder panic")
}
}()
img, _, err = image.Decode(file)
return img, err
}
+134
View File
@@ -0,0 +1,134 @@
package attachments
import (
"bytes"
"context"
"image"
"image/color"
"image/jpeg"
"image/png"
"os"
"path/filepath"
"testing"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
)
// writeTemp stores raw bytes in a temporary file and returns its path. The
// validator works on *os.File so it can seek, sniff and decode the same body
// without loading it into memory twice.
func writeTemp(t *testing.T, name string, data []byte) *os.File {
t.Helper()
path := filepath.Join(t.TempDir(), name)
if err := os.WriteFile(path, data, 0o600); err != nil {
t.Fatalf("write %s: %v", path, err)
}
file, err := os.Open(path)
if err != nil {
t.Fatalf("open %s: %v", path, err)
}
t.Cleanup(func() { _ = file.Close() })
return file
}
func encodePNG(t *testing.T, width, height int) []byte {
t.Helper()
img := image.NewRGBA(image.Rect(0, 0, width, height))
for x := 0; x < width; x++ {
for y := 0; y < height; y++ {
img.Set(x, y, color.RGBA{R: uint8(x), G: uint8(y), B: 0x40, A: 0xff})
}
}
var buf bytes.Buffer
if err := png.Encode(&buf, img); err != nil {
t.Fatalf("encode png: %v", err)
}
return buf.Bytes()
}
func TestValidateAcceptsPNGAndJPEG(t *testing.T) {
validator := NewValidator(40_000_000, 256<<20)
ctx := context.Background()
pngFile := writeTemp(t, "image.png", encodePNG(t, 8, 6))
info, err := validator.Validate(ctx, pngFile, "image/png")
if err != nil {
t.Fatalf("valid png rejected: %v", err)
}
if info.ContentType != "image/png" || info.Width != 8 || info.Height != 6 {
t.Fatalf("unexpected png info: %+v", info)
}
img := image.NewRGBA(image.Rect(0, 0, 4, 3))
var buf bytes.Buffer
if err = jpeg.Encode(&buf, img, nil); err != nil {
t.Fatalf("encode jpeg: %v", err)
}
jpegFile := writeTemp(t, "image.jpg", buf.Bytes())
info, err = validator.Validate(ctx, jpegFile, "image/jpeg; charset=binary")
if err != nil {
t.Fatalf("valid jpeg rejected: %v", err)
}
if info.ContentType != "image/jpeg" || info.Width != 4 || info.Height != 3 {
t.Fatalf("unexpected jpeg info: %+v", info)
}
}
func TestValidateRejectsMismatchedOrBrokenContent(t *testing.T) {
validator := NewValidator(40_000_000, 256<<20)
ctx := context.Background()
pngBytes := encodePNG(t, 8, 6)
cases := []struct {
name string
data []byte
declared string
code string
}{
{"png declared as jpeg", pngBytes, "image/jpeg", common.ErrorCodeUnsupportedMedia},
{"text declared as png", []byte("this is not an image at all"), "image/png", common.ErrorCodeUnsupportedMedia},
{"unsupported declared type", pngBytes, "image/gif", common.ErrorCodeUnsupportedMedia},
{"garbage declared type", pngBytes, "not a media type", common.ErrorCodeUnsupportedMedia},
{"truncated png", pngBytes[:len(pngBytes)/2], "image/png", common.ErrorCodeUnsupportedMedia},
{"empty body", nil, "image/png", common.ErrorCodeUnsupportedMedia},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
file := writeTemp(t, "upload", tc.data)
if _, err := validator.Validate(ctx, file, tc.declared); !common.IsCode(err, tc.code) {
t.Fatalf("Validate(%q) = %v, want code %s", tc.name, err, tc.code)
}
})
}
}
func TestValidateRejectsTooManyPixelsBeforeDecoding(t *testing.T) {
validator := NewValidator(100, 256<<20)
file := writeTemp(t, "big.png", encodePNG(t, 20, 20))
_, err := validator.Validate(context.Background(), file, "image/png")
if !common.IsCode(err, common.ErrorCodePayloadTooLarge) {
t.Fatalf("Validate = %v, want %s", err, common.ErrorCodePayloadTooLarge)
}
}
func TestValidateRejectsImageAboveDecodeBudget(t *testing.T) {
// A 100-pixel image needs ~400 bytes of RGBA memory, more than this budget.
validator := NewValidator(40_000_000, 100)
file := writeTemp(t, "budget.png", encodePNG(t, 10, 10))
_, err := validator.Validate(context.Background(), file, "image/png")
if !common.IsCode(err, common.ErrorCodePayloadTooLarge) {
t.Fatalf("Validate = %v, want %s", err, common.ErrorCodePayloadTooLarge)
}
}
func TestSafeDecodeConvertsPanicToError(t *testing.T) {
// image.Decode does not panic on arbitrary bytes, so this only guards the
// contract that safeDecode always returns an error instead of crashing.
file := writeTemp(t, "junk", []byte{0x89, 'P', 'N', 'G', 0x0d, 0x0a, 0x1a, 0x0a, 0xff})
if _, err := safeDecode(file); err == nil {
t.Fatal("safeDecode accepted junk bytes")
}
}
+127
View File
@@ -0,0 +1,127 @@
// Package auth implements password hashing, Basic login, Bearer session
// middleware and the authentication routes.
package auth
import (
"encoding/base64"
"errors"
"log/slog"
"strings"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/model"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/session"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/store"
"github.com/gofiber/fiber/v3"
)
const (
basicChallenge = `Basic realm="messenger", charset="UTF-8"`
bearerChallenge = `Bearer realm="messenger"`
)
type Handler struct {
store *store.Store
sessions *session.Store
}
func NewHandler(st *store.Store, sessions *session.Store) *Handler {
return &Handler{store: st, sessions: sessions}
}
func (h *Handler) CreateSession(c fiber.Ctx) error {
username, password, ok := parseBasic(c.Get(fiber.HeaderAuthorization))
if !ok {
return common.NewChallenge(basicChallenge, "invalid credentials")
}
user, err := h.store.GetUserByUsername(c.Context(), username)
if err != nil {
if common.IsCode(err, common.ErrorCodeNotFound) {
// An unknown username returns before Argon2 runs, while a known
// username with a wrong password runs VerifyPassword. The branches
// therefore take different time, so username enumeration through
// response timing is theoretically possible. The contract only
// requires the same 401 body and challenge.
return common.NewChallenge(basicChallenge, "invalid credentials")
}
return err
}
valid, err := VerifyPassword(user.PasswordHash, password)
if err != nil || !valid {
return common.NewChallenge(basicChallenge, "invalid credentials")
}
token, expiresAt, err := h.sessions.Create(c.Context(), user.ID)
if err != nil {
slog.Error("create session", "err", err)
return common.Unavailable("session store unavailable")
}
return c.Status(fiber.StatusCreated).JSON(model.Session{
Token: token,
TokenType: "Bearer",
ExpiresAt: common.RFCFormTime(expiresAt),
User: user,
})
}
func (h *Handler) RevokeSession(c fiber.Ctx) error {
token, ok := common.SessionToken(c)
if !ok {
return common.NewChallenge(bearerChallenge, "authentication required")
}
if err := h.sessions.Revoke(c.Context(), token); err != nil {
slog.Error("revoke session", "err", err)
return common.Unavailable("session store unavailable")
}
return c.SendStatus(fiber.StatusNoContent)
}
func BearerAuth(sessions *session.Store) fiber.Handler {
return func(c fiber.Ctx) error {
header := strings.TrimSpace(c.Get(fiber.HeaderAuthorization))
scheme, token, found := strings.Cut(header, " ")
token = strings.TrimSpace(token)
if !found || !strings.EqualFold(scheme, "Bearer") || token == "" || strings.ContainsAny(token, " \t") {
return common.NewChallenge(bearerChallenge, "authentication required")
}
userID, err := sessions.Lookup(c.Context(), token)
if err != nil {
if errors.Is(err, session.ErrNotFound) {
return common.NewChallenge(bearerChallenge, "session expired or revoked")
}
slog.Error("session lookup", "err", err)
return common.Unavailable("session store unavailable")
}
c.Locals(common.KeyUserID, userID)
c.Locals(common.KeySessionToken, token)
return c.Next()
}
}
func parseBasic(header string) (string, string, bool) {
scheme, encoded, found := strings.Cut(strings.TrimSpace(header), " ")
if !found || !strings.EqualFold(scheme, "Basic") || encoded == "" {
return "", "", false
}
decoded, err := base64.StdEncoding.DecodeString(encoded)
if err != nil {
if decoded, err = base64.RawStdEncoding.DecodeString(encoded); err != nil {
return "", "", false
}
}
// RFC 7617: split at the first colon; a colon is legal inside the password.
username, password, found := strings.Cut(string(decoded), ":")
if !found || username == "" || password == "" {
return "", "", false
}
return username, password, true
}
+73
View File
@@ -0,0 +1,73 @@
package auth
import (
"crypto/rand"
"crypto/subtle"
"encoding/base64"
"errors"
"fmt"
"strings"
"golang.org/x/crypto/argon2"
)
const (
argonMemory = 64 * 1024
argonIterations = 2
argonParallelism = 4
argonKeyLength = 32
argonSaltLength = 16
)
var ErrInvalidHash = errors.New("invalid password hash")
func HashPassword(password string) (string, error) {
salt := make([]byte, argonSaltLength)
if _, err := rand.Read(salt); err != nil {
return "", fmt.Errorf("generate salt: %w", err)
}
key := argon2.IDKey([]byte(password), salt, argonIterations, argonMemory, argonParallelism, argonKeyLength)
return fmt.Sprintf("$argon2id$v=%d$m=%d,t=%d,p=%d$%s$%s",
argon2.Version, argonMemory, argonIterations, argonParallelism,
base64.RawStdEncoding.EncodeToString(salt),
base64.RawStdEncoding.EncodeToString(key)), nil
}
func VerifyPassword(encoded, password string) (bool, error) {
parts := strings.Split(encoded, "$")
if len(parts) != 6 || parts[1] != "argon2id" {
return false, ErrInvalidHash
}
var version int
if _, err := fmt.Sscanf(parts[2], "v=%d", &version); err != nil || version != argon2.Version {
return false, ErrInvalidHash
}
var memory uint32
var iterations uint32
var parallelism uint8
if _, err := fmt.Sscanf(parts[3], "m=%d,t=%d,p=%d", &memory, &iterations, &parallelism); err != nil {
return false, ErrInvalidHash
}
// argon2.IDKey panics when the cost parameters are outside its supported
// range. A corrupted row or an attacker-supplied encoding must be rejected
// with ErrInvalidHash instead of taking the process down.
if iterations < 1 || parallelism < 1 || memory < 8*uint32(parallelism) {
return false, ErrInvalidHash
}
salt, err := base64.RawStdEncoding.DecodeString(parts[4])
if err != nil || len(salt) == 0 {
return false, ErrInvalidHash
}
expected, err := base64.RawStdEncoding.DecodeString(parts[5])
if err != nil || len(expected) == 0 {
return false, ErrInvalidHash
}
actual := argon2.IDKey([]byte(password), salt, iterations, memory, parallelism, uint32(len(expected)))
return subtle.ConstantTimeCompare(actual, expected) == 1, nil
}
+64
View File
@@ -0,0 +1,64 @@
package auth
import (
"strings"
"testing"
)
func TestPasswordRoundTripPreservesFullRange(t *testing.T) {
passwords := []string{
"simple-password-123",
"colon:inside:password",
"пароль-с-кириллицей-12",
strings.Repeat("😀", 128),
}
for _, password := range passwords {
hash, err := HashPassword(password)
if err != nil {
t.Fatalf("HashPassword: %v", err)
}
ok, err := VerifyPassword(hash, password)
if err != nil || !ok {
t.Fatalf("VerifyPassword rejected the correct password (ok=%v err=%v)", ok, err)
}
ok, err = VerifyPassword(hash, password+"x")
if err != nil || ok {
t.Fatalf("VerifyPassword accepted a different password (ok=%v err=%v)", ok, err)
}
}
}
func TestPasswordHashIsSalted(t *testing.T) {
first, err := HashPassword("same-password-123")
if err != nil {
t.Fatal(err)
}
second, err := HashPassword("same-password-123")
if err != nil {
t.Fatal(err)
}
if first == second {
t.Fatal("two hashes of the same password are identical")
}
if _, err := VerifyPassword("not-a-hash", "same-password-123"); err == nil {
t.Fatal("invalid encoded hash was accepted")
}
}
func TestVerifyPasswordRejectsMalformedCostAndEmptyDigest(t *testing.T) {
malformed := []string{
// Empty digest would make argon2 derive a zero-length key.
"$argon2id$v=19$m=65536,t=2,p=4$c2FsdA$",
// Zero cost parameters are outside argon2's supported range and would
// otherwise panic inside the KDF.
"$argon2id$v=19$m=65536,t=0,p=4$c2FsdA$YWJjZA",
"$argon2id$v=19$m=65536,t=2,p=0$c2FsdA$YWJjZA",
"$argon2id$v=19$m=1,t=2,p=4$c2FsdA$YWJjZA",
}
for _, encoded := range malformed {
ok, err := VerifyPassword(encoded, "same-password-123")
if err == nil || ok {
t.Fatalf("VerifyPassword(%q) = (%v, %v), want ErrInvalidHash", encoded, ok, err)
}
}
}
+254
View File
@@ -0,0 +1,254 @@
// Package broker wraps RabbitMQ delivery and publisher confirms. PostgreSQL is
// still the source of truth: a message carries only a job id and a snapshot of
// the durable job intent.
package broker
import (
"context"
"encoding/json"
"errors"
"fmt"
"log/slog"
"sync"
"time"
amqp "github.com/rabbitmq/amqp091-go"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/model"
)
// ErrRequeue asks the consumer to nack and requeue. The worker uses it only
// when its durable state could not be written (for example PostgreSQL is down).
var ErrRequeue = errors.New("requeue delivery")
func dial(url string) (*amqp.Connection, error) {
return amqp.DialConfig(url, amqp.Config{
Heartbeat: 10 * time.Second,
Locale: "en_US",
})
}
func declareQueue(ch *amqp.Channel, queue string) error {
// A quorum queue uses a durable replicated log on the broker. Delivery is
// still at least once; idempotency lives in the job effect key.
_, err := ch.QueueDeclare(queue, true, false, false, false, amqp.Table{
"x-queue-type": "quorum",
})
return err
}
// Publisher is safe for concurrent use. It keeps one confirm-mode channel and
// reconnects lazily after a broker restart.
type Publisher struct {
url string
queue string
mu sync.Mutex
conn *amqp.Connection
ch *amqp.Channel
}
func NewPublisher(url, queue string) *Publisher {
return &Publisher{url: url, queue: queue}
}
func (p *Publisher) ensure() error {
if p.ch != nil && !p.ch.IsClosed() {
return nil
}
p.closeLocked()
conn, err := dial(p.url)
if err != nil {
return err
}
ch, err := conn.Channel()
if err != nil {
_ = conn.Close()
return err
}
if err = ch.Confirm(false); err != nil {
_ = ch.Close()
_ = conn.Close()
return err
}
if err = declareQueue(ch, p.queue); err != nil {
_ = ch.Close()
_ = conn.Close()
return err
}
p.conn, p.ch = conn, ch
return nil
}
// Publish waits for the broker's publisher confirm. A nil return means the
// broker accepted the message, not that a worker executed it.
func (p *Publisher) Publish(ctx context.Context, job model.Job) error {
body, err := json.Marshal(job)
if err != nil {
return err
}
p.mu.Lock()
defer p.mu.Unlock()
if err = p.ensure(); err != nil {
return err
}
confirmation, err := p.ch.PublishWithDeferredConfirmWithContext(ctx, "", p.queue, true, false, amqp.Publishing{
DeliveryMode: amqp.Persistent,
ContentType: "application/json",
MessageId: job.JobID.String(),
Timestamp: time.Now().UTC(),
Type: job.Type,
Body: body,
})
if err != nil {
p.closeLocked()
return err
}
confirmed, err := confirmation.WaitContext(ctx)
if err != nil {
p.closeLocked()
return err
}
if !confirmed {
p.closeLocked()
return fmt.Errorf("publisher confirm was negative")
}
return nil
}
func (p *Publisher) Close() {
p.mu.Lock()
defer p.mu.Unlock()
p.closeLocked()
}
func (p *Publisher) closeLocked() {
if p.ch != nil {
_ = p.ch.Close()
}
if p.conn != nil {
_ = p.conn.Close()
}
p.ch, p.conn = nil, nil
}
// Handler must return nil after the durable effect (and optional ack); the
// consumer acks then. ErrRequeue leaves the message for redelivery.
type Handler func(context.Context, model.Job) error
type Consumer struct {
url string
queue string
logger *slog.Logger
}
func NewConsumer(url, queue string, logger *slog.Logger) *Consumer {
if logger == nil {
logger = slog.Default()
}
return &Consumer{url: url, queue: queue, logger: logger}
}
// Run reconnects until ctx is cancelled. concurrency bounds simultaneous
// handlers in this process; prefetch bounds unacked messages the broker may
// push. The two knobs are intentionally separate so throughput and in-flight
// memory can be measured independently.
func (c *Consumer) Run(ctx context.Context, concurrency, prefetch int, handler Handler) error {
for {
if ctx.Err() != nil {
return nil
}
err := c.consumeOnce(ctx, concurrency, prefetch, handler)
if ctx.Err() != nil {
return nil
}
c.logger.Error("rabbitmq consumer disconnected", "err", err)
select {
case <-ctx.Done():
return nil
case <-time.After(2 * time.Second):
}
}
}
func (c *Consumer) consumeOnce(ctx context.Context, concurrency, prefetch int, handler Handler) error {
conn, err := dial(c.url)
if err != nil {
return err
}
defer conn.Close()
ch, err := conn.Channel()
if err != nil {
return err
}
defer ch.Close()
if err = declareQueue(ch, c.queue); err != nil {
return err
}
if err = ch.Qos(prefetch, 0, false); err != nil {
return err
}
deliveries, err := ch.Consume(c.queue, "", false, false, false, false, nil)
if err != nil {
return err
}
sem := make(chan struct{}, concurrency)
var workers sync.WaitGroup
stop := make(chan struct{})
go func() {
select {
case <-ctx.Done():
case <-stop:
}
_ = ch.Cancel("", false)
}()
for {
select {
case <-ctx.Done():
close(stop)
workers.Wait()
return nil
case delivery, ok := <-deliveries:
if !ok {
close(stop)
workers.Wait()
return errors.New("delivery channel closed")
}
var job model.Job
if err = json.Unmarshal(delivery.Body, &job); err != nil {
c.logger.Error("drop malformed job message", "err", err)
_ = delivery.Ack(false)
continue
}
sem <- struct{}{}
workers.Add(1)
go func(delivery amqp.Delivery, job model.Job) {
defer workers.Done()
defer func() { <-sem }()
handlerErr := handler(ctx, job)
switch {
case handlerErr == nil:
if err := delivery.Ack(false); err != nil {
c.logger.Error("ack failed", "job_id", job.JobID, "err", err)
}
case errors.Is(handlerErr, ErrRequeue):
// A short sleep avoids a tight redelivery loop while a
// dependency is down. The lease/outbox still own the job.
time.Sleep(time.Second)
if err := delivery.Nack(false, true); err != nil {
c.logger.Error("nack failed", "job_id", job.JobID, "err", err)
}
default:
c.logger.Error("job handler failed", "job_id", job.JobID, "err", handlerErr)
_ = delivery.Ack(false)
}
}(delivery, job)
}
}
}
+86
View File
@@ -0,0 +1,86 @@
// Package chats implements chat creation, lookup and cursor-paginated listing.
package chats
import (
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/store"
"github.com/gofiber/fiber/v3"
"github.com/google/uuid"
)
type Handler struct {
store *store.Store
jsonBodyLimit int64
}
func NewHandler(st *store.Store, jsonBodyLimit int64) *Handler {
return &Handler{store: st, jsonBodyLimit: jsonBodyLimit}
}
type CreateChatRequest struct {
Title string `json:"title"`
MemberIDs []uuid.UUID `json:"member_ids"`
}
func (h *Handler) CreateChat(c fiber.Ctx) error {
userID := common.MustUserID(c)
var req CreateChatRequest
if err := common.BindJSON(c, &req, h.jsonBodyLimit); err != nil {
return err
}
if err := common.ValidateChatTitle(req.Title); err != nil {
return err
}
if err := common.ValidateMemberIDs(req.MemberIDs, userID); err != nil {
return err
}
chat, err := h.store.CreateChat(c.Context(), req.Title, req.MemberIDs)
if err != nil {
return err
}
return c.Status(fiber.StatusCreated).JSON(chat)
}
func (h *Handler) ListChats(c fiber.Ctx) error {
userID := common.MustUserID(c)
limit, err := common.ParseLimit(c, 50)
if err != nil {
return err
}
cursor, err := common.ParseCursor(c)
if err != nil {
return err
}
if cursor != nil && cursor.ContextID != userID {
return common.BadRequest("bad cursor value")
}
page, err := h.store.ListChats(c.Context(), userID, limit, cursor)
if err != nil {
return err
}
return c.JSON(page)
}
func (h *Handler) GetChat(c fiber.Ctx) error {
userID := common.MustUserID(c)
chatID, err := common.PathUUID(c, "chat_id")
if err != nil {
return err
}
chat, err := h.store.GetChatForUser(c.Context(), userID, chatID)
if err != nil {
return err
}
return c.JSON(chat)
}
@@ -0,0 +1,64 @@
package common
import (
"strings"
"testing"
"github.com/google/uuid"
)
func TestValidateFilename(t *testing.T) {
valid := []string{"photo.png", "фото 1.jpeg", "a", strings.Repeat("a", 255)}
for _, name := range valid {
if err := ValidateFilename(name); err != nil {
t.Errorf("ValidateFilename(%q) = %v, want nil", name, err)
}
}
invalid := []string{
"", " ", ".", "..", "dir/photo.png", `dir\photo.png`,
"nul\x00byte", "line\nbreak", strings.Repeat("a", 256),
}
for _, name := range invalid {
if err := ValidateFilename(name); err == nil {
t.Errorf("ValidateFilename(%q) = nil, want error", name)
}
}
}
func TestValidateAttachmentIDs(t *testing.T) {
ids := make([]uuid.UUID, 10)
for index := range ids {
ids[index] = uuid.New()
}
if err := ValidateAttachmentIDs(ids); err != nil {
t.Fatalf("ten unique ids rejected: %v", err)
}
if err := ValidateAttachmentIDs(append(ids, uuid.New())); err == nil {
t.Fatal("eleven ids accepted")
}
if err := ValidateAttachmentIDs([]uuid.UUID{ids[0], ids[0]}); err == nil {
t.Fatal("duplicate ids accepted")
}
if err := ValidateAttachmentIDs(nil); err != nil {
t.Fatalf("empty list rejected: %v", err)
}
}
func TestValidateMessageText(t *testing.T) {
if err := ValidateMessageText("", 1); err != nil {
t.Fatalf("empty text with attachment rejected: %v", err)
}
if err := ValidateMessageText(" ", 1); err != nil {
t.Fatalf("blank text with attachment rejected: %v", err)
}
if err := ValidateMessageText("", 0); err == nil {
t.Fatal("empty text without attachment accepted")
}
if err := ValidateMessageText(strings.Repeat("ж", 4000), 0); err != nil {
t.Fatalf("4000 code points rejected: %v", err)
}
if err := ValidateMessageText(strings.Repeat("ж", 4001), 1); err == nil {
t.Fatal("4001 code points accepted")
}
}
+63
View File
@@ -0,0 +1,63 @@
package common
import (
"bytes"
"encoding/json"
"io"
"strings"
"github.com/gofiber/fiber/v3"
)
// ReadBodyLimited reads at most limit bytes from the request body. The caller
// must use it instead of c.Body() when the endpoint has a smaller budget than
// the process-wide Fiber.BodyLimit. The limit itself belongs to the caller
// (config.MaxJSONBodyBytes for JSON endpoints).
//
// with StreamRequestBody enabled fasthttp exposes the network body as a
// stream, so io.ReadAll over a LimitReader allocates only the bytes the
// endpoint actually accepts instead of the whole upload limit.
func ReadBodyLimited(c fiber.Ctx, limit int64) ([]byte, error) {
var reader io.Reader = c.Request().BodyStream()
if reader == nil {
reader = bytes.NewReader(c.Body())
}
data, err := io.ReadAll(io.LimitReader(reader, limit+1))
if err != nil {
return nil, BadRequest("incomplete request body")
}
if int64(len(data)) > limit {
return nil, PayloadTooLarge("request body is too large")
}
return data, nil
}
func BindJSON(c fiber.Ctx, dst any, limit int64) error {
if contentType := c.Get(fiber.HeaderContentType); !strings.HasPrefix(contentType, "application/json") {
return UnsupportedMedia("invalid JSON body")
}
body, err := ReadBodyLimited(c, limit)
if err != nil {
return err
}
if len(body) == 0 {
return BadRequest("request body is empty")
}
decoder := json.NewDecoder(bytes.NewReader(body))
decoder.DisallowUnknownFields()
if err := decoder.Decode(dst); err != nil {
return BadRequest("invalid JSON body")
}
var extra json.RawMessage
if err := decoder.Decode(&extra); err != io.EOF {
return BadRequest("request body has extra trailing bytes")
}
return nil
}
+142
View File
@@ -0,0 +1,142 @@
// Package common contains cross-cutting HTTP helpers: contract errors, JSON
// binding, validation, cursor pagination and timestamp formatting.
package common
import (
"errors"
"net/http"
"github.com/gofiber/fiber/v3"
"github.com/gofiber/fiber/v3/middleware/requestid"
)
const (
ErrorCodeInvalidRequest = "invalid_request"
ErrorCodeUnauthorized = "unauthorized"
ErrorCodeNotFound = "not_found"
ErrorCodeConflict = "conflict"
ErrorCodePayloadTooLarge = "payload_too_large"
ErrorCodeUnsupportedMedia = "unsupported_media_type"
ErrorCodeRateLimited = "rate_limited"
ErrorCodeUnavailable = "unavailable"
ErrorCodeInternal = "internal_error"
)
type APIErrorResponse struct {
Error APIError `json:"error"`
}
type APIError struct {
Code string `json:"code"`
Message string `json:"message"`
RequestID string `json:"request_id"`
}
func NewError(code, message, requestID string) APIErrorResponse {
return APIErrorResponse{
Error: APIError{
Code: code,
Message: message,
RequestID: requestID,
},
}
}
// AppError is the single contract error type. Handlers and the store return it;
// the Fiber error handler is the only place that turns it into a response.
type AppError struct {
Status int
Code string
Message string
Challenge string // WWW-Authenticate value, set for 401 responses
}
func (e *AppError) Error() string {
return e.Message
}
func NewAppError(status int, code, message string) *AppError {
return &AppError{
Status: status,
Code: code,
Message: message,
}
}
func BadRequest(message string) *AppError {
return NewAppError(http.StatusBadRequest, ErrorCodeInvalidRequest, message)
}
func NotFound(message string) *AppError {
return NewAppError(http.StatusNotFound, ErrorCodeNotFound, message)
}
func Conflict(message string) *AppError {
return NewAppError(http.StatusConflict, ErrorCodeConflict, message)
}
func UnsupportedMedia(message string) *AppError {
return NewAppError(http.StatusUnsupportedMediaType, ErrorCodeUnsupportedMedia, message)
}
func PayloadTooLarge(message string) *AppError {
return NewAppError(http.StatusRequestEntityTooLarge, ErrorCodePayloadTooLarge, message)
}
func Internal() *AppError {
return NewAppError(http.StatusInternalServerError, ErrorCodeInternal, "Internal server error")
}
func Unavailable(message string) *AppError {
return NewAppError(http.StatusServiceUnavailable, ErrorCodeUnavailable, message)
}
// NewChallenge builds the 401 that carries a WWW-Authenticate header.
func NewChallenge(challenge, message string) *AppError {
return &AppError{
Status: http.StatusUnauthorized,
Code: ErrorCodeUnauthorized,
Message: message,
Challenge: challenge,
}
}
// IsCode reports whether err is an AppError with the given contract code.
func IsCode(err error, code string) bool {
appErr, ok := errors.AsType[*AppError](err)
return ok && appErr.Code == code
}
func ErrorCodeFromStatus(status int) string {
switch status {
case http.StatusBadRequest:
return ErrorCodeInvalidRequest
case http.StatusUnauthorized:
return ErrorCodeUnauthorized
case http.StatusNotFound:
return ErrorCodeNotFound
case http.StatusConflict:
return ErrorCodeConflict
case http.StatusRequestEntityTooLarge:
return ErrorCodePayloadTooLarge
case http.StatusUnsupportedMediaType:
return ErrorCodeUnsupportedMedia
case http.StatusTooManyRequests:
return ErrorCodeRateLimited
case http.StatusServiceUnavailable:
return ErrorCodeUnavailable
default:
return ErrorCodeInternal
}
}
func getRequestID(c fiber.Ctx) string {
if id := string(c.Response().Header.Peek("X-Request-Id")); id != "" {
return id
}
return requestid.FromContext(c)
}
func SendError(c fiber.Ctx, status int, code, message string) error {
return c.Status(status).JSON(NewError(code, message, getRequestID(c)))
}
@@ -0,0 +1,30 @@
package common
import (
"strings"
"testing"
)
func TestValidateIdempotencyKey(t *testing.T) {
tests := []struct {
name string
key string
ok bool
}{
{name: "hex", key: "4f8c2d1e", ok: true},
{name: "allowed punctuation", key: "load-1_a.b:c", ok: true},
{name: "empty", key: "", ok: false},
{name: "space", key: "two words", ok: false},
{name: "slash", key: "a/b", ok: false},
{name: "too long", key: strings.Repeat("a", 129), ok: false},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
got := ValidateIdempotencyKey(test.key) == nil
if got != test.ok {
t.Fatalf("ValidateIdempotencyKey(%q) valid = %v, want %v", test.key, got, test.ok)
}
})
}
}
+39
View File
@@ -0,0 +1,39 @@
package common
import (
"github.com/gofiber/fiber/v3"
"github.com/google/uuid"
)
const (
KeyUserID = "userID"
KeySessionToken = "sessionToken"
)
func UserID(c fiber.Ctx) (uuid.UUID, bool) {
userID, ok := c.Locals(KeyUserID).(uuid.UUID)
return userID, ok
}
func MustUserID(c fiber.Ctx) uuid.UUID {
userID, ok := UserID(c)
if !ok {
panic("common: authenticated user ID is missing from the request context")
}
return userID
}
func SessionToken(c fiber.Ctx) (string, bool) {
token, ok := c.Locals(KeySessionToken).(string)
return token, ok && token != ""
}
// PathUUID parses a UUID path parameter. A malformed value is a contract
// invalid_request, not a lookup failure.
func PathUUID(c fiber.Ctx, name string) (uuid.UUID, error) {
id, err := uuid.Parse(c.Params(name))
if err != nil {
return uuid.Nil, BadRequest("invalid " + name + " format")
}
return id, nil
}
+102
View File
@@ -0,0 +1,102 @@
package common
import (
"encoding/base64"
"strconv"
"strings"
"time"
"github.com/gofiber/fiber/v3"
"github.com/google/uuid"
)
type CursorKeyCtx struct {
Time time.Time
ID uuid.UUID
ContextID uuid.UUID
}
type Page[T any] struct {
Items []T `json:"items"`
NextCursor *string `json:"next_cursor"`
}
func ParseLimit(c fiber.Ctx, defaultLimit int) (int, error) {
limitStr, exists := c.Queries()["limit"]
if !exists {
return defaultLimit, nil
}
if strings.TrimSpace(limitStr) == "" {
return 0, BadRequest("limit param declared but empty")
}
limit, err := strconv.Atoi(limitStr)
if err != nil || limit < 1 || limit > 100 {
return 0, BadRequest("limit must be an integer between 1 and 100")
}
return limit, nil
}
func ParseCursor(c fiber.Ctx) (*CursorKeyCtx, error) {
cursorStr, exists := c.Queries()["cursor"]
if !exists {
return nil, nil
}
if strings.TrimSpace(cursorStr) == "" {
return nil, BadRequest("cursor param declared but empty")
}
cursor, ok := DecodeCursor(cursorStr)
if !ok {
return nil, BadRequest("invalid cursor format")
}
return cursor, nil
}
func EncodeCursor(key CursorKeyCtx) string {
raw := key.Time.UTC().Format(time.RFC3339Nano) + "|" + key.ID.String() + "|" + key.ContextID.String()
return base64.RawURLEncoding.EncodeToString([]byte(raw))
}
func DecodeCursor(cursor string) (*CursorKeyCtx, bool) {
data, err := base64.RawURLEncoding.DecodeString(cursor)
if err != nil {
return nil, false
}
parts := strings.SplitN(string(data), "|", 3)
if len(parts) != 3 {
return nil, false
}
parsedTime, err := time.Parse(time.RFC3339Nano, parts[0])
if err != nil {
return nil, false
}
parsedID, err := uuid.Parse(parts[1])
if err != nil {
return nil, false
}
parsedContext, err := uuid.Parse(parts[2])
if err != nil {
return nil, false
}
return &CursorKeyCtx{Time: parsedTime, ID: parsedID, ContextID: parsedContext}, true
}
func NextCursor[T interface {
GetID() uuid.UUID
GetCreatedAt() time.Time
}](items []T, hasMore bool, contextID uuid.UUID) *string {
if !hasMore || len(items) == 0 {
return nil
}
last := items[len(items)-1]
encoded := EncodeCursor(CursorKeyCtx{
Time: last.GetCreatedAt(),
ID: last.GetID(),
ContextID: contextID,
})
return &encoded
}
+31
View File
@@ -0,0 +1,31 @@
package common
import (
"testing"
"time"
"github.com/google/uuid"
)
func TestCursorRoundTrip(t *testing.T) {
want := CursorKeyCtx{
Time: time.Date(2026, 10, 20, 12, 30, 45, 123456789, time.UTC),
ID: uuid.MustParse("11111111-1111-4111-8111-111111111111"),
ContextID: uuid.MustParse("22222222-2222-4222-8222-222222222222"),
}
encoded := EncodeCursor(want)
got, ok := DecodeCursor(encoded)
if !ok {
t.Fatal("cursor did not decode")
}
if !got.Time.Equal(want.Time) || got.ID != want.ID || got.ContextID != want.ContextID {
t.Fatalf("cursor round trip changed the key: %+v", got)
}
for _, invalid := range []string{"not-base64!", "", "aGVsbG8"} {
if _, ok := DecodeCursor(invalid); ok {
t.Fatalf("invalid cursor %q was accepted", invalid)
}
}
}
+21
View File
@@ -0,0 +1,21 @@
package common
import "testing"
func TestValidatePasswordRange(t *testing.T) {
cases := []struct {
password string
valid bool
}{
{"short", false},
{"12345678901", false},
{"123456789012", true},
{"пароль-из-двенадцати", true},
{"😀", false},
}
for _, tc := range cases {
if got := ValidatePassword(tc.password) == nil; got != tc.valid {
t.Fatalf("ValidatePassword(%q) valid=%v, want %v", tc.password, got, tc.valid)
}
}
}
+49
View File
@@ -0,0 +1,49 @@
package common
import (
"encoding/json"
"fmt"
"time"
)
// PostgreSQL stores microseconds, so keeping the same boundary in Go
// prevents a create response from showing more precision
// than a later read can reproduce.
const storageTimePrecision = time.Microsecond
type RFCFormTime time.Time
func (t RFCFormTime) MarshalJSON() ([]byte, error) {
// RFC 3339 permits an optional fractional second.
utc := time.Time(t).UTC().Truncate(storageTimePrecision)
layout := "2006-01-02T15:04:05Z"
if utc.Nanosecond() != 0 {
layout = "2006-01-02T15:04:05.000000Z"
}
return json.Marshal(utc.Format(layout))
}
// UnmarshalJSON accepts the same RFC 3339 strings MarshalJSON emits. The
// broker round-trips Job values through JSON, and a named time.Time type does
// not inherit encoding/json's time.Time special case on decode.
func (t *RFCFormTime) UnmarshalJSON(data []byte) error {
if string(data) == "null" {
return nil
}
var raw string
if err := json.Unmarshal(data, &raw); err != nil {
return err
}
parsed, err := time.Parse(time.RFC3339Nano, raw)
if err != nil {
return fmt.Errorf("parse RFC form time: %w", err)
}
*t = RFCFormTime(parsed)
return nil
}
func NowUTC() RFCFormTime {
return RFCFormTime(time.Now().UTC().Truncate(storageTimePrecision))
}
+57
View File
@@ -0,0 +1,57 @@
package common
import (
"encoding/json"
"testing"
"time"
)
func TestNowUTCHasMicrosecondPrecision(t *testing.T) {
now := time.Time(NowUTC())
if now.Nanosecond()%1000 != 0 {
t.Fatalf("NowUTC() kept precision below a microsecond: %v", now)
}
if now.Location() != time.UTC {
t.Fatalf("NowUTC() location = %v, want UTC", now.Location())
}
}
func TestRFCFormTimeSerializesFractionalSeconds(t *testing.T) {
cases := []struct {
name string
value time.Time
want string
}{
{
name: "microseconds",
value: time.Date(2026, 10, 26, 12, 34, 56, 123456789, time.UTC),
want: `"2026-10-26T12:34:56.123456Z"`,
},
{
name: "exact second",
value: time.Date(2026, 10, 26, 12, 34, 56, 0, time.UTC),
want: `"2026-10-26T12:34:56Z"`,
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
encoded, err := json.Marshal(RFCFormTime(tc.value))
if err != nil {
t.Fatalf("marshal RFC form time: %v", err)
}
if string(encoded) != tc.want {
t.Fatalf("RFC form time = %s, want %s", encoded, tc.want)
}
var decoded RFCFormTime
if err = json.Unmarshal(encoded, &decoded); err != nil {
t.Fatalf("unmarshal RFC form time: %v", err)
}
want := tc.value.UTC().Truncate(storageTimePrecision)
if !time.Time(decoded).Equal(want) {
t.Fatalf("round trip = %v, want %v", time.Time(decoded), want)
}
})
}
}
+149
View File
@@ -0,0 +1,149 @@
package common
import (
"regexp"
"strings"
"unicode"
"unicode/utf8"
"github.com/google/uuid"
)
var (
usernameRegex = regexp.MustCompile(`^[a-z0-9_]{3,32}$`)
idempotencyKeyRegex = regexp.MustCompile(`^[A-Za-z0-9._:-]{1,128}$`)
)
func blank(s string) bool {
return strings.TrimSpace(s) == ""
}
func strLenBetween(s string, min, max int) bool {
n := utf8.RuneCountInString(s)
return n >= min && n <= max
}
func ValidateCreateUser(username, displayName string) error {
if !usernameRegex.MatchString(username) {
return BadRequest("username must match ^[a-z0-9_]{3,32}$")
}
if blank(displayName) {
return BadRequest("display_name must not be empty or only whitespace")
}
if !strLenBetween(displayName, 1, 100) {
return BadRequest("display_name must be between 1 and 100 Unicode code points")
}
return nil
}
func ValidatePassword(password string) error {
if blank(password) {
return BadRequest("password must not be empty or only whitespace")
}
if !strLenBetween(password, 12, 128) {
return BadRequest("password must be between 12 and 128 Unicode code points")
}
return nil
}
func ValidateChatTitle(title string) error {
if blank(title) {
return BadRequest("title must not be empty or only whitespace")
}
if !strLenBetween(title, 1, 100) {
return BadRequest("title must be between 1 and 100 Unicode code points")
}
return nil
}
func ValidateMessageText(text string, attachmentCount int) error {
if attachmentCount == 0 && blank(text) {
return BadRequest("text must not be empty or only whitespace")
}
if !strLenBetween(text, 0, 4000) {
return BadRequest("text must be between 0 and 4000 Unicode code points")
}
return nil
}
func ValidateAttachmentIDs(ids []uuid.UUID) error {
if len(ids) > 10 {
return BadRequest("attachment_ids must contain at most 10 items")
}
seen := make(map[uuid.UUID]bool, len(ids))
for _, id := range ids {
if seen[id] {
return BadRequest("attachment_ids must not contain duplicates")
}
seen[id] = true
}
return nil
}
func ValidateFilename(filename string) error {
if !strLenBetween(filename, 1, 255) {
return BadRequest("filename must be between 1 and 255 Unicode code points")
}
if blank(filename) {
return BadRequest("filename must not be empty or only whitespace")
}
if filename == "." || filename == ".." {
return BadRequest("filename is not allowed")
}
for _, r := range filename {
if r == '/' || r == '\\' || r == 0 || unicode.IsControl(r) {
return BadRequest("filename contains a forbidden character")
}
}
return nil
}
func ValidateIdempotencyKey(key string) error {
if key == "" {
return BadRequest("Idempotency-Key header is required")
}
if !idempotencyKeyRegex.MatchString(key) {
return BadRequest("Idempotency-Key must match ^[A-Za-z0-9._:-]{1,128}$")
}
return nil
}
func ValidateMemberIDs(ids []uuid.UUID, callerID uuid.UUID) error {
if len(ids) < 1 || len(ids) > 100 {
return BadRequest("member_ids must contain between 1 and 100 members")
}
seen := make(map[uuid.UUID]bool, len(ids))
callerFound := false
for _, id := range ids {
if seen[id] {
return BadRequest("member_ids must not contain duplicates")
}
seen[id] = true
if id == callerID {
callerFound = true
}
}
if !callerFound {
return BadRequest("caller must be included in member_ids")
}
return nil
}
+395
View File
@@ -0,0 +1,395 @@
// Package config loads and validates process configuration from environment
// variables and optional *_FILE secrets.
package config
import (
"fmt"
"os"
"path/filepath"
"strconv"
"strings"
"time"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/model"
)
const (
// Configuration may make a limit stricter, never looser.
MaxUploadBytesHardCap = 10 * 1024 * 1024
MaxImagePixelsHardCap = 40_000_000
// MaxJSONBodyBytes is the body budget of every JSON endpoint. It is also
// the slack between the global HTTP body limit and the upload limit, so a
// smaller MAX_UPLOAD_BYTES cannot strangle JSON requests.
MaxJSONBodyBytes = 1 * 1024 * 1024
// Image variant parameters. The validator and the worker must agree on the
// contract dimensions, and the experiment records these values, so they
// have exactly one definition.
ThumbnailMax = 256
CropSize = 128
OptimizedMax = 1280
JPEGQuality = 85
WebPQuality = 82
WebPMethod = 4
)
type Config struct {
Host string
Port string
InstanceID string
SessionTTL time.Duration
IdempotencyTTL time.Duration
DatabaseURL string
CacheURL string
DBMaxConns int32
DBTimeout time.Duration
RedisTimeout time.Duration
StartupWait time.Duration
RunMigrations bool
MigrateOnly bool
S3Endpoint string
S3Region string
S3Bucket string
S3AccessKey string
S3SecretKey string
UploadTimeout time.Duration
DownloadTimeout time.Duration
MaxUploadBytes int64
MaxImagePixels int64
ImageDecodeBytes int64
UploadTempDir string
// HTTP timeouts and body limits are derived once in Load so that app.go
// and the handlers never invent values of their own.
ReadTimeout time.Duration
WriteTimeout time.Duration
IdleTimeout time.Duration
UploadDrainLimit int64
HTTPBodyLimit int
ReconcileGrace time.Duration
ReconcileInterval time.Duration
// BrokerURL is required by the API (outbox
// publisher) and the worker (consumer), but not by the reconciler.
BrokerURL string
WorkerOnly bool
JobQueue string
JobMaxAttempts int
JobTimeout time.Duration
JobLease time.Duration
RetryBase time.Duration
RetryMax time.Duration
WorkerConcurrency int
WorkerPrefetch int
MaxPendingJobs int
LeaseSweep time.Duration
StagingRetention time.Duration
WorkTempDir string
WorkerFaultMode string
CrashAfterEffect bool
CrashAfterClaim bool
}
func Load() (Config, error) {
instance, err := instanceID()
if err != nil {
return Config{}, err
}
sessionTTL, err := envInt("SESSION_TTL_SECONDS", 3600, 1, 0)
if err != nil {
return Config{}, err
}
idempotencyTTL, err := envInt("IDEMPOTENCY_TTL_SECONDS", 86400, 1, 0)
if err != nil {
return Config{}, err
}
maxConns, err := envInt("DB_POOL_MAX_CONNS", 10, 1, 100)
if err != nil {
return Config{}, err
}
maxUpload, err := envInt[int64]("MAX_UPLOAD_BYTES", MaxUploadBytesHardCap, 1, MaxUploadBytesHardCap)
if err != nil {
return Config{}, err
}
maxPixels, err := envInt[int64]("MAX_IMAGE_PIXELS", MaxImagePixelsHardCap, 1, MaxImagePixelsHardCap)
if err != nil {
return Config{}, err
}
decodeMB, err := envInt("IMAGE_DECODE_MEMORY_MB", 256, 32, 4096)
if err != nil {
return Config{}, err
}
cfg := Config{
Host: envOr("APP_HOST", "0.0.0.0"),
Port: envOr("APP_PORT", "8080"),
InstanceID: instance,
SessionTTL: time.Duration(sessionTTL) * time.Second,
IdempotencyTTL: time.Duration(idempotencyTTL) * time.Second,
DBMaxConns: int32(maxConns),
DBTimeout: 5 * time.Second,
RedisTimeout: 2 * time.Second,
StartupWait: 30 * time.Second,
S3Region: envOr("S3_REGION", "us-east-1"),
S3Bucket: envOr("S3_BUCKET", "messenger-attachments"),
ReadTimeout: 30 * time.Second,
WriteTimeout: 60 * time.Second,
IdleTimeout: 60 * time.Second,
UploadTimeout: 30 * time.Second,
DownloadTimeout: 60 * time.Second,
MaxUploadBytes: maxUpload,
MaxImagePixels: maxPixels,
ImageDecodeBytes: int64(decodeMB) * 1024 * 1024,
UploadTempDir: envOr("UPLOAD_TEMP_DIR", filepath.Join(os.TempDir(), "messenger-uploads")),
}
cfg.deriveHTTPLimits()
if cfg.RunMigrations, err = envBool("RUN_MIGRATIONS", true); err != nil {
return Config{}, err
}
if cfg.MigrateOnly, err = envBool("MIGRATE_ONLY", false); err != nil {
return Config{}, err
}
reconcileOnly, err := envBool("RECONCILE_ONLY", false)
if err != nil {
return Config{}, err
}
reconcileGrace, err := envInt("RECONCILE_GRACE_SECONDS", 900, 60, 0)
if err != nil {
return Config{}, err
}
cfg.ReconcileGrace = time.Duration(reconcileGrace) * time.Second
reconcileInterval, err := envInt("RECONCILE_INTERVAL_SECONDS", 300, 10, 0)
if err != nil {
return Config{}, err
}
cfg.ReconcileInterval = time.Duration(reconcileInterval) * time.Second
if cfg.WorkerOnly, err = envBool("WORKER_ONLY", false); err != nil {
return Config{}, err
}
if cfg.CrashAfterEffect, err = envBool("WORKER_CRASH_AFTER_EFFECT", false); err != nil {
return Config{}, err
}
if cfg.CrashAfterClaim, err = envBool("WORKER_CRASH_AFTER_CLAIM", false); err != nil {
return Config{}, err
}
jobAttempts, err := envInt("JOB_MAX_ATTEMPTS", 5, 1, 100)
if err != nil {
return Config{}, err
}
jobTimeout, err := envInt("JOB_TIMEOUT_SECONDS", 60, 5, 3600)
if err != nil {
return Config{}, err
}
jobLease, err := envInt("JOB_LEASE_SECONDS", 90, 10, 7200)
if err != nil {
return Config{}, err
}
retryBase, err := envInt("JOB_RETRY_BASE_SECONDS", 2, 1, 3600)
if err != nil {
return Config{}, err
}
retryMax, err := envInt("JOB_RETRY_MAX_SECONDS", 30, 1, 7200)
if err != nil {
return Config{}, err
}
concurrency, err := envInt("WORKER_CONCURRENCY", 2, 1, 128)
if err != nil {
return Config{}, err
}
prefetch, err := envInt("WORKER_PREFETCH", 2, 1, 512)
if err != nil {
return Config{}, err
}
maxPending, err := envInt("MAX_PENDING_JOBS", 1000, 0, 0)
if err != nil {
return Config{}, err
}
leaseSweep, err := envInt("LEASE_SWEEP_SECONDS", 15, 1, 3600)
if err != nil {
return Config{}, err
}
stagingRetention, err := envInt("STAGING_RETENTION_SECONDS", 86400, 60, 0)
if err != nil {
return Config{}, err
}
cfg.JobQueue = envOr("JOB_QUEUE", model.JobTypeAttachmentProcessV1)
cfg.JobMaxAttempts = jobAttempts
cfg.JobTimeout = time.Duration(jobTimeout) * time.Second
cfg.JobLease = time.Duration(jobLease) * time.Second
cfg.RetryBase = time.Duration(retryBase) * time.Second
cfg.RetryMax = time.Duration(retryMax) * time.Second
cfg.WorkerConcurrency = concurrency
cfg.WorkerPrefetch = prefetch
cfg.MaxPendingJobs = maxPending
cfg.LeaseSweep = time.Duration(leaseSweep) * time.Second
cfg.StagingRetention = time.Duration(stagingRetention) * time.Second
cfg.WorkTempDir = envOr("WORK_TEMP_DIR", filepath.Join(os.TempDir(), "messenger-work"))
cfg.WorkerFaultMode = envOr("WORKER_FAULT_MODE", "none")
switch cfg.WorkerFaultMode {
case "none", "transient-once", "permanent-once":
default:
return Config{}, fmt.Errorf("WORKER_FAULT_MODE must be none, transient-once or permanent-once")
}
if cfg.DatabaseURL, err = secret("DATABASE_URL"); err != nil {
return Config{}, err
}
if cfg.DatabaseURL == "" {
return Config{}, fmt.Errorf("DATABASE_URL or DATABASE_URL_FILE is required")
}
if cfg.MigrateOnly {
return cfg, nil
}
if cfg.S3Endpoint, err = secret("S3_ENDPOINT"); err != nil {
return Config{}, err
}
if cfg.S3Endpoint == "" {
return Config{}, fmt.Errorf("S3_ENDPOINT or S3_ENDPOINT_FILE is required")
}
if cfg.S3AccessKey, err = secret("S3_ACCESS_KEY_ID"); err != nil {
return Config{}, err
}
if cfg.S3AccessKey == "" {
return Config{}, fmt.Errorf("S3_ACCESS_KEY_ID or S3_ACCESS_KEY_ID_FILE is required")
}
if cfg.S3SecretKey, err = secret("S3_SECRET_ACCESS_KEY"); err != nil {
return Config{}, err
}
if cfg.S3SecretKey == "" {
return Config{}, fmt.Errorf("S3_SECRET_ACCESS_KEY or S3_SECRET_ACCESS_KEY_FILE is required")
}
if reconcileOnly {
return cfg, nil
}
if cfg.BrokerURL, err = secret("BROKER_URL"); err != nil {
return Config{}, err
}
if cfg.BrokerURL == "" {
return Config{}, fmt.Errorf("BROKER_URL or BROKER_URL_FILE is required")
}
if cfg.WorkerOnly {
return cfg, nil
}
if cfg.CacheURL, err = secret("CACHE_URL"); err != nil {
return Config{}, err
}
if cfg.CacheURL == "" {
if cfg.CacheURL, err = secret("REDIS_URL"); err != nil {
return Config{}, err
}
}
if cfg.CacheURL == "" {
return Config{}, fmt.Errorf("CACHE_URL or CACHE_URL_FILE is required")
}
if value := strings.ToLower(envOr("SESSION_STORE", "redis")); value != "redis" {
return Config{}, fmt.Errorf("SESSION_STORE must be redis (got %q)", value)
}
return cfg, nil
}
// deriveHTTPLimits is the single place that relates the three HTTP budgets:
// the transport gate must admit both a full upload and a full JSON body, and it
// must leave room for the handler to drain an oversized declared body so the
// client receives a 413 instead of a connection reset.
func (c *Config) deriveHTTPLimits() {
c.UploadDrainLimit = c.MaxUploadBytes + MaxJSONBodyBytes
c.HTTPBodyLimit = int(c.UploadDrainLimit)
}
func instanceID() (string, error) {
if value := strings.TrimSpace(os.Getenv("INSTANCE_ID")); value != "" {
return value, nil
}
name, err := os.Hostname()
if err != nil {
return "", fmt.Errorf("determine INSTANCE_ID: %w", err)
}
if name = strings.TrimSpace(name); name == "" {
return "", fmt.Errorf("determine INSTANCE_ID: hostname is empty and INSTANCE_ID is not set")
}
return name, nil
}
func envBool(name string, fallback bool) (bool, error) {
value := os.Getenv(name)
if value == "" {
return fallback, nil
}
parsed, err := strconv.ParseBool(value)
if err != nil {
return false, fmt.Errorf("%s must be true or false", name)
}
return parsed, nil
}
// envInt parses an integer setting into int or int64; max <= 0 means "no upper
// bound". Both widths are needed by Config, and one helper keeps validation
// identical for every numeric variable.
func envInt[T ~int | ~int64](name string, fallback, min, max T) (T, error) {
value := os.Getenv(name)
if value == "" {
return fallback, nil
}
parsed, err := strconv.ParseInt(value, 10, 64)
if err != nil || parsed < int64(min) || (max > 0 && parsed > int64(max)) {
if max > 0 {
return 0, fmt.Errorf("%s must be an integer between %d and %d", name, min, max)
}
return 0, fmt.Errorf("%s must be an integer >= %d", name, min)
}
result := T(parsed)
if int64(result) != parsed {
return 0, fmt.Errorf("%s is out of range", name)
}
return result, nil
}
func envOr(name, fallback string) string {
if value := os.Getenv(name); value != "" {
return value
}
return fallback
}
func secret(name string) (string, error) {
if value := os.Getenv(name); value != "" {
return value, nil
}
path := os.Getenv(name + "_FILE")
if path == "" {
return "", nil
}
data, err := os.ReadFile(path)
if err != nil {
return "", fmt.Errorf("read %s_FILE: %w", name, err)
}
value := strings.TrimSpace(string(data))
if value == "" {
return "", fmt.Errorf("%s_FILE is empty", name)
}
return value, nil
}
+64
View File
@@ -0,0 +1,64 @@
package config
import "testing"
func TestInstanceIDUsesConfiguredValue(t *testing.T) {
t.Setenv("INSTANCE_ID", " api-1 ")
got, err := instanceID()
if err != nil {
t.Fatalf("instanceID: %v", err)
}
if got != "api-1" {
t.Fatalf("instanceID = %q, want api-1", got)
}
}
func TestInstanceIDFallsBackToHostname(t *testing.T) {
t.Setenv("INSTANCE_ID", " ")
got, err := instanceID()
if err != nil {
t.Fatalf("instanceID: %v", err)
}
if got == "" {
t.Fatal("instanceID returned an empty value")
}
}
func TestDerivedHTTPLimits(t *testing.T) {
cfg := Config{MaxUploadBytes: 2 * 1024 * 1024}
cfg.deriveHTTPLimits()
if cfg.UploadDrainLimit != cfg.MaxUploadBytes+MaxJSONBodyBytes {
t.Fatalf("drain limit = %d, want upload + JSON body", cfg.UploadDrainLimit)
}
if cfg.HTTPBodyLimit != int(cfg.UploadDrainLimit) {
t.Fatalf("body limit = %d, want %d", cfg.HTTPBodyLimit, cfg.UploadDrainLimit)
}
if cfg.HTTPBodyLimit-int(cfg.MaxUploadBytes) != MaxJSONBodyBytes {
t.Fatal("transport gate would strangle a full JSON body")
}
}
func TestEnvIntHandlesBothWidths(t *testing.T) {
t.Setenv("TEST_ENV_INT", "42")
small, err := envInt("TEST_ENV_INT", 1, 1, 100)
if err != nil || small != 42 {
t.Fatalf("envInt = %d, %v; want 42, nil", small, err)
}
large, err := envInt[int64]("TEST_ENV_INT", 1, 1, 100)
if err != nil || large != 42 {
t.Fatalf("envInt[int64] = %d, %v; want 42, nil", large, err)
}
}
func TestEnvIntRejectsOutOfRange(t *testing.T) {
for _, value := range []string{"0", "101", "not-a-number"} {
t.Setenv("TEST_ENV_INT", value)
if _, err := envInt("TEST_ENV_INT", 1, 1, 100); err == nil {
t.Fatalf("envInt accepted %q", value)
}
}
}
+44
View File
@@ -0,0 +1,44 @@
// Package database owns the PostgreSQL connection pool and schema migrations.
package database
import (
"context"
"fmt"
"time"
"github.com/jackc/pgx/v5/pgxpool"
)
func Connect(ctx context.Context, url string, maxConns int32, wait time.Duration) (*pgxpool.Pool, error) {
cfg, err := pgxpool.ParseConfig(url)
if err != nil {
return nil, fmt.Errorf("parse DATABASE_URL: %w", err)
}
cfg.MaxConns = maxConns
cfg.MinConns = 1
cfg.MaxConnLifetime = 30 * time.Minute
cfg.MaxConnIdleTime = 5 * time.Minute
cfg.HealthCheckPeriod = 30 * time.Second
cfg.ConnConfig.ConnectTimeout = 3 * time.Second
pool, err := pgxpool.NewWithConfig(ctx, cfg)
if err != nil {
return nil, fmt.Errorf("create pool: %w", err)
}
deadline := time.Now().Add(wait)
for {
pingCtx, cancel := context.WithTimeout(ctx, 2*time.Second)
err = pool.Ping(pingCtx)
cancel()
if err == nil {
return pool, nil
}
if time.Now().After(deadline) || ctx.Err() != nil {
pool.Close()
return nil, fmt.Errorf("postgres is not reachable: %w", err)
}
time.Sleep(500 * time.Millisecond)
}
}
+71
View File
@@ -0,0 +1,71 @@
package database
import (
"embed"
"errors"
"fmt"
"time"
"github.com/golang-migrate/migrate/v4"
pgxmigrate "github.com/golang-migrate/migrate/v4/database/pgx/v5"
"github.com/golang-migrate/migrate/v4/source/iofs"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/stdlib"
)
//go:embed migrations/*.sql
var migrationFiles embed.FS
const migrationsTable = "schema_migrations"
// Migrate applies every pending embedded migration. golang-migrate takes its
// own advisory lock, so concurrent processes serialise instead of racing the
// schema.
func Migrate(databaseURL string) error {
source, err := iofs.New(migrationFiles, "migrations")
if err != nil {
return fmt.Errorf("load embedded migrations: %w", err)
}
connConfig, err := pgx.ParseConfig(databaseURL)
if err != nil {
_ = source.Close()
return fmt.Errorf("parse DATABASE_URL for migrations: %w", err)
}
if connConfig.RuntimeParams == nil {
connConfig.RuntimeParams = make(map[string]string)
}
connConfig.RuntimeParams["lock_timeout"] = "30000"
connConfig.ConnectTimeout = 3 * time.Second
db := stdlib.OpenDB(*connConfig)
db.SetMaxOpenConns(1)
db.SetMaxIdleConns(1)
driver, err := pgxmigrate.WithInstance(db, &pgxmigrate.Config{
MigrationsTable: migrationsTable,
StatementTimeout: 30 * time.Second,
})
if err != nil {
_ = source.Close()
_ = db.Close()
return fmt.Errorf("create migration driver: %w", err)
}
migrator, err := migrate.NewWithInstance("iofs", source, "pgx5", driver)
if err != nil {
_ = source.Close()
_ = driver.Close()
return fmt.Errorf("create migrator: %w", err)
}
defer func() {
// Close closes both the embedded source and the database connection.
_, _ = migrator.Close()
}()
if err := migrator.Up(); err != nil && !errors.Is(err, migrate.ErrNoChange) {
return fmt.Errorf("apply migrations: %w", err)
}
return nil
}
@@ -0,0 +1,4 @@
DROP TABLE IF EXISTS messages;
DROP TABLE IF EXISTS chat_members;
DROP TABLE IF EXISTS chats;
DROP TABLE IF EXISTS users;
@@ -0,0 +1,38 @@
CREATE TABLE IF NOT EXISTS users (
id uuid PRIMARY KEY,
username text NOT NULL,
display_name text NOT NULL,
password_hash text NOT NULL,
created_at timestamptz NOT NULL,
CONSTRAINT users_username_key UNIQUE (username),
CONSTRAINT users_username_format CHECK (username ~ '^[a-z0-9_]{3,32}$'),
CONSTRAINT users_display_name_len CHECK (char_length(display_name) BETWEEN 1 AND 100)
);
CREATE TABLE IF NOT EXISTS chats (
id uuid PRIMARY KEY,
title text NOT NULL,
created_at timestamptz NOT NULL,
CONSTRAINT chats_title_len CHECK (char_length(title) BETWEEN 1 AND 100)
);
CREATE TABLE IF NOT EXISTS chat_members (
chat_id uuid NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
user_id uuid NOT NULL REFERENCES users(id) ON DELETE RESTRICT,
PRIMARY KEY (chat_id, user_id)
);
CREATE INDEX IF NOT EXISTS chat_members_user_idx
ON chat_members (user_id, chat_id);
CREATE TABLE IF NOT EXISTS messages (
id uuid PRIMARY KEY,
chat_id uuid NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
sender_id uuid NOT NULL REFERENCES users(id) ON DELETE RESTRICT,
text text NOT NULL,
created_at timestamptz NOT NULL,
CONSTRAINT messages_text_len CHECK (char_length(text) BETWEEN 1 AND 4000)
);
CREATE INDEX IF NOT EXISTS messages_chat_created_id_idx
ON messages (chat_id, created_at, id);
@@ -0,0 +1 @@
DROP TABLE IF EXISTS message_idempotency;
@@ -0,0 +1,14 @@
CREATE TABLE IF NOT EXISTS message_idempotency (
sender_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
chat_id uuid NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
idempotency_key text NOT NULL,
request_hash bytea NOT NULL,
message_id uuid NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
created_at timestamptz NOT NULL,
expires_at timestamptz NOT NULL,
PRIMARY KEY (sender_id, chat_id, idempotency_key),
CONSTRAINT message_idempotency_key_len CHECK (char_length(idempotency_key) BETWEEN 1 AND 128)
);
CREATE INDEX IF NOT EXISTS message_idempotency_expires_at_idx
ON message_idempotency (expires_at);
@@ -0,0 +1,6 @@
DROP TABLE IF EXISTS message_attachments;
DROP TABLE IF EXISTS attachments;
ALTER TABLE messages DROP CONSTRAINT IF EXISTS messages_text_len;
ALTER TABLE messages
ADD CONSTRAINT messages_text_len CHECK (char_length(text) BETWEEN 1 AND 4000);
@@ -0,0 +1,46 @@
ALTER TABLE messages DROP CONSTRAINT IF EXISTS messages_text_len;
ALTER TABLE messages
ADD CONSTRAINT messages_text_len CHECK (char_length(text) BETWEEN 0 AND 4000);
CREATE TABLE IF NOT EXISTS attachments (
id uuid PRIMARY KEY,
chat_id uuid NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
owner_id uuid NOT NULL REFERENCES users(id) ON DELETE RESTRICT,
filename text NOT NULL,
content_type text NOT NULL,
size_bytes bigint NOT NULL,
sha256 bytea NOT NULL,
object_key text NOT NULL,
status text NOT NULL,
error_code text,
created_at timestamptz NOT NULL,
updated_at timestamptz NOT NULL,
CONSTRAINT attachments_filename_len CHECK (char_length(filename) BETWEEN 1 AND 255),
CONSTRAINT attachments_content_type CHECK (content_type IN ('image/png', 'image/jpeg')),
CONSTRAINT attachments_size CHECK (size_bytes BETWEEN 1 AND 10485760),
CONSTRAINT attachments_sha256_len CHECK (octet_length(sha256) = 32),
CONSTRAINT attachments_status CHECK (status IN ('pending', 'ready', 'failed')),
CONSTRAINT attachments_error_state CHECK (
(status = 'pending' AND error_code IS NULL) OR
(status = 'ready' AND error_code IS NULL) OR
(status = 'failed' AND error_code IS NOT NULL)
),
CONSTRAINT attachments_object_key_key UNIQUE (object_key)
);
CREATE INDEX IF NOT EXISTS attachments_chat_created_idx
ON attachments (chat_id, created_at, id);
CREATE INDEX IF NOT EXISTS attachments_status_created_idx
ON attachments (status, created_at);
CREATE TABLE IF NOT EXISTS message_attachments (
message_id uuid NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
attachment_id uuid NOT NULL REFERENCES attachments(id) ON DELETE RESTRICT,
position integer NOT NULL,
PRIMARY KEY (message_id, attachment_id),
CONSTRAINT message_attachments_position CHECK (position >= 0),
CONSTRAINT message_attachments_position_key UNIQUE (message_id, position)
);
CREATE INDEX IF NOT EXISTS message_attachments_attachment_idx
ON message_attachments (attachment_id);
@@ -0,0 +1,27 @@
DROP TABLE IF EXISTS attachment_jobs;
ALTER TABLE attachments DROP CONSTRAINT IF EXISTS attachments_variants_array;
ALTER TABLE attachments DROP CONSTRAINT IF EXISTS attachments_pipeline_version;
ALTER TABLE attachments DROP CONSTRAINT IF EXISTS attachments_error_state;
ALTER TABLE attachments DROP CONSTRAINT IF EXISTS attachments_status;
-- Collapse asynchronous states back to the previous single in-flight state.
-- Ready and failed rows keep their meaning; queued/processing work is abandoned.
UPDATE attachments
SET status = 'pending', error_code = NULL
WHERE status IN ('queued', 'processing');
ALTER TABLE attachments
DROP COLUMN IF EXISTS variants,
DROP COLUMN IF EXISTS pipeline_version,
DROP COLUMN IF EXISTS staging_key,
DROP COLUMN IF EXISTS ready_at;
ALTER TABLE attachments
ADD CONSTRAINT attachments_status
CHECK (status IN ('pending', 'ready', 'failed')),
ADD CONSTRAINT attachments_error_state CHECK (
(status = 'pending' AND error_code IS NULL) OR
(status = 'ready' AND error_code IS NULL) OR
(status = 'failed' AND error_code IS NOT NULL)
);
@@ -0,0 +1,76 @@
-- Attachment processing is a durable asynchronous job.
-- The API stages bytes in S3, then in ONE transaction writes the attachment
-- snapshot and its outbox/job row. A separate publisher moves the intent to
-- RabbitMQ, so a broker outage cannot lose an accepted upload.
ALTER TABLE attachments DROP CONSTRAINT IF EXISTS attachments_status;
ALTER TABLE attachments DROP CONSTRAINT IF EXISTS attachments_error_state;
-- The previous "pending" state meant "object PUT in progress". Nothing can
-- finish it now, because acceptors write "queued" plus a job. Preserve the row
-- for the operator instead of silently deleting it.
UPDATE attachments
SET status = 'failed', error_code = 'legacy_interrupted', updated_at = now()
WHERE status = 'pending';
ALTER TABLE attachments
ADD COLUMN IF NOT EXISTS staging_key text,
ADD COLUMN IF NOT EXISTS pipeline_version integer NOT NULL DEFAULT 1,
ADD COLUMN IF NOT EXISTS variants jsonb NOT NULL DEFAULT '[]'::jsonb,
ADD COLUMN IF NOT EXISTS ready_at timestamptz;
UPDATE attachments SET ready_at = updated_at WHERE status = 'ready' AND ready_at IS NULL;
ALTER TABLE attachments
ADD CONSTRAINT attachments_status
CHECK (status IN ('queued', 'processing', 'ready', 'failed')),
ADD CONSTRAINT attachments_error_state CHECK (
(status IN ('queued', 'processing', 'ready') AND error_code IS NULL) OR
(status = 'failed' AND error_code IS NOT NULL)
),
ADD CONSTRAINT attachments_variants_array
CHECK (jsonb_typeof(variants) = 'array'),
ADD CONSTRAINT attachments_pipeline_version CHECK (pipeline_version >= 1);
-- One job per logical attachment effect. published_at IS NULL is the outbox
-- marker: a committed intent that has not received a publisher confirm.
CREATE TABLE IF NOT EXISTS attachment_jobs (
job_id uuid PRIMARY KEY,
attachment_id uuid NOT NULL UNIQUE REFERENCES attachments(id) ON DELETE CASCADE,
type text NOT NULL,
pipeline_version integer NOT NULL,
attempt integer NOT NULL DEFAULT 0,
max_attempts integer NOT NULL,
status text NOT NULL,
error_code text,
available_at timestamptz NOT NULL,
published_at timestamptz,
lease_until timestamptz,
processing_started_at timestamptz,
finished_at timestamptz,
replay_count integer NOT NULL DEFAULT 0,
last_replay_at timestamptz,
created_at timestamptz NOT NULL,
updated_at timestamptz NOT NULL,
CONSTRAINT attachment_jobs_status
CHECK (status IN ('queued', 'processing', 'ready', 'failed')),
CONSTRAINT attachment_jobs_attempts CHECK (attempt >= 0 AND max_attempts >= 1),
CONSTRAINT attachment_jobs_pipeline_version CHECK (pipeline_version >= 1),
CONSTRAINT attachment_jobs_error_state CHECK (
(status IN ('queued', 'processing', 'ready') AND error_code IS NULL) OR
(status = 'failed' AND error_code IS NOT NULL)
),
CONSTRAINT attachment_jobs_finished CHECK (
(status IN ('queued', 'processing') AND finished_at IS NULL) OR
(status IN ('ready', 'failed') AND finished_at IS NOT NULL)
)
);
CREATE INDEX IF NOT EXISTS attachment_jobs_outbox_idx
ON attachment_jobs (available_at, created_at)
WHERE published_at IS NULL AND status = 'queued';
CREATE INDEX IF NOT EXISTS attachment_jobs_lease_idx
ON attachment_jobs (lease_until)
WHERE status = 'processing';
CREATE INDEX IF NOT EXISTS attachment_jobs_status_idx
ON attachment_jobs (status, created_at);
@@ -0,0 +1,24 @@
-- Returning to the stricter 0004 shape requires clearing retry reasons first.
UPDATE attachment_jobs
SET error_code = NULL
WHERE status IN ('queued', 'processing');
UPDATE attachments
SET error_code = NULL
WHERE status IN ('queued', 'processing');
ALTER TABLE attachments
DROP CONSTRAINT IF EXISTS attachments_error_state;
ALTER TABLE attachments
ADD CONSTRAINT attachments_error_state CHECK (
(status IN ('queued', 'processing', 'ready') AND error_code IS NULL) OR
(status = 'failed' AND error_code IS NOT NULL)
);
ALTER TABLE attachment_jobs
DROP CONSTRAINT IF EXISTS attachment_jobs_error_state;
ALTER TABLE attachment_jobs
ADD CONSTRAINT attachment_jobs_error_state CHECK (
(status IN ('queued', 'processing', 'ready') AND error_code IS NULL) OR
(status = 'failed' AND error_code IS NOT NULL)
);
@@ -0,0 +1,23 @@
-- ScheduleRetry records the transient reason on the job while it waits for the
-- next attempt. The original 0004 constraint only allowed error_code on failed
-- rows, so a jittered retry update was rejected and the broker redelivery was
-- then acknowledged as "busy", leaving the job processing until lease expiry.
-- Ready stays clean; failed stays explainable; queued/processing may carry the
-- most recent transient reason.
ALTER TABLE attachments
DROP CONSTRAINT IF EXISTS attachments_error_state;
ALTER TABLE attachments
ADD CONSTRAINT attachments_error_state CHECK (
(status = 'ready' AND error_code IS NULL) OR
(status = 'failed' AND error_code IS NOT NULL) OR
(status IN ('queued', 'processing'))
);
ALTER TABLE attachment_jobs
DROP CONSTRAINT IF EXISTS attachment_jobs_error_state;
ALTER TABLE attachment_jobs
ADD CONSTRAINT attachment_jobs_error_state CHECK (
(status = 'ready' AND error_code IS NULL) OR
(status = 'failed' AND error_code IS NOT NULL) OR
(status IN ('queued', 'processing'))
);
+88
View File
@@ -0,0 +1,88 @@
package database
import (
"errors"
"io/fs"
"regexp"
"strconv"
"testing"
"github.com/golang-migrate/migrate/v4/source/iofs"
)
// latestMigrationVersion is the only value to bump when a migration is added.
// The rest of the test derives filenames and ordering from the directory.
const latestMigrationVersion = 5
var migrationFilePattern = regexp.MustCompile(`^(\d{4})_[a-z0-9_]+\.(up|down)\.sql$`)
func TestEmbeddedMigrationsAreCompleteAndContiguous(t *testing.T) {
entries, err := fs.ReadDir(migrationFiles, "migrations")
if err != nil {
t.Fatalf("read embedded migrations: %v", err)
}
up := make(map[int]string, latestMigrationVersion)
down := make(map[int]string, latestMigrationVersion)
for _, entry := range entries {
if entry.IsDir() {
continue
}
name := entry.Name()
match := migrationFilePattern.FindStringSubmatch(name)
if match == nil {
t.Fatalf("migration %q does not match NNNN_name.{up,down}.sql", name)
}
version, parseErr := strconv.Atoi(match[1])
if parseErr != nil || version < 1 || version > latestMigrationVersion {
t.Fatalf("migration %q has version outside 1..%d", name, latestMigrationVersion)
}
files := up
if match[2] == "down" {
files = down
}
if previous, ok := files[version]; ok {
t.Fatalf("version %d has two %s files: %q and %q", version, match[2], previous, name)
}
files[version] = name
}
for version := 1; version <= latestMigrationVersion; version++ {
if up[version] == "" {
t.Fatalf("version %d is missing an up migration", version)
}
if down[version] == "" {
t.Fatalf("version %d is missing a down migration", version)
}
}
source, err := iofs.New(migrationFiles, "migrations")
if err != nil {
t.Fatalf("parse migrations with golang-migrate: %v", err)
}
defer func() { _ = source.Close() }()
version, err := source.First()
if err != nil {
t.Fatalf("read first migration: %v", err)
}
for expected := 1; expected <= latestMigrationVersion; expected++ {
if version != uint(expected) {
t.Fatalf("migration order = %d, want %d", version, expected)
}
next, nextErr := source.Next(version)
if expected == latestMigrationVersion {
if !errors.Is(nextErr, fs.ErrNotExist) {
t.Fatalf("expected %d to be the last migration, got %v", version, nextErr)
}
break
}
if nextErr != nil {
t.Fatalf("read migration after %d: %v", version, nextErr)
}
version = next
}
}
+61
View File
@@ -0,0 +1,61 @@
// Package health exposes liveness and readiness endpoints plus the process
// shutdown flag used by readiness.
package health
import (
"sync/atomic"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/objectstore"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/session"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/store"
"github.com/gofiber/fiber/v3"
)
type Handler struct {
store *store.Store
sessions *session.Store
objects *objectstore.Client
}
func NewHandler(st *store.Store, sessions *session.Store, objects *objectstore.Client) *Handler {
return &Handler{store: st, sessions: sessions, objects: objects}
}
var shuttingDown atomic.Bool
func SignalShutdown() {
shuttingDown.Store(true)
}
type Status struct {
Status string `json:"status"`
}
func (h *Handler) Live(c fiber.Ctx) error {
return c.JSON(Status{Status: "ok"})
}
func (h *Handler) Ready(c fiber.Ctx) error {
if shuttingDown.Load() {
return common.SendError(c, fiber.StatusServiceUnavailable,
common.ErrorCodeUnavailable, "service is shutting down")
}
if err := h.store.Ping(c.Context()); err != nil {
return common.SendError(c, fiber.StatusServiceUnavailable,
common.ErrorCodeUnavailable, "database unavailable")
}
if err := h.sessions.Ping(c.Context()); err != nil {
return common.SendError(c, fiber.StatusServiceUnavailable,
common.ErrorCodeUnavailable, "session store unavailable")
}
if err := h.objects.Ping(c.Context()); err != nil {
return common.SendError(c, fiber.StatusServiceUnavailable,
common.ErrorCodeUnavailable, "object storage unavailable")
}
return c.JSON(Status{Status: "ok"})
}
+82
View File
@@ -0,0 +1,82 @@
// Package messages implements message creation with idempotency and message
// history pagination.
package messages
import (
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/store"
"github.com/gofiber/fiber/v3"
"github.com/google/uuid"
)
type Handler struct {
store *store.Store
jsonBodyLimit int64
}
func NewHandler(st *store.Store, jsonBodyLimit int64) *Handler {
return &Handler{store: st, jsonBodyLimit: jsonBodyLimit}
}
type CreateMessageRequest struct {
Text string `json:"text"`
AttachmentIDs []uuid.UUID `json:"attachment_ids"`
}
func (h *Handler) CreateMessage(c fiber.Ctx) error {
userID := common.MustUserID(c)
chatID, err := common.PathUUID(c, "chat_id")
if err != nil {
return err
}
idempotencyKey := c.Get("Idempotency-Key")
if err := common.ValidateIdempotencyKey(idempotencyKey); err != nil {
return err
}
var req CreateMessageRequest
if err := common.BindJSON(c, &req, h.jsonBodyLimit); err != nil {
return err
}
if err := common.ValidateAttachmentIDs(req.AttachmentIDs); err != nil {
return err
}
if err := common.ValidateMessageText(req.Text, len(req.AttachmentIDs)); err != nil {
return err
}
message, err := h.store.CreateMessage(c.Context(), chatID, userID, req.Text, idempotencyKey, req.AttachmentIDs)
if err != nil {
return err
}
return c.Status(fiber.StatusCreated).JSON(message)
}
func (h *Handler) ListMessages(c fiber.Ctx) error {
userID := common.MustUserID(c)
chatID, err := common.PathUUID(c, "chat_id")
if err != nil {
return err
}
limit, err := common.ParseLimit(c, 50)
if err != nil {
return err
}
cursor, err := common.ParseCursor(c)
if err != nil {
return err
}
page, err := h.store.ListMessages(c.Context(), userID, chatID, limit, cursor)
if err != nil {
return err
}
return c.JSON(page)
}
+47
View File
@@ -0,0 +1,47 @@
package model
import (
"time"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
"github.com/google/uuid"
)
// Job is the broker payload described by contracts/jobs.schema.json. It never
// contains image bytes, bucket keys or credentials: the worker reloads trusted
// metadata from PostgreSQL by attachment_id.
type Job struct {
SchemaVersion int `json:"schema_version"`
JobID uuid.UUID `json:"job_id"`
Type string `json:"type"`
AttachmentID uuid.UUID `json:"attachment_id"`
PipelineVersion int `json:"pipeline_version"`
Attempt int `json:"attempt"`
CreatedAt common.RFCFormTime `json:"created_at"`
}
const (
JobTypeAttachmentProcessV1 = "attachment.process.v1"
JobStatusQueued = "queued"
JobStatusProcessing = "processing"
JobStatusReady = "ready"
JobStatusFailed = "failed"
)
// JobRecord is the durable outbox row plus the operator-visible job state.
type JobRecord struct {
Job
Status string
// AvailableAt is the earliest time the outbox may publish a retry.
AvailableAt time.Time
// PublishedAt nil means the outbox has not received a publisher confirm
// yet. This is the recovery marker for a crash between COMMIT and publish.
PublishedAt *time.Time
LeaseUntil *time.Time
ProcessingStartedAt *time.Time
FinishedAt *time.Time
MaxAttempts int
ErrorCode *string
ReplayCount int
}
+82
View File
@@ -0,0 +1,82 @@
// Package model defines the JSON-facing domain types shared by handlers,
// storage and the public contract.
package model
import (
"time"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
"github.com/google/uuid"
)
type User struct {
ID uuid.UUID `json:"id"`
Username string `json:"username"`
DisplayName string `json:"display_name"`
PasswordHash string `json:"-"`
CreatedAt common.RFCFormTime `json:"created_at"`
}
func (u User) GetID() uuid.UUID { return u.ID }
func (u User) GetCreatedAt() time.Time { return time.Time(u.CreatedAt) }
type Chat struct {
ID uuid.UUID `json:"id"`
Title string `json:"title"`
MemberIDs []uuid.UUID `json:"member_ids"`
CreatedAt common.RFCFormTime `json:"created_at"`
}
func (c Chat) GetID() uuid.UUID { return c.ID }
func (c Chat) GetCreatedAt() time.Time { return time.Time(c.CreatedAt) }
type Message struct {
ID uuid.UUID `json:"id"`
ChatID uuid.UUID `json:"chat_id"`
SenderID uuid.UUID `json:"sender_id"`
Text string `json:"text"`
CreatedAt common.RFCFormTime `json:"created_at"`
AttachmentIDs []uuid.UUID `json:"attachment_ids"`
}
func (m Message) GetID() uuid.UUID { return m.ID }
func (m Message) GetCreatedAt() time.Time { return time.Time(m.CreatedAt) }
type Attachment struct {
ID uuid.UUID `json:"id"`
ChatID uuid.UUID `json:"chat_id"`
OwnerID uuid.UUID `json:"owner_id"`
Filename string `json:"filename"`
ContentType string `json:"content_type"`
SizeBytes int64 `json:"size_bytes"`
SHA256 string `json:"sha256"`
Status string `json:"status"`
Variants []Variant `json:"variants"`
ErrorCode *string `json:"error_code"`
CreatedAt common.RFCFormTime `json:"created_at"`
UpdatedAt common.RFCFormTime `json:"updated_at"`
// ObjectKey and StagingKey are internal and never serialized.
ObjectKey string `json:"-"`
StagingKey string `json:"-"`
PipelineVersion int `json:"-"`
}
func (a Attachment) GetID() uuid.UUID { return a.ID }
func (a Attachment) GetCreatedAt() time.Time { return time.Time(a.CreatedAt) }
type Variant struct {
Name string `json:"name"`
ContentType string `json:"content_type"`
SizeBytes int64 `json:"size_bytes"`
Width int `json:"width"`
Height int `json:"height"`
SHA256 string `json:"sha256"`
}
type Session struct {
Token string `json:"token"`
TokenType string `json:"token_type"`
ExpiresAt common.RFCFormTime `json:"expires_at"`
User User `json:"user"`
}
+176
View File
@@ -0,0 +1,176 @@
// Package objectstore is the only place that knows how to talk to the private S3
// bucket. PostgreSQL remains the source of truth for metadata; this package
// stores and returns opaque bytes only.
package objectstore
import (
"context"
"errors"
"io"
"net/http"
"strings"
"time"
"github.com/aws/aws-sdk-go-v2/aws"
awsconfig "github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/credentials"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/aws/smithy-go"
)
var ErrNotFound = errors.New("object not found")
type Config struct {
Endpoint string
Region string
Bucket string
AccessKey string
SecretKey string
}
type Client struct {
s3 *s3.Client
bucket string
}
// Object describes one key returned by List. It intentionally exposes only the
// fields the reconciler needs.
type Object struct {
Key string
Size int64
LastModified time.Time
}
func New(cfg Config) (*Client, error) {
awsCfg, err := awsconfig.LoadDefaultConfig(context.Background(),
awsconfig.WithRegion(cfg.Region),
awsconfig.WithCredentialsProvider(credentials.NewStaticCredentialsProvider(
cfg.AccessKey, cfg.SecretKey, "")),
)
if err != nil {
return nil, err
}
// Bounded retry count keeps a broken S3 dependency from turning an
// HTTP upload into an unbounded wait. Context deadlines in the handlers set
// the overall request budget.
awsCfg.RetryMaxAttempts = 2
transport := http.DefaultTransport.(*http.Transport).Clone()
transport.MaxIdleConnsPerHost = 20
transport.MaxIdleConns = 40
client := s3.NewFromConfig(awsCfg, func(options *s3.Options) {
// RustFS is addressed by its Compose service name, so virtual-host
// addressing cannot work without extra DNS entries. Path-style is the
// portable choice for every S3-compatible server used in this stand.
options.UsePathStyle = true
options.BaseEndpoint = aws.String(strings.TrimRight(cfg.Endpoint, "/"))
options.HTTPClient = &http.Client{Transport: transport}
})
return &Client{s3: client, bucket: cfg.Bucket}, nil
}
func (c *Client) Bucket() string { return c.bucket }
// Ping is used by /health/ready. HeadBucket is the cheapest authenticated call
// that proves both the endpoint and the scoped credentials work.
func (c *Client) Ping(ctx context.Context) error {
pingCtx, cancel := context.WithTimeout(ctx, 2*time.Second)
defer cancel()
_, err := c.s3.HeadBucket(pingCtx, &s3.HeadBucketInput{Bucket: aws.String(c.bucket)})
return mapError(err)
}
func (c *Client) Put(ctx context.Context, key string, body io.ReadSeeker, size int64, contentType string) error {
_, err := c.s3.PutObject(ctx, &s3.PutObjectInput{
Bucket: aws.String(c.bucket),
Key: aws.String(key),
Body: body,
ContentLength: aws.Int64(size),
ContentType: aws.String(contentType),
})
return mapError(err)
}
func (c *Client) Get(ctx context.Context, key string) (io.ReadCloser, int64, error) {
output, err := c.s3.GetObject(ctx, &s3.GetObjectInput{
Bucket: aws.String(c.bucket),
Key: aws.String(key),
})
if err != nil {
return nil, 0, mapError(err)
}
size := int64(0)
if output.ContentLength != nil {
size = *output.ContentLength
}
return output.Body, size, nil
}
func (c *Client) Delete(ctx context.Context, key string) error {
_, err := c.s3.DeleteObject(ctx, &s3.DeleteObjectInput{
Bucket: aws.String(c.bucket),
Key: aws.String(key),
})
return mapError(err)
}
func (c *Client) List(ctx context.Context, prefix string) ([]Object, error) {
var objects []Object
var continuation *string
for {
output, err := c.s3.ListObjectsV2(ctx, &s3.ListObjectsV2Input{
Bucket: aws.String(c.bucket),
Prefix: aws.String(prefix),
ContinuationToken: continuation,
})
if err != nil {
return nil, mapError(err)
}
for _, item := range output.Contents {
if item.Key == nil {
continue
}
object := Object{Key: *item.Key}
if item.Size != nil {
object.Size = *item.Size
}
if item.LastModified != nil {
object.LastModified = *item.LastModified
}
objects = append(objects, object)
}
if output.IsTruncated == nil || !*output.IsTruncated {
return objects, nil
}
continuation = output.NextContinuationToken
if continuation == nil {
return objects, nil
}
}
}
// mapError turns a missing object into a sentinel the HTTP layer can map to
// 404, while every other failure stays a dependency error (503).
func mapError(err error) error {
if err == nil {
return nil
}
var apiErr smithy.APIError
if errors.As(err, &apiErr) {
switch apiErr.ErrorCode() {
case "NoSuchKey", "NoSuchBucket", "NotFound", "XMinioInvalidObjectName":
return ErrNotFound
}
}
return err
}
+85
View File
@@ -0,0 +1,85 @@
// Package outbox publishes committed job intents to RabbitMQ. The database row
// is the outbox: published_at IS NULL means "this intent still needs a broker
// confirm". A crash between COMMIT and publish is recovered by the next loop.
package outbox
import (
"context"
"fmt"
"log/slog"
"time"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/broker"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/model"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/store"
)
const batchSize = 20
type Publisher struct {
store *store.Store
broker *broker.Publisher
logger *slog.Logger
}
func New(st *store.Store, b *broker.Publisher, logger *slog.Logger) *Publisher {
if logger == nil {
logger = slog.Default()
}
return &Publisher{store: st, broker: b, logger: logger}
}
func (p *Publisher) Run(ctx context.Context) {
ticker := time.NewTicker(250 * time.Millisecond)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
if err := p.PublishBatch(ctx); err != nil {
p.logger.Error("outbox publish pass failed", "err", err)
select {
case <-ctx.Done():
return
case <-time.After(time.Second):
}
}
}
}
}
func (p *Publisher) PublishBatch(ctx context.Context) error {
jobs, err := p.store.ListPublishableJobs(ctx, batchSize)
if err != nil {
return err
}
for _, record := range jobs {
if err = p.publish(ctx, record); err != nil {
return fmt.Errorf("publish job %s: %w", record.JobID, err)
}
}
return nil
}
func (p *Publisher) publish(ctx context.Context, record model.JobRecord) error {
publishCtx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
message := model.Job{
SchemaVersion: 1,
JobID: record.JobID,
Type: record.Type,
AttachmentID: record.AttachmentID,
PipelineVersion: record.PipelineVersion,
Attempt: record.Attempt + 1,
CreatedAt: record.CreatedAt,
}
if err := p.broker.Publish(publishCtx, message); err != nil {
return err
}
// Compare-and-set on attempt/status: a worker may have claimed it, and a
// retry may have been scheduled while the confirm was in flight.
return p.store.MarkJobPublished(publishCtx, record.JobID, record.Attempt, time.Now().UTC())
}
+162
View File
@@ -0,0 +1,162 @@
package pipeline
import (
"encoding/binary"
"image"
"io"
)
// jpegOrientation returns the EXIF orientation (1..8) of a JPEG, or 1 when the
// file has no usable EXIF block. Go's image/jpeg intentionally ignores EXIF,
// so the worker applies the transform after decoding. The original bytes are
// never rewritten.
func jpegOrientation(r io.ReadSeeker) int {
defer func() { _, _ = r.Seek(0, 0) }()
if _, err := r.Seek(0, 0); err != nil {
return 1
}
var soi [2]byte
if _, err := io.ReadFull(r, soi[:]); err != nil || soi[0] != 0xff || soi[1] != 0xd8 {
return 1
}
for {
var prefix [1]byte
if _, err := io.ReadFull(r, prefix[:]); err != nil || prefix[0] != 0xff {
return 1
}
marker := byte(0xff)
for marker == 0xff {
if _, err := io.ReadFull(r, prefix[:]); err != nil {
return 1
}
marker = prefix[0]
}
// Markers without a length field, and start-of-scan where metadata
// after it cannot be an APP1 segment we care about.
if marker == 0xd8 || marker == 0xd9 || marker == 0xda ||
marker == 0x01 || (marker >= 0xd0 && marker <= 0xd7) {
return 1
}
var lengthBytes [2]byte
if _, err := io.ReadFull(r, lengthBytes[:]); err != nil {
return 1
}
length := int(binary.BigEndian.Uint16(lengthBytes[:]))
if length < 2 {
return 1
}
payloadLength := length - 2
if marker != 0xe1 || payloadLength < 6 {
if _, err := r.Seek(int64(payloadLength), io.SeekCurrent); err != nil {
return 1
}
continue
}
payload := make([]byte, payloadLength)
if _, err := io.ReadFull(r, payload); err != nil {
return 1
}
if string(payload[:6]) == "Exif\x00\x00" {
return parseTIFFOrientation(payload[6:])
}
}
}
func parseTIFFOrientation(data []byte) int {
if len(data) < 8 {
return 1
}
var order binary.ByteOrder
switch string(data[:2]) {
case "II":
order = binary.LittleEndian
case "MM":
order = binary.BigEndian
default:
return 1
}
if order.Uint16(data[2:4]) != 42 {
return 1
}
ifdOffset := int(order.Uint32(data[4:8]))
if ifdOffset < 8 || ifdOffset+2 > len(data) {
return 1
}
count := int(order.Uint16(data[ifdOffset : ifdOffset+2]))
position := ifdOffset + 2
for index := 0; index < count; index++ {
if position+12 > len(data) {
return 1
}
entry := data[position : position+12]
tag := order.Uint16(entry[0:2])
fieldType := order.Uint16(entry[2:4])
items := order.Uint32(entry[4:8])
if tag == 0x0112 && fieldType == 3 && items >= 1 {
if items == 1 {
value := int(order.Uint16(entry[8:10]))
if value >= 1 && value <= 8 {
return value
}
return 1
}
offset := int(order.Uint32(entry[8:12]))
if offset+2 > len(data) {
return 1
}
value := int(order.Uint16(data[offset : offset+2]))
if value >= 1 && value <= 8 {
return value
}
return 1
}
position += 12
}
return 1
}
// orient applies the eight EXIF orientation transforms. Every branch returns a
// new image except orientation 1, so the caller can treat the result as
// read-only.
func orient(src image.Image, orientation int) image.Image {
if orientation <= 1 || orientation > 8 {
return src
}
bounds := src.Bounds()
width, height := bounds.Dx(), bounds.Dy()
swap := orientation >= 5
outWidth, outHeight := width, height
if swap {
outWidth, outHeight = height, width
}
dst := image.NewRGBA(image.Rect(0, 0, outWidth, outHeight))
for y := 0; y < height; y++ {
for x := 0; x < width; x++ {
var nx, ny int
switch orientation {
case 2: // mirror horizontal
nx, ny = width-1-x, y
case 3: // rotate 180
nx, ny = width-1-x, height-1-y
case 4: // mirror vertical
nx, ny = x, height-1-y
case 5: // transpose
nx, ny = y, x
case 6: // rotate 90 clockwise
nx, ny = height-1-y, x
case 7: // transverse
nx, ny = height-1-y, width-1-x
case 8: // rotate 270 clockwise
nx, ny = y, width-1-x
}
dst.Set(nx, ny, src.At(bounds.Min.X+x, bounds.Min.Y+y))
}
}
return dst
}
+288
View File
@@ -0,0 +1,288 @@
// Package pipeline turns one validated original image into the required
// variants. It is deterministic and side-effect free so a broker redelivery can
// simply overwrite the same keys with the same bytes.
package pipeline
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"image"
"image/color"
"image/jpeg"
_ "image/png"
"os"
"time"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/config"
"github.com/gen2brain/webp"
"golang.org/x/image/draw"
"golang.org/x/sync/semaphore"
)
// Re-encoding drops EXIF/XMP/ICC payloads from variants. The original is
// copied byte-for-byte and keeps its metadata and checksum.
type Processor struct {
maxPixels int64
budgetBytes int64
decodeBudget *semaphore.Weighted
}
// Encoded is one variant ready to be uploaded.
type Encoded struct {
Name string
ContentType string
Bytes []byte
Width int
Height int
SHA256 string
}
func New(maxPixels, decodeBudgetBytes int64) *Processor {
return &Processor{
maxPixels: maxPixels,
budgetBytes: decodeBudgetBytes,
decodeBudget: semaphore.NewWeighted(decodeBudgetBytes),
}
}
// Failure classifies pipeline errors for the retry policy. A corrupt image is
// permanent (retrying cannot help); I/O, lease expiry and dependency errors are
// transient.
type Failure struct {
Permanent bool
Code string
Err error
}
func (f *Failure) Error() string {
if f.Err == nil {
return f.Code
}
return f.Code + ": " + f.Err.Error()
}
func (f *Failure) Unwrap() error { return f.Err }
func permanent(code string, err error) *Failure {
return &Failure{Permanent: true, Code: code, Err: err}
}
func transient(code string, err error) *Failure {
return &Failure{Permanent: false, Code: code, Err: err}
}
// Stats separates decode, pixel scaling and the two encoders. A single
// processing_ms number cannot answer whether a large image costs more because
// of resampling or because of JPEG/WebP compression.
type Stats struct {
Decode time.Duration
Scale time.Duration
JPEGEncode time.Duration
WebPEncode time.Duration
}
func (s Stats) Total() time.Duration {
return s.Decode + s.Scale + s.JPEGEncode + s.WebPEncode
}
// Process decodes the file once and returns all variants in memory. Worker
// concurrency and IMAGE_DECODE_MEMORY_MB bound the number of simultaneous
// decoded images, which is the memory-heavy part of the pipeline.
func (p *Processor) Process(ctx context.Context, file *os.File) ([]Encoded, Stats, error) {
var stats Stats
started := time.Now()
img, err := p.decode(ctx, file)
stats.Decode = time.Since(started)
if err != nil {
return nil, stats, err
}
started = time.Now()
thumbnailPixels := fitOnWhite(img, config.ThumbnailMax, config.ThumbnailMax)
stats.Scale += time.Since(started)
thumbnail, encode, err := renderJPEGTimed("thumbnail", thumbnailPixels)
stats.JPEGEncode += encode
if err != nil {
return nil, stats, permanent("processing_failed", err)
}
started = time.Now()
cropPixels := centerSquare(img, config.CropSize)
stats.Scale += time.Since(started)
crop, encode, err := renderJPEGTimed("crop", cropPixels)
stats.JPEGEncode += encode
if err != nil {
return nil, stats, permanent("processing_failed", err)
}
started = time.Now()
optimizedPixels := fitOnWhite(img, config.OptimizedMax, config.OptimizedMax)
stats.Scale += time.Since(started)
optimized, encode, err := renderWebPTimed(optimizedPixels)
stats.WebPEncode += encode
if err != nil {
return nil, stats, permanent("processing_failed", err)
}
return []Encoded{thumbnail, crop, optimized}, stats, nil
}
func renderJPEGTimed(name string, img image.Image) (Encoded, time.Duration, error) {
started := time.Now()
encoded, err := renderJPEG(name, img)
return encoded, time.Since(started), err
}
func renderWebPTimed(img image.Image) (Encoded, time.Duration, error) {
started := time.Now()
encoded, err := renderWebP(img)
return encoded, time.Since(started), err
}
func (p *Processor) decode(ctx context.Context, file *os.File) (image.Image, error) {
if _, err := file.Seek(0, 0); err != nil {
return nil, transient("storage_read_failed", err)
}
cfg, format, err := image.DecodeConfig(file)
if err != nil {
return nil, permanent("invalid_image", err)
}
if format != "png" && format != "jpeg" {
return nil, permanent("invalid_image", fmt.Errorf("unsupported format %q", format))
}
if cfg.Width <= 0 || cfg.Height <= 0 {
return nil, permanent("invalid_image", errors.New("empty image bounds"))
}
pixels := int64(cfg.Width) * int64(cfg.Height)
if pixels > p.maxPixels {
return nil, permanent("invalid_image", errors.New("image exceeds MAX_IMAGE_PIXELS"))
}
// The semaphore is acquired before Decode, not after. A decoder
// allocates roughly width*height*4 bytes; without this gate several 40 MP
// files can OOM the worker even with a small HTTP concurrency.
weight := pixels * 4
if weight > p.budgetBytes {
return nil, permanent("invalid_image", errors.New("image exceeds decode memory budget"))
}
if err = p.decodeBudget.Acquire(ctx, weight); err != nil {
return nil, transient("processing_timeout", err)
}
defer p.decodeBudget.Release(weight)
orientation := jpegOrientation(file)
if _, err = file.Seek(0, 0); err != nil {
return nil, transient("storage_read_failed", err)
}
img, decodeErr := safeDecode(file)
if decodeErr != nil {
return nil, permanent("invalid_image", decodeErr)
}
bounds := img.Bounds()
if bounds.Dx() != cfg.Width || bounds.Dy() != cfg.Height {
return nil, permanent("invalid_image", errors.New("decoded bounds differ from header"))
}
return orient(img, orientation), nil
}
func safeDecode(file *os.File) (img image.Image, err error) {
defer func() {
if recovered := recover(); recovered != nil {
img = nil
err = fmt.Errorf("image decoder panic: %v", recovered)
}
}()
img, _, err = image.Decode(file)
return img, err
}
// fitOnWhite scales down to fit maxWidth x maxHeight, never up. The source is
// composited over white, which is the contract for JPEG variants with alpha.
func fitOnWhite(src image.Image, maxWidth, maxHeight int) *image.RGBA {
bounds := src.Bounds()
width, height := fitSize(bounds.Dx(), bounds.Dy(), maxWidth, maxHeight)
dst := image.NewRGBA(image.Rect(0, 0, width, height))
draw.Draw(dst, dst.Bounds(), &image.Uniform{C: color.White}, image.Point{}, draw.Src)
draw.CatmullRom.Scale(dst, dst.Bounds(), src, bounds, draw.Over, nil)
return dst
}
// centerSquare always produces config.CropSize x config.CropSize. Unlike thumbnails, the
// contract allows upscaling tiny sources for the crop variant.
func centerSquare(src image.Image, size int) *image.RGBA {
bounds := src.Bounds()
side := bounds.Dx()
if bounds.Dy() < side {
side = bounds.Dy()
}
x := bounds.Min.X + (bounds.Dx()-side)/2
y := bounds.Min.Y + (bounds.Dy()-side)/2
cropRect := image.Rect(x, y, x+side, y+side)
dst := image.NewRGBA(image.Rect(0, 0, size, size))
draw.Draw(dst, dst.Bounds(), &image.Uniform{C: color.White}, image.Point{}, draw.Src)
draw.CatmullRom.Scale(dst, dst.Bounds(), src, cropRect, draw.Over, nil)
return dst
}
// fitSize preserves aspect ratio, rounds down and clamps to at least 1 px.
// It deliberately never enlarges a small source.
func fitSize(width, height, maxWidth, maxHeight int) (int, int) {
if width <= maxWidth && height <= maxHeight {
return width, height
}
if width*maxHeight >= height*maxWidth {
newHeight := height * maxWidth / width
if newHeight < 1 {
newHeight = 1
}
return maxWidth, newHeight
}
newWidth := width * maxHeight / height
if newWidth < 1 {
newWidth = 1
}
return newWidth, maxHeight
}
func renderJPEG(name string, img image.Image) (Encoded, error) {
var buf bytes.Buffer
if err := jpeg.Encode(&buf, img, &jpeg.Options{Quality: config.JPEGQuality}); err != nil {
return Encoded{}, err
}
bounds := img.Bounds()
return Encoded{
Name: name,
ContentType: "image/jpeg",
Bytes: buf.Bytes(),
Width: bounds.Dx(),
Height: bounds.Dy(),
SHA256: digest(buf.Bytes()),
}, nil
}
func renderWebP(img image.Image) (Encoded, error) {
var buf bytes.Buffer
if err := webp.Encode(&buf, img, webp.Options{Quality: config.WebPQuality, Method: config.WebPMethod}); err != nil {
return Encoded{}, err
}
bounds := img.Bounds()
return Encoded{
Name: "optimized",
ContentType: "image/webp",
Bytes: buf.Bytes(),
Width: bounds.Dx(),
Height: bounds.Dy(),
SHA256: digest(buf.Bytes()),
}, nil
}
func digest(data []byte) string {
sum := sha256.Sum256(data)
return hex.EncodeToString(sum[:])
}
+194
View File
@@ -0,0 +1,194 @@
package pipeline
import (
"bytes"
"context"
"encoding/binary"
"image"
"image/color"
"image/jpeg"
"image/png"
"os"
"path/filepath"
"testing"
)
func TestFitSizePreservesAspectAndNeverUpscales(t *testing.T) {
cases := []struct {
w, h, maxW, maxH int
wantW, wantH int
}{
{320, 200, 256, 256, 256, 160},
{200, 320, 256, 256, 160, 256},
{100, 50, 256, 256, 100, 50},
{4000, 1000, 1280, 1280, 1280, 320},
{1000, 4000, 256, 256, 64, 256},
{1, 4000, 256, 256, 1, 256},
}
for _, tc := range cases {
w, h := fitSize(tc.w, tc.h, tc.maxW, tc.maxH)
if w != tc.wantW || h != tc.wantH {
t.Fatalf("fitSize(%d,%d,%d,%d) = %dx%d, want %dx%d",
tc.w, tc.h, tc.maxW, tc.maxH, w, h, tc.wantW, tc.wantH)
}
}
}
func TestProcessReportsTiming(t *testing.T) {
file := pngFile(t, 320, 200, false)
_, stats, err := New(40_000_000, 256<<20).Process(context.Background(), file)
if err != nil {
t.Fatalf("Process: %v", err)
}
if stats.Decode <= 0 || stats.Scale <= 0 || stats.JPEGEncode <= 0 || stats.WebPEncode <= 0 {
t.Fatalf("incomplete timing stats: %+v", stats)
}
if stats.Total() != stats.Decode+stats.Scale+stats.JPEGEncode+stats.WebPEncode {
t.Fatalf("total %v does not match its parts: %+v", stats.Total(), stats)
}
if stats.Total() <= 0 {
t.Fatalf("total timing must be positive: %+v", stats)
}
}
func TestProcessProducesContractVariants(t *testing.T) {
file := pngFile(t, 320, 200, false)
variants, _, err := New(40_000_000, 256<<20).Process(context.Background(), file)
if err != nil {
t.Fatalf("Process: %v", err)
}
if len(variants) != 3 {
t.Fatalf("got %d variants, want 3", len(variants))
}
byName := map[string]Encoded{}
for _, variant := range variants {
byName[variant.Name] = variant
}
thumbnail := byName["thumbnail"]
if thumbnail.ContentType != "image/jpeg" || thumbnail.Width != 256 || thumbnail.Height != 160 {
t.Fatalf("thumbnail = %s %dx%d, want image/jpeg 256x160",
thumbnail.ContentType, thumbnail.Width, thumbnail.Height)
}
decoded, err := jpeg.Decode(bytes.NewReader(thumbnail.Bytes))
if err != nil {
t.Fatalf("decode thumbnail: %v", err)
}
if decoded.Bounds().Dx() != 256 || decoded.Bounds().Dy() != 160 {
t.Fatalf("actual thumbnail bounds = %v", decoded.Bounds())
}
crop := byName["crop"]
if crop.ContentType != "image/jpeg" || crop.Width != 128 || crop.Height != 128 {
t.Fatalf("crop = %s %dx%d", crop.ContentType, crop.Width, crop.Height)
}
optimized := byName["optimized"]
if optimized.ContentType != "image/webp" || optimized.Width != 320 || optimized.Height != 200 {
t.Fatalf("optimized = %s %dx%d, want image/webp 320x200", optimized.ContentType, optimized.Width, optimized.Height)
}
}
func TestThumbnailCompositesAlphaOnWhite(t *testing.T) {
file := pngFile(t, 64, 64, true)
variants, _, err := New(40_000_000, 256<<20).Process(context.Background(), file)
if err != nil {
t.Fatalf("Process: %v", err)
}
decoded, err := jpeg.Decode(bytes.NewReader(variants[0].Bytes))
if err != nil {
t.Fatalf("decode thumbnail: %v", err)
}
r, g, b, _ := decoded.At(32, 32).RGBA()
if r < 0xf000 || g < 0xf000 || b < 0xf000 {
t.Fatalf("transparent pixel is not white: %04x %04x %04x", r, g, b)
}
}
func TestEXIFOrientationRotatesBeforeResize(t *testing.T) {
file := jpegFileWithOrientation(t, 320, 200, 6)
if got := jpegOrientation(file); got != 6 {
t.Fatalf("jpegOrientation = %d, want 6", got)
}
variants, _, err := New(40_000_000, 256<<20).Process(context.Background(), file)
if err != nil {
t.Fatalf("Process: %v", err)
}
if variants[0].Width != 160 || variants[0].Height != 256 {
t.Fatalf("oriented thumbnail = %dx%d, want 160x256", variants[0].Width, variants[0].Height)
}
}
func TestParseTIFFOrientationBigEndian(t *testing.T) {
tiff := []byte{
'M', 'M', 0x00, 0x2a, 0x00, 0x00, 0x00, 0x08,
0x00, 0x01,
0x01, 0x12, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x00, 0x08, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00,
}
if got := parseTIFFOrientation(tiff); got != 8 {
t.Fatalf("parseTIFFOrientation = %d, want 8", got)
}
}
func pngFile(t *testing.T, width, height int, transparent bool) *os.File {
t.Helper()
img := image.NewRGBA(image.Rect(0, 0, width, height))
for y := 0; y < height; y++ {
for x := 0; x < width; x++ {
alpha := uint8(0xff)
if transparent {
alpha = 0
}
img.Set(x, y, color.RGBA{R: 10, G: 120, B: 200, A: alpha})
}
}
var buf bytes.Buffer
if err := png.Encode(&buf, img); err != nil {
t.Fatalf("encode png: %v", err)
}
return tempFile(t, "image.png", buf.Bytes())
}
func jpegFileWithOrientation(t *testing.T, width, height int, orientation uint16) *os.File {
t.Helper()
img := image.NewRGBA(image.Rect(0, 0, width, height))
var buf bytes.Buffer
if err := jpeg.Encode(&buf, img, nil); err != nil {
t.Fatalf("encode jpeg: %v", err)
}
payload := []byte{'E', 'x', 'i', 'f', 0, 0}
tiff := make([]byte, 26)
copy(tiff, []byte{'I', 'I'})
binary.LittleEndian.PutUint16(tiff[2:4], 42)
binary.LittleEndian.PutUint32(tiff[4:8], 8)
binary.LittleEndian.PutUint16(tiff[8:10], 1)
binary.LittleEndian.PutUint16(tiff[10:12], 0x0112)
binary.LittleEndian.PutUint16(tiff[12:14], 3)
binary.LittleEndian.PutUint32(tiff[14:18], 1)
binary.LittleEndian.PutUint16(tiff[18:20], orientation)
payload = append(payload, tiff...)
segment := append([]byte{0xff, 0xe1}, 0, 0)
binary.BigEndian.PutUint16(segment[2:4], uint16(len(payload)+2))
segment = append(segment, payload...)
data := append([]byte{}, buf.Bytes()[:2]...)
data = append(data, segment...)
data = append(data, buf.Bytes()[2:]...)
return tempFile(t, "oriented.jpg", data)
}
func tempFile(t *testing.T, name string, data []byte) *os.File {
t.Helper()
path := filepath.Join(t.TempDir(), name)
if err := os.WriteFile(path, data, 0o600); err != nil {
t.Fatalf("write %s: %v", path, err)
}
file, err := os.Open(path)
if err != nil {
t.Fatalf("open %s: %v", path, err)
}
t.Cleanup(func() { _ = file.Close() })
return file
}
+110
View File
@@ -0,0 +1,110 @@
// Package reconcile repairs the two storage/metadata gaps that a crash can
// leave behind:
//
// - a staging object whose metadata row never committed;
// - a terminal attachment whose staging copy was not deleted yet.
//
// It never touches the final original or a variant while PostgreSQL still owns
// it. The grace period protects in-flight work.
package reconcile
import (
"context"
"errors"
"log/slog"
"time"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/objectstore"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/store"
)
var prefixes = []string{"attachments/", "staging/"}
type Reconciler struct {
store *store.Store
objects *objectstore.Client
grace time.Duration
logger *slog.Logger
}
type Stats struct {
StaleStaging int `json:"stale_staging"`
OrphanObjects int `json:"orphan_objects"`
DeletedStaging int `json:"deleted_staging"`
DeletedObjects int `json:"deleted_objects"`
Skipped int `json:"skipped"`
}
func New(st *store.Store, objects *objectstore.Client, grace time.Duration, logger *slog.Logger) *Reconciler {
if logger == nil {
logger = slog.Default()
}
return &Reconciler{store: st, objects: objects, grace: grace, logger: logger}
}
func (r *Reconciler) RunOnce(ctx context.Context, dryRun bool) (Stats, error) {
var stats Stats
cutoff := time.Now().UTC().Add(-r.grace)
// Phase 1: ready/failed rows eventually lose their staging copy. The
// worker usually deletes it immediately after ready commits; this covers a
// crash or a transient S3 error.
stale, err := r.store.ListStaleStagingAttachments(ctx, cutoff, 500)
if err != nil {
return stats, err
}
for _, attachment := range stale {
stats.StaleStaging++
if dryRun {
r.logger.Info("reconcile: would remove stale staging",
"attachment_id", attachment.ID, "staging_key", attachment.StagingKey)
continue
}
if err = r.objects.Delete(ctx, attachment.StagingKey); err != nil && !errors.Is(err, objectstore.ErrNotFound) {
r.logger.Error("reconcile: delete staging",
"err", err, "attachment_id", attachment.ID)
stats.Skipped++
continue
}
if err = r.store.ClearStagingKey(ctx, attachment.ID); err != nil {
r.logger.Error("reconcile: clear staging key", "err", err, "attachment_id", attachment.ID)
stats.Skipped++
continue
}
stats.DeletedStaging++
}
// Phase 2: anything under our prefixes with no database owner. The API
// writes staging before the metadata commit, so this window is real.
known, err := r.store.ListStorageKeys(ctx)
if err != nil {
return stats, err
}
for _, prefix := range prefixes {
listed, listErr := r.objects.List(ctx, prefix)
if listErr != nil {
return stats, listErr
}
for _, object := range listed {
if _, ok := known[object.Key]; ok {
continue
}
if object.LastModified.IsZero() || object.LastModified.After(cutoff) {
continue
}
stats.OrphanObjects++
if dryRun {
r.logger.Info("reconcile: would delete orphan object", "object_key", object.Key)
continue
}
if err = r.objects.Delete(ctx, object.Key); err != nil && !errors.Is(err, objectstore.ErrNotFound) {
r.logger.Error("reconcile: delete orphan object", "err", err, "object_key", object.Key)
stats.Skipped++
continue
}
stats.DeletedObjects++
}
}
return stats, nil
}
+103
View File
@@ -0,0 +1,103 @@
// Package session stores bearer sessions in Redis. Tokens are hashed before
// they become keys, and the configured TTL is absolute.
package session
import (
"context"
"crypto/rand"
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"errors"
"fmt"
"strings"
"time"
"github.com/google/uuid"
"github.com/redis/go-redis/v9"
)
var ErrNotFound = errors.New("session not found")
const keyPrefix = "messenger:session:"
type Store struct {
client *redis.Client
ttl time.Duration
timeout time.Duration
}
func New(rawURL string, ttl, timeout time.Duration) (*Store, error) {
options, err := redis.ParseURL(rawURL)
if err != nil {
return nil, fmt.Errorf("parse CACHE_URL: %w", err)
}
options.PoolSize = 10
options.DialTimeout = timeout
options.ReadTimeout = timeout
options.WriteTimeout = timeout
options.MaxRetries = 0
return &Store{
client: redis.NewClient(options),
ttl: ttl,
timeout: timeout,
}, nil
}
func (s *Store) Close() error {
return s.client.Close()
}
func (s *Store) Ping(ctx context.Context) error {
pingCtx, cancel := context.WithTimeout(ctx, s.timeout)
defer cancel()
return s.client.Ping(pingCtx).Err()
}
func (s *Store) Create(ctx context.Context, userID uuid.UUID) (string, time.Time, error) {
raw := make([]byte, 32)
if _, err := rand.Read(raw); err != nil {
return "", time.Time{}, fmt.Errorf("generate token: %w", err)
}
token := base64.RawURLEncoding.EncodeToString(raw)
expiresAt := time.Now().UTC().Add(s.ttl)
opCtx, cancel := context.WithTimeout(ctx, s.timeout)
defer cancel()
if err := s.client.Set(opCtx, keyFor(token), userID.String(), s.ttl).Err(); err != nil {
return "", time.Time{}, err
}
return token, expiresAt, nil
}
func (s *Store) Lookup(ctx context.Context, token string) (uuid.UUID, error) {
opCtx, cancel := context.WithTimeout(ctx, s.timeout)
defer cancel()
value, err := s.client.Get(opCtx, keyFor(token)).Result()
if errors.Is(err, redis.Nil) {
return uuid.Nil, ErrNotFound
}
if err != nil {
return uuid.Nil, err
}
userID, err := uuid.Parse(value)
if err != nil {
return uuid.Nil, ErrNotFound
}
return userID, nil
}
func (s *Store) Revoke(ctx context.Context, token string) error {
opCtx, cancel := context.WithTimeout(ctx, s.timeout)
defer cancel()
return s.client.Del(opCtx, keyFor(token)).Err()
}
func keyFor(token string) string {
sum := sha256.Sum256([]byte(strings.TrimSpace(token)))
return keyPrefix + hex.EncodeToString(sum[:])
}
+250
View File
@@ -0,0 +1,250 @@
package store
import (
"context"
"encoding/hex"
"encoding/json"
"errors"
"time"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/model"
"github.com/google/uuid"
"github.com/jackc/pgx/v5"
)
// CreateQueuedAttachment is the durability boundary of the upload endpoint.
// The attachment snapshot and the outbox job commit together, so there is no
// observable state in which the API promised work but PostgreSQL lost the
// intent. Publication to RabbitMQ happens later and may be retried safely.
func (s *Store) CreateQueuedAttachment(ctx context.Context, att model.Attachment, job model.JobRecord) error {
ctx, cancel := s.withTimeout(ctx)
defer cancel()
hash, err := hex.DecodeString(att.SHA256)
if err != nil {
return common.BadRequest("invalid sha256")
}
if att.PipelineVersion < 1 {
att.PipelineVersion = 1
}
variants, err := json.Marshal([]model.Variant{})
if err != nil {
return common.Internal()
}
tx, err := s.pool.Begin(ctx)
if err != nil {
return dbError(err)
}
defer rollback(tx)
_, err = tx.Exec(ctx, `
INSERT INTO attachments
(id, chat_id, owner_id, filename, content_type, size_bytes, sha256,
object_key, staging_key, pipeline_version, variants, status, error_code,
created_at, updated_at)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, 'queued', NULL, $12, $12)`,
att.ID, att.ChatID, att.OwnerID, att.Filename, att.ContentType, att.SizeBytes,
hash, att.ObjectKey, att.StagingKey, att.PipelineVersion, variants, time.Time(att.CreatedAt))
if err != nil {
return dbError(err)
}
_, err = tx.Exec(ctx, `
INSERT INTO attachment_jobs
(job_id, attachment_id, type, pipeline_version, attempt, max_attempts,
status, error_code, available_at, published_at, lease_until,
processing_started_at, finished_at, replay_count, created_at, updated_at)
VALUES ($1, $2, $3, $4, 0, $5, 'queued', NULL, $6, NULL, NULL,
NULL, NULL, 0, $6, $6)`,
job.JobID, att.ID, model.JobTypeAttachmentProcessV1, att.PipelineVersion,
job.MaxAttempts, time.Time(att.CreatedAt))
if err != nil {
return dbError(err)
}
if err = tx.Commit(ctx); err != nil {
return dbError(err)
}
return nil
}
// CountPendingJobs is an admission control signal, not a queue size proof.
// Two API replicas may observe slightly stale counts, but the durable cap is
// still useful: it stops an overload from growing staging and PostgreSQL
// without limit. Rejections are 503 because the client may retry later.
func (s *Store) CountPendingJobs(ctx context.Context) (int, error) {
ctx, cancel := s.withTimeout(ctx)
defer cancel()
var count int
err := s.pool.QueryRow(ctx, `
SELECT count(*) FROM attachment_jobs
WHERE status IN ('queued', 'processing')`).Scan(&count)
if err != nil {
return 0, dbError(err)
}
return count, nil
}
func (s *Store) GetAttachmentForUser(ctx context.Context, userID, attachmentID uuid.UUID) (model.Attachment, error) {
ctx, cancel := s.withTimeout(ctx)
defer cancel()
att, err := scanAttachment(s.pool.QueryRow(ctx, attachmentSelect+`
JOIN chat_members cm ON cm.chat_id = a.chat_id AND cm.user_id = $2
WHERE a.id = $1`, attachmentID, userID))
if err != nil {
return model.Attachment{}, err
}
return att, nil
}
// GetAttachment is the worker's trusted read. The worker never accepts bucket
// keys or filenames from the broker message.
func (s *Store) GetAttachment(ctx context.Context, attachmentID uuid.UUID) (model.Attachment, error) {
ctx, cancel := s.withTimeout(ctx)
defer cancel()
att, err := scanAttachment(s.pool.QueryRow(ctx, attachmentSelect+` WHERE a.id = $1`, attachmentID))
if err != nil {
return model.Attachment{}, err
}
return att, nil
}
const attachmentSelect = `
SELECT a.id, a.chat_id, a.owner_id, a.filename, a.content_type,
a.size_bytes, a.sha256, a.status, a.error_code,
a.created_at, a.updated_at, a.object_key, COALESCE(a.staging_key, ''),
a.pipeline_version, a.variants
FROM attachments a`
func scanAttachment(row rowScanner) (model.Attachment, error) {
var (
att model.Attachment
hash []byte
errCode *string
createdAt time.Time
updatedAt time.Time
variants []byte
)
if err := row.Scan(
&att.ID, &att.ChatID, &att.OwnerID, &att.Filename, &att.ContentType,
&att.SizeBytes, &hash, &att.Status, &errCode,
&createdAt, &updatedAt, &att.ObjectKey, &att.StagingKey,
&att.PipelineVersion, &variants,
); err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return model.Attachment{}, common.NotFound("attachment not found")
}
return model.Attachment{}, dbError(err)
}
att.SHA256 = hex.EncodeToString(hash)
att.ErrorCode = errCode
att.CreatedAt = common.RFCFormTime(createdAt.UTC())
att.UpdatedAt = common.RFCFormTime(updatedAt.UTC())
att.Variants = []model.Variant{}
if len(variants) > 0 {
if err := json.Unmarshal(variants, &att.Variants); err != nil {
return model.Attachment{}, dbError(err)
}
}
if att.Variants == nil {
att.Variants = []model.Variant{}
}
return att, nil
}
// ListStaleStagingAttachments returns terminal attachments whose staging copy
// may be removed. The grace period is applied to updated_at so a just-finished
// job is never cleaned while its commit is still settling.
func (s *Store) ListStaleStagingAttachments(ctx context.Context, cutoff time.Time, limit int) ([]model.Attachment, error) {
ctx, cancel := s.withTimeout(ctx)
defer cancel()
rows, err := s.pool.Query(ctx, attachmentSelect+`
WHERE a.status IN ('ready', 'failed')
AND a.updated_at < $1
AND a.staging_key IS NOT NULL
ORDER BY a.updated_at
LIMIT $2`, cutoff, limit)
if err != nil {
return nil, dbError(err)
}
defer rows.Close()
var result []model.Attachment
for rows.Next() {
att, scanErr := scanAttachment(rows)
if scanErr != nil {
return nil, scanErr
}
result = append(result, att)
}
if err = rows.Err(); err != nil {
return nil, dbError(err)
}
return result, nil
}
func (s *Store) ClearStagingKey(ctx context.Context, attachmentID uuid.UUID) error {
ctx, cancel := s.withTimeout(ctx)
defer cancel()
_, err := s.pool.Exec(ctx,
`UPDATE attachments SET staging_key = NULL WHERE id = $1`, attachmentID)
return dbError(err)
}
// ListStorageKeys returns every object the database still owns. It includes
// variant keys derived from the original key; the reconciler must never delete
// a thumbnail that has no separate row.
func (s *Store) ListStorageKeys(ctx context.Context) (map[string]struct{}, error) {
ctx, cancel := s.withTimeout(ctx)
defer cancel()
rows, err := s.pool.Query(ctx,
`SELECT object_key, COALESCE(staging_key, ''), variants FROM attachments`)
if err != nil {
return nil, dbError(err)
}
defer rows.Close()
keys := make(map[string]struct{})
for rows.Next() {
var (
objectKey string
staging string
variants []byte
)
if err = rows.Scan(&objectKey, &staging, &variants); err != nil {
return nil, dbError(err)
}
keys[objectKey] = struct{}{}
if staging != "" {
keys[staging] = struct{}{}
}
var list []model.Variant
if err = json.Unmarshal(variants, &list); err != nil {
return nil, dbError(err)
}
for _, variant := range list {
keys[VariantObjectKey(objectKey, variant.Name)] = struct{}{}
}
}
if err = rows.Err(); err != nil {
return nil, dbError(err)
}
return keys, nil
}
// VariantObjectKey is the one place that names variant objects. Keeping it
// deterministic makes a retried job overwrite the same keys instead of
// creating a second variant set.
func VariantObjectKey(originalKey, variant string) string {
return originalKey + "/" + variant
}
+142
View File
@@ -0,0 +1,142 @@
package store
import (
"context"
"errors"
"time"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/model"
"github.com/google/uuid"
"github.com/jackc/pgx/v5"
)
func (s *Store) CreateChat(ctx context.Context, title string, memberIDs []uuid.UUID) (model.Chat, error) {
ctx, cancel := s.withTimeout(ctx)
defer cancel()
tx, err := s.pool.Begin(ctx)
if err != nil {
return model.Chat{}, dbError(err)
}
defer rollback(tx)
var existing int
if err = tx.QueryRow(ctx,
`SELECT count(*) FROM users WHERE id = ANY($1::uuid[])`,
memberIDs).Scan(&existing); err != nil {
return model.Chat{}, dbError(err)
}
if existing != len(memberIDs) {
return model.Chat{}, common.BadRequest("member_id refers to non-existent user")
}
chat := model.Chat{
ID: uuid.New(),
Title: title,
MemberIDs: append([]uuid.UUID(nil), memberIDs...),
CreatedAt: common.NowUTC(),
}
if _, err = tx.Exec(ctx,
`INSERT INTO chats (id, title, created_at) VALUES ($1, $2, $3)`,
chat.ID, chat.Title, time.Time(chat.CreatedAt)); err != nil {
return model.Chat{}, dbError(err)
}
for _, memberID := range memberIDs {
if _, err = tx.Exec(ctx,
`INSERT INTO chat_members (chat_id, user_id) VALUES ($1, $2)`,
chat.ID, memberID); err != nil {
return model.Chat{}, dbError(err)
}
}
if err = tx.Commit(ctx); err != nil {
return model.Chat{}, dbError(err)
}
return chat, nil
}
func (s *Store) GetChatForUser(ctx context.Context, userID, chatID uuid.UUID) (model.Chat, error) {
ctx, cancel := s.withTimeout(ctx)
defer cancel()
return scanChat(s.pool.QueryRow(ctx, `
SELECT c.id, c.title, c.created_at,
ARRAY(SELECT user_id FROM chat_members WHERE chat_id = c.id ORDER BY user_id)
FROM chats c
JOIN chat_members me ON me.chat_id = c.id AND me.user_id = $2
WHERE c.id = $1`, chatID, userID))
}
func (s *Store) ListChats(ctx context.Context, userID uuid.UUID, limit int, cursor *common.CursorKeyCtx) (*common.Page[model.Chat], error) {
ctx, cancel := s.withTimeout(ctx)
defer cancel()
cursorTime, cursorID := cursorValues(cursor)
rows, err := s.pool.Query(ctx, `
SELECT c.id, c.title, c.created_at,
ARRAY(SELECT user_id FROM chat_members WHERE chat_id = c.id ORDER BY user_id)
FROM chats c
JOIN chat_members me ON me.chat_id = c.id AND me.user_id = $1
WHERE (c.created_at, c.id) > ($2, $3)
ORDER BY c.created_at, c.id
LIMIT $4`, userID, cursorTime, cursorID, limit+1)
if err != nil {
return nil, dbError(err)
}
defer rows.Close()
chats := make([]model.Chat, 0, limit)
for rows.Next() {
chat, scanErr := scanChat(rows)
if scanErr != nil {
return nil, scanErr
}
chats = append(chats, chat)
}
if err = rows.Err(); err != nil {
return nil, dbError(err)
}
hasMore := len(chats) > limit
if hasMore {
chats = chats[:limit]
}
return &common.Page[model.Chat]{Items: chats, NextCursor: common.NextCursor(chats, hasMore, userID)}, nil
}
func (s *Store) IsMember(ctx context.Context, userID, chatID uuid.UUID) (bool, error) {
ctx, cancel := s.withTimeout(ctx)
defer cancel()
var exists int
err := s.pool.QueryRow(ctx,
`SELECT 1 FROM chat_members WHERE chat_id = $1 AND user_id = $2`,
chatID, userID).Scan(&exists)
if errors.Is(err, pgx.ErrNoRows) {
return false, nil
}
if err != nil {
return false, dbError(err)
}
return true, nil
}
// scanChat owns the "row is missing" meaning for single-row lookups; list
// queries never see ErrNoRows.
func scanChat(row rowScanner) (model.Chat, error) {
var chat model.Chat
var createdAt time.Time
if err := row.Scan(&chat.ID, &chat.Title, &createdAt, &chat.MemberIDs); err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return model.Chat{}, common.NotFound("chat not found")
}
return model.Chat{}, dbError(err)
}
chat.CreatedAt = common.RFCFormTime(createdAt.UTC())
return chat, nil
}
+44
View File
@@ -0,0 +1,44 @@
package store
import (
"context"
"errors"
"log/slog"
"net"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgconn"
)
func isUniqueViolation(err error) bool {
pgErr, ok := errors.AsType[*pgconn.PgError](err)
return ok && pgErr.Code == "23505"
}
// dbError maps driver failures to contract errors. Unexpected failures are
// logged with their cause so a 500 is never silent.
func dbError(err error) error {
if err == nil {
return nil
}
if pgErr, ok := errors.AsType[*pgconn.PgError](err); ok {
switch pgErr.Code {
case "23505":
return common.Conflict("resource already exists")
case "23503", "23514", "22P02":
return common.BadRequest("request conflicts with stored data")
}
}
if _, ok := errors.AsType[net.Error](err); ok || errors.Is(err, context.DeadlineExceeded) ||
errors.Is(err, context.Canceled) ||
pgconn.SafeToRetry(err) ||
errors.Is(err, pgx.ErrTxClosed) {
return common.Unavailable("database unavailable")
}
slog.Error("unhandled database error", "err", err)
return common.Internal()
}
+445
View File
@@ -0,0 +1,445 @@
package store
import (
"context"
"encoding/json"
"errors"
"time"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/model"
"github.com/google/uuid"
"github.com/jackc/pgx/v5"
)
// Job state machine. Every transition is a compare-and-set on
// (job_id, attempt, status), so broker duplicates and stale workers are
// harmless.
//
// queued --ClaimJob--> processing --CompleteJob--> ready
// |
// +--ScheduleRetry--> queued (future available_at)
// +--FailJob--------> failed
// +--attempt >= max-> failed/retry_exhausted
// processing with an expired lease --RecoverExpiredLeases--> queued/failed
// failed --ReplayFailed--> queued (attempt=0, replay_count+1)
//
// ClaimResult tells the consumer whether this delivery owns the work.
type ClaimResult struct {
Job model.JobRecord
Attachment model.Attachment
// Terminal means the logical work is already ready/failed. Ack it.
Terminal bool
// Busy means another attempt owns an active lease, or a retry is not due.
// The publisher/lease sweeper will make the job visible again.
Busy bool
}
// ClaimJob acquires or renews the logical job lease in one transaction.
// A broker redelivery of the same job_id therefore cannot become two workers
// writing results at the same time.
func (s *Store) ClaimJob(ctx context.Context, jobID uuid.UUID, lease time.Duration) (ClaimResult, error) {
ctx, cancel := s.withTimeout(ctx)
defer cancel()
tx, err := s.pool.Begin(ctx)
if err != nil {
return ClaimResult{}, dbError(err)
}
defer rollback(tx)
job, err := scanJob(tx.QueryRow(ctx, `
SELECT job_id, attachment_id, type, pipeline_version, attempt, max_attempts,
status, error_code, created_at, available_at, published_at,
lease_until, processing_started_at, finished_at, replay_count
FROM attachment_jobs WHERE job_id = $1 FOR UPDATE`, jobID))
if err != nil {
return ClaimResult{}, err
}
att, err := scanAttachment(tx.QueryRow(ctx, attachmentSelect+` WHERE a.id = $1 FOR UPDATE`, job.AttachmentID))
if err != nil {
return ClaimResult{}, err
}
now := time.Now().UTC()
if job.Status == model.JobStatusReady || job.Status == model.JobStatusFailed || att.Status == "ready" {
if err = tx.Commit(ctx); err != nil {
return ClaimResult{}, dbError(err)
}
return ClaimResult{Job: job, Attachment: att, Terminal: true}, nil
}
if att.Status == "failed" {
if err = tx.Commit(ctx); err != nil {
return ClaimResult{}, dbError(err)
}
return ClaimResult{Job: job, Attachment: att, Terminal: true}, nil
}
if job.Status == model.JobStatusProcessing && job.LeaseUntil != nil && job.LeaseUntil.After(now) {
if err = tx.Commit(ctx); err != nil {
return ClaimResult{}, dbError(err)
}
return ClaimResult{Job: job, Attachment: att, Busy: true}, nil
}
if job.Status == model.JobStatusQueued && job.AvailableAt.After(now) {
if err = tx.Commit(ctx); err != nil {
return ClaimResult{}, dbError(err)
}
return ClaimResult{Job: job, Attachment: att, Busy: true}, nil
}
if job.Attempt >= job.MaxAttempts {
_, failErr := tx.Exec(ctx, `
UPDATE attachment_jobs
SET status = 'failed', error_code = 'retry_exhausted',
finished_at = $2, lease_until = NULL, updated_at = $2
WHERE job_id = $1`, jobID, now)
if failErr != nil {
return ClaimResult{}, dbError(failErr)
}
_, failErr = tx.Exec(ctx, `
UPDATE attachments
SET status = 'failed', error_code = 'retry_exhausted', updated_at = $2
WHERE id = $1 AND status <> 'ready'`, job.AttachmentID, now)
if failErr != nil {
return ClaimResult{}, dbError(failErr)
}
if err = tx.Commit(ctx); err != nil {
return ClaimResult{}, dbError(err)
}
return ClaimResult{Job: job, Attachment: att, Terminal: true}, nil
}
attempt := job.Attempt + 1
leaseUntil := now.Add(lease)
_, err = tx.Exec(ctx, `
UPDATE attachment_jobs
SET status = 'processing', attempt = $2, error_code = NULL,
lease_until = $3, processing_started_at = COALESCE(processing_started_at, $4),
published_at = COALESCE(published_at, $4), updated_at = $4
WHERE job_id = $1`, jobID, attempt, leaseUntil, now)
if err != nil {
return ClaimResult{}, dbError(err)
}
_, err = tx.Exec(ctx, `
UPDATE attachments
SET status = 'processing', error_code = NULL, updated_at = $2
WHERE id = $1 AND status IN ('queued', 'processing')`, job.AttachmentID, now)
if err != nil {
return ClaimResult{}, dbError(err)
}
if err = tx.Commit(ctx); err != nil {
return ClaimResult{}, dbError(err)
}
job.Attempt = attempt
job.Status = model.JobStatusProcessing
job.ErrorCode = nil
job.LeaseUntil = &leaseUntil
if job.ProcessingStartedAt == nil {
job.ProcessingStartedAt = &now
}
att.Status = "processing"
att.ErrorCode = nil
att.UpdatedAt = common.RFCFormTime(now)
return ClaimResult{Job: job, Attachment: att}, nil
}
// CompleteJob commits the ready snapshot only for the attempt/version that
// still owns the lease. An older duplicate can therefore never roll a newer
// result back or overwrite it.
func (s *Store) CompleteJob(ctx context.Context, jobID uuid.UUID, attempt int, variants []model.Variant) (bool, error) {
ctx, cancel := s.withTimeout(ctx)
defer cancel()
if variants == nil {
variants = []model.Variant{}
}
encoded, err := json.Marshal(variants)
if err != nil {
return false, common.Internal()
}
tx, err := s.pool.Begin(ctx)
if err != nil {
return false, dbError(err)
}
defer rollback(tx)
var attachmentID uuid.UUID
var pipelineVersion int
var status string
err = tx.QueryRow(ctx, `
SELECT attachment_id, pipeline_version, status
FROM attachment_jobs WHERE job_id = $1 FOR UPDATE`, jobID).
Scan(&attachmentID, &pipelineVersion, &status)
if errors.Is(err, pgx.ErrNoRows) {
return false, common.NotFound("job not found")
}
if err != nil {
return false, dbError(err)
}
if status != model.JobStatusProcessing {
return false, nil
}
att, err := scanAttachment(tx.QueryRow(ctx, attachmentSelect+` WHERE a.id = $1 FOR UPDATE`, attachmentID))
if err != nil {
return false, err
}
if att.PipelineVersion != pipelineVersion || att.Status == "ready" || att.Status == "failed" {
return false, nil
}
now := time.Now().UTC()
tag, err := tx.Exec(ctx, `
UPDATE attachment_jobs
SET status = 'ready', error_code = NULL, finished_at = $3,
lease_until = NULL, updated_at = $3
WHERE job_id = $1 AND attempt = $2 AND status = 'processing'`,
jobID, attempt, now)
if err != nil {
return false, dbError(err)
}
if tag.RowsAffected() == 0 {
return false, nil
}
_, err = tx.Exec(ctx, `
UPDATE attachments
SET status = 'ready', variants = $2, error_code = NULL,
ready_at = $3, updated_at = $3
WHERE id = $1 AND pipeline_version = $4 AND status IN ('queued', 'processing')`,
attachmentID, encoded, now, pipelineVersion)
if err != nil {
return false, dbError(err)
}
if err = tx.Commit(ctx); err != nil {
return false, dbError(err)
}
return true, nil
}
// ScheduleRetry keeps the job in the durable outbox with a jittered delay.
// The caller acks the broker only after this commit, so the retry intent is
// never lost between "try later" and ack.
func (s *Store) ScheduleRetry(ctx context.Context, jobID uuid.UUID, attempt int, code string, delay time.Duration) (bool, error) {
ctx, cancel := s.withTimeout(ctx)
defer cancel()
tx, err := s.pool.Begin(ctx)
if err != nil {
return false, dbError(err)
}
defer rollback(tx)
now := time.Now().UTC()
tag, err := tx.Exec(ctx, `
UPDATE attachment_jobs
SET status = 'queued', error_code = $3, available_at = $4,
published_at = NULL, lease_until = NULL, updated_at = $5
WHERE job_id = $1 AND status = 'processing' AND attempt = $2`,
jobID, attempt, code, now.Add(delay), now)
if err != nil {
return false, dbError(err)
}
if tag.RowsAffected() == 0 {
return false, nil
}
_, err = tx.Exec(ctx, `
UPDATE attachments
SET status = 'queued', error_code = NULL, updated_at = $2
WHERE id = (SELECT attachment_id FROM attachment_jobs WHERE job_id = $1)
AND status <> 'ready'`, jobID, now)
if err != nil {
return false, dbError(err)
}
return true, dbError(tx.Commit(ctx))
}
// FailJob records a terminal reason. It is idempotent: a later stale attempt
// that no longer matches the current attempt is ignored.
func (s *Store) FailJob(ctx context.Context, jobID uuid.UUID, attempt int, code string) error {
ctx, cancel := s.withTimeout(ctx)
defer cancel()
tx, err := s.pool.Begin(ctx)
if err != nil {
return dbError(err)
}
defer rollback(tx)
now := time.Now().UTC()
tag, err := tx.Exec(ctx, `
UPDATE attachment_jobs
SET status = 'failed', error_code = $3, finished_at = $4,
lease_until = NULL, updated_at = $4
WHERE job_id = $1 AND status = 'processing' AND attempt = $2`,
jobID, attempt, code, now)
if err != nil {
return dbError(err)
}
if tag.RowsAffected() == 0 {
return nil
}
_, err = tx.Exec(ctx, `
UPDATE attachments
SET status = 'failed', error_code = $2, updated_at = $3
WHERE id = (SELECT attachment_id FROM attachment_jobs WHERE job_id = $1)
AND status <> 'ready'`, jobID, code, now)
if err != nil {
return dbError(err)
}
return dbError(tx.Commit(ctx))
}
// RecoverExpiredLeases is the stuck-processing guard. A dead worker leaves a
// processing row whose lease expires; the sweeper requeues it or marks it
// retry_exhausted. It never touches ready work.
func (s *Store) RecoverExpiredLeases(ctx context.Context) (int, error) {
ctx, cancel := s.withTimeout(ctx)
defer cancel()
tag, err := s.pool.Exec(ctx, `
WITH expired AS (
UPDATE attachment_jobs j
SET status = CASE WHEN j.attempt >= j.max_attempts THEN 'failed' ELSE 'queued' END,
error_code = CASE WHEN j.attempt >= j.max_attempts THEN 'retry_exhausted' ELSE NULL END,
available_at = now(),
published_at = NULL,
lease_until = NULL,
finished_at = CASE WHEN j.attempt >= j.max_attempts THEN now() ELSE NULL END,
updated_at = now()
WHERE j.status = 'processing'
AND j.lease_until IS NOT NULL
AND j.lease_until < now()
RETURNING j.attachment_id, j.status, j.error_code
)
UPDATE attachments a
SET status = CASE WHEN e.status = 'failed' THEN 'failed' ELSE 'queued' END,
error_code = CASE WHEN e.status = 'failed' THEN e.error_code ELSE NULL END,
updated_at = now()
FROM expired e
WHERE a.id = e.attachment_id AND a.status <> 'ready'`)
if err != nil {
return 0, dbError(err)
}
return int(tag.RowsAffected()), nil
}
// ListPublishableJobs is the outbox read side. It intentionally does not lock
// rows: two API replicas may publish the same message, and the idempotent
// claim/effect makes that duplicate harmless.
func (s *Store) ListPublishableJobs(ctx context.Context, limit int) ([]model.JobRecord, error) {
ctx, cancel := s.withTimeout(ctx)
defer cancel()
rows, err := s.pool.Query(ctx, `
SELECT job_id, attachment_id, type, pipeline_version, attempt, max_attempts,
status, error_code, created_at, available_at, published_at,
lease_until, processing_started_at, finished_at, replay_count
FROM attachment_jobs
WHERE published_at IS NULL AND status = 'queued' AND available_at <= now()
ORDER BY available_at, created_at
LIMIT $1`, limit)
if err != nil {
return nil, dbError(err)
}
defer rows.Close()
var jobs []model.JobRecord
for rows.Next() {
job, scanErr := scanJob(rows)
if scanErr != nil {
return nil, scanErr
}
jobs = append(jobs, job)
}
if err = rows.Err(); err != nil {
return nil, dbError(err)
}
return jobs, nil
}
// MarkJobPublished uses status+attempt as a compare-and-set. If a worker has
// already claimed the job, or a retry has superseded this publish, the update
// is a no-op and the outbox will publish the current intent later.
func (s *Store) MarkJobPublished(ctx context.Context, jobID uuid.UUID, attempt int, at time.Time) error {
ctx, cancel := s.withTimeout(ctx)
defer cancel()
_, err := s.pool.Exec(ctx, `
UPDATE attachment_jobs
SET published_at = $3, updated_at = $3
WHERE job_id = $1 AND attempt = $2 AND status = 'queued' AND published_at IS NULL`,
jobID, attempt, at.UTC())
return dbError(err)
}
// ReplayFailed is the explicit operator action. The status guard means a
// second click, or a replay after success, cannot duplicate a ready effect.
func (s *Store) ReplayFailed(ctx context.Context, attachmentID uuid.UUID) (bool, error) {
ctx, cancel := s.withTimeout(ctx)
defer cancel()
tx, err := s.pool.Begin(ctx)
if err != nil {
return false, dbError(err)
}
defer rollback(tx)
now := time.Now().UTC()
tag, err := tx.Exec(ctx, `
UPDATE attachment_jobs
SET status = 'queued', attempt = 0, error_code = NULL,
available_at = $2, published_at = NULL, lease_until = NULL,
processing_started_at = NULL, finished_at = NULL,
replay_count = replay_count + 1, last_replay_at = $2, updated_at = $2
WHERE attachment_id = $1 AND status = 'failed'
AND EXISTS (SELECT 1 FROM attachments
WHERE id = $1 AND status = 'failed')`,
attachmentID, now)
if err != nil {
return false, dbError(err)
}
if tag.RowsAffected() == 0 {
return false, nil
}
_, err = tx.Exec(ctx, `
UPDATE attachments
SET status = 'queued', error_code = NULL, updated_at = $2
WHERE id = $1 AND status = 'failed'`, attachmentID, now)
if err != nil {
return false, dbError(err)
}
if err = tx.Commit(ctx); err != nil {
return false, dbError(err)
}
return true, nil
}
func scanJob(row rowScanner) (model.JobRecord, error) {
var (
job model.JobRecord
created time.Time
errCode *string
)
if err := row.Scan(
&job.JobID, &job.AttachmentID, &job.Type, &job.PipelineVersion,
&job.Attempt, &job.MaxAttempts, &job.Status, &errCode, &created,
&job.AvailableAt, &job.PublishedAt, &job.LeaseUntil,
&job.ProcessingStartedAt, &job.FinishedAt, &job.ReplayCount,
); err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return model.JobRecord{}, common.NotFound("job not found")
}
return model.JobRecord{}, dbError(err)
}
job.SchemaVersion = 1
job.CreatedAt = common.RFCFormTime(created.UTC())
job.ErrorCode = errCode
return job, nil
}
+287
View File
@@ -0,0 +1,287 @@
package store
import (
"bytes"
"context"
"crypto/sha256"
"errors"
"sort"
"time"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/model"
"github.com/google/uuid"
"github.com/jackc/pgx/v5"
)
func (s *Store) CreateMessage(ctx context.Context, chatID, senderID uuid.UUID, text, idempotencyKey string, attachmentIDs []uuid.UUID) (model.Message, error) {
ctx, cancel := s.withTimeout(ctx)
defer cancel()
// Sorting makes the idempotency fingerprint independent of the order
// in which the client listed the same set of attachment IDs. The response
// uses the same deterministic order.
attachmentIDs = canonicalAttachmentIDs(attachmentIDs)
fingerprint := messageFingerprint(text, attachmentIDs)
if message, found, err := s.findIdempotentMessage(ctx, senderID, chatID, idempotencyKey, fingerprint); err != nil {
return model.Message{}, err
} else if found {
return message, nil
}
message := model.Message{
ID: uuid.New(),
ChatID: chatID,
SenderID: senderID,
Text: text,
CreatedAt: common.NowUTC(),
AttachmentIDs: attachmentIDs,
}
tx, err := s.pool.Begin(ctx)
if err != nil {
return model.Message{}, dbError(err)
}
defer rollback(tx)
// Check membership and attachment ownership in the same transaction as the
// insert. A foreign chat or a foreign attachment is intentionally the same
// 404 as a missing object, so the API does not leak existence.
var member int
err = tx.QueryRow(ctx,
`SELECT 1 FROM chat_members WHERE chat_id = $1 AND user_id = $2`,
chatID, senderID).Scan(&member)
if errors.Is(err, pgx.ErrNoRows) {
return model.Message{}, common.NotFound("chat not found")
}
if err != nil {
return model.Message{}, dbError(err)
}
if len(attachmentIDs) > 0 {
rows, queryErr := tx.Query(ctx, `
SELECT id, status FROM attachments
WHERE chat_id = $1 AND id = ANY($2::uuid[])`, chatID, attachmentIDs)
if queryErr != nil {
return model.Message{}, dbError(queryErr)
}
found := 0
for rows.Next() {
var (
id uuid.UUID
status string
)
if scanErr := rows.Scan(&id, &status); scanErr != nil {
rows.Close()
return model.Message{}, dbError(scanErr)
}
found++
// A queued/processing attachment is already linked to its
// chat and may be sent. Only a terminal failed one is refused.
if status == "failed" {
rows.Close()
return model.Message{}, common.Conflict("attachment failed processing")
}
}
if err = rows.Err(); err != nil {
rows.Close()
return model.Message{}, dbError(err)
}
rows.Close()
if found != len(attachmentIDs) {
return model.Message{}, common.NotFound("attachment not found")
}
}
_, err = tx.Exec(ctx, `
INSERT INTO messages (id, chat_id, sender_id, text, created_at)
VALUES ($1, $2, $3, $4, $5)`,
message.ID, message.ChatID, message.SenderID, message.Text, time.Time(message.CreatedAt))
if err != nil {
return model.Message{}, dbError(err)
}
for position, attachmentID := range attachmentIDs {
if _, err = tx.Exec(ctx, `
INSERT INTO message_attachments (message_id, attachment_id, position)
VALUES ($1, $2, $3)`, message.ID, attachmentID, position); err != nil {
return model.Message{}, dbError(err)
}
}
expiresAt := time.Time(message.CreatedAt).Add(s.idempotencyTTL)
tag, err := tx.Exec(ctx, `
INSERT INTO message_idempotency
(sender_id, chat_id, idempotency_key, request_hash, message_id, created_at, expires_at)
VALUES ($1, $2, $3, $4, $5, $6, $7)
ON CONFLICT (sender_id, chat_id, idempotency_key) DO UPDATE
SET request_hash = EXCLUDED.request_hash,
message_id = EXCLUDED.message_id,
created_at = EXCLUDED.created_at,
expires_at = EXCLUDED.expires_at
WHERE message_idempotency.expires_at <= now()`,
senderID, chatID, idempotencyKey, fingerprint, message.ID,
time.Time(message.CreatedAt), expiresAt)
if err != nil {
return model.Message{}, dbError(err)
}
if tag.RowsAffected() == 0 {
rollback(tx)
stored, found, lookupErr := s.findIdempotentMessage(ctx, senderID, chatID, idempotencyKey, fingerprint)
if lookupErr != nil {
return model.Message{}, lookupErr
}
if !found {
return model.Message{}, common.Internal()
}
return stored, nil
}
if err = tx.Commit(ctx); err != nil {
return model.Message{}, dbError(err)
}
return message, nil
}
func canonicalAttachmentIDs(ids []uuid.UUID) []uuid.UUID {
result := append([]uuid.UUID(nil), ids...)
sort.Slice(result, func(i, j int) bool {
return bytes.Compare(result[i][:], result[j][:]) < 0
})
if result == nil {
return []uuid.UUID{}
}
return result
}
// messageFingerprint keeps the legacy hash for messages without attachments,
// so an idempotency key stored before this migration still replays correctly.
// For messages with attachments the sorted IDs are part of the canonical body.
func messageFingerprint(text string, attachmentIDs []uuid.UUID) []byte {
if len(attachmentIDs) == 0 {
sum := sha256.Sum256([]byte(text))
return sum[:]
}
hash := sha256.New()
hash.Write([]byte(text))
hash.Write([]byte{0})
for _, id := range attachmentIDs {
hash.Write(id[:])
}
return hash.Sum(nil)
}
func (s *Store) findIdempotentMessage(ctx context.Context, senderID, chatID uuid.UUID, idempotencyKey string, expectedHash []byte) (model.Message, bool, error) {
var message model.Message
var createdAt time.Time
var storedHash []byte
err := s.pool.QueryRow(ctx, `
SELECT m.id, m.chat_id, m.sender_id, m.text, m.created_at,
ARRAY(SELECT ma.attachment_id
FROM message_attachments ma
WHERE ma.message_id = m.id
ORDER BY ma.position),
i.request_hash
FROM message_idempotency i
JOIN messages m ON m.id = i.message_id
JOIN chat_members cm ON cm.chat_id = i.chat_id AND cm.user_id = i.sender_id
WHERE i.sender_id = $1
AND i.chat_id = $2
AND i.idempotency_key = $3
AND i.expires_at > now()`,
senderID, chatID, idempotencyKey).Scan(
&message.ID, &message.ChatID, &message.SenderID, &message.Text, &createdAt,
&message.AttachmentIDs, &storedHash)
if errors.Is(err, pgx.ErrNoRows) {
return model.Message{}, false, nil
}
if err != nil {
return model.Message{}, false, dbError(err)
}
if !bytes.Equal(storedHash, expectedHash) {
return model.Message{}, false, common.Conflict("idempotency key reused with a different message")
}
if message.AttachmentIDs == nil {
message.AttachmentIDs = []uuid.UUID{}
}
message.CreatedAt = common.RFCFormTime(createdAt.UTC())
return message, true, nil
}
func (s *Store) ListMessages(ctx context.Context, userID, chatID uuid.UUID, limit int, cursor *common.CursorKeyCtx) (*common.Page[model.Message], error) {
ctx, cancel := s.withTimeout(ctx)
defer cancel()
isMember, err := s.IsMember(ctx, userID, chatID)
if err != nil {
return nil, err
}
if !isMember {
return nil, common.NotFound("chat not found")
}
// AGENT: Cursor context is checked only after membership, so a foreign
// caller gets 404 and cannot probe cursor validity against someone else's chat.
if cursor != nil && cursor.ContextID != chatID {
return nil, common.BadRequest("bad cursor value")
}
cursorTime, cursorID := cursorValues(cursor)
rows, err := s.pool.Query(ctx, `
SELECT m.id, m.chat_id, m.sender_id, m.text, m.created_at,
ARRAY(SELECT ma.attachment_id
FROM message_attachments ma
WHERE ma.message_id = m.id
ORDER BY ma.position)
FROM messages m
WHERE m.chat_id = $1 AND (m.created_at, m.id) > ($2, $3)
ORDER BY m.created_at, m.id
LIMIT $4`, chatID, cursorTime, cursorID, limit+1)
if err != nil {
return nil, dbError(err)
}
defer rows.Close()
messages := make([]model.Message, 0, limit)
for rows.Next() {
message, scanErr := scanMessage(rows)
if scanErr != nil {
return nil, dbError(scanErr)
}
messages = append(messages, message)
}
if err = rows.Err(); err != nil {
return nil, dbError(err)
}
hasMore := len(messages) > limit
if hasMore {
messages = messages[:limit]
}
return &common.Page[model.Message]{
Items: messages,
NextCursor: common.NextCursor(messages, hasMore, chatID),
}, nil
}
func scanMessage(row rowScanner) (model.Message, error) {
var message model.Message
var createdAt time.Time
if err := row.Scan(&message.ID, &message.ChatID, &message.SenderID, &message.Text,
&createdAt, &message.AttachmentIDs); err != nil {
return model.Message{}, err
}
if message.AttachmentIDs == nil {
message.AttachmentIDs = []uuid.UUID{}
}
message.CreatedAt = common.RFCFormTime(createdAt.UTC())
return message, nil
}
+80
View File
@@ -0,0 +1,80 @@
package store
import (
"bytes"
"crypto/sha256"
"testing"
"github.com/google/uuid"
)
func TestCanonicalAttachmentIDsSortsAndCopies(t *testing.T) {
first, second, third := uuid.MustParse("00000000-0000-0000-0000-000000000003"),
uuid.MustParse("00000000-0000-0000-0000-000000000001"),
uuid.MustParse("00000000-0000-0000-0000-000000000002")
input := []uuid.UUID{first, second, third}
got := canonicalAttachmentIDs(input)
want := []uuid.UUID{second, third, first}
if !equalIDs(got, want) {
t.Fatalf("canonicalAttachmentIDs = %v, want %v", got, want)
}
if !equalIDs(input, []uuid.UUID{first, second, third}) {
t.Fatalf("canonicalAttachmentIDs mutated its input: %v", input)
}
}
func TestCanonicalAttachmentIDsReturnsEmptySliceForNil(t *testing.T) {
got := canonicalAttachmentIDs(nil)
if got == nil {
t.Fatal("canonicalAttachmentIDs(nil) = nil, want empty slice")
}
if len(got) != 0 {
t.Fatalf("canonicalAttachmentIDs(nil) = %v, want empty", got)
}
}
// TestMessageFingerprintWithoutAttachmentsPreservesLab3Hash is the
// compatibility guard: idempotency rows written before attachments existed
// must still match on replay.
func TestMessageFingerprintWithoutAttachmentsPreservesLab3Hash(t *testing.T) {
expected := sha256.Sum256([]byte("hello"))
if got := messageFingerprint("hello", nil); !bytes.Equal(got, expected[:]) {
t.Fatalf("fingerprint = %x, want %x", got, expected)
}
if got := messageFingerprint("hello", []uuid.UUID{}); !bytes.Equal(got, expected[:]) {
t.Fatalf("empty slice fingerprint = %x, want %x", got, expected)
}
}
func TestMessageFingerprintDependsOnSortedAttachments(t *testing.T) {
first := uuid.MustParse("00000000-0000-0000-0000-000000000001")
second := uuid.MustParse("00000000-0000-0000-0000-000000000002")
// CreateMessage canonicalizes before hashing; the pair must therefore be
// order independent even though messageFingerprint itself takes the list
// as given.
one := messageFingerprint("hello", canonicalAttachmentIDs([]uuid.UUID{first, second}))
reordered := messageFingerprint("hello", canonicalAttachmentIDs([]uuid.UUID{second, first}))
if !bytes.Equal(one, reordered) {
t.Fatal("fingerprint depends on attachment order")
}
other := messageFingerprint("hello", canonicalAttachmentIDs([]uuid.UUID{first}))
if bytes.Equal(one, other) {
t.Fatal("fingerprint ignores attachment identity")
}
}
func equalIDs(a, b []uuid.UUID) bool {
if len(a) != len(b) {
return false
}
for index := range a {
if a[index] != b[index] {
return false
}
}
return true
}
+17
View File
@@ -0,0 +1,17 @@
package store
import (
"time"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
"github.com/google/uuid"
)
// cursorValues converts an optional page cursor to the two keyset columns used
// by every paginated query. A nil cursor means "start from the beginning".
func cursorValues(cursor *common.CursorKeyCtx) (time.Time, uuid.UUID) {
if cursor == nil {
return time.Time{}, uuid.Nil
}
return cursor.Time, cursor.ID
}
+36
View File
@@ -0,0 +1,36 @@
// Package store contains the PostgreSQL persistence layer. It uses plain
// parameterized SQL, one bounded pool and explicit transaction boundaries.
// Contract errors are returned as common.AppError so HTTP status mapping stays
// in one place.
package store
import (
"context"
"time"
"github.com/jackc/pgx/v5/pgxpool"
)
type Store struct {
pool *pgxpool.Pool
timeout time.Duration
idempotencyTTL time.Duration
}
func New(pool *pgxpool.Pool, timeout, idempotencyTTL time.Duration) *Store {
return &Store{pool: pool, timeout: timeout, idempotencyTTL: idempotencyTTL}
}
func (s *Store) withTimeout(ctx context.Context) (context.Context, context.CancelFunc) {
return context.WithTimeout(ctx, s.timeout)
}
func (s *Store) Ping(ctx context.Context) error {
pingCtx, cancel := context.WithTimeout(ctx, 2*time.Second)
defer cancel()
return s.pool.Ping(pingCtx)
}
type rowScanner interface {
Scan(dest ...any) error
}
+16
View File
@@ -0,0 +1,16 @@
package store
import (
"context"
"time"
"github.com/jackc/pgx/v5"
)
const rollbackTimeout = 5 * time.Second
func rollback(tx pgx.Tx) {
ctx, cancel := context.WithTimeout(context.Background(), rollbackTimeout)
defer cancel()
_ = tx.Rollback(ctx)
}
+69
View File
@@ -0,0 +1,69 @@
package store
import (
"context"
"errors"
"time"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/model"
"github.com/google/uuid"
"github.com/jackc/pgx/v5"
)
func (s *Store) CreateUser(ctx context.Context, username, displayName, passwordHash string) (model.User, error) {
ctx, cancel := s.withTimeout(ctx)
defer cancel()
user := model.User{
ID: uuid.New(),
Username: username,
DisplayName: displayName,
PasswordHash: passwordHash,
CreatedAt: common.NowUTC(),
}
_, err := s.pool.Exec(ctx, `
INSERT INTO users (id, username, display_name, password_hash, created_at)
VALUES ($1, $2, $3, $4, $5)`,
user.ID, user.Username, user.DisplayName, user.PasswordHash, time.Time(user.CreatedAt))
if err != nil {
if isUniqueViolation(err) {
return model.User{}, common.Conflict("username already taken")
}
return model.User{}, dbError(err)
}
return user, nil
}
func (s *Store) GetUserByID(ctx context.Context, id uuid.UUID) (model.User, error) {
ctx, cancel := s.withTimeout(ctx)
defer cancel()
return s.scanUser(s.pool.QueryRow(ctx, `
SELECT id, username, display_name, password_hash, created_at
FROM users WHERE id = $1`, id))
}
func (s *Store) GetUserByUsername(ctx context.Context, username string) (model.User, error) {
ctx, cancel := s.withTimeout(ctx)
defer cancel()
return s.scanUser(s.pool.QueryRow(ctx, `
SELECT id, username, display_name, password_hash, created_at
FROM users WHERE username = $1`, username))
}
func (s *Store) scanUser(row rowScanner) (model.User, error) {
var user model.User
var createdAt time.Time
if err := row.Scan(&user.ID, &user.Username, &user.DisplayName, &user.PasswordHash, &createdAt); err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return model.User{}, common.NotFound("user not found")
}
return model.User{}, dbError(err)
}
user.CreatedAt = common.RFCFormTime(createdAt.UTC())
return user, nil
}
+62
View File
@@ -0,0 +1,62 @@
// Package users implements registration and the current-user endpoint.
package users
import (
"log/slog"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/auth"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/store"
"github.com/gofiber/fiber/v3"
)
type Handler struct {
store *store.Store
jsonBodyLimit int64
}
func NewHandler(st *store.Store, jsonBodyLimit int64) *Handler {
return &Handler{store: st, jsonBodyLimit: jsonBodyLimit}
}
type CreateUserRequest struct {
Username string `json:"username"`
DisplayName string `json:"display_name"`
Password string `json:"password"`
}
func (h *Handler) CreateUser(c fiber.Ctx) error {
var req CreateUserRequest
if err := common.BindJSON(c, &req, h.jsonBodyLimit); err != nil {
return err
}
if err := common.ValidateCreateUser(req.Username, req.DisplayName); err != nil {
return err
}
if err := common.ValidatePassword(req.Password); err != nil {
return err
}
hash, err := auth.HashPassword(req.Password)
if err != nil {
slog.Error("hash password", "err", err)
return common.Internal()
}
user, err := h.store.CreateUser(c.Context(), req.Username, req.DisplayName, hash)
if err != nil {
return err
}
return c.Status(fiber.StatusCreated).JSON(user)
}
func (h *Handler) GetCurrentUser(c fiber.Ctx) error {
user, err := h.store.GetUserByID(c.Context(), common.MustUserID(c))
if err != nil {
return err
}
return c.JSON(user)
}
+308
View File
@@ -0,0 +1,308 @@
// Package worker runs the asynchronous attachment pipeline. It consumes one
// logical job, reads trusted metadata from PostgreSQL, loads the staged object,
// writes the original and variants, commits ready atomically, and only then
// lets the broker ack the message.
package worker
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"io"
"log/slog"
"math/rand/v2"
"os"
"sync/atomic"
"time"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/broker"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/config"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/model"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/objectstore"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/pipeline"
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/store"
)
type Worker struct {
cfg config.Config
store *store.Store
objects *objectstore.Client
pipeline *pipeline.Processor
logger *slog.Logger
faultOnce atomic.Bool
}
func New(cfg config.Config, st *store.Store, objects *objectstore.Client, logger *slog.Logger) *Worker {
if logger == nil {
logger = slog.Default()
}
w := &Worker{
cfg: cfg,
store: st,
objects: objects,
pipeline: pipeline.New(cfg.MaxImagePixels, cfg.ImageDecodeBytes),
logger: logger,
}
w.faultOnce.Store(true)
return w
}
// StartLeaseSweeper recovers a worker that died after claiming a job. Without
// this, the database would show "processing" forever and no message would be
// left to redeliver.
func (w *Worker) StartLeaseSweeper(ctx context.Context) {
ticker := time.NewTicker(w.cfg.LeaseSweep)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
count, err := w.store.RecoverExpiredLeases(ctx)
if err != nil {
w.logger.Error("lease sweep failed", "err", err)
} else if count > 0 {
w.logger.Warn("recovered expired job leases", "count", count)
}
}
}
}
// Handle is the broker callback. Returning nil means "durable effect or
// deliberate no-op; ack". ErrRequeue means the database itself was unavailable.
func (w *Worker) Handle(ctx context.Context, message model.Job) error {
claim, err := w.store.ClaimJob(ctx, message.JobID, w.cfg.JobLease)
if err != nil {
if common.IsCode(err, common.ErrorCodeNotFound) {
w.logger.Warn("job no longer exists; acking", "job_id", message.JobID)
return nil
}
w.logger.Error("claim job failed", "job_id", message.JobID, "err", err)
return broker.ErrRequeue
}
if claim.Terminal || claim.Busy {
w.logger.Info("job is already finished or owned",
"job_id", message.JobID, "terminal", claim.Terminal, "busy", claim.Busy,
"attempt", claim.Job.Attempt)
return nil
}
if claim.Attachment.ID != message.AttachmentID {
w.logger.Error("message attachment mismatch; acking poison message",
"job_id", message.JobID, "message_attachment", message.AttachmentID,
"database_attachment", claim.Attachment.ID)
return nil
}
log := w.logger.With(
"job_id", claim.Job.JobID,
"attachment_id", claim.Attachment.ID,
"attempt", claim.Job.Attempt,
"image_bytes", claim.Attachment.SizeBytes,
)
if w.cfg.CrashAfterClaim {
// Deterministic fault injection for the "claimed but never finished"
// window. The lease sweeper requeues the durable job after it expires;
// it does not depend on the broker redelivery still being around.
w.logger.Error("fault injection: exiting after claim before processing",
"job_id", claim.Job.JobID, "lease_until", claim.Job.LeaseUntil)
os.Exit(10)
}
log.Info("processing attachment")
jobCtx, cancel := context.WithTimeout(ctx, w.cfg.JobTimeout)
defer cancel()
if err = w.process(jobCtx, claim, log); err != nil {
if jobCtx.Err() != nil && errors.Is(jobCtx.Err(), context.DeadlineExceeded) {
err = &pipeline.Failure{Permanent: false, Code: "processing_timeout", Err: err}
}
return w.handleFailure(ctx, claim, err, log)
}
if w.cfg.CrashAfterEffect {
// Deterministic fault injection for the "written but not acked" window.
// The unacked delivery is redelivered after the connection dies.
w.logger.Error("fault injection: exiting after durable effect before ack",
"job_id", claim.Job.JobID)
os.Exit(9)
}
return nil
}
func (w *Worker) process(ctx context.Context, claim store.ClaimResult, log *slog.Logger) error {
if mode := w.cfg.WorkerFaultMode; mode != "none" && w.faultOnce.CompareAndSwap(true, false) {
if mode == "transient-once" {
return &pipeline.Failure{Permanent: false, Code: "processing_failed",
Err: errors.New("injected transient failure")}
}
return &pipeline.Failure{Permanent: true, Code: "invalid_image",
Err: errors.New("injected permanent failure")}
}
att := claim.Attachment
if att.StagingKey == "" {
return &pipeline.Failure{Permanent: true, Code: "staging_missing",
Err: errors.New("attachment has no staging key")}
}
reader, size, err := w.objects.Get(ctx, att.StagingKey)
if err != nil {
if errors.Is(err, objectstore.ErrNotFound) {
return &pipeline.Failure{Permanent: true, Code: "staging_missing", Err: err}
}
return &pipeline.Failure{Permanent: false, Code: "storage_unavailable", Err: err}
}
defer reader.Close()
if size != att.SizeBytes {
return &pipeline.Failure{Permanent: true, Code: "staging_corrupt",
Err: fmt.Errorf("staged size %d != metadata %d", size, att.SizeBytes)}
}
if err = os.MkdirAll(w.cfg.WorkTempDir, 0o700); err != nil {
return &pipeline.Failure{Permanent: false, Code: "processing_failed", Err: err}
}
file, err := os.CreateTemp(w.cfg.WorkTempDir, "job-*.part")
if err != nil {
return &pipeline.Failure{Permanent: false, Code: "processing_failed", Err: err}
}
path := file.Name()
defer func() {
_ = file.Close()
_ = os.Remove(path)
}()
hasher := sha256.New()
// The same 10 MiB contract limit protects the worker from a corrupt row
// pointing at an unexpectedly large object.
written, err := io.CopyBuffer(io.MultiWriter(file, hasher), io.LimitReader(reader, w.cfg.MaxUploadBytes+1), make([]byte, 32*1024))
if err != nil {
return &pipeline.Failure{Permanent: false, Code: "storage_unavailable", Err: err}
}
if written > w.cfg.MaxUploadBytes {
return &pipeline.Failure{Permanent: true, Code: "invalid_image", Err: errors.New("staged object too large")}
}
if written != att.SizeBytes || hex.EncodeToString(hasher.Sum(nil)) != att.SHA256 {
return &pipeline.Failure{Permanent: true, Code: "staging_corrupt", Err: errors.New("staged checksum mismatch")}
}
encoded, stats, err := w.pipeline.Process(ctx, file)
if err != nil {
return err
}
// Original bytes are copied unchanged from staging to the final key.
if _, err = file.Seek(0, 0); err != nil {
return &pipeline.Failure{Permanent: false, Code: "processing_failed", Err: err}
}
if err = w.objects.Put(ctx, att.ObjectKey, file, att.SizeBytes, att.ContentType); err != nil {
return &pipeline.Failure{Permanent: false, Code: "storage_unavailable", Err: err}
}
variants := make([]model.Variant, 0, len(encoded))
for _, item := range encoded {
key := store.VariantObjectKey(att.ObjectKey, item.Name)
if err = w.objects.Put(ctx, key, bytes.NewReader(item.Bytes), int64(len(item.Bytes)), item.ContentType); err != nil {
return &pipeline.Failure{Permanent: false, Code: "storage_unavailable", Err: err}
}
variants = append(variants, model.Variant{
Name: item.Name,
ContentType: item.ContentType,
SizeBytes: int64(len(item.Bytes)),
Width: item.Width,
Height: item.Height,
SHA256: item.SHA256,
})
}
committed, err := w.store.CompleteJob(ctx, claim.Job.JobID, claim.Job.Attempt, variants)
if err != nil {
return &pipeline.Failure{Permanent: false, Code: "processing_failed", Err: err}
}
if !committed {
log.Warn("ready snapshot was superseded; leaving newer state alone")
return nil
}
// The timing line is the experiment's source for compression overhead. It
// is emitted only after the ready snapshot commits, so a redelivered job
// cannot produce two successful samples.
log.Info("attachment ready",
"decode_us", stats.Decode.Microseconds(),
"scale_us", stats.Scale.Microseconds(),
"jpeg_encode_us", stats.JPEGEncode.Microseconds(),
"webp_encode_us", stats.WebPEncode.Microseconds(),
"pipeline_us", stats.Total().Microseconds(),
)
// Staging is removed only after ready commits. A crash before this delete
// leaves an object the reconciler will clean after retention.
if err = w.objects.Delete(ctx, att.StagingKey); err != nil && !errors.Is(err, objectstore.ErrNotFound) {
log.Warn("staging delete failed; reconciler will retry", "err", err)
return nil
}
if err = w.store.ClearStagingKey(ctx, att.ID); err != nil {
log.Warn("clear staging key failed; reconciler will retry", "err", err)
}
return nil
}
func (w *Worker) handleFailure(ctx context.Context, claim store.ClaimResult, err error, log *slog.Logger) error {
var failure *pipeline.Failure
if !errors.As(err, &failure) {
failure = &pipeline.Failure{Permanent: false, Code: "processing_failed", Err: err}
}
if failure.Permanent {
log.Error("permanent job failure", "code", failure.Code, "err", failure.Err)
if failErr := w.store.FailJob(ctx, claim.Job.JobID, claim.Job.Attempt, failure.Code); failErr != nil {
log.Error("record permanent failure failed; requeueing", "err", failErr)
return broker.ErrRequeue
}
return nil
}
if claim.Job.Attempt >= claim.Job.MaxAttempts {
log.Error("retry budget exhausted", "code", failure.Code, "err", failure.Err)
if failErr := w.store.FailJob(ctx, claim.Job.JobID, claim.Job.Attempt, "retry_exhausted"); failErr != nil {
log.Error("record exhausted retry failed; requeueing", "err", failErr)
return broker.ErrRequeue
}
return nil
}
delay := w.backoff(claim.Job.Attempt)
log.Warn("transient job failure; scheduling retry",
"code", failure.Code, "err", failure.Err, "retry_in", delay)
scheduled, retryErr := w.store.ScheduleRetry(ctx, claim.Job.JobID, claim.Job.Attempt, failure.Code, delay)
if retryErr != nil {
log.Error("schedule retry failed; requeueing", "err", retryErr)
return broker.ErrRequeue
}
if !scheduled {
// This attempt no longer owns the row (for example a lease sweep or a
// replay changed it). Requeue rather than ack a job whose future work
// was not durably recorded.
log.Warn("retry scheduling lost ownership; requeueing")
return broker.ErrRequeue
}
return nil
}
// backoff is exponential with +/-50% jitter, capped by JOB_RETRY_MAX_SECONDS.
// Jitter prevents many retries from hitting a recovered dependency together.
func (w *Worker) backoff(attempt int) time.Duration {
delay := w.cfg.RetryBase
for i := 1; i < attempt && delay < w.cfg.RetryMax; i++ {
delay *= 2
}
if delay > w.cfg.RetryMax {
delay = w.cfg.RetryMax
}
jitter := 0.5 + rand.Float64() // 0.5 .. 1.5
return time.Duration(float64(delay) * jitter)
}
+335
View File
@@ -0,0 +1,335 @@
name: mtusi2026lab
x-api-build: &api-build
image: mtusi2026lab-api:local
build:
context: .
dockerfile: Dockerfile
# API receives only the attachments/ scoped S3 credential
x-api-environment: &api-environment
APP_HOST: 0.0.0.0
APP_PORT: "8080"
DB_POOL_MAX_CONNS: ${DB_POOL_MAX_CONNS:-10}
DATABASE_URL: postgres://${POSTGRES_USER:-messenger}:${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}@postgres:5432/${POSTGRES_DB:-messenger}?sslmode=disable
CACHE_URL: redis://redis:6379/0
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
IDEMPOTENCY_TTL_SECONDS: ${IDEMPOTENCY_TTL_SECONDS:-86400}
S3_ENDPOINT: ${S3_ENDPOINT:-http://rustfs:9000}
S3_REGION: ${S3_REGION:-us-east-1}
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID:?set S3_ACCESS_KEY_ID in .env}
S3_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY:?set S3_SECRET_ACCESS_KEY in .env}
MAX_UPLOAD_BYTES: ${MAX_UPLOAD_BYTES:-10485760}
MAX_IMAGE_PIXELS: ${MAX_IMAGE_PIXELS:-40000000}
IMAGE_DECODE_MEMORY_MB: ${IMAGE_DECODE_MEMORY_MB:-256}
UPLOAD_TEMP_DIR: /tmp/messenger-uploads
BROKER_URL: "amqp://${RABBITMQ_USER:-messenger}:${RABBITMQ_PASSWORD:?set RABBITMQ_PASSWORD in .env}@rabbitmq:5672/"
JOB_QUEUE: ${JOB_QUEUE:-attachment.process.v1}
JOB_MAX_ATTEMPTS: ${JOB_MAX_ATTEMPTS:-5}
MAX_PENDING_JOBS: ${MAX_PENDING_JOBS:-1000}
services:
postgres:
image: postgres:16.10-alpine
restart: unless-stopped
environment:
POSTGRES_USER: ${POSTGRES_USER:-messenger}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}
POSTGRES_DB: ${POSTGRES_DB:-messenger}
volumes:
- pgdata:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
interval: 3s
timeout: 3s
retries: 20
start_period: 5s
deploy:
resources:
limits:
cpus: "${POSTGRES_CPU_LIMIT:-1.0}"
memory: "${POSTGRES_MEM_LIMIT:-512M}"
redis:
image: redis:7.4.7-alpine
restart: unless-stopped
command:
[
"redis-server",
"--appendonly",
"yes",
"--appendfsync",
"everysec",
"--save",
"",
]
volumes:
- redisdata:/data
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 3s
timeout: 3s
retries: 20
start_period: 3s
deploy:
resources:
limits:
cpus: "${REDIS_CPU_LIMIT:-0.50}"
memory: "${REDIS_MEM_LIMIT:-256M}"
rabbitmq:
image: rabbitmq:4.2-management-alpine
restart: unless-stopped
environment:
RABBITMQ_DEFAULT_USER: ${RABBITMQ_USER:-messenger}
RABBITMQ_DEFAULT_PASS: ${RABBITMQ_PASSWORD:?set RABBITMQ_PASSWORD in .env}
volumes:
- rabbitmqdata:/var/lib/rabbitmq
ports:
# Management UI is only for the local learning stand.
- "127.0.0.1:${RABBITMQ_MGMT_PORT:-15672}:15672"
healthcheck:
test: ["CMD", "rabbitmq-diagnostics", "-q", "ping"]
interval: 3s
timeout: 5s
retries: 20
start_period: 10s
deploy:
resources:
limits:
cpus: "${RABBITMQ_CPU_LIMIT:-1.0}"
memory: "${RABBITMQ_MEM_LIMIT:-512M}"
rustfs:
image: rustfs/rustfs:1.0.4
restart: unless-stopped
# No published ports. The S3 API is reachable only on the Compose
# network, and the admin console is disabled. For inspection, see the
# one-shot rc client pattern in scripts/check-private-bucket.sh.
environment:
RUSTFS_ACCESS_KEY: ${S3_ROOT_ACCESS_KEY:?set S3_ROOT_ACCESS_KEY in .env}
RUSTFS_SECRET_KEY: ${S3_ROOT_SECRET_KEY:?set S3_ROOT_SECRET_KEY in .env}
RUSTFS_ADDRESS: ":9000"
RUSTFS_CONSOLE_ENABLE: "false"
RUSTFS_OBS_LOGGER_LEVEL: "error"
RUSTFS_OBS_LOG_DIRECTORY: "/logs"
command: ["/data"]
volumes:
- rustfsdata:/data
expose:
- "9000"
healthcheck:
test:
[
"CMD-SHELL",
"wget -q -O - http://127.0.0.1:9000/health/ready >/dev/null",
]
interval: 3s
timeout: 3s
retries: 20
start_period: 5s
stop_grace_period: 15s
deploy:
resources:
limits:
cpus: "${RUSTFS_CPU_LIMIT:-1.0}"
memory: "${RUSTFS_MEM_LIMIT:-512M}"
# Converges the bucket and the scoped IAM identity on every
# run. Create is idempotent (ignore-existing), policy create replaces the
# named document, and user add updates the password if the user exists.
s3-init:
image: rustfs/rc:v0.1.36
restart: "no"
depends_on:
rustfs:
condition: service_healthy
environment:
HOME: /tmp
S3_ENDPOINT: ${S3_ENDPOINT:-http://rustfs:9000}
S3_REGION: ${S3_REGION:-us-east-1}
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
S3_ROOT_ACCESS_KEY: ${S3_ROOT_ACCESS_KEY:?set S3_ROOT_ACCESS_KEY in .env}
S3_ROOT_SECRET_KEY: ${S3_ROOT_SECRET_KEY:?set S3_ROOT_SECRET_KEY in .env}
S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID:?set S3_ACCESS_KEY_ID in .env}
S3_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY:?set S3_SECRET_ACCESS_KEY in .env}
S3_APP_USER: ${S3_APP_USER:-messenger-api}
S3_APP_POLICY: ${S3_APP_POLICY:-messenger-attachments-rw}
S3_POLICY_TEMPLATE: /config/api-policy.json.template
volumes:
- ./infra/rustfs/bootstrap.sh:/bootstrap.sh:ro
- ./infra/rustfs/api-policy.json.template:/config/api-policy.json.template:ro
entrypoint: ["/bin/sh", "/bootstrap.sh"]
migrate:
<<: *api-build
restart: "no"
depends_on:
postgres:
condition: service_healthy
environment:
<<: *api-environment
MIGRATE_ONLY: "true"
api:
<<: *api-build
restart: unless-stopped
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_healthy
rustfs:
condition: service_healthy
rabbitmq:
condition: service_healthy
s3-init:
condition: service_completed_successfully
migrate:
condition: service_completed_successfully
environment:
<<: *api-environment
RUN_MIGRATIONS: "false"
expose:
- "8080"
healthcheck:
test:
["CMD", "wget", "-q", "-O", "-", "http://127.0.0.1:8080/health/ready"]
interval: 2s
timeout: 2s
retries: 10
start_period: 10s
stop_grace_period: 15s
deploy:
replicas: ${API_REPLICAS:-2}
resources:
limits:
cpus: "${API_CPU_LIMIT:-0.50}"
memory: "${API_MEM_LIMIT:-512M}"
worker:
<<: *api-build
restart: unless-stopped
depends_on:
postgres:
condition: service_healthy
rustfs:
condition: service_healthy
rabbitmq:
condition: service_healthy
s3-init:
condition: service_completed_successfully
migrate:
condition: service_completed_successfully
environment:
<<: *api-environment
RUN_MIGRATIONS: "false"
WORKER_ONLY: "true"
WORKER_CONCURRENCY: ${WORKER_CONCURRENCY:-2}
WORKER_PREFETCH: ${WORKER_PREFETCH:-2}
JOB_TIMEOUT_SECONDS: ${JOB_TIMEOUT_SECONDS:-60}
JOB_LEASE_SECONDS: ${JOB_LEASE_SECONDS:-90}
JOB_RETRY_BASE_SECONDS: ${JOB_RETRY_BASE_SECONDS:-2}
JOB_RETRY_MAX_SECONDS: ${JOB_RETRY_MAX_SECONDS:-30}
LEASE_SWEEP_SECONDS: ${LEASE_SWEEP_SECONDS:-15}
STAGING_RETENTION_SECONDS: ${STAGING_RETENTION_SECONDS:-86400}
WORK_TEMP_DIR: /tmp/messenger-work
# Test hooks, off by default. They make the grade-4 failure windows
# deterministic; see evidence/ and REPORT.md.
WORKER_FAULT_MODE: ${WORKER_FAULT_MODE:-none}
WORKER_CRASH_AFTER_EFFECT: ${WORKER_CRASH_AFTER_EFFECT:-false}
WORKER_CRASH_AFTER_CLAIM: ${WORKER_CRASH_AFTER_CLAIM:-false}
entrypoint: ["/usr/local/bin/worker"]
healthcheck:
test: ["CMD-SHELL", "pgrep -f /usr/local/bin/worker >/dev/null"]
interval: 3s
timeout: 2s
retries: 10
start_period: 5s
stop_grace_period: 20s
deploy:
replicas: ${WORKER_REPLICAS:-1}
resources:
limits:
cpus: "${WORKER_CPU_LIMIT:-1.0}"
memory: "${WORKER_MEM_LIMIT:-1G}"
# The reconciler uses the same scoped credential as the API. It never
# touches ready attachments; it only removes stale pending rows/objects and
# objects with no metadata row at all.
reconciler:
<<: *api-build
restart: unless-stopped
depends_on:
postgres:
condition: service_healthy
rustfs:
condition: service_healthy
s3-init:
condition: service_completed_successfully
migrate:
condition: service_completed_successfully
environment:
<<: *api-environment
RUN_MIGRATIONS: "false"
RECONCILE_ONLY: "true"
RECONCILE_GRACE_SECONDS: ${RECONCILE_GRACE_SECONDS:-900}
RECONCILE_INTERVAL_SECONDS: ${RECONCILE_INTERVAL_SECONDS:-300}
entrypoint: ["/usr/local/bin/reconciler"]
deploy:
resources:
limits:
cpus: "${RECONCILER_CPU_LIMIT:-0.25}"
memory: "${RECONCILER_MEM_LIMIT:-128M}"
haproxy:
image: haproxy:3.2.6-alpine
restart: unless-stopped
depends_on:
api:
condition: service_healthy
ports:
- "${LB_PORT:-8080}:8080"
- "${HAPROXY_STATS_PORT:-8404}:8404"
volumes:
- ./infra/haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
healthcheck:
test:
["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:8404/stats"]
interval: 2s
timeout: 2s
retries: 10
start_period: 5s
deploy:
resources:
limits:
cpus: "${HAPROXY_CPU_LIMIT:-0.50}"
memory: "${HAPROXY_MEM_LIMIT:-128M}"
k6:
image: grafana/k6:1.3.0
profiles: ["load"]
depends_on:
haproxy:
condition: service_healthy
user: "0:0"
environment:
BASE_URL: http://haproxy:8080
VUS: ${K6_VUS:-10}
DURATION: ${K6_DURATION:-60s}
volumes:
- ./load:/scripts:ro
- ./evidence:/results
command: ["run", "/scripts/messages.js"]
deploy:
resources:
limits:
cpus: "${K6_CPU_LIMIT:-1.0}"
memory: "${K6_MEM_LIMIT:-512M}"
volumes:
pgdata:
redisdata:
rustfsdata:
rabbitmqdata:
+4
View File
@@ -0,0 +1,4 @@
{"uploaded_by": "ad8141b5e17c", "attachment_id": "4395a996-db6d-4568-8bfe-805a926b54c8", "size_bytes": 111817}
stopping API replica ad8141b5e17c
{"downloaded_by": "2229551f45ac", "stopped_replica": "ad8141b5e17c", "attempts": 1, "transient_errors": 0, "size_bytes": 111817, "sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75"}
PASS: attachment survived the stop of the replica that accepted it
+2
View File
@@ -0,0 +1,2 @@
python3 tests/custom/attachments.py --base-url "http://haproxy:8080"
PASS: graded negative cases (413/415/404/400), no 5xx, cross-member download checksum matches
+4
View File
@@ -0,0 +1,4 @@
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://haproxy:8080"
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 118, "failed_attempts": 1, "first_recovery_seconds": 2.557}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
+2
View File
@@ -0,0 +1,2 @@
python3 tests/custom/idempotency.py --base-url "http://haproxy:8080"
PASS: 12 concurrent requests used 2 replicas and stored one message
+4
View File
@@ -0,0 +1,4 @@
attachments rows: 61 -> 61
stored objects: 61 -> 61
temp files left: 0
PASS: interrupted upload left no row, object or temp file
+11
View File
@@ -0,0 +1,11 @@
$ make test-failover ACTION_SCRIPT=scripts/stop-one.sh BASE_URL=http://172.18.0.1:8080
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://172.18.0.1:8080"
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 118, "failed_attempts": 1, "first_recovery_seconds": 2.659}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
Operator scope evidence:
- before hook: docker compose ps showed two healthy api replicas.
- hook: scripts/stop-one.sh stopped mtusi2026lab-api-2 and left it stopped.
- during failover the load balancer served the surviving replica; the test observed 118 successful writes, 1 failed attempt and first recovery in 2.659 s.
- after the test: `docker start mtusi2026lab-api-2`; the replica returned to healthy, so the stand is restored to two API replicas.
+208
View File
@@ -0,0 +1,208 @@
{
"generated_at": "2026-11-14T21:23:08.494056+00:00",
"fixture": {
"bytes": 111817,
"sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
"width": 320,
"height": 200
},
"scenarios": {
"worker_stopped": {
"attachment_id": "3d1153db-d279-447c-8cda-e24fe2079119",
"queued_status": "queued",
"content_while_queued": 409,
"ready_after_worker_start": true,
"queue_seconds": 2.316,
"evidence": {
"original": {
"content_type": "image/png",
"size_bytes": 111817,
"sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
"width": 320,
"height": 200
},
"thumbnail": {
"content_type": "image/jpeg",
"size_bytes": 5653,
"sha256": "4462c506c73f692d1a8da1f5a5d217bf29e7c7af0fe19f2c03a021055c8751fe",
"width": 256,
"height": 160
},
"crop": {
"content_type": "image/jpeg",
"size_bytes": 2441,
"sha256": "802a8482e119909b5716aea9f502185f9604d3992656e5e5baf72bc79afd31f3",
"width": 128,
"height": 128
},
"optimized": {
"content_type": "image/webp",
"size_bytes": 2576,
"sha256": "b0021a56abe58c6346f8fcae3a3e9820fa230e40286de35b1c50127d8f04e49c",
"width": 320,
"height": 200
}
},
"row": {
"attempt": 1,
"published": true,
"staging_cleared": true
}
},
"broker_outage": {
"attachment_id": "bce30804-16d3-477c-a732-40bd336a0873",
"202_accepted_without_broker": true,
"unpublished_during_outage": true,
"api_ready_during_broker_outage": true,
"published_after_recovery": true,
"attempt": 1,
"queue_seconds": 9.005
},
"crash_after_effect": {
"attachment_id": "afc5a48d-7781-4627-a2b2-cf9be1d92ac5",
"worker_restart_count_before": 0,
"worker_restart_count_after": 1,
"ready_after_redelivery": true,
"variant_objects": 3,
"evidence": {
"original": {
"content_type": "image/png",
"size_bytes": 111817,
"sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
"width": 320,
"height": 200
},
"thumbnail": {
"content_type": "image/jpeg",
"size_bytes": 5653,
"sha256": "4462c506c73f692d1a8da1f5a5d217bf29e7c7af0fe19f2c03a021055c8751fe",
"width": 256,
"height": 160
},
"crop": {
"content_type": "image/jpeg",
"size_bytes": 2441,
"sha256": "802a8482e119909b5716aea9f502185f9604d3992656e5e5baf72bc79afd31f3",
"width": 128,
"height": 128
},
"optimized": {
"content_type": "image/webp",
"size_bytes": 2576,
"sha256": "b0021a56abe58c6346f8fcae3a3e9820fa230e40286de35b1c50127d8f04e49c",
"width": 320,
"height": 200
}
},
"log_excerpt": [
"worker-1 | time=2026-11-14T21:23:37.203Z level=INFO msg=\"processing attachment\" job_id=fdd5c786-ca1e-4d04-98b6-75ce15ccda14 attachment_id=afc5a48d-7781-4627-a2b2-cf9be1d92ac5 attempt=1 image_bytes=111817",
"worker-1 | time=2026-11-14T21:23:37.392Z level=INFO msg=\"attachment ready\" job_id=fdd5c786-ca1e-4d04-98b6-75ce15ccda14 attachment_id=afc5a48d-7781-4627-a2b2-cf9be1d92ac5 attempt=1 image_bytes=111817",
"worker-1 | time=2026-11-14T21:23:37.393Z level=ERROR msg=\"fault injection: exiting after durable effect before ack\" job_id=fdd5c786-ca1e-4d04-98b6-75ce15ccda14",
"worker-1 | time=2026-11-14T21:23:37.932Z level=INFO msg=\"job is already finished or owned\" job_id=fdd5c786-ca1e-4d04-98b6-75ce15ccda14 terminal=true busy=false attempt=1"
]
},
"transient_retry": {
"attachment_id": "3b46dc74-f4cc-4c1b-89fc-f0e4023df603",
"attempts": 2,
"status": "ready",
"log_excerpt": [
"worker-1 | time=2026-11-14T21:23:54.733Z level=INFO msg=\"processing attachment\" job_id=c18ef120-24a5-4344-899b-fcfc826bfbe7 attachment_id=3b46dc74-f4cc-4c1b-89fc-f0e4023df603 attempt=1 image_bytes=111817",
"worker-1 | time=2026-11-14T21:23:54.733Z level=WARN msg=\"transient job failure; scheduling retry\" job_id=c18ef120-24a5-4344-899b-fcfc826bfbe7 attachment_id=3b46dc74-f4cc-4c1b-89fc-f0e4023df603 attempt=1 image_bytes=111817 code=processing_failed err=\"injected transient failure\" retry_in=2.25377422s",
"worker-1 | time=2026-11-14T21:23:57.029Z level=INFO msg=\"processing attachment\" job_id=c18ef120-24a5-4344-899b-fcfc826bfbe7 attachment_id=3b46dc74-f4cc-4c1b-89fc-f0e4023df603 attempt=2 image_bytes=111817",
"worker-1 | time=2026-11-14T21:23:57.219Z level=INFO msg=\"attachment ready\" job_id=c18ef120-24a5-4344-899b-fcfc826bfbe7 attachment_id=3b46dc74-f4cc-4c1b-89fc-f0e4023df603 attempt=2 image_bytes=111817"
]
},
"permanent_failure_replay": {
"attachment_id": "a1417cf6-4d8b-48f7-9767-e5b876f0bdd1",
"failed_code": "invalid_image",
"failed_content_http": 409,
"failed_message_link_http": 409,
"replay_exit": 0,
"replay_output": [
"2026/11/14 21:24:43 replay: attachment a1417cf6-4d8b-48f7-9767-e5b876f0bdd1 requeued"
],
"replay_count": 1,
"variant_objects_after_replay": 3,
"original_object_present": true,
"evidence": {
"original": {
"content_type": "image/png",
"size_bytes": 111817,
"sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
"width": 320,
"height": 200
},
"thumbnail": {
"content_type": "image/jpeg",
"size_bytes": 5653,
"sha256": "4462c506c73f692d1a8da1f5a5d217bf29e7c7af0fe19f2c03a021055c8751fe",
"width": 256,
"height": 160
},
"crop": {
"content_type": "image/jpeg",
"size_bytes": 2441,
"sha256": "802a8482e119909b5716aea9f502185f9604d3992656e5e5baf72bc79afd31f3",
"width": 128,
"height": 128
},
"optimized": {
"content_type": "image/webp",
"size_bytes": 2576,
"sha256": "b0021a56abe58c6346f8fcae3a3e9820fa230e40286de35b1c50127d8f04e49c",
"width": 320,
"height": 200
}
},
"log_excerpt": [
"worker-1 | time=2026-11-14T21:24:33.736Z level=INFO msg=\"processing attachment\" job_id=3f9c3f24-841e-44cd-8f8f-4adfef88d3ea attachment_id=a1417cf6-4d8b-48f7-9767-e5b876f0bdd1 attempt=1 image_bytes=111817",
"worker-1 | time=2026-11-14T21:24:33.736Z level=ERROR msg=\"permanent job failure\" job_id=3f9c3f24-841e-44cd-8f8f-4adfef88d3ea attachment_id=a1417cf6-4d8b-48f7-9767-e5b876f0bdd1 attempt=1 image_bytes=111817 code=invalid_image err=\"injected permanent failure\"",
"worker-1 | time=2026-11-14T21:24:43.247Z level=INFO msg=\"processing attachment\" job_id=3f9c3f24-841e-44cd-8f8f-4adfef88d3ea attachment_id=a1417cf6-4d8b-48f7-9767-e5b876f0bdd1 attempt=1 image_bytes=111817",
"worker-1 | time=2026-11-14T21:24:43.469Z level=INFO msg=\"attachment ready\" job_id=3f9c3f24-841e-44cd-8f8f-4adfef88d3ea attachment_id=a1417cf6-4d8b-48f7-9767-e5b876f0bdd1 attempt=1 image_bytes=111817"
]
},
"lease_recovery": {
"attachment_id": "6d5091ae-474d-4e94-a4a9-f673aa16c9c1",
"one_shot_exit": 10,
"processing_before_expiry": true,
"attempts": 2,
"ready_after_sweep": true,
"evidence": {
"original": {
"content_type": "image/png",
"size_bytes": 111817,
"sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
"width": 320,
"height": 200
},
"thumbnail": {
"content_type": "image/jpeg",
"size_bytes": 5653,
"sha256": "4462c506c73f692d1a8da1f5a5d217bf29e7c7af0fe19f2c03a021055c8751fe",
"width": 256,
"height": 160
},
"crop": {
"content_type": "image/jpeg",
"size_bytes": 2441,
"sha256": "802a8482e119909b5716aea9f502185f9604d3992656e5e5baf72bc79afd31f3",
"width": 128,
"height": 128
},
"optimized": {
"content_type": "image/webp",
"size_bytes": 2576,
"sha256": "b0021a56abe58c6346f8fcae3a3e9820fa230e40286de35b1c50127d8f04e49c",
"width": 320,
"height": 200
}
},
"log_excerpt": [
"worker-1 | time=2026-11-14T21:24:51.033Z level=INFO msg=\"job is already finished or owned\" job_id=9201154a-7ddd-404b-8cc0-4ce24688c04f terminal=false busy=true attempt=1",
"worker-1 | time=2026-11-14T21:24:59.019Z level=WARN msg=\"recovered expired job leases\" count=1",
"worker-1 | time=2026-11-14T21:24:59.229Z level=INFO msg=\"processing attachment\" job_id=9201154a-7ddd-404b-8cc0-4ce24688c04f attachment_id=6d5091ae-474d-4e94-a4a9-f673aa16c9c1 attempt=2 image_bytes=111817",
"worker-1 | time=2026-11-14T21:24:59.420Z level=INFO msg=\"attachment ready\" job_id=9201154a-7ddd-404b-8cc0-4ce24688c04f attachment_id=6d5091ae-474d-4e94-a4a9-f673aa16c9c1 attempt=2 image_bytes=111817"
]
}
}
}
+53
View File
@@ -0,0 +1,53 @@
# Lab5 grade-4 runtime evidence
Generated: `2026-11-14T21:23:08.494056+00:00`
Synthetic fixture: 320x200 PNG, public contract fixture; no tokens or credentials are stored.
| Scenario | Observation |
| --- | --- |
| Worker stopped during upload | 202 queued, content 409, message linked while queued, ready after worker start; queue 2.316 s, attempt 1 |
| Broker outage (COMMIT/publish gap) | 202 accepted with `published_at IS NULL`, API readiness stayed 200, job published and ready after RabbitMQ restart |
| Crash after effect, before ack | worker restart count 0 -> 1, 3 deterministic variant objects, ready |
| Transient error + jittered retry | failed once, attempt 2, ready |
| Permanent error + explicit replay | failed `invalid_image`, content/link 409, replay_count 1, ready with 3 variant objects |
| Crash after claim / expired lease | one-shot exit 10, processing, sweeper recovered, attempt 2, ready |
## Sanitized log excerpts
### crash_after_effect
```text
worker-1 | time=2026-11-14T21:23:37.203Z level=INFO msg="processing attachment" job_id=fdd5c786-ca1e-4d04-98b6-75ce15ccda14 attachment_id=afc5a48d-7781-4627-a2b2-cf9be1d92ac5 attempt=1 image_bytes=111817
worker-1 | time=2026-11-14T21:23:37.392Z level=INFO msg="attachment ready" job_id=fdd5c786-ca1e-4d04-98b6-75ce15ccda14 attachment_id=afc5a48d-7781-4627-a2b2-cf9be1d92ac5 attempt=1 image_bytes=111817
worker-1 | time=2026-11-14T21:23:37.393Z level=ERROR msg="fault injection: exiting after durable effect before ack" job_id=fdd5c786-ca1e-4d04-98b6-75ce15ccda14
worker-1 | time=2026-11-14T21:23:37.932Z level=INFO msg="job is already finished or owned" job_id=fdd5c786-ca1e-4d04-98b6-75ce15ccda14 terminal=true busy=false attempt=1
```
### transient_retry
```text
worker-1 | time=2026-11-14T21:23:54.733Z level=INFO msg="processing attachment" job_id=c18ef120-24a5-4344-899b-fcfc826bfbe7 attachment_id=3b46dc74-f4cc-4c1b-89fc-f0e4023df603 attempt=1 image_bytes=111817
worker-1 | time=2026-11-14T21:23:54.733Z level=WARN msg="transient job failure; scheduling retry" job_id=c18ef120-24a5-4344-899b-fcfc826bfbe7 attachment_id=3b46dc74-f4cc-4c1b-89fc-f0e4023df603 attempt=1 image_bytes=111817 code=processing_failed err="injected transient failure" retry_in=2.25377422s
worker-1 | time=2026-11-14T21:23:57.029Z level=INFO msg="processing attachment" job_id=c18ef120-24a5-4344-899b-fcfc826bfbe7 attachment_id=3b46dc74-f4cc-4c1b-89fc-f0e4023df603 attempt=2 image_bytes=111817
worker-1 | time=2026-11-14T21:23:57.219Z level=INFO msg="attachment ready" job_id=c18ef120-24a5-4344-899b-fcfc826bfbe7 attachment_id=3b46dc74-f4cc-4c1b-89fc-f0e4023df603 attempt=2 image_bytes=111817
```
### permanent_failure_replay
```text
worker-1 | time=2026-11-14T21:24:33.736Z level=INFO msg="processing attachment" job_id=3f9c3f24-841e-44cd-8f8f-4adfef88d3ea attachment_id=a1417cf6-4d8b-48f7-9767-e5b876f0bdd1 attempt=1 image_bytes=111817
worker-1 | time=2026-11-14T21:24:33.736Z level=ERROR msg="permanent job failure" job_id=3f9c3f24-841e-44cd-8f8f-4adfef88d3ea attachment_id=a1417cf6-4d8b-48f7-9767-e5b876f0bdd1 attempt=1 image_bytes=111817 code=invalid_image err="injected permanent failure"
worker-1 | time=2026-11-14T21:24:43.247Z level=INFO msg="processing attachment" job_id=3f9c3f24-841e-44cd-8f8f-4adfef88d3ea attachment_id=a1417cf6-4d8b-48f7-9767-e5b876f0bdd1 attempt=1 image_bytes=111817
worker-1 | time=2026-11-14T21:24:43.469Z level=INFO msg="attachment ready" job_id=3f9c3f24-841e-44cd-8f8f-4adfef88d3ea attachment_id=a1417cf6-4d8b-48f7-9767-e5b876f0bdd1 attempt=1 image_bytes=111817
```
### lease_recovery
```text
worker-1 | time=2026-11-14T21:24:51.033Z level=INFO msg="job is already finished or owned" job_id=9201154a-7ddd-404b-8cc0-4ce24688c04f terminal=false busy=true attempt=1
worker-1 | time=2026-11-14T21:24:59.019Z level=WARN msg="recovered expired job leases" count=1
worker-1 | time=2026-11-14T21:24:59.229Z level=INFO msg="processing attachment" job_id=9201154a-7ddd-404b-8cc0-4ce24688c04f attachment_id=6d5091ae-474d-4e94-a4a9-f673aa16c9c1 attempt=2 image_bytes=111817
worker-1 | time=2026-11-14T21:24:59.420Z level=INFO msg="attachment ready" job_id=9201154a-7ddd-404b-8cc0-4ce24688c04f attachment_id=6d5091ae-474d-4e94-a4a9-f673aa16c9c1 attempt=2 image_bytes=111817
```
+1
View File
@@ -0,0 +1 @@
PASS: graded negative cases (413 byte/pixel, 415/404/400), no 5xx, cross-member download checksum matches
+38
View File
@@ -0,0 +1,38 @@
# Persistence hook scope: container StartedAt before and after
$ docker compose ps --format "{{.Service}} {{.Status}}"
api Up 7 seconds (healthy)
api Up 7 seconds (healthy)
haproxy Up 7 seconds (healthy)
postgres Up 7 seconds (healthy)
rabbitmq Up 6 seconds (health: starting)
reconciler Up 5 minutes
redis Up 7 seconds (healthy)
rustfs Up 7 seconds (healthy)
worker Up 7 seconds (healthy)
StartedAt before:
/mtusi2026lab-api-1 2026-11-14T21:25:41.77929768Z
/mtusi2026lab-api-2 2026-11-14T21:25:41.637807913Z
/mtusi2026lab-haproxy-1 2026-11-14T21:25:41.555990612Z
/mtusi2026lab-postgres-1 2026-11-14T21:25:41.820919775Z
/mtusi2026lab-rabbitmq-1 2026-11-14T21:25:42.573125104Z
/mtusi2026lab-reconciler-1 2026-11-14T21:19:51.25100733Z
/mtusi2026lab-redis-1 2026-11-14T21:25:41.663107884Z
/mtusi2026lab-rustfs-1 2026-11-14T21:25:41.292286108Z
/mtusi2026lab-worker-1 2026-11-14T21:25:41.248247598Z
$ make test-persistence ACTION_SCRIPT=scripts/restart.sh BASE_URL=http://172.18.0.1:8080
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://172.18.0.1:8080"
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
StartedAt after:
/mtusi2026lab-api-1 2026-11-14T21:25:53.837523775Z
/mtusi2026lab-api-2 2026-11-14T21:25:54.252693201Z
/mtusi2026lab-haproxy-1 2026-11-14T21:25:54.223324484Z
/mtusi2026lab-postgres-1 2026-11-14T21:25:54.026612072Z
/mtusi2026lab-rabbitmq-1 2026-11-14T21:25:55.048458301Z
/mtusi2026lab-reconciler-1 2026-11-14T21:19:51.25100733Z
/mtusi2026lab-redis-1 2026-11-14T21:25:54.361607562Z
/mtusi2026lab-rustfs-1 2026-11-14T21:25:53.897730396Z
/mtusi2026lab-worker-1 2026-11-14T21:25:53.965619121Z
+4
View File
@@ -0,0 +1,4 @@
$ make test-persistence ACTION_SCRIPT=scripts/restart.sh BASE_URL=http://172.18.0.1:8080
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://172.18.0.1:8080"
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
+3
View File
@@ -0,0 +1,3 @@
$ make test BASE_URL=http://172.18.0.1:8080
python3 tests/smoke.py --base-url "http://172.18.0.1:8080"
PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.
+17
View File
@@ -0,0 +1,17 @@
{
"lab": 4,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 15.0,
"elapsed_seconds": 15.216,
"concurrency": 8,
"successful_requests": 6249,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 410.689,
"successful_latency_ms": {
"p50": 18.104,
"p95": 31.058,
"p99": 46.769
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
+17
View File
@@ -0,0 +1,17 @@
{
"lab": 4,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 15.0,
"elapsed_seconds": 15.216,
"concurrency": 8,
"successful_requests": 6249,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 410.689,
"successful_latency_ms": {
"p50": 18.104,
"p95": 31.058,
"p99": 46.769
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
+17
View File
@@ -0,0 +1,17 @@
before:
/mtusi2026lab-api-1 started=2026-10-31T17:51:06.750192429Z
/mtusi2026lab-api-2 started=2026-10-31T17:51:49.665340634Z
/mtusi2026lab-haproxy-1 started=2026-10-31T17:48:59.281606934Z
/mtusi2026lab-postgres-1 started=2026-10-31T17:48:59.351033349Z
/mtusi2026lab-redis-1 started=2026-10-31T17:48:59.306542581Z
/mtusi2026lab-rustfs-1 started=2026-10-31T17:48:59.424216765Z
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://haproxy:8080"
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
after:
/mtusi2026lab-api-1 started=2026-10-31T17:52:30.339455795Z
/mtusi2026lab-api-2 started=2026-10-31T17:52:30.289832188Z
/mtusi2026lab-haproxy-1 started=2026-10-31T17:52:30.089676215Z
/mtusi2026lab-postgres-1 started=2026-10-31T17:52:30.233422161Z
/mtusi2026lab-redis-1 started=2026-10-31T17:52:30.195015955Z
/mtusi2026lab-rustfs-1 started=2026-10-31T17:52:29.875749955Z
+3
View File
@@ -0,0 +1,3 @@
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://haproxy:8080"
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
+9
View File
@@ -0,0 +1,9 @@
1/4 create a probe object with the root credential
2/4 anonymous object GET must be denied
anonymous GET -> 403
3/4 scoped credential works inside attachments/ and is denied elsewhere
scoped PUT/HEAD inside attachments/ -> allowed
scoped PUT outside attachments/ -> denied
scoped CreateBucket -> denied
4/4 cleanup the probe object on exit
PASS: bucket is private and the API credential is limited to attachments/*
+18
View File
@@ -0,0 +1,18 @@
== before ==
X-Instance-Id: 2229551f45ac
Connection: close
== stopping postgres ==
Container mtusi2026lab-postgres-1 Stopping
Container mtusi2026lab-postgres-1 Stopped
HTTP/1.1 503 Service Unavailable
wget: server returned error: HTTP/1.1 503 Service Unavailable
== starting postgres ==
Container mtusi2026lab-postgres-1 Starting
Container mtusi2026lab-postgres-1 Started
== after ==
X-Request-Id: SixnEMLrvZMdpzQ7KEnqWSq0qLmSnx02kMfwNT39fiA
X-Instance-Id: 2229551f45ac
Connection: close
{"status":"ok"}
+15
View File
@@ -0,0 +1,15 @@
== before ==
Connection: close
{"status":"ok"}== stopping redis ==
Container mtusi2026lab-redis-1 Stopping
Container mtusi2026lab-redis-1 Stopped
HTTP/1.1 503 Service Unavailable
wget: server returned error: HTTP/1.1 503 Service Unavailable
== starting redis ==
Container mtusi2026lab-redis-1 Starting
Container mtusi2026lab-redis-1 Started
== after ==
X-Instance-Id: 2229551f45ac
Connection: close
+21
View File
@@ -0,0 +1,21 @@
== before stopping rustfs ==
X-Request-Id: JlYsHSjQDyVkf9Sfo3ifA3L0_2k4DXB4ONl34YgKC40
X-Instance-Id: 2229551f45ac
Connection: close
{"status":"ok"}exit=0
== stopping rustfs ==
Container mtusi2026lab-rustfs-1 Stopping
Container mtusi2026lab-rustfs-1 Stopped
HTTP/1.1 503 Service Unavailable
wget: server returned error: HTTP/1.1 503 Service Unavailable
exit=0
== starting rustfs ==
Container mtusi2026lab-rustfs-1 Starting
Container mtusi2026lab-rustfs-1 Started
== after rustfs recovery ==
X-Request-Id: KeusU6cz8m0fpFPz5NpDzvQU3XD0E6N-bLLo410ePG8
X-Instance-Id: 2229551f45ac
Connection: close
{"status":"ok"}exit=0
+24
View File
@@ -0,0 +1,24 @@
Container mtusi2026lab-reconciler-1 Stopping
Container mtusi2026lab-reconciler-1 Stopped
1/4 old pending row is visible to dry-run but unchanged
dry-run reported it and changed neither store
2/4 old pending row and object are removed by a real pass
Container mtusi2026lab-reconciler-run-d207ba83e1d7 Creating
Container mtusi2026lab-reconciler-run-d207ba83e1d7 Created
2026/11/06 15:57:35 INFO reconcile pass complete dry_run=false stale_pending=1 orphan_objects=0 deleted_rows=1 deleted_objects=1 skipped=0
row and object removed
3/4 fresh pending row survives the grace period
Container mtusi2026lab-reconciler-run-c71fa380c411 Creating
Container mtusi2026lab-reconciler-run-c71fa380c411 Created
2026/11/06 15:57:39 INFO reconcile pass complete dry_run=false stale_pending=0 orphan_objects=0 deleted_rows=0 deleted_objects=0 skipped=0
fresh row and object kept
Container mtusi2026lab-reconciler-run-2666c645577b Creating
Container mtusi2026lab-reconciler-run-2666c645577b Created
2026/11/06 15:57:41 INFO reconcile pass complete dry_run=false stale_pending=0 orphan_objects=1 deleted_rows=0 deleted_objects=1 skipped=0
row removed, orphan object cleaned up
4/4 ready attachment is not touched
Container mtusi2026lab-reconciler-run-5f830168e512 Creating
Container mtusi2026lab-reconciler-run-5f830168e512 Created
2026/11/06 15:57:47 INFO reconcile pass complete dry_run=false stale_pending=0 orphan_objects=0 deleted_rows=0 deleted_objects=0 skipped=0
ready metadata and bytes kept
PASS: reconciler removed only abandoned data and respected the grace period
+8
View File
@@ -0,0 +1,8 @@
run 1:
Container mtusi2026lab-s3-init-run-6ff27dc7d5c8 Creating
Container mtusi2026lab-s3-init-run-6ff27dc7d5c8 Created
S3 bootstrap complete: bucket=messenger-attachments user=messenger-api policy=messenger-attachments-rw
run 2:
Container mtusi2026lab-s3-init-run-83790bb5cf48 Creating
Container mtusi2026lab-s3-init-run-83790bb5cf48 Created
S3 bootstrap complete: bucket=messenger-attachments user=messenger-api policy=messenger-attachments-rw
+2
View File
@@ -0,0 +1,2 @@
python3 tests/smoke.py --base-url "http://haproxy:8080"
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
+359
View File
@@ -0,0 +1,359 @@
<!doctype html>
<html lang="ru">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Lab5 track 1 — очередь под перегрузкой</title>
<style>
:root { color-scheme: light; }
* { box-sizing: border-box; }
body { margin: 0; padding: 28px; font: 15px/1.55 -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
color: #0f172a; background: #f8fafc; }
main { max-width: 980px; margin: 0 auto; }
h1 { font-size: 28px; margin: 0 0 6px; }
h2 { font-size: 20px; margin: 28px 0 10px; }
p.lead { color: #475569; margin: 0 0 18px; }
.card { background: #fff; border: 1px solid #e2e8f0; border-radius: 12px; padding: 16px 18px;
box-shadow: 0 1px 2px rgba(15,23,42,.04); margin: 0 0 16px; }
.hypothesis { border-left: 4px solid #0284c7; }
.hypothesis strong { color: #075985; }
.metrics { display: grid; grid-template-columns: repeat(auto-fit, minmax(170px, 1fr)); gap: 12px; }
.metric { background: #f8fafc; border: 1px solid #e2e8f0; border-radius: 10px; padding: 12px; }
.metric .value { font-size: 22px; font-weight: 700; }
.metric .label { color: #64748b; font-size: 13px; }
table { border-collapse: collapse; width: 100%; font-size: 14px; }
th, td { text-align: right; padding: 8px 9px; border-bottom: 1px solid #e2e8f0; }
th:first-child, td:first-child, th:nth-child(2), td:nth-child(2) { text-align: left; }
th { color: #475569; font-weight: 600; background: #f8fafc; }
tr:last-child td { border-bottom: 0; }
.ok { color: #047857; font-weight: 700; }
.blocked { color: #b91c1c; font-weight: 700; }
.chart { padding: 10px 12px 4px; }
svg { width: 100%; height: auto; display: block; }
.chart-title { font-size: 17px; font-weight: 700; fill: #0f172a; }
.chart-sub { font-size: 12px; fill: #64748b; }
.axis { font-size: 11px; fill: #475569; }
.bar-label { font-size: 10px; fill: #0f172a; font-weight: 600; }
.grid { stroke: #e2e8f0; stroke-width: 1; }
ul { margin: 8px 0 0; padding-left: 22px; }
li { margin: 5px 0; }
.note { color: #475569; font-size: 13px; }
code { background: #f1f5f9; padding: 1px 5px; border-radius: 4px; }
@media (max-width: 640px) {
body { padding: 16px; }
.card { padding: 12px; }
table { font-size: 12px; }
}
</style>
</head>
<body>
<main>
<h1>Lab5, трек 1: очередь под перегрузкой</h1>
<p class="lead">Сгенерировано 2026-11-20T16:02:07.856909+00:00. Стенд: http://172.18.0.1:8080,
4 CPU, Docker 5.6.1.</p>
<section class="card hypothesis">
<strong>Гипотеза.</strong> Стоимость задания складывается из decode, масштабирования
и сжатия. <em>exactly_right</em> (1280×1280) не масштабирует optimized-вариант,
поэтому показывает нижнюю границу стоимости сжатия; <em>huge_few</em> добавляет
downscale, а <em>small_many</em> почти не масштабируется, но и пикселей в нём
мало. Большие изображения при том же числе worker-ов увеличивают очередь,
p95 time-to-ready и RSS. Одновременно проверяем компромисс: <em>eager</em>
уменьшает ожидание, но повышает память; <em>serial</em> экономит память, но
ограничивает throughput.
</section>
<section class="card">
<h2>Профили и сценарии</h2>
<table>
<thead><tr><th>Профиль</th><th>Concurrency</th><th>Prefetch</th><th>Смысл</th></tr></thead>
<tbody>
<tr><td>serial</td><td>1</td><td>1</td><td style='text-align:left'>one job at a time; minimum memory, maximum queueing</td></tr><tr><td>balanced</td><td>4</td><td>4</td><td style='text-align:left'>moderate in-process parallelism and broker window</td></tr><tr><td>eager</td><td>8</td><td>32</td><td style='text-align:left'>wide broker window; highest memory, lowest latency</td></tr>
</tbody>
</table>
<p class="note">Сценарии:
small_many — 192×128, 30 jobs, many small images; almost no scaling; exactly_right — 1280×1280, 18 jobs, largest variant encodes at native 1280, so no scaling on the WebP path; huge_few — 2400×1800, 12 jobs, few 4.32 MP images; optimized variant must scale down to 1280.</p>
</section>
<section class="card">
<h2>Ключевые числа</h2>
<div class="metrics">
<div class="metric"><div class="value">9</div><div class="label">прогонов матрицы</div></div>
<div class="metric"><div class="value">950 MiB</div><div class="label">максимальный RSS worker</div></div>
<div class="metric"><div class="value">7.7 s</div><div class="label">восстановление после backpressure</div></div>
</div>
</section>
<section class="card chart"><svg viewBox="0 0 760 320" role="img" aria-label="Throughput: готовые задания в секунду">
<text x="64" y="24" class="chart-title">Throughput: готовые задания в секунду</text>
<text x="64" y="42" class="chart-sub">Чем выше, тем быстрее pipeline обрабатывает очередь.</text>
<line x1="64" y1="258.0" x2="740" y2="258.0" class="grid"/>
<text x="54" y="262.0" class="axis" text-anchor="end">0 jobs/s</text>
<line x1="64" y1="206.5" x2="740" y2="206.5" class="grid"/>
<text x="54" y="210.5" class="axis" text-anchor="end">5 jobs/s</text>
<line x1="64" y1="155.0" x2="740" y2="155.0" class="grid"/>
<text x="54" y="159.0" class="axis" text-anchor="end">9 jobs/s</text>
<line x1="64" y1="103.5" x2="740" y2="103.5" class="grid"/>
<text x="54" y="107.5" class="axis" text-anchor="end">14 jobs/s</text>
<line x1="64" y1="52.0" x2="740" y2="52.0" class="grid"/>
<text x="54" y="56.0" class="axis" text-anchor="end">18 jobs/s</text>
<rect x="83.2" y="181.4" width="45.1" height="76.6" fill="#64748b" rx="3"/>
<text x="105.8" y="176.4" class="bar-label" text-anchor="middle">6.84 jobs/s</text>
<rect x="138.2" y="104.5" width="45.1" height="153.5" fill="#0284c7" rx="3"/>
<text x="160.7" y="99.5" class="bar-label" text-anchor="middle">13.69 jobs/s</text>
<rect x="193.2" y="83.4" width="45.1" height="174.6" fill="#f59e0b" rx="3"/>
<text x="215.7" y="78.4" class="bar-label" text-anchor="middle">15.58 jobs/s</text>
<text x="176.7" y="282" class="axis" text-anchor="middle">small_many</text>
<rect x="308.6" y="252.7" width="45.1" height="5.3" fill="#64748b" rx="3"/>
<text x="331.1" y="247.7" class="bar-label" text-anchor="middle">0.47 jobs/s</text>
<rect x="363.5" y="252.5" width="45.1" height="5.5" fill="#0284c7" rx="3"/>
<text x="386.1" y="247.5" class="bar-label" text-anchor="middle">0.49 jobs/s</text>
<rect x="418.5" y="252.5" width="45.1" height="5.5" fill="#f59e0b" rx="3"/>
<text x="441.0" y="247.5" class="bar-label" text-anchor="middle">0.49 jobs/s</text>
<text x="402.0" y="282" class="axis" text-anchor="middle">exactly_right</text>
<rect x="533.9" y="253.1" width="45.1" height="4.9" fill="#64748b" rx="3"/>
<text x="556.4" y="248.1" class="bar-label" text-anchor="middle">0.44 jobs/s</text>
<rect x="588.9" y="253.1" width="45.1" height="4.9" fill="#0284c7" rx="3"/>
<text x="611.4" y="248.1" class="bar-label" text-anchor="middle">0.44 jobs/s</text>
<rect x="643.8" y="253.0" width="45.1" height="5.0" fill="#f59e0b" rx="3"/>
<text x="666.4" y="248.0" class="bar-label" text-anchor="middle">0.44 jobs/s</text>
<text x="627.3" y="282" class="axis" text-anchor="middle">huge_few</text>
<rect x="64" y="298" width="11" height="11" fill="#64748b" rx="2"/>
<text x="80" y="308" class="axis">serial</text>
<rect x="154" y="298" width="11" height="11" fill="#0284c7" rx="2"/>
<text x="170" y="308" class="axis">balanced</text>
<rect x="244" y="298" width="11" height="11" fill="#f59e0b" rx="2"/>
<text x="260" y="308" class="axis">eager</text>
<text x="740" y="308" class="axis" text-anchor="end">higher is better</text>
</svg></section>
<section class="card chart"><svg viewBox="0 0 760 320" role="img" aria-label="p95 time-to-ready: от 202 до ready">
<text x="64" y="24" class="chart-title">p95 time-to-ready: от 202 до ready</text>
<text x="64" y="42" class="chart-sub">95% заданий стали ready быстрее этого времени (меньше — лучше).</text>
<line x1="64" y1="258.0" x2="740" y2="258.0" class="grid"/>
<text x="54" y="262.0" class="axis" text-anchor="end">0 ms</text>
<line x1="64" y1="206.5" x2="740" y2="206.5" class="grid"/>
<text x="54" y="210.5" class="axis" text-anchor="end">10560 ms</text>
<line x1="64" y1="155.0" x2="740" y2="155.0" class="grid"/>
<text x="54" y="159.0" class="axis" text-anchor="end">21120 ms</text>
<line x1="64" y1="103.5" x2="740" y2="103.5" class="grid"/>
<text x="54" y="107.5" class="axis" text-anchor="end">31680 ms</text>
<line x1="64" y1="52.0" x2="740" y2="52.0" class="grid"/>
<text x="54" y="56.0" class="axis" text-anchor="end">42240 ms</text>
<rect x="83.2" y="241.7" width="45.1" height="16.3" fill="#64748b" rx="3"/>
<text x="105.8" y="236.7" class="bar-label" text-anchor="middle">3338 ms</text>
<rect x="138.2" y="251.8" width="45.1" height="6.2" fill="#0284c7" rx="3"/>
<text x="160.7" y="246.8" class="bar-label" text-anchor="middle">1274 ms</text>
<rect x="193.2" y="252.3" width="45.1" height="5.7" fill="#f59e0b" rx="3"/>
<text x="215.7" y="247.3" class="bar-label" text-anchor="middle">1166 ms</text>
<text x="176.7" y="282" class="axis" text-anchor="middle">small_many</text>
<rect x="308.6" y="83.8" width="45.1" height="174.2" fill="#64748b" rx="3"/>
<text x="331.1" y="78.8" class="bar-label" text-anchor="middle">35714 ms</text>
<rect x="363.5" y="83.6" width="45.1" height="174.4" fill="#0284c7" rx="3"/>
<text x="386.1" y="78.6" class="bar-label" text-anchor="middle">35764 ms</text>
<rect x="418.5" y="83.4" width="45.1" height="174.6" fill="#f59e0b" rx="3"/>
<text x="441.0" y="78.4" class="bar-label" text-anchor="middle">35797 ms</text>
<text x="402.0" y="282" class="axis" text-anchor="middle">exactly_right</text>
<rect x="533.9" y="131.7" width="45.1" height="126.3" fill="#64748b" rx="3"/>
<text x="556.4" y="126.7" class="bar-label" text-anchor="middle">25896 ms</text>
<rect x="588.9" y="128.4" width="45.1" height="129.6" fill="#0284c7" rx="3"/>
<text x="611.4" y="123.4" class="bar-label" text-anchor="middle">26576 ms</text>
<rect x="643.8" y="128.1" width="45.1" height="129.9" fill="#f59e0b" rx="3"/>
<text x="666.4" y="123.1" class="bar-label" text-anchor="middle">26641 ms</text>
<text x="627.3" y="282" class="axis" text-anchor="middle">huge_few</text>
<rect x="64" y="298" width="11" height="11" fill="#64748b" rx="2"/>
<text x="80" y="308" class="axis">serial</text>
<rect x="154" y="298" width="11" height="11" fill="#0284c7" rx="2"/>
<text x="170" y="308" class="axis">balanced</text>
<rect x="244" y="298" width="11" height="11" fill="#f59e0b" rx="2"/>
<text x="260" y="308" class="axis">eager</text>
<text x="740" y="308" class="axis" text-anchor="end">lower is better</text>
</svg></section>
<section class="card chart"><svg viewBox="0 0 760 320" role="img" aria-label="p95 queue lag: ожидание до старта обработки">
<text x="64" y="24" class="chart-title">p95 queue lag: ожидание до старта обработки</text>
<text x="64" y="42" class="chart-sub">Время между durable 202 и первым claim worker-а (меньше — лучше).</text>
<line x1="64" y1="258.0" x2="740" y2="258.0" class="grid"/>
<text x="54" y="262.0" class="axis" text-anchor="end">0 ms</text>
<line x1="64" y1="206.5" x2="740" y2="206.5" class="grid"/>
<text x="54" y="210.5" class="axis" text-anchor="end">9909 ms</text>
<line x1="64" y1="155.0" x2="740" y2="155.0" class="grid"/>
<text x="54" y="159.0" class="axis" text-anchor="end">19819 ms</text>
<line x1="64" y1="103.5" x2="740" y2="103.5" class="grid"/>
<text x="54" y="107.5" class="axis" text-anchor="end">29728 ms</text>
<line x1="64" y1="52.0" x2="740" y2="52.0" class="grid"/>
<text x="54" y="56.0" class="axis" text-anchor="end">39637 ms</text>
<rect x="83.2" y="241.1" width="45.1" height="16.9" fill="#64748b" rx="3"/>
<text x="105.8" y="236.1" class="bar-label" text-anchor="middle">3256 ms</text>
<rect x="138.2" y="252.0" width="45.1" height="6.0" fill="#0284c7" rx="3"/>
<text x="160.7" y="247.0" class="bar-label" text-anchor="middle">1147 ms</text>
<rect x="193.2" y="253.2" width="45.1" height="4.8" fill="#f59e0b" rx="3"/>
<text x="215.7" y="248.2" class="bar-label" text-anchor="middle">924 ms</text>
<text x="176.7" y="282" class="axis" text-anchor="middle">small_many</text>
<rect x="308.6" y="83.4" width="45.1" height="174.6" fill="#64748b" rx="3"/>
<text x="331.1" y="78.4" class="bar-label" text-anchor="middle">33591 ms</text>
<rect x="363.5" y="94.2" width="45.1" height="163.8" fill="#0284c7" rx="3"/>
<text x="386.1" y="89.2" class="bar-label" text-anchor="middle">31510 ms</text>
<rect x="418.5" y="93.1" width="45.1" height="164.9" fill="#f59e0b" rx="3"/>
<text x="441.0" y="88.1" class="bar-label" text-anchor="middle">31735 ms</text>
<text x="402.0" y="282" class="axis" text-anchor="middle">exactly_right</text>
<rect x="533.9" y="135.0" width="45.1" height="123.0" fill="#64748b" rx="3"/>
<text x="556.4" y="130.0" class="bar-label" text-anchor="middle">23668 ms</text>
<rect x="588.9" y="163.6" width="45.1" height="94.4" fill="#0284c7" rx="3"/>
<text x="611.4" y="158.6" class="bar-label" text-anchor="middle">18161 ms</text>
<rect x="643.8" y="166.3" width="45.1" height="91.7" fill="#f59e0b" rx="3"/>
<text x="666.4" y="161.3" class="bar-label" text-anchor="middle">17648 ms</text>
<text x="627.3" y="282" class="axis" text-anchor="middle">huge_few</text>
<rect x="64" y="298" width="11" height="11" fill="#64748b" rx="2"/>
<text x="80" y="308" class="axis">serial</text>
<rect x="154" y="298" width="11" height="11" fill="#0284c7" rx="2"/>
<text x="170" y="308" class="axis">balanced</text>
<rect x="244" y="298" width="11" height="11" fill="#f59e0b" rx="2"/>
<text x="260" y="308" class="axis">eager</text>
<text x="740" y="308" class="axis" text-anchor="end">lower is better</text>
</svg></section>
<section class="card chart"><svg viewBox="0 0 760 320" role="img" aria-label="Пиковый RSS worker">
<text x="64" y="24" class="chart-title">Пиковый RSS worker</text>
<text x="64" y="42" class="chart-sub">VmRSS процесса worker во время задания (меньше — экономнее).</text>
<line x1="64" y1="258.0" x2="740" y2="258.0" class="grid"/>
<text x="54" y="262.0" class="axis" text-anchor="end">0 MiB</text>
<line x1="64" y1="206.5" x2="740" y2="206.5" class="grid"/>
<text x="54" y="210.5" class="axis" text-anchor="end">280 MiB</text>
<line x1="64" y1="155.0" x2="740" y2="155.0" class="grid"/>
<text x="54" y="159.0" class="axis" text-anchor="end">560 MiB</text>
<line x1="64" y1="103.5" x2="740" y2="103.5" class="grid"/>
<text x="54" y="107.5" class="axis" text-anchor="end">840 MiB</text>
<line x1="64" y1="52.0" x2="740" y2="52.0" class="grid"/>
<text x="54" y="56.0" class="axis" text-anchor="end">1121 MiB</text>
<rect x="83.2" y="252.8" width="45.1" height="5.2" fill="#64748b" rx="3"/>
<text x="105.8" y="247.8" class="bar-label" text-anchor="middle">28 MiB</text>
<rect x="138.2" y="252.7" width="45.1" height="5.3" fill="#0284c7" rx="3"/>
<text x="160.7" y="247.7" class="bar-label" text-anchor="middle">29 MiB</text>
<rect x="193.2" y="252.1" width="45.1" height="5.9" fill="#f59e0b" rx="3"/>
<text x="215.7" y="247.1" class="bar-label" text-anchor="middle">32 MiB</text>
<text x="176.7" y="282" class="axis" text-anchor="middle">small_many</text>
<rect x="308.6" y="226.0" width="45.1" height="32.0" fill="#64748b" rx="3"/>
<text x="331.1" y="221.0" class="bar-label" text-anchor="middle">174 MiB</text>
<rect x="363.5" y="178.2" width="45.1" height="79.8" fill="#0284c7" rx="3"/>
<text x="386.1" y="173.2" class="bar-label" text-anchor="middle">434 MiB</text>
<rect x="418.5" y="107.5" width="45.1" height="150.5" fill="#f59e0b" rx="3"/>
<text x="441.0" y="102.5" class="bar-label" text-anchor="middle">818 MiB</text>
<text x="402.0" y="282" class="axis" text-anchor="middle">exactly_right</text>
<rect x="533.9" y="218.9" width="45.1" height="39.1" fill="#64748b" rx="3"/>
<text x="556.4" y="213.9" class="bar-label" text-anchor="middle">213 MiB</text>
<rect x="588.9" y="118.9" width="45.1" height="139.1" fill="#0284c7" rx="3"/>
<text x="611.4" y="113.9" class="bar-label" text-anchor="middle">756 MiB</text>
<rect x="643.8" y="83.4" width="45.1" height="174.6" fill="#f59e0b" rx="3"/>
<text x="666.4" y="78.4" class="bar-label" text-anchor="middle">950 MiB</text>
<text x="627.3" y="282" class="axis" text-anchor="middle">huge_few</text>
<rect x="64" y="298" width="11" height="11" fill="#64748b" rx="2"/>
<text x="80" y="308" class="axis">serial</text>
<rect x="154" y="298" width="11" height="11" fill="#0284c7" rx="2"/>
<text x="170" y="308" class="axis">balanced</text>
<rect x="244" y="298" width="11" height="11" fill="#f59e0b" rx="2"/>
<text x="260" y="308" class="axis">eager</text>
<text x="740" y="308" class="axis" text-anchor="end">lower is better</text>
</svg></section>
<section class="card chart"><svg viewBox="0 0 760 320" role="img" aria-label="Compression overhead: доля encode в pipeline CPU">
<text x="64" y="24" class="chart-title">Compression overhead: доля encode в pipeline CPU</text>
<text x="64" y="42" class="chart-sub">encode / (decode + scale + encode), % (меньше — дешевле сжатие).</text>
<line x1="64" y1="258.0" x2="740" y2="258.0" class="grid"/>
<text x="54" y="262.0" class="axis" text-anchor="end">0 %</text>
<line x1="64" y1="206.5" x2="740" y2="206.5" class="grid"/>
<text x="54" y="210.5" class="axis" text-anchor="end">25 %</text>
<line x1="64" y1="155.0" x2="740" y2="155.0" class="grid"/>
<text x="54" y="159.0" class="axis" text-anchor="end">49 %</text>
<line x1="64" y1="103.5" x2="740" y2="103.5" class="grid"/>
<text x="54" y="107.5" class="axis" text-anchor="end">74 %</text>
<line x1="64" y1="52.0" x2="740" y2="52.0" class="grid"/>
<text x="54" y="56.0" class="axis" text-anchor="end">99 %</text>
<rect x="83.2" y="83.4" width="45.1" height="174.6" fill="#64748b" rx="3"/>
<text x="105.8" y="78.4" class="bar-label" text-anchor="middle">83.8 %</text>
<rect x="138.2" y="83.4" width="45.1" height="174.6" fill="#0284c7" rx="3"/>
<text x="160.7" y="78.4" class="bar-label" text-anchor="middle">83.8 %</text>
<rect x="193.2" y="91.5" width="45.1" height="166.5" fill="#f59e0b" rx="3"/>
<text x="215.7" y="86.5" class="bar-label" text-anchor="middle">79.9 %</text>
<text x="176.7" y="282" class="axis" text-anchor="middle">small_many</text>
<rect x="308.6" y="87.2" width="45.1" height="170.8" fill="#64748b" rx="3"/>
<text x="331.1" y="82.2" class="bar-label" text-anchor="middle">82.0 %</text>
<rect x="363.5" y="84.9" width="45.1" height="173.1" fill="#0284c7" rx="3"/>
<text x="386.1" y="79.9" class="bar-label" text-anchor="middle">83.1 %</text>
<rect x="418.5" y="84.9" width="45.1" height="173.1" fill="#f59e0b" rx="3"/>
<text x="441.0" y="79.9" class="bar-label" text-anchor="middle">83.1 %</text>
<text x="402.0" y="282" class="axis" text-anchor="middle">exactly_right</text>
<rect x="533.9" y="125.1" width="45.1" height="132.9" fill="#64748b" rx="3"/>
<text x="556.4" y="120.1" class="bar-label" text-anchor="middle">63.8 %</text>
<rect x="588.9" y="130.1" width="45.1" height="127.9" fill="#0284c7" rx="3"/>
<text x="611.4" y="125.1" class="bar-label" text-anchor="middle">61.4 %</text>
<rect x="643.8" y="130.1" width="45.1" height="127.9" fill="#f59e0b" rx="3"/>
<text x="666.4" y="125.1" class="bar-label" text-anchor="middle">61.4 %</text>
<text x="627.3" y="282" class="axis" text-anchor="middle">huge_few</text>
<rect x="64" y="298" width="11" height="11" fill="#64748b" rx="2"/>
<text x="80" y="308" class="axis">serial</text>
<rect x="154" y="298" width="11" height="11" fill="#0284c7" rx="2"/>
<text x="170" y="308" class="axis">balanced</text>
<rect x="244" y="298" width="11" height="11" fill="#f59e0b" rx="2"/>
<text x="260" y="308" class="axis">eager</text>
<text x="740" y="308" class="axis" text-anchor="end">lower is better</text>
</svg></section>
<section class="card">
<h2>Полная матрица</h2>
<table>
<thead><tr><th>Профиль</th><th>Сценарий</th><th>Jobs</th><th>Jobs/s</th><th>Queue lag p95, ms</th><th>Ready p95, ms</th><th>Processing p95, ms</th><th>Encode p95, ms</th><th>Encode share, %</th><th>RSS peak, MiB</th><th>Max parallel</th><th>Max backlog</th></tr></thead>
<tbody><tr><td>serial</td><td>small_many</td><td>30</td><td>6.84</td><td>3256</td><td>3338</td><td>170</td><td>80</td><td>83.8</td><td>28</td><td>1</td><td>27</td></tr><tr><td>serial</td><td>exactly_right</td><td>18</td><td>0.47</td><td>33591</td><td>35714</td><td>2180</td><td>1722</td><td>82.0</td><td>174</td><td>1</td><td>18</td></tr><tr><td>serial</td><td>huge_few</td><td>12</td><td>0.44</td><td>23668</td><td>25896</td><td>2359</td><td>1508</td><td>63.8</td><td>213</td><td>1</td><td>12</td></tr><tr><td>balanced</td><td>small_many</td><td>30</td><td>13.69</td><td>1147</td><td>1274</td><td>396</td><td>90</td><td>83.8</td><td>29</td><td>4</td><td>24</td></tr><tr><td>balanced</td><td>exactly_right</td><td>18</td><td>0.49</td><td>31510</td><td>35764</td><td>8174</td><td>6562</td><td>83.1</td><td>434</td><td>4</td><td>18</td></tr><tr><td>balanced</td><td>huge_few</td><td>12</td><td>0.44</td><td>18161</td><td>26576</td><td>9046</td><td>5606</td><td>61.4</td><td>756</td><td>4</td><td>12</td></tr><tr><td>eager</td><td>small_many</td><td>30</td><td>15.58</td><td>924</td><td>1166</td><td>410</td><td>95</td><td>79.9</td><td>32</td><td>8</td><td>22</td></tr><tr><td>eager</td><td>exactly_right</td><td>18</td><td>0.49</td><td>31735</td><td>35797</td><td>16363</td><td>13330</td><td>83.1</td><td>818</td><td>8</td><td>18</td></tr><tr><td>eager</td><td>huge_few</td><td>12</td><td>0.44</td><td>17648</td><td>26641</td><td>17678</td><td>11124</td><td>61.4</td><td>950</td><td>8</td><td>12</td></tr></tbody>
</table>
<p class="note">Queue lag = <code>processing_started_at - created_at</code>.
Time-to-ready = <code>finished_at - created_at</code>. Processing = <code>finished_at - processing_started_at</code>.
Encode = JPEG thumbnail + JPEG crop + WebP optimized; encode share =
<code>encode / (decode + scale + encode)</code> из worker-лога.
Суммы байт вариантов и оригинала остаются в JSON, но не выносятся в заголовок:
синтетический PNG сжимается слишком хорошо, чтобы отношение байт было честной
метрикой эффективности кодека.
RSS — VmRSS процесса worker, сэмплирование примерно каждые 100 мс.
Max parallel — максимум одновременно обрабатываемых заданий; max backlog —
максимум принятых, но ещё не готовых заданий.</p>
</section>
<section class="card">
<h2>Backpressure и восстановление</h2>
<p>Перед этим прогоном API пересоздан с <code>MAX_PENDING_JOBS=3</code>, worker
остановлен. Cap выбран намеренно: важно не число принятых заданий, а то, что
API возвращает <code>503</code> с <code>Retry-After</code> и не теряет уже
подтверждённую работу. Затем worker запускается, принятые задания должны дойти
до ready.</p>
<table>
<thead><tr><th>#</th><th>HTTP</th><th>Latency, ms</th><th>Retry-After</th><th>Error code</th></tr></thead>
<tbody><tr><td>1</td><td class="ok">202</td><td>26.7</td><td>—</td><td>—</td></tr><tr><td>2</td><td class="ok">202</td><td>20.3</td><td>—</td><td>—</td></tr><tr><td>3</td><td class="ok">202</td><td>13.8</td><td>—</td><td>—</td></tr><tr><td>4</td><td class="blocked">503</td><td>2.4</td><td>5</td><td>unavailable</td></tr></tbody>
</table>
<p class="note">Recovery: 3 принятых заданий
стали ready за 7.65 с после старта worker.
Финальный probe upload: HTTP 202 → ready.</p>
</section>
<section class="card">
<h2>Что это значит</h2>
<ul><li>small_many: eager быстрее serial в 2.3× по throughput.</li><li>small_many: пиковый RSS eager относится к serial как 1.1×.</li><li>exactly_right: throughput eager и serial практически одинаков.</li><li>exactly_right: пиковый RSS eager относится к serial как 4.7×.</li><li>huge_few: throughput eager и serial практически одинаков.</li><li>huge_few: пиковый RSS eager относится к serial как 4.5×.</li><li>При balanced p95 time-to-ready для huge_few в 20.9× больше, чем для small_many: размер изображения меняет стоимость задания, а не только число заданий.</li><li>Compression vs scaling (balanced): exactly_right — encode share 83.1%, scale p95 1527 ms; huge_few — encode share 61.4%, scale p95 3448 ms. На exactly_right optimized WebP кодируется в native 1280, поэтому это нижняя граница стоимости сжатия; на huge_few к ней добавляется downscale.</li><li>На huge_few eager почти не поднял throughput (1.02×): worker ограничен 1.0 CPU, а декодирование и масштабирование упираются в CPU. Concurrency снижает queue lag, но при этом растёт RSS; это и есть компромисс latency/ресурсы, а не бесплатное ускорение.</li><li>serial ограничивает одновременно обрабатываемые задания одним, поэтому queue lag растёт, но RSS остаётся минимальным. eager держит больше заданий в работе и в prefetch-окне брокера.</li><li>Backpressure: cap=3 выставлен намеренно, поэтому число принятых заданий — следствие настройки, а не открытие. Значимо то, что API вернул 503 с Retry-After=5, а после старта worker очередь восстановилась за 7.7 с.</li></ul>
</section>
<section class="card">
<h2>Ограничения</h2>
<ul>
<li>Один worker, один RabbitMQ и один PostgreSQL: это сравнение настроек, а не
HA-кластер. Числа зависят от CPU/диска конкретной машины.</li>
<li>PNG-фикстуры синтетические и хорошо сжимаются; absolute encode time зависит
от паттерна, но сравнение сценариев при одном паттерне остаётся корректным.
Размер в байтах указан в JSON.</li>
<li>Timing-поля читаются из worker-лога. Это wall-clock внутри процесса, поэтому
при concurrency &gt; 1 в них входит и планировщик, что как раз и нужно для
сравнения профилей.</li>
<li>exactly_right кодирует 1.64 MP WebP, а huge_few после downscale — около
1.23 MP. Сценарий убирает scaling, но не уравнивает объём encode; это указано,
чтобы не сравнивать throughput как «одинаковую работу».</li>
<li>RSS сэмплируется из <code>/proc/1/status</code>; короткий прогон не заменяет
длительный memory profile и p99.</li>
<li>Backpressure cap приблизительный: каждая API-реплика считает pending-задания
по общей БД, но проверка и вставка не являются одной транзакцией.</li>
</ul>
</section>
</main>
</body>
</html>
+2923
View File
File diff suppressed because it is too large Load Diff
+19
View File
@@ -0,0 +1,19 @@
# Lab5 track 1 — queue under load
Generated: `2026-11-20T16:02:07.856909+00:00`
Synthetic images; no tokens or credentials are stored.
| Profile | Scenario | Jobs | Jobs/s | Queue lag p95, ms | Ready p95, ms | Encode p95, ms | Encode share, % | RSS peak, MiB |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| serial | small_many | 30 | 6.84 | 3256 | 3338 | 80 | 83.8 | 28 |
| serial | exactly_right | 18 | 0.47 | 33591 | 35714 | 1722 | 82.0 | 174 |
| serial | huge_few | 12 | 0.44 | 23668 | 25896 | 1508 | 63.8 | 213 |
| balanced | small_many | 30 | 13.69 | 1147 | 1274 | 90 | 83.8 | 29 |
| balanced | exactly_right | 18 | 0.49 | 31510 | 35764 | 6562 | 83.1 | 434 |
| balanced | huge_few | 12 | 0.44 | 18161 | 26576 | 5606 | 61.4 | 756 |
| eager | small_many | 30 | 15.58 | 924 | 1166 | 95 | 79.9 | 32 |
| eager | exactly_right | 18 | 0.49 | 31735 | 35797 | 13330 | 83.1 | 818 |
| eager | huge_few | 12 | 0.44 | 17648 | 26641 | 11124 | 61.4 | 950 |
Backpressure: cap=3 (deliberate), first 503 Retry-After=5, recovery=7.65s, probe HTTP 202.
+25
View File
@@ -0,0 +1,25 @@
{
"workload": {
"uploads": 10,
"width": 1800,
"height": 1400,
"image_bytes": 7563763,
"sha256": "e42f6283ac02b27d81700f861fc943a1bcb2b17a638edde130d37625315907f5",
"elapsed_seconds": 2.6,
"instances": [
"2229551f45ac",
"ad8141b5e17c"
],
"download_verified": true
},
"sampling_interval_seconds": 0.05,
"samples": 121,
"wall_seconds": 6.884,
"baseline_total_mib": 14.7,
"peak_total_mib": 354.1,
"peak_per_container_mib": {
"mtusi2026lab-api-1": 174.1,
"mtusi2026lab-api-2": 180.1
},
"api_memory_limit_mib": 512.0
}
+24
View File
@@ -0,0 +1,24 @@
{
"workload": {
"uploads": 1,
"width": 1800,
"height": 1400,
"image_bytes": 7563763,
"sha256": "be9606c01fdb972e9d60ffdacc03d0208cdca825a0797e727647639b14c0e35c",
"elapsed_seconds": 0.435,
"instances": [
"2229551f45ac"
],
"download_verified": true
},
"sampling_interval_seconds": 0.05,
"samples": 82,
"wall_seconds": 4.616,
"baseline_total_mib": 16.7,
"peak_total_mib": 280.4,
"peak_per_container_mib": {
"mtusi2026lab-api-1": 142.9,
"mtusi2026lab-api-2": 137.5
},
"api_memory_limit_mib": 512.0
}

Some files were not shown because too many files have changed in this diff Show More